ColinLeeo opened a new issue, #1077: URL: https://github.com/apache/tsfile/issues/1077
## Summary The C++/Python reader can silently drop a whole time series when the device path contains non-ASCII characters. No error is reported: metadata APIs still list the series, but the query returns an all-NULL column (or omits it). ## Root cause `Murmur128Hash` differs between Java and C++ for the hash tail: - Java (`java/.../utils/Murmur128Hash.java`) uses `((long) key[i]) << shift` on `byte`, which is **signed** and sign-extends bytes >= 0x80. - C++ (`cpp/src/common/container/murmur_hash3.cc`) uses `((uint64_t)(uint8_t)buf[i]) << shift`, which **zero-extends**. The two only diverge when the hashed bytes `device + "." + measurement` have a byte >= 0x80 in the tail (`len % 16`), i.e. exactly for multi-byte UTF-8 device path segments. The bloom filter layout is persisted in the file. When C++ reads a Java-written file (or vice versa), it computes different bit positions, so `BloomFilter::contains()` returns false and `TsFileIOReader::alloc_ssi()` returns `E_NO_MORE_DATA` (`cpp/src/file/tsfile_io_reader.cc:79`). The series is then dropped without any error. ## Reproduction A Java-written tree-model file with device `root.sensors.宿舍.TH` and measurements `humidity` / `temperature`: - `root.sensors.宿舍.TH.humidity` (31 bytes, tail contains the bytes of `舍`) -> 0 rows / all NULL - `root.sensors.宿舍.TH.temperature` -> reads correctly Renaming the device path to ASCII (e.g. `root.sensors.dorm.TH`) and re-writing the same data makes both series readable. The reverse direction is symmetric: a C++-written file with such a path loses the series when read from Java. ## Proposed fix 1. Writer: compute bloom positions with Java semantics (signed tail), so newly written C++ files are interoperable with Java. 2. Reader: accept a hit under either variant (Java signed or legacy C++ unsigned). A bloom filter may yield false positives, never false negatives, so checking both is safe and keeps existing C++-written files readable. I have a patch with unit tests ready and will link the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
