ColinLeeo opened a new issue, #1077:
URL: https://github.com/apache/tsfile/issues/1077

   ## Summary
   
   The C++/Python reader can silently drop a whole time series when the device 
path contains non-ASCII characters. No error is reported: metadata APIs still 
list the series, but the query returns an all-NULL column (or omits it).
   
   ## Root cause
   
   `Murmur128Hash` differs between Java and C++ for the hash tail:
   
   - Java (`java/.../utils/Murmur128Hash.java`) uses `((long) key[i]) << shift` 
on `byte`, which is **signed** and sign-extends bytes >= 0x80.
   - C++ (`cpp/src/common/container/murmur_hash3.cc`) uses 
`((uint64_t)(uint8_t)buf[i]) << shift`, which **zero-extends**.
   
   The two only diverge when the hashed bytes `device + "." + measurement` have 
a byte >= 0x80 in the tail (`len % 16`), i.e. exactly for multi-byte UTF-8 
device path segments.
   
   The bloom filter layout is persisted in the file. When C++ reads a 
Java-written file (or vice versa), it computes different bit positions, so 
`BloomFilter::contains()` returns false and `TsFileIOReader::alloc_ssi()` 
returns `E_NO_MORE_DATA` (`cpp/src/file/tsfile_io_reader.cc:79`). The series is 
then dropped without any error.
   
   ## Reproduction
   
   A Java-written tree-model file with device `root.sensors.宿舍.TH` and 
measurements `humidity` / `temperature`:
   
   - `root.sensors.宿舍.TH.humidity` (31 bytes, tail contains the bytes of `舍`) 
-> 0 rows / all NULL
   - `root.sensors.宿舍.TH.temperature` -> reads correctly
   
   Renaming the device path to ASCII (e.g. `root.sensors.dorm.TH`) and 
re-writing the same data makes both series readable. The reverse direction is 
symmetric: a C++-written file with such a path loses the series when read from 
Java.
   
   ## Proposed fix
   
   1. Writer: compute bloom positions with Java semantics (signed tail), so 
newly written C++ files are interoperable with Java.
   2. Reader: accept a hit under either variant (Java signed or legacy C++ 
unsigned). A bloom filter may yield false positives, never false negatives, so 
checking both is safe and keeps existing C++-written files readable.
   
   I have a patch with unit tests ready and will link the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to