ColinLeeo opened a new issue, #1057: URL: https://github.com/apache/tsfile/issues/1057
## Motivation Cross-language compatibility is currently tested with a table-model fixture matrix that covers only part of what each implementation can write. The gaps let encoding and compression bugs reach releases and push maintainers into ad-hoc manual testing. ## Current state (verified) - Java: `java/tsfile/src/test/java/org/apache/tsfile/compatibility/TableModelEncodingCompressionCompatibilityTest.java` - C++: `cpp/test/reader/table_view/table_model_encoding_compression_compatibility_test.cc` - Orchestration: `.github/workflows/compatibility-test.yml` — each language generates fixtures, and the other language validates them (both directions). Coverage today: - **Table model only.** - Data types: INT32, DATE, INT64, TIMESTAMP, FLOAT, DOUBLE, plus BOOLEAN on the Java side. - Encodings: CHIMP and RLBE for the six numeric types; TS_2DIFF for FLOAT/DOUBLE (plus an explicit `max_point_number=0` variant); CAMEL for DOUBLE; PLAIN for null-pattern fixtures; Java-only BOOLEAN RLE default fixtures. - C++ ships encoders for PLAIN, DICTIONARY, RLE, TS_2DIFF, GORILLA, ZIGZAG, CHIMP, SPRINTZ, RLBE, and CAMEL; Java supports the same set plus deprecated DIFF/REGULAR/BITMAP/FREQ. DICTIONARY for STRING/TEXT, GORILLA, SPRINTZ, ZIGZAG, RLE for integer types, and the STRING/TEXT/BLOB data types are not cross-validated. - The Java and C++ matrices are maintained separately and already differ (Java-only BOOLEAN RLE fixtures; C++ writes PLAIN null fixtures). - The tree model and v3 files are not covered by the shared fixture flow; Java's `CompatibilityTest` reads repo fixtures for v3 only. ## Proposal 1. **Single source of truth**: define the matrix in one checked-in spec (for example a small YAML/CSV file or a shared generated manifest) consumed by both the Java and C++ generators, so the two cannot drift. 2. **Cover the full supported cartesian space**, skipping cases explicitly with a reason rather than silently omitting them: every (data type × encoding) pair supported by each side, every compression both can build (UNCOMPRESSED, SNAPPY, GZIP, LZ4, ZSTD, LZMA2 as available in CI), and both models (tree and table). 3. **Add boundary and scale coverage**: 1 row; exactly one page; page + 1; the TS_2DIFF 129-value block boundary; multiple pages; multiple chunks; multiple chunk groups; all-null and mixed-null pages; long RLE runs; dictionary cardinality extremes; string edge cases (empty, embedded NUL, UTF-8); numeric extremes; and precision properties. 4. **Keep both directions** (Java → C++, C++ → Java) and add Python as a reader where the binding supports it. 5. **Decide the policy for encodings one side cannot write**: implement them, or record them as known gaps that fail deliberately. 6. **Make CI report the covered/skipped matrix** so gaps are visible in every run. ## Acceptance criteria - Java and C++ generate an identical fixture set from one shared definition; language-specific fixtures can only exist when explicitly marked as such. - Every (type, encoding, compression) pair supported by both sides is generated and validated in both directions. - Tree-model and table-model files are both covered. - Known unsupported combinations are listed explicitly and tracked instead of missing. - CI fails when a writer cannot produce a declared fixture or a reader cannot read it. ## Related - #1021 — expand cross-language compatibility coverage across scales and nullable BOOLEAN columns - #1019 — C++ support for reading BOOLEAN RLE written with Java defaults -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
