ColinLeeo opened a new issue, #1069: URL: https://github.com/apache/tsfile/issues/1069
## Summary TsFile's `DICTIONARY` encoding is only wired for `STRING`/`TEXT`. Numeric types (`DOUBLE`/`FLOAT`/`INT32`/`INT64`) fall through to `return nullptr` in `cpp/src/encoding/encoder_factory.h`, so low-cardinality numeric data cannot use dictionary encoding. ## Motivation Parquet auto-detects low cardinality and dictionary-encodes float64 columns (`RLE_DICTIONARY`), storing indices instead of 8-byte values. This is the dominant reason Parquet compresses low-cardinality numeric data better than TsFile. Benchmark on a 5214-column wide table of minute-level stock features (discrete/quantized values, ~1% distinct — e.g. 288k values with only ~2–4k distinct), all 1-chunk + zstd: | format / encoding | size | vs Parquet | |---|---|---| | Parquet (dictionary) | 511 MB | 1.00× | | TsFile DOUBLE PLAIN | 587 MB | 1.15× | | TsFile DOUBLE RLE | 563 MB | 1.10× | | TsFile DOUBLE GORILLA / TS_2DIFF / SPRINTZ / CAMEL | 607–991 MB | worse | None of TsFile's existing numeric encodings match Parquet: they target smoothness/delta (Gorilla, TS_2DIFF, Chimp, Camel) or runs (RLE, RLBE), while this data is low-cardinality but scattered (no smoothness, no long runs). ## Proposal Extend `DICTIONARY` (or add a numeric dictionary encoding) to support `DOUBLE`/`FLOAT`/`INT32`/`INT64`. A numeric dictionary builds a map from each distinct value (bit pattern, or scaled integer) to a compact index, stores the dictionary once, then stores indices per value — the same mechanism Parquet uses. ## Related - #1068 added FLOAT/DOUBLE RLE (run-length), but RLE only compresses consecutive repeats, not scattered low-cardinality repetition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
