ColinLeeo opened a new issue, #1069:
URL: https://github.com/apache/tsfile/issues/1069

   ## Summary
   
   TsFile's `DICTIONARY` encoding is only wired for `STRING`/`TEXT`. Numeric 
types (`DOUBLE`/`FLOAT`/`INT32`/`INT64`) fall through to `return nullptr` in 
`cpp/src/encoding/encoder_factory.h`, so low-cardinality numeric data cannot 
use dictionary encoding.
   
   ## Motivation
   
   Parquet auto-detects low cardinality and dictionary-encodes float64 columns 
(`RLE_DICTIONARY`), storing indices instead of 8-byte values. This is the 
dominant reason Parquet compresses low-cardinality numeric data better than 
TsFile.
   
   Benchmark on a 5214-column wide table of minute-level stock features 
(discrete/quantized values, ~1% distinct — e.g. 288k values with only ~2–4k 
distinct), all 1-chunk + zstd:
   
   | format / encoding | size | vs Parquet |
   |---|---|---|
   | Parquet (dictionary) | 511 MB | 1.00× |
   | TsFile DOUBLE PLAIN | 587 MB | 1.15× |
   | TsFile DOUBLE RLE | 563 MB | 1.10× |
   | TsFile DOUBLE GORILLA / TS_2DIFF / SPRINTZ / CAMEL | 607–991 MB | worse |
   
   None of TsFile's existing numeric encodings match Parquet: they target 
smoothness/delta (Gorilla, TS_2DIFF, Chimp, Camel) or runs (RLE, RLBE), while 
this data is low-cardinality but scattered (no smoothness, no long runs).
   
   ## Proposal
   
   Extend `DICTIONARY` (or add a numeric dictionary encoding) to support 
`DOUBLE`/`FLOAT`/`INT32`/`INT64`. A numeric dictionary builds a map from each 
distinct value (bit pattern, or scaled integer) to a compact index, stores the 
dictionary once, then stores indices per value — the same mechanism Parquet 
uses.
   
   ## Related
   
   - #1068 added FLOAT/DOUBLE RLE (run-length), but RLE only compresses 
consecutive repeats, not scattered low-cardinality repetition.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to