Hi all,

I would like to propose that we evaluate ALP/FALP as an optional floating-point 
encoding for TsFile, alongside Gorilla and Chimp. The goal is straightforward: 
address the compression-ratio and encode/decode-efficiency gaps that Gorilla 
shows on some training datasets, without replacing the current defaults.


On some training data, we have already seen Gorilla produce a lower compression 
ratio than expected and offer less consistent encode/decode efficiency. More 
specifically, after cleaning, normalization, scaling, alignment, concatenation, 
or feature transformation, the original smoothness, local continuity, and XOR 
prefix reuse of floating-point columns are often weakened or broken, so the 
data characteristics no longer fit Gorilla's assumptions. This limits how much 
we can gain from further Gorilla tuning. At the same time, the current 
encode/decode path cannot fully turn the SIMD width, pipeline parallelism, and 
memory bandwidth of modern hardware into throughput and latency improvements, 
making it increasingly difficult to meet performance requirements as training 
data continues to grow. Chimp improves part of the problem, but it does not 
change these two core limitations.


The effects I care about most are practical ones. First, for decimal-like 
features, stable precision, or repeated scaling patterns, ALP should improve 
compression over Gorilla, while ALP-RD can cover high-precision or more complex 
values. Second, on suitable data, it should improve encode/decode throughput 
and reduce decode tail latency, which matters for training and inference 
pipelines. Third, reusable kernels cover AVX2, AVX-512BW, NEON, SVE, and WASM 
SIMD, with a scalar fallback. Fourth, ALP/FALP are lossless floating-point 
codecs; exceptions and the ALP-RD path preserve the exact original values 
rather than trading away precision, and columns or row groups that do not 
benefit from ALP can continue to use Gorilla or Chimp.


I would not suggest replacing the default codec all at once. A safer approach 
is to make ALP/FALP an opt-in codec: allocate a new codec id, let the writer 
choose it per column or row group after sampling, and leave the existing 
Gorilla/Chimp formats and read paths unchanged. That way, data with no benefit 
does not affect existing users or files. It is also worth being clear that ALP 
will not win everywhere; high-entropy, non-decimal, or random floating-point 
data may still be better served by Gorilla or Chimp.


If we pursue this, I suggest first defining the new codec id, metadata, and 
version-compatibility strategy, and agreeing on the scope of cross-language 
compatibility, boundary-value handling, and error recovery. ALP and FastLanes 
are both MIT-licensed and compatible with TsFile's Apache-2.0 licensing, but we 
would need to retain the copyright, license, and source notices. One 
distinction is also worth keeping clear: ALP is the floating-point codec, while 
FastLanes provides a general SIMD-oriented layout and kernel framework. We 
would introduce ALP/FALP's floating-point encoding capability and use 
FastLanes/FALP-style SIMD kernels as the acceleration layer.


I would like to hear the team's view first. Do we agree that this should be 
evaluated as a new opt-in floating-point encoding? If the direction makes 
sense, I would also welcome help reviewing the format, SIMD kernels, or 
cross-language compatibility.


If the direction looks reasonable, I can prepare a format draft as the next 
step.


Thanks,
Colin

Reply via email to