Hi all,
I would like to propose that we evaluate ALP/FALP as an optional floating-point encoding for TsFile, alongside Gorilla and Chimp. The goal is straightforward: address the compression-ratio and encode/decode-efficiency gaps that Gorilla shows on some training datasets, without replacing the current defaults. On some training data, we have already seen Gorilla produce a lower compression ratio than expected and offer less consistent encode/decode efficiency. More specifically, after cleaning, normalization, scaling, alignment, concatenation, or feature transformation, the original smoothness, local continuity, and XOR prefix reuse of floating-point columns are often weakened or broken, so the data characteristics no longer fit Gorilla's assumptions. This limits how much we can gain from further Gorilla tuning. At the same time, the current encode/decode path cannot fully turn the SIMD width, pipeline parallelism, and memory bandwidth of modern hardware into throughput and latency improvements, making it increasingly difficult to meet performance requirements as training data continues to grow. Chimp improves part of the problem, but it does not change these two core limitations. The effects I care about most are practical ones. First, for decimal-like features, stable precision, or repeated scaling patterns, ALP should improve compression over Gorilla, while ALP-RD can cover high-precision or more complex values. Second, on suitable data, it should improve encode/decode throughput and reduce decode tail latency, which matters for training and inference pipelines. Third, reusable kernels cover AVX2, AVX-512BW, NEON, SVE, and WASM SIMD, with a scalar fallback. Fourth, ALP/FALP are lossless floating-point codecs; exceptions and the ALP-RD path preserve the exact original values rather than trading away precision, and columns or row groups that do not benefit from ALP can continue to use Gorilla or Chimp. I would not suggest replacing the default codec all at once. A safer approach is to make ALP/FALP an opt-in codec: allocate a new codec id, let the writer choose it per column or row group after sampling, and leave the existing Gorilla/Chimp formats and read paths unchanged. That way, data with no benefit does not affect existing users or files. It is also worth being clear that ALP will not win everywhere; high-entropy, non-decimal, or random floating-point data may still be better served by Gorilla or Chimp. If we pursue this, I suggest first defining the new codec id, metadata, and version-compatibility strategy, and agreeing on the scope of cross-language compatibility, boundary-value handling, and error recovery. ALP and FastLanes are both MIT-licensed and compatible with TsFile's Apache-2.0 licensing, but we would need to retain the copyright, license, and source notices. One distinction is also worth keeping clear: ALP is the floating-point codec, while FastLanes provides a general SIMD-oriented layout and kernel framework. We would introduce ALP/FALP's floating-point encoding capability and use FastLanes/FALP-style SIMD kernels as the acceleration layer. I would like to hear the team's view first. Do we agree that this should be evaluated as a new opt-in floating-point encoding? If the direction makes sense, I would also welcome help reviewing the format, SIMD kernels, or cross-language compatibility. If the direction looks reasonable, I can prepare a format draft as the next step. Thanks, Colin
