airborne12 opened a new pull request, #421:
URL: https://github.com/apache/doris-thirdparty/pull/421
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`src/ext/for` is a vendored copy of powturbo's TurboPFor, which is licensed
under GPL v2. It is built into libic.a, linked into clucene-core and thereby
into every Doris BE, so a GPL-licensed library sat in an Apache-2.0 / LGPL code
base.
This replaces it with `CLucene/util/PForCodec.{h,cpp}`, a new implementation
of the same on-disk block format, and deletes `src/ext/for` together with the
`ic` library target, its cmake glue and the `ic` link dependencies of the demo
and test targets.
The block format is what existing indexes contain (CodeMode kPfor, kPfor128,
kPfor256), so it is kept bit compatible, in both directions: indexes written
before this change are read by the new decoders, and indexes written by the new
encoders are read by the previous decoders (important for rollbacks). The
format is documented in PForCodec.h.
Provenance: the code was written without copying or adapting any TurboPFor
source. The format was derived from the observable behaviour of the previous
library (its encoded output for crafted inputs) and then implemented from that
description; the packing kernels are a generic template over the bit width
instead of the generated per-width code.
Implementation notes:
- 4 variants as before: zigzag delta (frequencies, positions) and delta-1
(doc ids), each in the 128 value block layout (arm64) and the 256 value block
layout with vertical bit packing (x86_64 AVX2).
- The encoder picks, per block, plain / exception list / exception bitmap /
constant by exact size; for the same input it produces the same size as the
previous encoder or a slightly smaller one (0.6% smaller over the random test
corpus).
- The decoders are portable C++ and bounds checked: a truncated or malformed
stream raises CL_ERR_CorruptIndex instead of reading past the buffer (the
previous decoders took no length).
- Decoding is no longer tied to the build platform. kPfor128 / kPfor256 data
that used to throw "not supported on this platform" in builds without AVX2 /
NEON can now be read; what is written per platform is unchanged.
- Decoding is within about 1.0x-1.9x of the previous hand written SIMD
decoders (up to 1.4 ns per value), encoding is on par.
Tests: testPFOR gets golden vectors produced by the previous encoder (212
cases, all four variants, every block mode and variable length integer form),
the two index files written by older versions decoded with the explicit 128 /
256 decoders on every platform, round trips over many lengths and data shapes,
and truncation / corruption checks. Two existing tests read their input buffer
after `pfor_encode` / `encodePos` had emptied it (an out-of-bounds read that
only fails when libstdc++ assertions are on); they now keep a copy.
### Release note
Removed the GPL v2 licensed TurboPFor library from the inverted index code.
The index format is unchanged.
### Check List (For Author)
- Test: Unit Test / Manual test
- cl_test (full, 89 tests) with USE_AVX2=ON and cl_test testPFOR without
AVX2, both on x86_64 with ASAN.
- Differential tests against the previous library (not part of the
repository): over 100k random and exhaustive-length cases in each direction
(old encoder to new decoder, new encoder to old decoders including the 256
scalar decoder), no mismatch.
- ASAN + UBSAN fuzzing of the new decoders with exact size buffers (over
70k round trips, over 1M truncated or corrupted streams).
- The arm64 branches were only compile checked.
- Behavior changed: No (written format unchanged; builds without AVX2 / NEON
can additionally read kPfor128 / kPfor256 data)
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
