zhuxiangyi opened a new pull request, #9274:
URL: https://github.com/apache/paimon/pull/9274
### Purpose
This PR introduces the Python `FieldHllSketchAgg` implementation and aligns
the
Python HyperLogLog aggregation behavior with the Java implementation.
The changes include:
- Add `hll_sketch` field aggregation support, mirroring Java's
`HllSketchUtil.union` (the union is seeded from the input sketch, and the
result is emitted as a compact `HLL_4` sketch).
- Add `VARBINARY` type validation, matching `FieldHllSketchAggFactory`.
- Keep `datasketches` an optional dependency: the import is lazy and raises a
clear `pypaimon[hll-sketch]` hint, following the `theta_sketch` precedent.
- Register `hll_sketch` in
`merge_engine_support._AGGREGATION_SUPPORTED_AGG_FUNCS`
and narrow the "unsupported aggregators" message to `rbm64`.
- Pin cross-language compatibility with fixtures produced by Java's
`HllSketchUtil` (datasketches-java 4.2.0, lgK=12), verified in both
directions.
- Extend the `RegistrationTest.EXPECTED` list, which was still pinned to the
10
aggregators from the initial scope in #7952, to all 20, and add a test
asserting the registry and the merge-engine guard list stay in sync.
`rbm64` remains deferred: Java serializes it with `Roaring64Bitmap`'s
private ART
format, which has no portable counterpart in `pyroaring`, so the bytes are
not
interchangeable in either direction. The module docstring now records that
reason.
### Tests
- `python -m pytest pypaimon/tests/test_field_aggregators.py -q`
- `python -m pytest pypaimon/tests/test_aggregation_e2e.py -q`
- `python -m pytest pypaimon/tests/test_aggregation_merge_function.py -q`
- `python -m pytest pypaimon/tests/test_field_aggregator_registry.py -q`
- `python -m pytest pypaimon/tests/test_merge_engine_dispatch.py -q`
- `python -m pytest pypaimon/tests/test_optional_datasketches_dependency.py
-q`
- `flake8 --config=dev/cfg.ini`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
