rahulbsw opened a new issue, #3331:
URL: https://github.com/apache/fluss/issues/3331
### Motivation
Fluss aggregation merge-engine tables currently support exact numeric,
string, boolean, value-selection, and RoaringBitmap byte aggregations. The
aggregation documentation does not currently expose a native approximate
distinct-count aggregation, such as HLL/HyperLogLog, for incremental real-time
analytics.
Approximate distinct counting is useful for workloads such as daily active
users, unique devices, unique sessions, and feature-store metrics. In these
workloads, exact `COUNT(DISTINCT ...)` can be expensive to maintain
continuously, while sketches provide compact mergeable state.
Apache DataSketches provides sketch implementations with defined
serialization and merge semantics. HLL sketch support fits Fluss' aggregation
merge engine model because Fluss can store serialized sketch bytes and merge
them per primary key, similar to the existing `rbm32` and `rbm64` serialized
RoaringBitmap aggregation functions.
### Solution
Add a new aggregation merge function named `hll_sketch`.
The function should:
- Support only `BYTES` columns.
- Treat input values as serialized Apache DataSketches HLL sketches.
- Merge two non-null sketches using the DataSketches HLL union API.
- Return compact serialized sketch bytes.
- Ignore nulls consistently with existing byte aggregations such as `rbm32`
and `rbm64`.
- Validate function options through the existing aggregation function
parameter validation path.
Example DDL:
```sql
CREATE TABLE daily_user_metrics (
metric_day STRING,
user_hll BYTES,
PRIMARY KEY (metric_day) NOT ENFORCED
) WITH (
'table.merge-engine' = 'aggregation',
'fields.user_hll.agg' = 'hll_sketch'
);
```
This first change should focus on storage-level sketch merging. Users would
still write pre-serialized HLL sketch bytes from a client or compute engine.
Potential follow-up work:
- Add Flink SQL helper functions such as `fluss_hll_build(value)` and
`fluss_hll_estimate(bytes)`.
- Add Spark SQL helper functions with the same semantics.
- Add KLL-based percentile support with a `kll_double_sketch` merge function.
### Anything else?
The implementation can follow the existing RoaringBitmap aggregation pattern:
- Add `HLL_SKETCH` to `AggFunctionType`.
- Add a Java API factory method such as `AggFunctions.HLL_SKETCH()`.
- Add a server-side `FieldHllSketchAgg` and SPI factory.
- Add unit tests in the aggregation row merger test suite.
- Document the function in the aggregation merge-engine docs.
Dependency and packaging notes:
- DataSketches Java depends on DataSketches Memory, so the build and binary
license metadata should be checked.
- `LICENSE-bin` already mentions Apache DataSketches, but the actual module
dependencies still need to be added and verified.
I searched existing issues for DataSketches, HLL, KLL,
`approx_count_distinct`, and `approx_percentile` and did not find a matching
Fluss feature request.
I am willing to submit a PR for the first `hll_sketch` implementation after
maintainers confirm the API name and scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
