Zerolzj commented on issue #5436:
URL: https://github.com/apache/paimon/issues/5436#issuecomment-5550987251
Thanks for the work in this issue and in #5445. I am exploring another
interoperability use
case and would like to clarify the intended scope of `bucket-function.type`.
The use case is to adopt files from an existing fixed-bucket table into
Paimon snapshots without
rewriting or reshuffling the data first. The legacy table already has a
stable bucket-key encoding
and deterministic bucket algorithm, but that algorithm is not one of
`default`, `mod`, or `hive`.
The same algorithm is also needed for subsequent Paimon writes, key lookup,
bucket predicate
pruning, and bucket-aware joins.
I noticed that #5436 originally proposed making the bucket hash function
pluggable. During the
review of #5445, a reviewer asked where a custom `HashFunction`
implementation would be loaded,
and the answer was that Paimon could add other built-in functions instead,
so loading an end-user
class was unnecessary. Later `mod` and `hive` were indeed added as built-ins.
Is built-in-only still the intended long-term boundary, or would the
community consider a
cross-engine, identifier-based extension point for this legacy-layout use
case?
The extension I have in mind would not persist a Java class name in table
metadata:
1. A table stores an immutable semantic identifier for its bucket function.
2. Each runtime maps that identifier to a native implementation (for example
Java through the
existing Factory discovery mechanism, and Python/Rust/C++ through their
own registries).
3. The algorithm contract is defined by cross-language golden vectors,
including logical types,
null handling, key encoding, signed arithmetic, and bucket-count behavior.
4. Writers and every operation that computes a bucket use the same resolved
implementation.
5. A runtime without the implementation must not write data or perform
lookup, bucket pruning,
bucketed joins, or report compatible partitioning. A full scan may
conservatively continue
without a bucket optimization.
6. Changing an algorithm requires a new semantic identifier; an existing
table cannot mutate it.
I see three possible directions:
- continue accepting only generally useful algorithms as Paimon built-ins;
- add the identifier-based extension point above;
- keep custom routing entirely in an execution-engine integration and
disable all bucket-aware
optimizations for such tables.
The second direction enables zero-copy adoption and keeps vendor-specific
algorithms outside
Paimon Core, but it also creates a deployment and portability contract
across runtimes. The first
direction has the strongest portability, but requires every legacy algorithm
to become a Paimon
built-in. The third direction avoids a Core API, but cannot provide portable
SDK writes and key
lookups.
Would maintainers prefer one of these directions? If an identifier-based
provider is considered
reasonable, would this public configuration/API change require a PIP before
a prototype PR?
I can provide a small prototype and compatibility tests after the preferred
boundary is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
