Zerolzj commented on PR #10662: URL: https://github.com/apache/paimon/pull/10662#issuecomment-6414762016
@JingsongLi Yes. The requirement comes from production data stored in our in-house database. It uses a fixed hash-based bucket assignment with an already well-distributed layout. We would like to adopt this data in Paimon while preserving that layout, rather than reshuffling the entire dataset solely to match a different built-in bucket algorithm. Avoiding that re-bucketing shuffle would reduce migration and ingestion overhead. Subsequent writes and bucket-aware reads also need to use the same deterministic function to keep bucket assignment consistent and pruning correct. To clarify, this is about avoiding redistribution solely for re-bucketing, not guaranteeing a shuffle-free ingestion pipeline or shuffle-free joins. This PR provides the bucket-function extension; importing existing files is outside its scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
