naivedogger commented on PR #3675: URL: https://github.com/apache/fluss/pull/3675#issuecomment-5000659039
Thanks @zuston for your comment! For some context: I'm trying to help enhance Fluss's Flink connector, and this feature is similar to Paimon's [Large Scale Lookup (Fixed Bucket)](https://paimon.apache.org/docs/master/flink/sql-lookup/#large-scale-lookup-fixed-bucket) — it only kicks in on Flink 2.0+ and when the join keys contain all bucket keys. **What shuffle lookup buys us:** rows of the same bucket are routed to the same lookup subtask, so each subtask only deals with its own buckets. This way, when partial caching is enabled, each subtask only needs to cache its own slice of the table, giving a potentially better cache hit ratio and less duplicated cached data across subtasks. Here the cache refers to [Flink's built-in lookup caching](https://nightlies.apache.org/flink/flink-docs-stable/docs/sql/reference/queries/hints/), not a Fluss-specific one — that's probably why it was hard to spot. `FULL` caching isn't supported yet; I'm planning to add it as a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
