Guosmilesmile commented on PR #17280: URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5070863548
In our use case, dimension tables are synced from MySQL to Iceberg. Currently, we do lookup joins directly against MySQL. But this creates too many concurrent connections to MySQL, putting a lot of pressure on it. Since the same data already exists in Iceberg, we could do lookup joins directly against Iceberg instead.In our case, the dimension tables are small, so a full-cache (or partial cache) based lookup join should work well enough. For larger dimension tables where full caching isn't feasible, we may need indexing features that are being discussed for Iceberg V4. This can be explored later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
