swapna267 commented on PR #17280: URL: https://github.com/apache/iceberg/pull/17280#issuecomment-5075092656
Thanks @Guosmilesmile . Yeah , i see some scenarios where this feature can be useful on source in our use cases when CDC is enabled on MySQL tables. I am not fully aware of how Lookup source interacts with sources in general. I have concerns about the table size. In brief, as i understand full-cache is loading all data into memory on each TM right ? Can you include how the Flink Dag looks in this case. Looking to understand, 1. What is the behavior of lookup joins for stores like Mysql ? Is the cache built incrementally, by doing a key lookup ? 2. Lookup keys are converted into Iceberg filter expressions. => Can this be predetermined ? If number of keys is too large, can there be issues with filter expressions ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
