gaborkaszab commented on PR #17219: URL: https://github.com/apache/iceberg/pull/17219#issuecomment-4990682484
Hi @HaoXuAI , As I see the new functionality is not wired in to `TableMetada`/`TableMetadataParser`. It's not entirely clear how this is going to be used. Random questions: - I assume this new parser opens a file descriptor to the json input file, one per TableMetadata, and keeps them open until the TableMetadata object is destroyed. If we cache tables does this means we keep the file descriptors open for forever? - For operations that need all the snapshots, like expire snapshots, does this design do 1000 IO to read 1000 snapshots? - This is meant to be an enhancement for memory usage. Would be nice to see numbers what is the order of magnitude we gain here. Also this comes with a tradeoff, would be nice to see what is the read overhead based on the different use cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
