vaquarkhan commented on PR #17284: URL: https://github.com/apache/iceberg/pull/17284#issuecomment-5125665406
@RussellSpitzer , This makes sense. One observation from the numbers: as written the gate is size-based (<=1 MB), not format-based, so "always on" would also wrap Avro and local-filesystem reads, where I saw no benefit and it still buffers the whole manifest into memory during planning. So our "check whether it's Parquet first" is the right call and matches the data. That Parquet check also covers V3 and below on its own, since those are Avro-only, and it handles mixed manifests in a V4 table correctly, so I don't think a separate version check is needed. Only caution: local filesystem showed no gain even for Parquet, and my numbers are single-shot on one instance, so a slightly broader run would be worth it before flipping the default. Happy to help validate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
