terrytlu commented on PR #17317: URL: https://github.com/apache/iceberg/pull/17317#issuecomment-5053595131
> How many tables do you have to get an OOM? How many tables do you have to get a timeout, and which component returns the timeout? Why do we want to have a new config for this? Thanks for the questions, @pvary Let me address them point by point: **1. OOM** — Theoretical so far; we haven't hit it in production. The old path loads a full `Table` object per table into client memory via `getTableObjectsByName`, so memory grows unbounded with namespace size — that's the risk. **2. Timeout** — Covered in #17312: a namespace with ~46,000 tables still timed out even at `hive.metastore.client.socket.timeout=900s`. The timeout is raised on the HMS client side, since serializing/transferring that many full `Table` objects exceeds the socket timeout. **3. New config** — `list-iceberg-tables-legacy` is just a defensive fallback. The old path has run stably in production for a long time, so the flag lets users revert if the new server-side-filter path hits an unexpected issue. It's off by default. Happy to drop the flag and keep only the new path if you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
