vanliu-tx opened a new issue #3907: URL: https://github.com/apache/iceberg/issues/3907
when we update iceberg 0.9.0 to 0.12.1, we found an issue that HiveCatalog#listTables(Namespace) takes minutes in 0.12.1 in our production env, compared to ~0.1s in 0.9.0. In our production, this method takes 28s for one namespace contains 517 tables, 75s for one namespace contains 1301 tables, more than half an hour for one namespace with 27000+ tables. In this issue(apache#1835), client.getTableObjectsByName was added to filter iceberg tables, but this method is really slow if there are thousands tables. In our product env, we separate different table types in different namespace, so there is no need to filter iceberg tables. ``` List<Table> tableObjects = clients.run(client -> client.getTableObjectsByName(database, tableNames)); ``` We should add a flag(hive properties) to indicate whether we need to filter iceberg tables in namespace. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
