[
https://issues.apache.org/jira/browse/IMPALA-13314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108964#comment-18108964
]
ASF subversion and git services commented on IMPALA-13314:
----------------------------------------------------------
Commit 51b57a77c9f60a70c7b194386d3e1b406e78a643 in impala's branch
refs/heads/master from Nandor Kollar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=51b57a77c ]
IMPALA-13314: Cache HadoopCatalog instances and use holder pattern for
singletons
Previously, a new HadoopCatalog (and its FileIO/filesystem resources)
was created on every table operation via getIcebergCatalog(). This
caused excessive resource consumption when many Iceberg tables shared
the same catalog location.
Changes:
- Introduce a Guava Cache with soft values in IcebergHadoopCatalog
keyed by catalog location string. Tables sharing the same
iceberg.catalog_location now reuse a single HadoopCatalog instance.
Entries are evicted under memory pressure via soft references.
- Replace synchronized lazy initialization with the
initialization-on-demand holder idiom for IcebergHiveCatalog,
IcebergHadoopTables, and IcebergCatalogUtil. The JVM guarantees
thread-safe lazy initialization without synchronization overhead on
every access.
- Convert getIcebergCatalog() to a switch expression, removing the
throws ImpalaRuntimeException declaration. Simplify loadTable()
by dropping the now-unnecessary try-catch.
- Remove unused getLastComputeStatsTimeMs() and
collectFieldIdsWithNdvStats() from IcebergTable.
- Fix logger in IcebergHadoopCatalog (was incorrectly using
IcebergHadoopTables.class).
- Make hadoopTables field final in IcebergHadoopTables.
Testing: Ran existing Iceberg catalog tests.
Assisted-by: Claude Opus 4.6 (Claude Code)
Change-Id: Ibb3a6c8e4f1d2a9e7c5b0f8d3e6a4c2b1d0e9f7a
Reviewed-on: http://gerrit.cloudera.org:8080/24712
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Create a store for HadoopCatalogs to avoid creating a new one for each table
> ----------------------------------------------------------------------------
>
> Key: IMPALA-13314
> URL: https://issues.apache.org/jira/browse/IMPALA-13314
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Gabor Kaszab
> Assignee: Nándor Kollár
> Priority: Minor
> Labels: impala-iceberg, ramp-up
>
> Currently when we create a new Iceberg table in HadoopCatalog we create a new
> HadoopCatalog instance for each of these tables
> [here|https://github.com/apache/impala/blob/4b500a55cbfcdd311a1c766e33849f7ae05a1a8e/fe/src/main/java/org/apache/impala/util/IcebergUtil.java#L145]
> The issue with this is that a catalog object such as HadoopCatalog holds an
> Iceberg FileIO instance where the size of such an instance can be measured in
> MBs in terms of memory consumption. This can blow up the catalog/localCatalog
> memory even if we have empty Iceberg tables in HadoopCatalog.
> So as a solution we should have a kind of HadoopCatalog store, where based on
> a location string we could cache HadoopCatalog objects for later use or cache
> a new HadoopCatalog in the store. With this approach tables under the sane
> HadoopCatalog location would be in the same HadoopCatalog instance and we
> won't end up having as many FileIO instance as many tables we have in
> HadoopCatalog.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
