Subject: Re: [Bug?] Impala 4.5 Iceberg table loading failure on Tencent 
Cloud COS (cosn://)

Hi Zoltan,

Thank you for the reminder. I have just filed a Jira ticket to track this 
issue:https://issues.apache.org/jira/browse/IMPALA-14659




For the convenience of the mailing list, I am copy-pasting the core stack trace 
from the catalogd logs below. As mentioned earlier, this exception occurs 
even when parallelism is restricted to 1:










I0105 22:28:30.371518 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:28:30.371537 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
I0105 22:28:33.177121 45789 jni-util.cc:321] 
org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata 
for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
        at 
org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and 
block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: 
failed to load 1 paths. Check the catalog server log for more details.
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
        at 
org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
        ... 7 more
I0105 22:28:33.177153 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:28:33.177165 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
W0105 22:28:35.111603 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:28:35.111624 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:28:35.111640 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:28:58.031069 45789 jni-util.cc:321] 
org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata 
for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
        at 
org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and 
block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: 
failed to load 1 paths. Check the catalog server log for more details.
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
        at 
org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
        ... 7 more
I0105 22:28:58.031085 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:28:58.031098 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
W0105 22:29:05.124416 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:29:05.124442 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:29:05.124451 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:29:08.762989 45789 jni-util.cc:321] 
org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata 
for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
        at 
org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and 
block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: 
failed to load 1 paths. Check the catalog server log for more details.
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
        at 
org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
        ... 7 more
I0105 22:29:08.763005 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:29:08.763021 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
W0105 22:29:35.137226 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:29:35.137250 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:29:35.137259 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
W0105 22:30:05.150115 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:30:05.150141 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:30:05.150151 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
W0105 22:30:35.162889 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:30:35.162916 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:30:35.162925 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:30:42.056529 45789 jni-util.cc:321] 
org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata 
for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
        at 
org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and 
block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: 
failed to load 1 paths. Check the catalog server log for more details.
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
        at 
org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
        ... 7 more
I0105 22:30:42.056545 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:30:42.056563 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
I0105 22:31:00.136520 45789 jni-util.cc:321] 
org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata 
for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
        at 
org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and 
block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: 
failed to load 1 paths. Check the catalog server log for more details.
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
        at 
org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
        ... 7 more
I0105 22:31:00.136550 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:31:00.136565 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
W0105 22:31:05.175676 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:31:05.175701 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:31:05.175710 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.
I0105 22:31:14.446766 45789 jni-util.cc:321] 
org.apache.impala.catalog.IcebergTableLoadingException: Error loading metadata 
for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:460)
        at 
org.apache.impala.catalog.TableLoader.load(TableLoader.java:145)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:251)
        at 
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:247)
        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.impala.catalog.TableLoadingException: Loading file and 
block metadata for 1 paths for table db_on_cos_test.iceberg_cos_employee_test: 
failed to load 1 paths. Check the catalog server log for more details.
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.loadInternal(ParallelFileMetadataLoader.java:211)
        at 
org.apache.impala.catalog.ParallelFileMetadataLoader.load(ParallelFileMetadataLoader.java:140)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:827)
        at 
org.apache.impala.catalog.HdfsTable.loadFileMetadataForPartitions(HdfsTable.java:780)
        at 
org.apache.impala.catalog.HdfsTable.loadAllPartitions(HdfsTable.java:754)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1371)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1262)
        at 
org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1251)
        at 
org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:449)
        ... 7 more
I0105 22:31:14.446784 45789 status.cc:129] IcebergTableLoadingException: Error 
loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
    @          0x10a31d4
    @          0x1c1a17a
    @          0x104e830
    @           0xff5fd7
    @           0xfa8511
    @           0xf83d33
    @           0xf8b7fb
    @          0x14a8a42
    @          0x1cff659
    @          0x264d407
    @     0x7faf31128ea5
    @     0x7faf2dfb29fd
E0105 22:31:14.446801 45789 catalog-server.cc:377] 
IcebergTableLoadingException: Error loading metadata for Iceberg table 
cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths 
for table db_on_cos_test.iceberg_cos_employee_test: failed to load 1 paths. 
Check the catalog server log for more details.
W0105 22:31:35.188406 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/7e4ed6a5-0c24-4a24-4c0147b7-ab48e83f.dmp". Removing it.
W0105 22:31:35.188434 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/931cac7e-3669-48bc-bc68b185-77bd28e4.dmp". Removing it.
W0105 22:31:35.188443 25279 minidump.cc:202] Failed to open file 
"/tmp/minidumps/catalogd/9a535c7e-c762-4df8-0a6e1689-de0fbea0.dmp". Removing it.


         原始邮件
         
       
发件人:Zoltán Borók-Nagy <[email protected]&gt;
发件时间:2026年1月5日 23:19
收件人:dev <[email protected]&gt;
主题:Re: Impala 4.5 Iceberg table on Tencent Cloud COS fails with"failedto load 1 
paths" during metadata refresh.



       
Thanks&nbsp;for&nbsp;your&nbsp;reply.&nbsp;Yeah,&nbsp;if&nbsp;Impala&nbsp;cannot&nbsp;load&nbsp;a&nbsp;table&nbsp;for&nbsp;some&nbsp;reason,
it&nbsp;cannot&nbsp;really&nbsp;do&nbsp;anything&nbsp;with&nbsp;it.

Please&nbsp;note&nbsp;that&nbsp;attachments&nbsp;don't&nbsp;get&nbsp;through&nbsp;Apache&nbsp;mailing&nbsp;lists.

Please&nbsp;copy-paste&nbsp;the&nbsp;stack&nbsp;trace&nbsp;of&nbsp;the&nbsp;exception&nbsp;from&nbsp;the&nbsp;CatalogD&nbsp;logs.
Or,&nbsp;feel&nbsp;free&nbsp;to&nbsp;file 
a&nbsp;Jira&nbsp;ticket&nbsp;about&nbsp;this&nbsp;issue.

Cheers,
&nbsp;&nbsp;&nbsp;&nbsp;Zoltan

On&nbsp;Mon,&nbsp;Jan&nbsp;5,&nbsp;2026&nbsp;at&nbsp;3:51 
PM&nbsp;汲广熙&nbsp;<[email protected]&gt;&nbsp;wrote:

&gt;&nbsp;*Subject:*&nbsp;Re:&nbsp;[Bug?]&nbsp;Impala&nbsp;4.5&nbsp;Iceberg&nbsp;table&nbsp;loading&nbsp;failure&nbsp;on&nbsp;Tencent
&gt;&nbsp;Cloud&nbsp;COS&nbsp;(cosn://)
&gt;
&gt;&nbsp;*Hi,*
&gt;
&gt;&nbsp;Thank&nbsp;you&nbsp;for&nbsp;the&nbsp;guidance.&nbsp;I&nbsp;have&nbsp;an&nbsp;important&nbsp;update&nbsp;regarding&nbsp;the
&gt;&nbsp;backend&nbsp;flags&nbsp;you&nbsp;suggested.
&gt;
&gt;&nbsp;I&nbsp;applied&nbsp;--max_nonhdfs_partitions_parallel_load=1&nbsp;and
&gt;&nbsp;--max_hdfs_partitions_parallel_load=1&nbsp;to&nbsp;my&nbsp;catalogd_flags&nbsp;and&nbsp;restarted
&gt;&nbsp;the&nbsp;service.&nbsp;Unfortunately,&nbsp;the&nbsp;issue&nbsp;persists&nbsp;even&nbsp;with&nbsp;single-threaded
&gt;&nbsp;loading.
&gt;
&gt;&nbsp;It&nbsp;seems&nbsp;that&nbsp;in&nbsp;Impala&nbsp;4.5,&nbsp;ParallelFileMetadataLoader&nbsp;encounters
&gt;&nbsp;difficulties&nbsp;when&nbsp;processing&nbsp;metadata&nbsp;returned&nbsp;by&nbsp;the&nbsp;cosn://&nbsp;driver,
&gt;&nbsp;regardless&nbsp;of&nbsp;the&nbsp;thread&nbsp;count.&nbsp;This&nbsp;suggests&nbsp;there&nbsp;might&nbsp;be&nbsp;a&nbsp;specific
&gt;&nbsp;compatibility&nbsp;issue&nbsp;with&nbsp;how&nbsp;file/block&nbsp;metadata&nbsp;is&nbsp;interpreted&nbsp;for&nbsp;this
&gt;&nbsp;storage&nbsp;scheme.
&gt;
&gt;&nbsp;Below&nbsp;are&nbsp;the&nbsp;raw&nbsp;logs&nbsp;from&nbsp;my&nbsp;session&nbsp;showing&nbsp;that&nbsp;*SHOW&nbsp;METADATA*,
&gt;&nbsp;*INSERT*,&nbsp;and&nbsp;*SELECT*&nbsp;all&nbsp;trigger&nbsp;the&nbsp;same&nbsp;metadata&nbsp;loading&nbsp;error:
&gt;
&gt;&nbsp;======================================
&gt;&nbsp;=======================================================================================================
&gt;&nbsp;[9.192.163.17:21050]&nbsp;db_on_cos_test&gt;&nbsp;SHOW&nbsp;METADATA&nbsp;TABLES&nbsp;IN
&gt;&nbsp;iceberg_cos_employee_test;
&gt;&nbsp;Query:&nbsp;SHOW&nbsp;METADATA&nbsp;TABLES&nbsp;IN&nbsp;iceberg_cos_employee_test
&gt;&nbsp;2026-01-05&nbsp;22:28:58&nbsp;[Exception]&nbsp;&nbsp;ERROR:&nbsp;Query
&gt;&nbsp;9349f0ef1c788ec2:8548051e00000000&nbsp;failed:
&gt;&nbsp;AnalysisException:&nbsp;Could&nbsp;not&nbsp;load&nbsp;table
&gt;&nbsp;db_on_cos_test.iceberg_cos_employee_test&nbsp;from&nbsp;catalog
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TableLoadingException:&nbsp;Could&nbsp;not&nbsp;load&nbsp;table
&gt;&nbsp;db_on_cos_test.iceberg_cos_employee_test&nbsp;from&nbsp;catalog
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TException:
&gt;&nbsp;TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
&gt;&nbsp;error_msgs:[IcebergTableLoadingException:&nbsp;Error&nbsp;loading&nbsp;metadata&nbsp;for
&gt;&nbsp;Iceberg&nbsp;table
&gt;&nbsp;cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TableLoadingException:&nbsp;Loading&nbsp;file&nbsp;and&nbsp;block&nbsp;metadata&nbsp;for&nbsp;1
&gt;&nbsp;paths&nbsp;for&nbsp;table&nbsp;db_on_cos_test.iceberg_cos_employee_test:&nbsp;failed&nbsp;to&nbsp;load&nbsp;1
&gt;&nbsp;paths.&nbsp;Check&nbsp;the&nbsp;catalog&nbsp;server&nbsp;log&nbsp;for&nbsp;more&nbsp;details.]),&nbsp;lookup_status:OK)
&gt;
&gt;&nbsp;[9.192.163.17:21050]&nbsp;db_on_cos_test&gt;&nbsp;INSERT&nbsp;INTO
&gt;&nbsp;iceberg_cos_employee_test&nbsp;VALUES&nbsp;(1,&nbsp;'张三',&nbsp;'研发部',&nbsp;15001.5,&nbsp;now());
&gt;&nbsp;Query:&nbsp;INSERT&nbsp;INTO&nbsp;iceberg_cos_employee_test&nbsp;VALUES&nbsp;(1,&nbsp;'张三',&nbsp;'研发部',
&gt;&nbsp;15001.5,&nbsp;now())
&gt;&nbsp;Query&nbsp;submitted&nbsp;at:&nbsp;2026-01-05&nbsp;22:29:08&nbsp;(Coordinator:
&gt;&nbsp;http://9.192.163.17:8080)
&gt;&nbsp;2026-01-05&nbsp;22:29:08&nbsp;[Exception]&nbsp;&nbsp;ERROR:&nbsp;Query
&gt;&nbsp;ad4f18bf73bf1157:772c7c7600000000&nbsp;failed:
&gt;&nbsp;AnalysisException:&nbsp;org.apache.impala.catalog.TableLoadingException:&nbsp;Could
&gt;&nbsp;not&nbsp;load&nbsp;table&nbsp;db_on_cos_test.iceberg_cos_employee_test&nbsp;from&nbsp;catalog
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TableLoadingException:&nbsp;Could&nbsp;not&nbsp;load&nbsp;table
&gt;&nbsp;db_on_cos_test.iceberg_cos_employee_test&nbsp;from&nbsp;catalog
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TException:
&gt;&nbsp;TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
&gt;&nbsp;error_msgs:[IcebergTableLoadingException:&nbsp;Error&nbsp;loading&nbsp;metadata&nbsp;for
&gt;&nbsp;Iceberg&nbsp;table
&gt;&nbsp;cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TableLoadingException:&nbsp;Loading&nbsp;file&nbsp;and&nbsp;block&nbsp;metadata&nbsp;for&nbsp;1
&gt;&nbsp;paths&nbsp;for&nbsp;table&nbsp;db_on_cos_test.iceberg_cos_employee_test:&nbsp;failed&nbsp;to&nbsp;load&nbsp;1
&gt;&nbsp;paths.&nbsp;Check&nbsp;the&nbsp;catalog&nbsp;server&nbsp;log&nbsp;for&nbsp;more&nbsp;details.]),&nbsp;lookup_status:OK)
&gt;
&gt;&nbsp;[9.192.163.17:21050]&nbsp;db_on_cos_test&gt;&nbsp;select&nbsp;*&nbsp;from
&gt;&nbsp;iceberg_cos_employee_test;
&gt;&nbsp;Query:&nbsp;select&nbsp;*&nbsp;from&nbsp;iceberg_cos_employee_test
&gt;&nbsp;Query&nbsp;submitted&nbsp;at:&nbsp;2026-01-05&nbsp;22:30:42&nbsp;(Coordinator:
&gt;&nbsp;http://9.192.163.17:8080)
&gt;&nbsp;2026-01-05&nbsp;22:30:42&nbsp;[Exception]&nbsp;&nbsp;ERROR:&nbsp;Query
&gt;&nbsp;274fea8afd3f9dde:8d9847ab00000000&nbsp;failed:
&gt;&nbsp;AnalysisException:&nbsp;Failed&nbsp;to&nbsp;load&nbsp;metadata&nbsp;for&nbsp;table:
&gt;&nbsp;'iceberg_cos_employee_test'
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TableLoadingException:&nbsp;Could&nbsp;not&nbsp;load&nbsp;table
&gt;&nbsp;db_on_cos_test.iceberg_cos_employee_test&nbsp;from&nbsp;catalog
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TException:
&gt;&nbsp;TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
&gt;&nbsp;error_msgs:[IcebergTableLoadingException:&nbsp;Error&nbsp;loading&nbsp;metadata&nbsp;for
&gt;&nbsp;Iceberg&nbsp;table
&gt;&nbsp;cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
&gt;&nbsp;CAUSED&nbsp;BY:&nbsp;TableLoadingException:&nbsp;Loading&nbsp;file&nbsp;and&nbsp;block&nbsp;metadata&nbsp;for&nbsp;1
&gt;&nbsp;paths&nbsp;for&nbsp;table&nbsp;db_on_cos_test.iceberg_cos_employee_test:&nbsp;failed&nbsp;to&nbsp;load&nbsp;1
&gt;&nbsp;paths.&nbsp;Check&nbsp;the&nbsp;catalog&nbsp;server&nbsp;log&nbsp;for&nbsp;more&nbsp;details.]),&nbsp;lookup_status:OK)
&gt;&nbsp;======================================
&gt;&nbsp;=======================================================================================================
&gt;
&gt;&nbsp;I&nbsp;have&nbsp;attached&nbsp;the&nbsp;full&nbsp;*catalogd&nbsp;logs*&nbsp;to&nbsp;this&nbsp;email&nbsp;for&nbsp;further
&gt;&nbsp;investigation
&gt;
&gt;
&gt;&nbsp;Best&nbsp;regards,
&gt;&nbsp;原始邮件
&gt;&nbsp;------------------------------
&gt;&nbsp;发件人:Zoltán&nbsp;Borók-Nagy&nbsp;<[email protected]&gt;
&gt;&nbsp;发件时间:2026年1月5日&nbsp;21:10
&gt;&nbsp;收件人:dev&nbsp;<[email protected]&gt;
&gt;&nbsp;主题:Re:&nbsp;Impala&nbsp;4.5&nbsp;Iceberg&nbsp;table&nbsp;on&nbsp;Tencent&nbsp;Cloud&nbsp;COS&nbsp;fails&nbsp;with&nbsp;"failedto
&gt;&nbsp;load&nbsp;1&nbsp;paths"&nbsp;during&nbsp;metadata&nbsp;refresh.
&gt;
&gt;&nbsp;Thanks&nbsp;for&nbsp;reaching&nbsp;out.
&gt;
&gt;&nbsp;Please&nbsp;file&nbsp;a&nbsp;ticket&nbsp;about&nbsp;the&nbsp;error&nbsp;with&nbsp;stack&nbsp;traces&nbsp;and&nbsp;all
&gt;&nbsp;necessary&nbsp;information&nbsp;(snippets&nbsp;from&nbsp;cagalogd&nbsp;logs)&nbsp;to&nbsp;further&nbsp;debug&nbsp;this.
&gt;
&gt;&nbsp;&gt;&nbsp;I&nbsp;tried&nbsp;to&nbsp;apply&nbsp;a&nbsp;workaround&nbsp;by&nbsp;setting&nbsp;metadata_loader_parallelism=1
&gt;&nbsp;via&nbsp;ALTER&nbsp;TABLE
&gt;&nbsp;We&nbsp;don't&nbsp;have&nbsp;such&nbsp;a&nbsp;table&nbsp;property.&nbsp;Was&nbsp;it&nbsp;suggested&nbsp;by&nbsp;Gemini?
&gt;
&gt;&nbsp;&gt;&nbsp;Question:&nbsp;Is&nbsp;there&nbsp;a&nbsp;way&nbsp;to&nbsp;globally&nbsp;set&nbsp;metadata_loader_parallelism=1
&gt;&nbsp;for&nbsp;the&nbsp;entire&nbsp;Catalogd,&nbsp;or&nbsp;perhaps&nbsp;a&nbsp;flag&nbsp;to&nbsp;force&nbsp;Impala&nbsp;to&nbsp;ignore&nbsp;block
&gt;&nbsp;location&nbsp;errors&nbsp;for&nbsp;object&nbsp;storage&nbsp;during&nbsp;the&nbsp;initial&nbsp;load?
&gt;&nbsp;I&nbsp;suspect&nbsp;a&nbsp;different&nbsp;issue&nbsp;than&nbsp;a&nbsp;concurrency&nbsp;bug,&nbsp;but&nbsp;FWIW&nbsp;Catalogd&nbsp;has
&gt;&nbsp;backend&nbsp;flags&nbsp;"max_hdfs_partitions_parallel_load"&nbsp;and
&gt;&nbsp;"max_nonhdfs_partitions_parallel_load"&nbsp;which&nbsp;you&nbsp;can&nbsp;set:
&gt;
&gt;&nbsp;https://github.com/apache/impala/blob/master/be/src/catalog/catalog.cc#L39-L45
&gt;&nbsp;This&nbsp;affects&nbsp;file&nbsp;metadata&nbsp;loading&nbsp;of&nbsp;non-Iceberg&nbsp;tables&nbsp;as&nbsp;well,&nbsp;but&nbsp;it
&gt;&nbsp;can&nbsp;be&nbsp;useful&nbsp;to&nbsp;validate&nbsp;the&nbsp;bug.
&gt;
&gt;&nbsp;We&nbsp;improved&nbsp;Iceberg&nbsp;table&nbsp;loading&nbsp;since&nbsp;4.5,&nbsp;and&nbsp;fixed&nbsp;a&nbsp;few&nbsp;issues,&nbsp;so
&gt;&nbsp;there's&nbsp;a&nbsp;chance&nbsp;that&nbsp;the&nbsp;issue&nbsp;you&nbsp;ran&nbsp;into&nbsp;is&nbsp;already&nbsp;fixed.
&gt;
&gt;&nbsp;Would&nbsp;it&nbsp;be&nbsp;possible&nbsp;for&nbsp;you&nbsp;to&nbsp;try&nbsp;out&nbsp;the&nbsp;current&nbsp;Impala&nbsp;master?
&gt;&nbsp;I&nbsp;think&nbsp;it&nbsp;is&nbsp;time&nbsp;for&nbsp;us&nbsp;to&nbsp;do&nbsp;another&nbsp;upstream&nbsp;release,&nbsp;but&nbsp;it&nbsp;would&nbsp;be
&gt;&nbsp;nice&nbsp;if&nbsp;we&nbsp;can&nbsp;make&nbsp;sure&nbsp;your&nbsp;use&nbsp;case&nbsp;will&nbsp;work&nbsp;well&nbsp;with&nbsp;the&nbsp;newest
&gt;&nbsp;release.
&gt;
&gt;&nbsp;Cheers,
&gt;&nbsp;&nbsp;&nbsp;&nbsp;Zoltan
&gt;
&gt;
&gt;&nbsp;On&nbsp;Mon,&nbsp;Jan&nbsp;5,&nbsp;2026&nbsp;at&nbsp;1:35 
PM&nbsp;汲广熙&nbsp;<[email protected]&gt;&nbsp;wrote:
&gt;
&gt;&nbsp;&gt;&nbsp;Subject:&amp;nbsp;Re:&nbsp;[Bug?]&nbsp;Impala&nbsp;4.5&nbsp;Iceberg&nbsp;table&nbsp;loading&nbsp;failure&nbsp;on
&gt;&nbsp;&gt;&nbsp;Tencent&nbsp;Cloud&nbsp;COS&nbsp;(cosn://)
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Hi,
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Thank&nbsp;you&nbsp;for&nbsp;the&nbsp;suggestions.&nbsp;I&nbsp;have&nbsp;some&nbsp;critical&nbsp;updates&nbsp;regarding&nbsp;the
&gt;&nbsp;&gt;&nbsp;TableLoadingException:&nbsp;failed&nbsp;to&nbsp;load&nbsp;1&nbsp;paths&amp;nbsp;issue&nbsp;on&nbsp;Tencent&nbsp;Cloud
&gt;&nbsp;&gt;&nbsp;COS&nbsp;(cosn://).
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;I&nbsp;have&nbsp;already&nbsp;enabled&nbsp;Local&nbsp;Catalog
&gt;
&gt;&nbsp;&gt;&nbsp;Mode&amp;nbsp;(--use_local_catalog=true&amp;nbsp;and&nbsp;minimal&amp;nbsp;topic&nbsp;mode),&nbsp;but
&gt;&nbsp;&gt;&nbsp;the&nbsp;issue&nbsp;persists.
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;Based&nbsp;on&nbsp;an&nbsp;analysis&nbsp;provided&nbsp;by&nbsp;Gemini&nbsp;(Google's&nbsp;AI&nbsp;model)&amp;nbsp;regarding
&gt;&nbsp;&gt;&nbsp;the&nbsp;stack&nbsp;trace,&nbsp;it&nbsp;was&nbsp;suggested&nbsp;that&nbsp;the&nbsp;error&nbsp;in
&gt;
&gt;&nbsp;&gt;&nbsp;ParallelFileMetadataLoader.loadInternal&amp;nbsp;might&nbsp;be&nbsp;caused&nbsp;by&nbsp;concurrency
&gt;
&gt;&nbsp;&gt;&nbsp;issues&nbsp;when&nbsp;handling&nbsp;object&nbsp;storage&nbsp;paths.&nbsp;Following&nbsp;this&nbsp;logic,&nbsp;I&nbsp;tried&nbsp;to
&gt;
&gt;&nbsp;&gt;&nbsp;apply&nbsp;a&nbsp;workaround&nbsp;by&nbsp;setting&nbsp;metadata_loader_parallelism=1&amp;nbsp;via&nbsp;ALTER
&gt;&nbsp;&gt;&nbsp;TABLE.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;However,&nbsp;the&nbsp;command&nbsp;failed&nbsp;with&nbsp;the&nbsp;same&nbsp;AnalysisException&amp;nbsp;and
&gt;&nbsp;&gt;&nbsp;TableLoadingException.&nbsp;It&nbsp;seems&nbsp;Impala&nbsp;falls&nbsp;into&nbsp;a&nbsp;deadlock&nbsp;state:&nbsp;I
&gt;
&gt;&nbsp;&gt;&nbsp;cannot&nbsp;modify&nbsp;the&nbsp;table&nbsp;properties&nbsp;to&nbsp;fix&nbsp;the&nbsp;loading&nbsp;logic&nbsp;because&nbsp;Impala
&gt;
&gt;&nbsp;&gt;&nbsp;fails&nbsp;to&nbsp;load&nbsp;the&nbsp;table&nbsp;metadata&nbsp;even&nbsp;during&nbsp;the&nbsp;ALTER&nbsp;TABLE&amp;nbsp;analysis
&gt;&nbsp;&gt;&nbsp;phase.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Observations:
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;The&nbsp;error&nbsp;originates&nbsp;from&nbsp;ParallelFileMetadataLoader.loadInternal.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Even&nbsp;with&nbsp;Local&nbsp;Catalog&nbsp;enabled,&nbsp;the&nbsp;loader&nbsp;seems&nbsp;unable&nbsp;to&nbsp;handle&nbsp;the
&gt;&nbsp;&gt;&nbsp;missing&nbsp;block&nbsp;location&nbsp;info&nbsp;from&nbsp;the&nbsp;cosn://&amp;nbsp;driver.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Standard&nbsp;Parquet&nbsp;tables&nbsp;work&nbsp;fine;&nbsp;this&nbsp;appears&nbsp;specific&nbsp;to&nbsp;the
&gt;&nbsp;&gt;&nbsp;Iceberg&amp;nbsp;metadata&nbsp;loading&nbsp;path&nbsp;in&nbsp;Impala&nbsp;4.5.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Question:&nbsp;Is&nbsp;there&nbsp;a&nbsp;way&nbsp;to&nbsp;globally&nbsp;set
&gt;&nbsp;&gt;&nbsp;metadata_loader_parallelism=1&amp;nbsp;for&nbsp;the&nbsp;entire&nbsp;Catalogd,&nbsp;or&nbsp;perhaps&nbsp;a
&gt;&nbsp;&gt;&nbsp;flag&nbsp;to&nbsp;force&nbsp;Impala&nbsp;to&nbsp;ignore&nbsp;block&nbsp;location&nbsp;errors&nbsp;for&nbsp;object&nbsp;storage
&gt;&nbsp;&gt;&nbsp;during&nbsp;the&nbsp;initial&nbsp;load?
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;I&nbsp;am&nbsp;happy&nbsp;to&nbsp;provide&nbsp;a&nbsp;full&nbsp;stack&nbsp;trace&nbsp;or&nbsp;open&nbsp;a&nbsp;JIRA&nbsp;ticket,&nbsp;as&nbsp;this
&gt;&nbsp;&gt;&nbsp;currently&nbsp;makes&nbsp;Iceberg&nbsp;on&nbsp;COS&nbsp;unusable&nbsp;in&nbsp;our&nbsp;environment.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;============================================================================================================
&gt;&nbsp;&gt;&nbsp;When&nbsp;I&nbsp;run&nbsp;a&nbsp;simple&nbsp;SELECT&nbsp;*or&nbsp;INSERT&nbsp;INTOquery&nbsp;on&nbsp;this&nbsp;table,&nbsp;I&nbsp;get&nbsp;the
&gt;&nbsp;&gt;&nbsp;following&nbsp;error:
&gt;&nbsp;&gt;&nbsp;AnalysisException:&nbsp;Failed&nbsp;to&nbsp;load&nbsp;metadata&nbsp;for&nbsp;table:
&gt;&nbsp;&gt;&nbsp;'iceberg_cos_employee_test'
&gt;&nbsp;&gt;&nbsp;...
&gt;&nbsp;&gt;&nbsp;IcebergTableLoadingException:&nbsp;Error&nbsp;loading&nbsp;metadata&nbsp;for&nbsp;Iceberg&nbsp;table
&gt;&nbsp;&gt;&nbsp;cosn://cdnlogtest-1252412955/impala_test_db/iceberg_cos_employee_test
&gt;&nbsp;&gt;&nbsp;...
&gt;&nbsp;&gt;&nbsp;Loading&nbsp;file&nbsp;and&nbsp;block&nbsp;metadata&nbsp;for&nbsp;1&nbsp;paths&nbsp;for&nbsp;table&nbsp;...:&nbsp;failed&nbsp;to&nbsp;load
&gt;&nbsp;&gt;&nbsp;1&nbsp;paths.
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;============================================================================================================
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Best&nbsp;regards,
&gt;&nbsp;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;原始邮件
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;发件人:Zoltán&nbsp;Borók-Nagy&nbsp;<[email protected]&amp;gt;
&gt;&nbsp;&gt;&nbsp;发件时间:2025年12月23日&nbsp;23:52
&gt;&nbsp;&gt;&nbsp;收件人:dev&nbsp;<[email protected]&amp;gt;
&gt;&nbsp;&gt;&nbsp;主题:Re:&nbsp;Advice&nbsp;and&nbsp;Considerations&nbsp;for&nbsp;Building&nbsp;an&nbsp;Impala
&gt;&nbsp;&gt;&nbsp;Compute-StorageSeparated&nbsp;Architecture
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Hi,
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;Just&amp;nbsp;a&amp;nbsp;few&amp;nbsp;tips&amp;nbsp;off&amp;nbsp;the&amp;nbsp;top&amp;nbsp;of&amp;nbsp;my&amp;nbsp;head:
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;-&amp;nbsp;use&amp;nbsp;dedicated&amp;nbsp;coordinators&amp;nbsp;and&amp;nbsp;executors,&amp;nbsp;rule&amp;nbsp;of&amp;nbsp;thumb&amp;nbsp;for
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;coordinator:executor&amp;nbsp;ratio&amp;nbsp;is&amp;nbsp;1:50.&amp;nbsp;Though&amp;nbsp;for&amp;nbsp;HA&amp;nbsp;you&amp;nbsp;probably&amp;nbsp;want&amp;nbsp;&amp;gt;1
&gt;&nbsp;&gt;&nbsp;coordinators.
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;-&amp;nbsp;use&amp;nbsp;local&amp;nbsp;catalog&amp;nbsp;mode&amp;nbsp;(aka&amp;nbsp;On-demand&amp;nbsp;metadata):
&gt;&nbsp;&gt;&nbsp;https://impala.apache.org/docs/build/html/topics/impala_metadata.html
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;-&amp;nbsp;enabling&amp;nbsp;remote&amp;nbsp;data&amp;nbsp;cache&amp;nbsp;(with&amp;nbsp;SSD&amp;nbsp;disks)&amp;nbsp;is&amp;nbsp;essential&amp;nbsp;in
&gt;&nbsp;&gt;&nbsp;compute-storage&amp;nbsp;separated&amp;nbsp;setup:
&gt;&nbsp;&gt;&nbsp;https://impala.apache.org/docs/build/html/topics/impala_data_cache.html
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;What&amp;nbsp;table&amp;nbsp;format&amp;nbsp;/&amp;nbsp;file&amp;nbsp;format&amp;nbsp;are&amp;nbsp;you&amp;nbsp;planning&amp;nbsp;to&amp;nbsp;use?
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;If&amp;nbsp;table&amp;nbsp;format&amp;nbsp;is&amp;nbsp;Iceberg,&amp;nbsp;make&amp;nbsp;sure&amp;nbsp;you&amp;nbsp;use&amp;nbsp;the&amp;nbsp;latest&amp;nbsp;Impala&amp;nbsp;as&amp;nbsp;we
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;continuously&amp;nbsp;improve&amp;nbsp;Impala's&amp;nbsp;performance&amp;nbsp;on&amp;nbsp;Iceberg.
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;File&amp;nbsp;format:&amp;nbsp;Impala&amp;nbsp;most&amp;nbsp;efficiently&amp;nbsp;works&amp;nbsp;on&amp;nbsp;Parquet&amp;nbsp;files.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Avoid&amp;nbsp;small&amp;nbsp;file&nbsp;issues:
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;-&amp;nbsp;choose&amp;nbsp;proper&amp;nbsp;partitioning&amp;nbsp;for&amp;nbsp;your&amp;nbsp;data,&amp;nbsp;i.e.&amp;nbsp;avoid&amp;nbsp;too&amp;nbsp;coarse-grained
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;and&amp;nbsp;too&amp;nbsp;fine-grained&amp;nbsp;partitioning.&amp;nbsp;I.e.&amp;nbsp;you&amp;nbsp;probably&amp;nbsp;want&amp;nbsp;more&amp;nbsp;than&amp;nbsp;200&amp;nbsp;MB
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;data&amp;nbsp;per&amp;nbsp;partition,&amp;nbsp;but&amp;nbsp;probably&amp;nbsp;less&amp;nbsp;than&amp;nbsp;20&amp;nbsp;GB.
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;-&amp;nbsp;compact&amp;nbsp;your&amp;nbsp;tables&amp;nbsp;regularly,&amp;nbsp;for&amp;nbsp;Iceberg&amp;nbsp;tables&amp;nbsp;Impala&amp;nbsp;has&amp;nbsp;the
&gt;&nbsp;&gt;&nbsp;OPTIMIZE&amp;nbsp;statement:
&gt;&nbsp;&gt;&nbsp;https://impala.apache.org/docs/build/html/topics/impala_iceberg.html
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;I&amp;nbsp;hope&amp;nbsp;others&amp;nbsp;chime&amp;nbsp;in&amp;nbsp;as&amp;nbsp;well.
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;We&amp;nbsp;would&amp;nbsp;love&amp;nbsp;to&amp;nbsp;hear&amp;nbsp;back&amp;nbsp;about&amp;nbsp;your&amp;nbsp;experiences,&amp;nbsp;and&amp;nbsp;feel&amp;nbsp;free&amp;nbsp;to&amp;nbsp;open
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;tickets&amp;nbsp;for&amp;nbsp;Impala&amp;nbsp;if&amp;nbsp;you&amp;nbsp;run&amp;nbsp;into&amp;nbsp;any&amp;nbsp;issue:
&gt;&nbsp;&gt;&nbsp;https://issues.apache.org/jira/projects/IMPALA/issues
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;Cheers,
&gt;&nbsp;&gt;&nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;Zoltan
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;On&amp;nbsp;Tue,&amp;nbsp;Dec&amp;nbsp;23,&amp;nbsp;2025&amp;nbsp;at&amp;nbsp;8:23
 AM&amp;nbsp;汲广熙&amp;nbsp;<
&gt;&nbsp;&gt;&nbsp;[email protected]&amp;gt;&amp;nbsp;wrote:
&gt;&nbsp;&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Dear&amp;nbsp;Impala&amp;nbsp;Team,
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;I&amp;nbsp;hope&amp;nbsp;this&amp;nbsp;message&amp;nbsp;finds&amp;nbsp;you&amp;nbsp;well.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;I&amp;nbsp;am&amp;nbsp;currently&amp;nbsp;planning&amp;nbsp;to&amp;nbsp;build&amp;nbsp;a&amp;nbsp;compute-storage&amp;nbsp;separated&amp;nbsp;architecture​
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;based&amp;nbsp;on&amp;nbsp;Apache&amp;nbsp;Impala.&amp;nbsp;In&amp;nbsp;this&amp;nbsp;setup:
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Compute&amp;nbsp;layer:&amp;nbsp;Apache&amp;nbsp;Impala&amp;nbsp;will&amp;nbsp;be&amp;nbsp;used&amp;nbsp;for&amp;nbsp;SQL&amp;nbsp;query&amp;nbsp;execution.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Storage&amp;nbsp;layer:&amp;nbsp;Tencent&amp;nbsp;Cloud&amp;nbsp;Object&amp;nbsp;Storage&amp;nbsp;(COS)&amp;nbsp;will&amp;nbsp;serve&amp;nbsp;as&amp;nbsp;the
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;backend&amp;nbsp;storage.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Data&amp;nbsp;ingestion:&amp;nbsp;Kafka&amp;nbsp;will&amp;nbsp;be&amp;nbsp;used&amp;nbsp;to&amp;nbsp;stream&amp;nbsp;data&amp;nbsp;into&amp;nbsp;the&amp;nbsp;system.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Monitoring&amp;nbsp;&amp;amp;amp;&amp;nbsp;visualization:&amp;nbsp;Grafana&amp;nbsp;will&amp;nbsp;be&amp;nbsp;used&amp;nbsp;to&amp;nbsp;display
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;operational&amp;nbsp;and&amp;nbsp;performance&amp;nbsp;metrics.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Could&amp;nbsp;you&amp;nbsp;please&amp;nbsp;provide&amp;nbsp;some&amp;nbsp;recommendations&amp;nbsp;and&amp;nbsp;key&amp;nbsp;considerations​&amp;nbsp;for
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;such&amp;nbsp;an&amp;nbsp;architecture?&amp;nbsp;Specifically,&amp;nbsp;I&amp;nbsp;would&amp;nbsp;appreciate&amp;nbsp;guidance&amp;nbsp;on:
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Best&amp;nbsp;practices&amp;nbsp;for&amp;nbsp;integrating&amp;nbsp;Impala&amp;nbsp;with&amp;nbsp;cloud&amp;nbsp;object&amp;nbsp;storage&amp;nbsp;(COS).
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Performance&amp;nbsp;tuning&amp;nbsp;tips&amp;nbsp;for&amp;nbsp;Impala&amp;nbsp;in&amp;nbsp;a&amp;nbsp;disaggregated&amp;nbsp;environment.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Any&amp;nbsp;known&amp;nbsp;limitations&amp;nbsp;or&amp;nbsp;compatibility&amp;nbsp;issues&amp;nbsp;when&amp;nbsp;using&amp;nbsp;COS&amp;nbsp;as&amp;nbsp;storage.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Recommended&amp;nbsp;configurations&amp;nbsp;for&amp;nbsp;Kafka-to-Impala&amp;nbsp;data&amp;nbsp;pipelines.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Monitoring&amp;nbsp;strategies&amp;nbsp;for&amp;nbsp;tracking&amp;nbsp;query&amp;nbsp;performance&amp;nbsp;and&amp;nbsp;resource&amp;nbsp;usage&amp;nbsp;in
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Grafana.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;
&gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Thank&amp;nbsp;you&amp;nbsp;very&amp;nbsp;much&amp;nbsp;for&amp;nbsp;your&amp;nbsp;support&amp;nbsp;and&amp;nbsp;advice.&amp;nbsp;I&amp;nbsp;look&amp;nbsp;forward&amp;nbsp;to&amp;nbsp;your
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;reply.
&gt;&nbsp;&gt;&nbsp;&amp;gt;
&gt;&nbsp;&gt;&nbsp;&amp;gt;&amp;nbsp;Best&amp;nbsp;regards
&gt;
&gt;
&gt;

Reply via email to