Yida Wu created IMPALA-15331:
--------------------------------
Summary: Queries fail due to catalog in standby mode in
GetNullPartitionName
Key: IMPALA-15331
URL: https://issues.apache.org/jira/browse/IMPALA-15331
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Yida Wu
Assignee: Yida Wu
In IMPALA-15121, we fixed an issue where queries would fail during a Catalogd
HA failover if an Impalad coordinator tried to fetch metadata from a restarted
catalogd that was currently in standby mode.
However, we are still seeing query failures during failovers with the following
logging:
{code:java}
I20260831 23:48:12.869620 451880 jni-util.cc:335]
de452853deebd4c8:9946356900000000] java.lang.RuntimeException:
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
org.apache.impala.common.InternalException: Error requesting
GetNullPartitionName: Request for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
at
org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2493)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2153)
at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:181)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException:
org.apache.impala.common.InternalException: Error requesting
GetNullPartitionName: Request for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:650)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadNullPartitionKeyValue(CatalogdMetaProvider.java:1439)
at
org.apache.impala.catalog.local.LocalCatalog.getNullPartitionKeyValue(LocalCatalog.java:325)
at
org.apache.impala.catalog.local.LocalFsTable.getNullPartitionKeyValue(LocalFsTable.java:243)
at
org.apache.impala.catalog.local.LocalFsTable.toTHdfsTable(LocalFsTable.java:345)
at
org.apache.impala.catalog.local.LocalFsTable.toThriftDescriptor(LocalFsTable.java:318)
at
org.apache.impala.analysis.DescriptorTable.toThrift(DescriptorTable.java:238)
at
org.apache.impala.analysis.DescriptorTable.toSerializedThrift(DescriptorTable.java:259)
at
org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:3556)
at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:3235)
at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2625)
at
org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2470)
... 2 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.impala.common.InternalException: Error requesting
GetNullPartitionName: Request for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
at
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at
com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:246)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:641)
... 13 more
Caused by: org.apache.impala.common.InternalException: Error requesting
GetNullPartitionName: Request for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
at
org.apache.impala.service.FeSupport.GetNullPartitionName(FeSupport.java:549)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$9.call(CatalogdMetaProvider.java:1446)
at
org.apache.impala.catalog.local.CatalogdMetaProvider$9.call(CatalogdMetaProvider.java:1442)
at
org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:625)
... 13 more
I20260831 23:48:12.869776 451880 status.cc:123]
de452853deebd4c8:9946356900000000] RuntimeException:
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
org.apache.impala.common.InternalException: Error requesting
GetNullPartitionName: Request for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
CAUSED BY: RuntimeException: java.util.concurrent.ExecutionException:
org.apache.impala.common.InternalException: Error requesting
GetNullPartitionName: Request for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
CAUSED BY: ExecutionException: org.apache.impala.common.InternalException:
Error requesting GetNullPartitionName: Request for Catalog service is rejected
since catalogd quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
CAUSED BY: InternalException: Error requesting GetNullPartitionName: Request
for Catalog service is rejected since catalogd
quasar-rcprbj-2.vpc.eng.cloudera.com:26000 is in standby mode
{code}
This error comes from a different code path for GetNullPartitionName than the
previous issue which was on GetPartialCatalogObject. Probably we can introduce
a similar retry idea to fix it, allowing the frontend to retry the fetch
against the new active catalogd without failing the query.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
