[
https://issues.apache.org/jira/browse/IMPALA-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Gaal resolved IMPALA-13724.
----------------------------------
Fix Version/s: Impala 4.6.0
Resolution: Fixed
> Impala fails to start up in Dockerised tests when the Docker gateway name is
> resolved as _gateway
> -------------------------------------------------------------------------------------------------
>
> Key: IMPALA-13724
> URL: https://issues.apache.org/jira/browse/IMPALA-13724
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Laszlo Gaal
> Assignee: Laszlo Gaal
> Priority: Critical
> Fix For: Impala 4.6.0
>
>
> When Impala daemons are started inside a Docker container (using the
> dockerised form of bin/start-impala-cluster.py like this):{code}
> $ export DOCKER_NETWORK=impala-dockernet # or use your own name
> $ bin/start-impala-cluster.py --docker_network=${DOCKER_NETWORK}
> {code}
> the gateway generated by Docker networking receives an IP address from
> Docker, but the way this IP address is resolved to a DNS name is
> platform-dependent.
> Unfortunately some newer platforms (e.g. Rocky 9 with Docker 27.5) resolve
> the gateway's name as {code}
> _gateway{code}
> with a leading underscore. This is a problem for Impala: on startup the
> catalog and all coordinators attempt to connect to HMS over Thrift, using a
> Java implementation of Thrift. This Java code relies on a communication
> library that considers the the underscore character invalid within DNS names,
> throws an exception complaining about it, and refuses to connect to HMS
> because of the exception. Failing to connect to HMS is fatal during Impala
> startup, so this problem prevents impala catalog and coordinator roles from
> starting up in such an environment.
> Example coordinator log sequence:
> {code}
> I0201 18:13:30.205049 1 HiveMetaStoreClient.java:279] HMS client
> filtering is enabled.
> E0201 18:13:30.288259 1 MetaStoreUtils.java:259] Got exception:
> java.net.URISyntaxException Illegal character in hostname at index 9:
> thrift://_gateway:9083
> Java exception follows:
> java.net.URISyntaxException: Illegal character in hostname at index 9:
> thrift://_gateway:9083
> at java.net.URI$Parser.fail(URI.java:2847)
> at java.net.URI$Parser.parseHostname(URI.java:3386)
> at java.net.URI$Parser.parseServer(URI.java:3235)
> at java.net.URI$Parser.parseAuthority(URI.java:3154)
> at java.net.URI$Parser.parseHierarchical(URI.java:3096)
> at java.net.URI$Parser.parse(URI.java:3052)
> at java.net.URI.<init>(URI.java:673)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:325)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:232)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:85)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:98)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:151)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:122)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:115)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:103)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:82)
> at
> org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:182)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:171)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:539)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:493)
> at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:153)
> E0201 18:13:30.288360 1 MetaStoreUtils.java:260] Converting exception to
> MetaException
> W0201 18:13:30.289749 1 MetaStoreClientPool.java:113] Failed to connect
> to Hive MetaStore. Retrying.
> Java exception follows:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:87)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:98)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:151)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:122)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:115)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:103)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:82)
> at
> org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:182)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:171)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:539)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:493)
> at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:153)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:85)
> ... 12 more
> Caused by: MetaException(message:Got exception: java.net.URISyntaxException
> Illegal character in hostname at index 9: thrift://_gateway:9083)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:261)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:342)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:232)
> ... 17 more
> {code}
> This is repeated several times, then eventually Impala exits, reporting this
> as a FATAL error:
> {code}
> F0201 18:15:30.194324 1 frontend.cc:188]
> MetastoreClientInstantiationException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: InvocationTargetException: null
> CAUSED BY: MetaException: Got exception: java.net.URISyntaxException Illegal
> character in hostname at index 9: thrift://_gateway:9083
> . Impalad exiting.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
