Laszlo Gaal created IMPALA-13724:
------------------------------------
Summary: Impala fails to start up in Dockerised tests when the
Docker gateway name is resolved as _gateway
Key: IMPALA-13724
URL: https://issues.apache.org/jira/browse/IMPALA-13724
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 4.5.0
Reporter: Laszlo Gaal
Assignee: Laszlo Gaal
When Impala daemons are started inside a Docker container (using the dockerised
form of bin/start-impala-cluster.py like this):{code}
$ export DOCKER_NETWORK=impala-dockernet # or use your own name
$ bin/start-impala-cluster.py --docker_network=${DOCKER_NETWORK}
{code}
the gateway generated by Docker networking receives an IP address from Docker,
but the way this IP address is resolved to a DNS name is platform-dependent.
Unfortunately some newer platforms (e.g. Rocky 9 with Docker 27.5) resolve the
gateway's name as {code}
_gateway{code}
with a leading underscore. This is a problem for Impala: on startup the catalog
and all coordinators attempt to connect to HMS over Thrift, using a Java
implementation of Thrift. This Java code relies on a communication library that
considers the the underscore character invalid within DNS names, throws an
exception complaining about it, and refuses to connect to HMS because of the
exception. Failing to connect to HMS is fatal during Impala startup, so this
problem prevents impala catalog and coordinator roles from starting up in such
an environment.
Example coordinator log sequence:
{code}
I0201 18:13:30.205049 1 HiveMetaStoreClient.java:279] HMS client filtering
is enabled.
E0201 18:13:30.288259 1 MetaStoreUtils.java:259] Got exception:
java.net.URISyntaxException Illegal character in hostname at index 9:
thrift://_gateway:9083
Java exception follows:
java.net.URISyntaxException: Illegal character in hostname at index 9:
thrift://_gateway:9083
at java.net.URI$Parser.fail(URI.java:2847)
at java.net.URI$Parser.parseHostname(URI.java:3386)
at java.net.URI$Parser.parseServer(URI.java:3235)
at java.net.URI$Parser.parseAuthority(URI.java:3154)
at java.net.URI$Parser.parseHierarchical(URI.java:3096)
at java.net.URI$Parser.parse(URI.java:3052)
at java.net.URI.<init>(URI.java:673)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:325)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:232)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:85)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:98)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:151)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:122)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:115)
at
org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:103)
at
org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:82)
at
org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:182)
at
org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:171)
at
org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
at org.apache.impala.service.Frontend.<init>(Frontend.java:539)
at org.apache.impala.service.Frontend.<init>(Frontend.java:493)
at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:153)
E0201 18:13:30.288360 1 MetaStoreUtils.java:260] Converting exception to
MetaException
W0201 18:13:30.289749 1 MetaStoreClientPool.java:113] Failed to connect to
Hive MetaStore. Retrying.
Java exception follows:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:87)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:98)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:151)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:122)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:115)
at
org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:103)
at
org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:82)
at
org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:182)
at
org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:171)
at
org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
at org.apache.impala.service.Frontend.<init>(Frontend.java:539)
at org.apache.impala.service.Frontend.<init>(Frontend.java:493)
at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:153)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:85)
... 12 more
Caused by: MetaException(message:Got exception: java.net.URISyntaxException
Illegal character in hostname at index 9: thrift://_gateway:9083)
at
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:261)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:342)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:232)
... 17 more
{code}
This is repeated several times, then eventually Impala exits, reporting this as
a FATAL error:
{code}
F0201 18:15:30.194324 1 frontend.cc:188]
MetastoreClientInstantiationException: java.lang.RuntimeException: Unable to
instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
CAUSED BY: RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
CAUSED BY: InvocationTargetException: null
CAUSED BY: MetaException: Got exception: java.net.URISyntaxException Illegal
character in hostname at index 9: thrift://_gateway:9083
. Impalad exiting.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
