[
https://issues.apache.org/jira/browse/IMPALA-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928903#comment-17928903
]
ASF subversion and git services commented on IMPALA-13724:
----------------------------------------------------------
Commit 26b7116c3bb296615d297af518f29bddcb2b2bd9 in impala's branch
refs/heads/master from Laszlo Gaal
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=26b7116c3 ]
IMPALA-13724: Add hostnames for Docker host and gateway to Impala containers
Reverse DNS lookup for the Docker container's internal gateway (routing
traffic between code running inside the container and code runnning
natively on the Docker host) happens differently on various operating
systems. Some recent platforms, like RHEL 9 resolve this address to the
default name _gateway. Unfortunately the Java Thrift library within
Impala's frontend considers the underscore character invalid in DNS
names, so it throws an error, preventing the Impala coordinator from
connecting to HMS. This kills Impala on startup, blocking any testing
efforts inside containers.
To avoid this problem this patch adds explicit entries to the container's
/etc/hosts file for the gateway's address as well as the Docker host network.
The name doesn't really matter, as it is used only for Thrift's logging
code, so the mapping uses constant generic name 'gateway'.
The IP address of the gateway is retrieved from the environment variable
INTERNAL_LISTEN_HOST, which is set up by docker/configure_test_network.sh
before the Impala containers are launched.
Tested by a dockerised test run executed on Rocky Linux 9.2, using Rocky
9.2 Docker base images for the Impala containers.
Change-Id: I545607c0bb32f8043a0d3f6045710f28a47bab99
Reviewed-on: http://gerrit.cloudera.org:8080/22438
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Impala fails to start up in Dockerised tests when the Docker gateway name is
> resolved as _gateway
> -------------------------------------------------------------------------------------------------
>
> Key: IMPALA-13724
> URL: https://issues.apache.org/jira/browse/IMPALA-13724
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.5.0
> Reporter: Laszlo Gaal
> Assignee: Laszlo Gaal
> Priority: Critical
>
> When Impala daemons are started inside a Docker container (using the
> dockerised form of bin/start-impala-cluster.py like this):{code}
> $ export DOCKER_NETWORK=impala-dockernet # or use your own name
> $ bin/start-impala-cluster.py --docker_network=${DOCKER_NETWORK}
> {code}
> the gateway generated by Docker networking receives an IP address from
> Docker, but the way this IP address is resolved to a DNS name is
> platform-dependent.
> Unfortunately some newer platforms (e.g. Rocky 9 with Docker 27.5) resolve
> the gateway's name as {code}
> _gateway{code}
> with a leading underscore. This is a problem for Impala: on startup the
> catalog and all coordinators attempt to connect to HMS over Thrift, using a
> Java implementation of Thrift. This Java code relies on a communication
> library that considers the the underscore character invalid within DNS names,
> throws an exception complaining about it, and refuses to connect to HMS
> because of the exception. Failing to connect to HMS is fatal during Impala
> startup, so this problem prevents impala catalog and coordinator roles from
> starting up in such an environment.
> Example coordinator log sequence:
> {code}
> I0201 18:13:30.205049 1 HiveMetaStoreClient.java:279] HMS client
> filtering is enabled.
> E0201 18:13:30.288259 1 MetaStoreUtils.java:259] Got exception:
> java.net.URISyntaxException Illegal character in hostname at index 9:
> thrift://_gateway:9083
> Java exception follows:
> java.net.URISyntaxException: Illegal character in hostname at index 9:
> thrift://_gateway:9083
> at java.net.URI$Parser.fail(URI.java:2847)
> at java.net.URI$Parser.parseHostname(URI.java:3386)
> at java.net.URI$Parser.parseServer(URI.java:3235)
> at java.net.URI$Parser.parseAuthority(URI.java:3154)
> at java.net.URI$Parser.parseHierarchical(URI.java:3096)
> at java.net.URI$Parser.parse(URI.java:3052)
> at java.net.URI.<init>(URI.java:673)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:325)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:232)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:85)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:98)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:151)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:122)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:115)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:103)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:82)
> at
> org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:182)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:171)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:539)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:493)
> at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:153)
> E0201 18:13:30.288360 1 MetaStoreUtils.java:260] Converting exception to
> MetaException
> W0201 18:13:30.289749 1 MetaStoreClientPool.java:113] Failed to connect
> to Hive MetaStore. Retrying.
> Java exception follows:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:87)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:98)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:151)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:122)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:115)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:103)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:82)
> at
> org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:182)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:171)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:539)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:493)
> at org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:153)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:85)
> ... 12 more
> Caused by: MetaException(message:Got exception: java.net.URISyntaxException
> Illegal character in hostname at index 9: thrift://_gateway:9083)
> at
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:261)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:342)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:232)
> ... 17 more
> {code}
> This is repeated several times, then eventually Impala exits, reporting this
> as a FATAL error:
> {code}
> F0201 18:15:30.194324 1 frontend.cc:188]
> MetastoreClientInstantiationException: java.lang.RuntimeException: Unable to
> instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: InvocationTargetException: null
> CAUSED BY: MetaException: Got exception: java.net.URISyntaxException Illegal
> character in hostname at index 9: thrift://_gateway:9083
> . Impalad exiting.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
