[
https://issues.apache.org/jira/browse/IMPALA-9097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084362#comment-17084362
]
ASF subversion and git services commented on IMPALA-9097:
---------------------------------------------------------
Commit 34018f62751b01c21e869ca7ff38e6d7d4a808d9 in impala's branch
refs/heads/master from Laszlo Gaal
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=34018f6 ]
IMPALA-9629: Add CentOS 8.1 support to bootstrap_system.sh
CentOS 8.1 is a new major version of the CentOS family.
It is now stable and popular enough to start supporting it for Impala
development.
Prepare a raw CentOS 8.1 system to support Impala development and testing.
This should work on a standalone computer, on a virtual machine,
or inside a Docker container.
Details:
- snappy-devel moved to the PowerTools repo, so it needs to be installed
from there
- CentOS 8 has no default Python version. The bootstrap script installs
(or configures) Python2 with pip2, then makes them the default via the
"alternatives" mechanism. The installer is adaptive, it performs only
the necessary steps, so it works in various environments.
The installer logic is also shared between bin/bootstrap_system.sh and
docker/entrypoint.sh
- The toolchain package tag "ec2-centos-8" is added to
bootstrap_toolchain.py
- For some unknown reason, when the downloaded Maven tarball is extracted
in a Docker-based test, the "bin" and "boot" directories are created
with owner-only permissions. The 'impdev' users has no access to the
maven executable, which then breaks the build.
This patch forcibly restores the correct permissions on these
directories; this is a no-op when the extraction happens correctly.
- TOOLCHAIN_ID is bumped to a build that already has CentOS 8 binaries.
- Centos8-specific bootstrap code was added to the Docker-based tests.
Tested:
- ran the Docker-based tests with --base-image=centos:8 to verify the following
build
phases are successful:
* system prep
* build
* dataload
and that test can start. Passing all tests is was not a requirement for this
step,
although plausible test results (i.e. not all of the tests fail) were.
- ran the Docker-based tests to verify nonregression with --base-image set to
the
following: centos:7, ubuntu:16.04, ubuntu:18.04.
On centos:7 and ubuntu:16.04 the only failure was IMPALA-9097 (BE tests fail
without
the minicluster running); ubuntu:18.04 showed the same failures as the
current upstream
code.
- passed a core-mode test run on private infrastructure on Centos 7.4
- ran buildall.sh in core mode manually inside a Docker container, simulating a
developer
workflow (prep-build-dataload-test). There were several observed test
failures, but
the workflow itself was run to completion with no problems.
Change-Id: I3df5d48eca7a10219264e3604a4f05f072188e6e
Reviewed-on: http://gerrit.cloudera.org:8080/15623
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Some backend tests fail if the Hive Metastore is not running
> ------------------------------------------------------------
>
> Key: IMPALA-9097
> URL: https://issues.apache.org/jira/browse/IMPALA-9097
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Priority: Major
>
> In our docker-based tests (i.e. docker/test_with_docker.py), we run the
> backend tests without starting a minicluster. This is now failing due to a
> new dependency on the Hive Metastore. This applies to a bunch of tests, which
> all fail with this error:
> {noformat}
> F0917 00:37:47.849447 7660 frontend.cc:134] IllegalStateException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClientF0917
> 00:37:47.849447 7660 frontend.cc:134] IllegalStateException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> CAUSED BY: InvocationTargetException: null
> CAUSED BY: MetaException: Could not connect to meta store using any of the
> URIs provided. Most recent failure:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:545)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1773)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:94)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:99)
> at
> org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:78)
> at
> org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:174)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:163)
> at
> org.apache.impala.catalog.MetaStoreClientPool.<init>(MetaStoreClientPool.java:155)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:301)
> at org.apache.impala.service.Frontend.<init>(Frontend.java:270)
> at
> org.apache.impala.service.JniFrontend.<init>(JniFrontend.java:141){noformat}
> The list of tests that currently fail in this configuration are:
>
> {noformat}
> 2019-10-25 14:21:36.059346 The following tests FAILED:
> 2019-10-25 14:21:36.059378 3 - llvm-codegen-test (Child aborted)
> 2019-10-25 14:21:36.059404 8 - hash-table-test (Failed)
> 2019-10-25 14:21:36.059436 11 - row-batch-list-test (Child aborted)
> 2019-10-25 14:21:36.059468 19 - hdfs-parquet-scanner-test (Failed)
> 2019-10-25 14:21:36.059491 20 - expr-test (Failed)
> 2019-10-25 14:21:36.059521 21 - expr-codegen-test (Child aborted)
> 2019-10-25 14:21:36.059551 28 - data-stream-test (Child aborted)
> 2019-10-25 14:21:36.059586 39 - buffered-tuple-stream-test (Child aborted)
> 2019-10-25 14:21:36.059613 41 - tmp-file-mgr-test (Failed)
> 2019-10-25 14:21:36.059644 42 - row-batch-serialize-test (Failed)
> 2019-10-25 14:21:36.059673 43 - row-batch-test (Child aborted)
> 2019-10-25 14:21:36.059710 44 - collection-value-builder-test (Child
> aborted)
> 2019-10-25 14:21:36.059741 45 - runtime-state-test (Child aborted)
> 2019-10-25 14:21:36.059774 46 - buffer-allocator-test (Child aborted)
> 2019-10-25 14:21:36.059804 47 - buffer-pool-test (Child aborted)
> 2019-10-25 14:21:36.059829 48 - free-list-test (Failed)
> 2019-10-25 14:21:36.059863 49 - reservation-tracker-test (Child aborted)
> 2019-10-25 14:21:36.059890 50 - suballocator-test (Failed)
> 2019-10-25 14:21:36.059917 51 - disk-io-mgr-test (Failed)
> 2019-10-25 14:21:36.059946 52 - data-cache-test (Child aborted)
> 2019-10-25 14:21:36.059977 53 - admission-controller-test (Failed)
> 2019-10-25 14:21:36.060015 59 - session-expiry-test (Child aborted)
> 2019-10-25 14:21:36.060043 67 - rpc-mgr-test (Child aborted)
> 2019-10-25 14:21:36.060076 68 - rpc-mgr-kerberized-test (Child aborted)
> 2019-10-25 14:21:36.060102 77 - bloom-filter-test (Failed)
> 2019-10-25 14:21:36.060127 85 - hdfs-util-test (Failed)
> 2019-10-25 14:21:36.060154 91 - min-max-filter-test (Failed){noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
