Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12189 )
Change subject: IMPALA-7988: support loading data with dockerized Impalas ...................................................................... IMPALA-7988: support loading data with dockerized Impalas This patch does the work to load data and run some end-to-end query tests on a dockerised cluster. Changes were required in start-impala-cluster.py/ImpalaCluster and in some configuration files. ImpalaCluster is used for various things, including discovering service ports and testing for cluster readiness. This patch adds basic support and uses it from start-impala-cluster.py to check for cluster readiness. Some logic is moved from start-impala-cluster.py to ImpalaCluster. Limitations: * We're fairly inconsistent about whether services listen only on a single interface (e.g. loopback, traditionally) or whether it listens on all interfaces. This doesn't fix all of those issues. E.g. HDFS datanodes listen on all interfaces to work around some issues. * Many tests don't pass yet, particularly those using ImpalaCluster(), which isn't initialised with the appropriate docker arguments. Testing: Did a full data load locally using a dockerised Impala cluster: START_CLUSTER_ARGS="--docker_network=impala-cluster" \ TEST_START_CLUSTER_ARGS="--docker_network=impala-cluster" \ ./buildall.sh -format -testdata -ninja -notests -skiptests -noclean Ran a selection of end-to-end tests touching HDFS, Kudu and HBase tables after I loaded data locally. Ran exhaustive tests with non-dockerised impala cluster. Change-Id: I98fb9c4f5a3a3bb15c7809eab28ec8e5f63ff517 Reviewed-on: http://gerrit.cloudera.org:8080/12189 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M bin/impala-config.sh M bin/start-impala-cluster.py M fe/src/test/resources/hbase-site.xml.template M fe/src/test/resources/hive-default.xml M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template M testdata/bin/create-load-data.sh M testdata/bin/run-all.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl M testdata/cluster/node_templates/common/etc/kudu/tserver.conf.tmpl M tests/common/impala_cluster.py 11 files changed, 358 insertions(+), 229 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12189 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I98fb9c4f5a3a3bb15c7809eab28ec8e5f63ff517 Gerrit-Change-Number: 12189 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
