Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 ...................................................................... IMPALA-8345 : Add option to set up minicluster to use Hive 3 As a first step to integrate Impala with Hive 3.1.0 this patch modifies the minicluster scripts to use Hive 3.1.0 instead of CDH Hive 2.1.1. In order to make sure that existing setups don't break this option is enabled via a command line argument to bin/impala-config.sh. This command line argument (-use-hive3) sets up certain environment variables such that Hive 3.1.0 based binaries can be used to instantiate Hive service (Hiveserver2 and metastore). The default is still Hive 2.1.1 Also, since Hive 3.1.1 uses a upgraded metastore schema, this patch makes use of a different database name so that it is easy to switch from working from one environment which uses Hive 2.1.1 metastore to another which usese Hive 3.1.0 metastore. In order to do so users should follow the below steps: 1. Open a new terminal 2. Run bin/bootstrap_toolchain.py 2. source bin/impala-config.sh -use-hive3 3. source bin/create-test-configuration.sh -create-metastore The above step should provide "-create-metastore" only the first time so that a new metastore db is created and the Hive 3.1.0 schema is initialized. For all subsequent invocations, the "-create-metastore" argument can be skipped. We should still source this script since the hive-site.xml of Hive 3.1.0 is slightly different than Hive 2.1.0 and needs to be regenerated. 4. Start services using the testdata/bin/run-all.sh Note that the testing was performed locally by downloading the Hive 3.1 binaries into toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once the binaries are available in S3 bucket, the bootstrap_toolchain script should automatically do this for you. Testing Done: 1. Made sure that the cluster comes up with Hive 3.1 when the steps above are performed. 2. Made sure that existing scripts work as they do currently when argument is not provided. 3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala still uses Hive 2.1.1 client. Upgrading client libraries in Impala will be done as a separate change) Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh A fe/src/test/resources/postgresql-hive-site.xml.cdp.template M testdata/bin/run-hive-server.sh 5 files changed, 372 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/4 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar <vih...@cloudera.com> Gerrit-Reviewer: Fredy Wijaya <fwij...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Vihang Karajgaonkar <vih...@cloudera.com>
