[
https://issues.apache.org/jira/browse/IMPALA-12115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778481#comment-17778481
]
ASF subversion and git services commented on IMPALA-12115:
----------------------------------------------------------
Commit 24a8a29120606cdc7205b09a5bd53d52d357539a in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=24a8a2912 ]
IMPALA-12115: Put each filesystem in a different directory
Uses different node directories and databases for each filesystem so we
don't need to recreate them from scratch when switching.
Preserves current defaults so developers with a default HDFS minicluster
don't need to recreate it after this patch. Any other cluster (Ozone,
erasure-coded, S3) will need to be recreated.
Starting with a new filesystem requires running
0. ./testdata/bin/kill-all.sh if an old cluster is running
1. create-test-configuration.sh. Add -create_metastore
-create_ranger_policy_db for first run.
2. ./testdata/bin/run-all.sh
3. start-impala-cluster.py
or "buildall.sh -noclean -start_minicluster -start_impala_cluster". Add
"-format" for the first run to create HMS and Ranger DBs.
IMPALA_CLUSTER_LOGS_DIR is shared for all clusters. Symlinks to the
minicluster are recreated by create-test-configuration.sh.
Change-Id: I8c89156fd1cefbb752fee3070e10bb08fbf80e07
Reviewed-on: http://gerrit.cloudera.org:8080/19841
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Michael Smith <[email protected]>
> Must rebuild minicluster to switch filesystems
> ----------------------------------------------
>
> Key: IMPALA-12115
> URL: https://issues.apache.org/jira/browse/IMPALA-12115
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
> Fix For: Impala 4.4.0
>
>
> When switching between HDFS, Ozone, S3, and erasure-coded variants, Impala's
> minicluster uses the same directory for config and data. That means any time
> you want to test a different one, you need to replace the current minicluster
> config/data (particularly data load) which can take awhile. When switching
> from erasure-coding, you usually need to wipe out the directory because it
> has too many nodes configured and build scripts don't know to clean them up.
> It would be useful to be able to switch between these configurations more
> quickly in a dev environment by preserving each. You can manually do that by
> setting {{IMPALA_CLUSTER_NODES_DIR}}, but that's an extra step to remember to
> do every time. We should set the directory based on {{TARGET_FILESYSTEM}} and
> {{ERASURE_CODING}} so they're automatically preserved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
