Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18738 )
Change subject: IMPALA-9442: Add Ozone to minicluster ...................................................................... IMPALA-9442: Add Ozone to minicluster Adds Ozone as an alternative to hdfs in the minicluster. Select by setting `export TARGET_FILESYSTEM=ozone`. With that flag, run-mini-dfs.sh will start Ozone instead of HDFS. Requires a snapshot because Ozone does not support HBase (HDDS-3589); snapshot loading doesn't work yet primarily due to HDDS-5502. Uses the o3fs interface because Ozone puts specific restrictions on bucket names (no underscores, for instance), and it was a lot easier to use an interface where everything is written to a single bucket than to update all Impala's use of HDFS-style paths to make `test-warehouse` a bucket inside a volume. Specifies reduced Ozone client retries during shutdown where Ozone may not be available. Passes tests with FE_TEST=false BE_TEST=false. Change-Id: Ibf8b0f7b2d685d8b011df1926e12bf5434b5a2be Reviewed-on: http://gerrit.cloudera.org:8080/18738 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> --- M be/src/runtime/io/disk-io-mgr.cc M bin/bootstrap_toolchain.py M bin/clean.sh M bin/create-test-configuration.sh M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/resources/.gitignore M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/kill-hbase.sh M testdata/bin/run-all.sh M testdata/cluster/admin M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py A testdata/cluster/node_templates/common/etc/hadoop/conf/ozone-site.xml.py M testdata/cluster/node_templates/common/etc/init.d/common.tmpl A testdata/cluster/node_templates/common/etc/init.d/ozone-common A testdata/cluster/node_templates/common/etc/init.d/ozone-datanode A testdata/cluster/node_templates/common/etc/init.d/ozone-manager A testdata/cluster/node_templates/common/etc/init.d/ozone-scm M tests/authorization/test_ranger.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_events_custom_configs.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_hive_text_codec_interop.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_lineage.py M tests/custom_cluster/test_local_catalog.py M tests/custom_cluster/test_local_tz_conversion.py M tests/custom_cluster/test_metadata_no_events_processing.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_runtime_profile.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_event_processing.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_stats_extrapolation.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_acid.py M tests/query_test/test_date_queries.py M tests/query_test/test_hbase_queries.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_acid_stress.py M tests/stress/test_ddl_stress.py M tests/util/filesystem_utils.py 65 files changed, 436 insertions(+), 105 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18738 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibf8b0f7b2d685d8b011df1926e12bf5434b5a2be Gerrit-Change-Number: 18738 Gerrit-PatchSet: 16 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
