mackror...@apache.org has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11630
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. ...................................................................... IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. URI schemes are configured separately, however, so new functions are needed. This also allows behavior to be tweaked independently should any quirks be found in subsequent performance testing. Was not able to get a full test run completed due to snags in the data loading on non-HDFS storage. Consequently, abfs_util.py has not yet been through a full test run either. Patch was tested against TPC-DS with no apparent problems. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 45 files changed, 326 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/1 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 1 Gerrit-Owner: mackror...@apache.org
