[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/11630/8/tests/query_test/test_insert.py File tests/query_test/test_insert.py: http://gerrit.cloudera.org:8080/#/c/11630/8/tests/query_test/test_insert.py@116 PS8, Line 116: S flake8: F821 undefined name 'SkipIfABFS' -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 10:46:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. Tests skipped on ADLS Gen1 due to eventual consistency of the Python client can be run against ADLS Gen2. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Reviewed-on: http://gerrit.cloudera.org:8080/11630 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 48 files changed, 360 insertions(+), 32 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 12 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 11 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 06:42:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/11630/7/tests/query_test/test_insert.py File tests/query_test/test_insert.py: http://gerrit.cloudera.org:8080/#/c/11630/7/tests/query_test/test_insert.py@116 PS7, Line 116: S flake8: F821 undefined name 'SkipIfABFS' -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 06:29:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1122/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 10 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 03:19:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 11 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 02:48:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3341/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 11 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 02:48:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 10: Code-Review+2 Fixed disk-io-mgr-test.cc and some typos in a couple tests. Carrying +2 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 10 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 02:46:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has uploaded a new patch set (#10) to the change originally created by mackror...@apache.org. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. Tests skipped on ADLS Gen1 due to eventual consistency of the Python client can be run against ADLS Gen2. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 48 files changed, 360 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/10 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 10 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3339/ -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 02:45:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1120/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 00:48:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3340/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 00:13:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3339/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 00:09:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 00:09:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 8: Code-Review+2 Thanks! -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 20 Oct 2018 00:06:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Jim Apple has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11630/7/be/src/runtime/io/scan-range.cc
File be/src/runtime/io/scan-range.cc:
http://gerrit.cloudera.org:8080/#/c/11630/7/be/src/runtime/io/scan-range.cc@31
PS7, Line 31:
> Nit: stray line
Done
http://gerrit.cloudera.org:8080/#/c/11630/7/be/src/runtime/io/scan-range.cc@447
PS7, Line 447: if (disk_id_ == io_mgr_->RemoteAdlsDiskId()) {
: DCHECK(IsADLSPath(file()));
: return FLAGS_adls_read_chunk_size;
: }
> What should we do for this commit regarding ABFS chunk size?
Yes, I agree that #3 is best.
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d
Gerrit-Change-Number: 11630
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Fri, 19 Oct 2018 23:59:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Jim Apple has uploaded a new patch set (#8) to the change originally created by mackror...@apache.org. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. Tests skipped on ADLS Gen1 due to eventual consistency of the Python client can be run against ADLS Gen2. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 47 files changed, 357 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/8 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 7:
(2 comments)
I'm basically ready to +2 this. I'd like to settle what we are doing with the
chunk size.
http://gerrit.cloudera.org:8080/#/c/11630/7/be/src/runtime/io/scan-range.cc
File be/src/runtime/io/scan-range.cc:
http://gerrit.cloudera.org:8080/#/c/11630/7/be/src/runtime/io/scan-range.cc@31
PS7, Line 31:
Nit: stray line
http://gerrit.cloudera.org:8080/#/c/11630/7/be/src/runtime/io/scan-range.cc@447
PS7, Line 447: if (disk_id_ == io_mgr_->RemoteAdlsDiskId()) {
: DCHECK(IsADLSPath(file()));
: return FLAGS_adls_read_chunk_size;
: }
What should we do for this commit regarding ABFS chunk size?
Some options:
1. Don't customize for ABFS and add parameters later. File a JIRA to customize
it.
2. Use adls_read_chunk_size for both ABFS and ADLS
3. Add a parameter now and set the default the same as adls_read_chunk_size
4. Hard code ABFS chunk size as 128k for now. File a JIRA to do perf tests and
customize it.
If we use 128k for S3 and ADLS, it seems reasonable to start out using it for
ABFS until we can do performance tests (or until we can convince ourselves that
ABFS is different).
#3 seems reasonable to me. What do you think?
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d
Gerrit-Change-Number: 11630
Gerrit-PatchSet: 7
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Fri, 19 Oct 2018 23:38:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1117/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 19 Oct 2018 22:50:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Jim Apple has uploaded a new patch set (#7) to the change originally created by mackror...@apache.org. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. Tests skipped on ADLS Gen1 due to eventual consistency of the Python client can be run against ADLS Gen2. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 47 files changed, 352 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/7 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 6: Code-Review+1 (2 comments) Two nits. Joe, Phil, would appreciate you taking a quick look - I'm going to post a new patchset, since I know Sean is otherwise occupied. http://gerrit.cloudera.org:8080/#/c/11630/6/tests/query_test/test_insert.py File tests/query_test/test_insert.py: http://gerrit.cloudera.org:8080/#/c/11630/6/tests/query_test/test_insert.py@112 PS6, Line 112: # This fails against ABFS due HADOOP-15860. It succeeds if the partition name Can you add a SkipIfABFS to this test, too? http://gerrit.cloudera.org:8080/#/c/11630/6/tests/util/abfs_util.py File tests/util/abfs_util.py: http://gerrit.cloudera.org:8080/#/c/11630/6/tests/util/abfs_util.py@32 PS6, Line 32: def hadoop_fs_shell(self, command): Methods used only within the class are traditionally prefixed by an underscore; here and below. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 19 Oct 2018 22:11:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1103/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 19 Oct 2018 05:07:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 6: Skipping drop_table_with_purge and drop_partition_with_purge with a reference to the JIRA to look into that. As discussed on the JIRA, I cannot find any flaw in trash on ABFS. Also fixing the latest round of style feedback from Jenkins. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 19 Oct 2018 04:33:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Hello Jim Apple, Philip Zeyliger, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11630 to look at the new patch set (#6). Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. Tests skipped on ADLS Gen1 due to eventual consistency of the Python client can be run against ADLS Gen2. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 47 files changed, 353 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/6 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/11630/5/tests/util/abfs_util.py File tests/util/abfs_util.py: http://gerrit.cloudera.org:8080/#/c/11630/5/tests/util/abfs_util.py@34 PS5, Line 34: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/11630/5/tests/util/abfs_util.py@108 PS5, Line 108: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/11630/5/tests/util/abfs_util.py@114 PS5, Line 114: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 18 Oct 2018 20:06:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1091/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 18 Oct 2018 16:19:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 5:
> > Thanks, Joe. All of those tests have now run after some bugfixing
> > in abfs_util.py. There are 4 tests failing with 2 root causes.
> >
> > 2 are failing because of HADOOP-15860, wherein files ending with
> > periods don't get listed. I temporarily modified the test to
> suffix
> > the partition name with another character and they passed. A note
> > is included about that test instead.
> >
> > 2 are failing because of files being or not being in the Trash
> > after dropping a table with or without purge. I'm still looking
> > into that one.
>
> Those are metadata.test_ddl.TestDdlStatements.test_drop_table_with_purge,
> metadata.test_ddl.TestDdlStatements.test_drop_partition_with_purge[protocol:
> beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0,
> 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action':
> None, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none], and
> metadata.test_ddl.TestDdlStatements.test_drop_cleans_hdfs_dirs?
Yes, that's correct, thank you.
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d
Gerrit-Change-Number: 11630
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Thu, 18 Oct 2018 15:56:24 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Jim Apple has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 5:
> Thanks, Joe. All of those tests have now run after some bugfixing
> in abfs_util.py. There are 4 tests failing with 2 root causes.
>
> 2 are failing because of HADOOP-15860, wherein files ending with
> periods don't get listed. I temporarily modified the test to suffix
> the partition name with another character and they passed. A note
> is included about that test instead.
>
> 2 are failing because of files being or not being in the Trash
> after dropping a table with or without purge. I'm still looking
> into that one.
Those are metadata.test_ddl.TestDdlStatements.test_drop_table_with_purge,
metadata.test_ddl.TestDdlStatements.test_drop_partition_with_purge[protocol:
beeswax | exec_option: {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: text/none], and
metadata.test_ddl.TestDdlStatements.test_drop_cleans_hdfs_dirs?
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d
Gerrit-Change-Number: 11630
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Thu, 18 Oct 2018 15:55:24 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 5: Thanks, Joe. All of those tests have now run after some bugfixing in abfs_util.py. There are 4 tests failing with 2 root causes. 2 are failing because of HADOOP-15860, wherein files ending with periods don't get listed. I temporarily modified the test to suffix the partition name with another character and they passed. A note is included about that test instead. 2 are failing because of files being or not being in the Trash after dropping a table with or without purge. I'm still looking into that one. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 18 Oct 2018 15:50:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Hello Jim Apple, Philip Zeyliger, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11630 to look at the new patch set (#5). Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. URI schemes are configured separately, however, so new functions are needed. This also allows behavior to be tweaked independently should any quirks be found in subsequent performance testing. Was not able to get a full test run completed due to snags in the data loading on non-HDFS storage. Consequently, abfs_util.py has not yet been through a full test run either. Patch was tested against TPC-DS with no apparent problems. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 46 files changed, 349 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/5 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 5 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/util/abfs_util.py
File tests/util/abfs_util.py:
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/util/abfs_util.py@36
PS4, Line 36: output = subprocess.output("hadoop fs -%s" % command)
> metadata/test_ddl.py:43: in test_drop_table_with_purge
Also, there are two ways for this to work:
1. You generate a string that is a shell command, and you pass shell=True to
check_output(). Without shell, you'll get a file not found error.
2. You generate a list with the command and arguments (i.e. ["hadoop", "fs",
... args ... ] and use shell=False (the default).
I prefer #2.
Nit: I think it is more intuitive for the callers to include the "-" in the
first arg. i.e. "-ls" rather than "ls" with the "-" appended here.
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/util/abfs_util.py@52
PS4, Line 52: path
The path passed in by our test framework does not have a leading '/'. i.e.
"test-warehouse/alltypes" rather than
"/test-warehouse/alltypes"
HDFS command line needs the leading "/", so all these paths need to be fixed up
before being passed to the HDFS command line.
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/util/abfs_util.py@65
PS4, Line 65: entries = output.split("\n")[1:] # Trim the "Found X items" line
The hadoop shell output has a trailing newline, so the last entry will be an
empty string. We need to handle that here. Maybe [1:-1]
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d
Gerrit-Change-Number: 11630
Gerrit-PatchSet: 4
Gerrit-Owner: mackror...@apache.org
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: mackror...@apache.org
Gerrit-Comment-Date: Fri, 12 Oct 2018 17:05:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 4: (4 comments) Looking through the existing skips, I don't think the combined ADLS skips vs ADLS+ABFS skips issue should impact this code change. That applies to more than just ADLS and can be tracked in a separate JIRA. I think this review is getting close. http://gerrit.cloudera.org:8080/#/c/11630/4/be/src/runtime/io/scan-range.cc File be/src/runtime/io/scan-range.cc: http://gerrit.cloudera.org:8080/#/c/11630/4/be/src/runtime/io/scan-range.cc@31 PS4, Line 31: Nit: remove stray line http://gerrit.cloudera.org:8080/#/c/11630/4/be/src/runtime/io/scan-range.cc@38 PS4, Line 38: // TODO: Decide if this should be done for ABFS Do we want to default to using this for ABFS as well? We'll need to do perf tests to check, but it seems reasonable to use what we are using for ADLS/S3 for ABFS until we know otherwise. http://gerrit.cloudera.org:8080/#/c/11630/4/tests/common/skip.py File tests/common/skip.py: http://gerrit.cloudera.org:8080/#/c/11630/4/tests/common/skip.py@63 PS4, Line 63: ADLS ADLS -> ABFS (same for other skips) http://gerrit.cloudera.org:8080/#/c/11630/4/tests/custom_cluster/test_permanent_udfs.py File tests/custom_cluster/test_permanent_udfs.py: http://gerrit.cloudera.org:8080/#/c/11630/4/tests/custom_cluster/test_permanent_udfs.py@310 PS4, Line 310: @SkipIfIsilon.hive : @SkipIfS3.hive : @SkipIfLocal.hive Same here -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 4 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Thu, 11 Oct 2018 17:35:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 4:
(2 comments)
Add some comments on stuff found by running the tests:
tests/query_test/test_resource_limits.py - Add SkipIfADLS/SkipIfABFS.hbase to
test_resource_limits_hbase().
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/custom_cluster/test_permanent_udfs.py@249
PS4, Line 249: @SkipIfIsilon.hive
: @SkipIfS3.hive
: @SkipIfLocal.hive
Add SkipIfADLS.hive and SkipIfABFS.hive
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/util/abfs_util.py
File tests/util/abfs_util.py:
http://gerrit.cloudera.org:8080/#/c/11630/4/tests/util/abfs_util.py@36
PS4, Line 36: output = subprocess.output("hadoop fs -%s" % command)
metadata/test_ddl.py:43: in test_drop_table_with_purge
format(unique_database), file_data='t1')
util/abfs_util.py:49: in create_file
(status, output) = self.hadoop_fs_shell("put %s %s" % (tmp_path, path))
util/abfs_util.py:36: in hadoop_fs_shell
output = subprocess.output("hadoop fs -%s" % command)
E AttributeError: 'module' object has no attribute 'output'
I think this should be subprocess.check_output()
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d
Gerrit-Change-Number: 11630
Gerrit-PatchSet: 4
Gerrit-Owner: mackror...@apache.org
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: mackror...@apache.org
Gerrit-Comment-Date: Thu, 11 Oct 2018 16:28:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1018/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 4 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Wed, 10 Oct 2018 22:46:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Hello Philip Zeyliger, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11630 to look at the new patch set (#4). Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. URI schemes are configured separately, however, so new functions are needed. This also allows behavior to be tweaked independently should any quirks be found in subsequent performance testing. Was not able to get a full test run completed due to snags in the data loading on non-HDFS storage. Consequently, abfs_util.py has not yet been through a full test run either. Patch was tested against TPC-DS with no apparent problems. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 44 files changed, 320 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/4 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 4 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 3: > Quick comment: I think we are going to be skipping the same tests > for ADLS and ABFS. I would rather smash them together and reuse the > existing skipping condition we have for ADLS to cover both. That > should eliminate the need for a bunch of test changes. We can > always split it out later if we need to. There are already multiple cases where they diverge, actually: ADLS's Python client was apparently eventually consistent for some reason. That should not be the case with ABFS, however. So tests tagged with SkipIfADLS.eventually_consistent should not be skipped if it's ABFS. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Wed, 10 Oct 2018 17:47:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 3: > (1 comment) No, thank you. Will include this tweak in my re-testing of all the other style changes and resubmit. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Wed, 10 Oct 2018 17:45:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 3: Quick comment: I think we are going to be skipping the same tests for ADLS and ABFS. I would rather smash them together and reuse the existing skipping condition we have for ADLS to cover both. That should eliminate the need for a bunch of test changes. We can always split it out later if we need to. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Tue, 09 Oct 2018 23:15:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11630/3/be/src/util/hdfs-util.cc File be/src/util/hdfs-util.cc: http://gerrit.cloudera.org:8080/#/c/11630/3/be/src/util/hdfs-util.cc@94 PS3, Line 94: return strncmp(path, "adl://", 6) == 0; Is this right? -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Tue, 09 Oct 2018 22:59:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1002/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Tue, 09 Oct 2018 22:47:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1000/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 2 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Tue, 09 Oct 2018 22:34:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 3: I will begin re-testing with the changes shown here. Other than style, the changes in these iterations include not skipping tests that were previously skipped for IMPALA-5335, as that shouldn't apply to the abfs_util.py (of course we don't empirically know that yet). I also removed the parameters passed to the ABFSClient constructor because this simply works through the Hadoop CLI which should already be configured with that information and the default FS (because there is not native Python client for ABFS right now). -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: mackror...@apache.org Gerrit-Comment-Date: Tue, 09 Oct 2018 22:09:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11630 to look at the new patch set (#3). Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. URI schemes are configured separately, however, so new functions are needed. This also allows behavior to be tweaked independently should any quirks be found in subsequent performance testing. Was not able to get a full test run completed due to snags in the data loading on non-HDFS storage. Consequently, abfs_util.py has not yet been through a full test run either. Patch was tested against TPC-DS with no apparent problems. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 44 files changed, 320 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/3 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 3 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/11630/2/tests/metadata/test_ddl.py File tests/metadata/test_ddl.py: http://gerrit.cloudera.org:8080/#/c/11630/2/tests/metadata/test_ddl.py@28 PS2, Line 28: from tests.common.skip import SkipIf, SkipIfABFS, SkipIfADLS, SkipIfLocal flake8: F401 'tests.common.skip.SkipIfABFS' imported but unused http://gerrit.cloudera.org:8080/#/c/11630/2/tests/util/abfs_util.py File tests/util/abfs_util.py: http://gerrit.cloudera.org:8080/#/c/11630/2/tests/util/abfs_util.py@29 PS2, Line 29: class ABFSClient(BaseFilesystem): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/11630/2/tests/util/abfs_util.py@64 PS2, Line 64: flake8: E261 at least two spaces before inline comment http://gerrit.cloudera.org:8080/#/c/11630/2/tests/util/abfs_util.py@76 PS2, Line 76: = flake8: E225 missing whitespace around operator -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 2 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Oct 2018 22:02:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11630 to look at the new patch set (#2). Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. URI schemes are configured separately, however, so new functions are needed. This also allows behavior to be tweaked independently should any quirks be found in subsequent performance testing. Was not able to get a full test run completed due to snags in the data loading on non-HDFS storage. Consequently, abfs_util.py has not yet been through a full test run either. Patch was tested against TPC-DS with no apparent problems. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 45 files changed, 320 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/2 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 2 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11630 ) Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/999/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 1 Gerrit-Owner: mackror...@apache.org Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Oct 2018 21:45:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/11630 )
Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
..
Patch Set 1:
(20 comments)
http://gerrit.cloudera.org:8080/#/c/11630/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/11630/1/bin/impala-config.sh@376
PS1, Line 376:
DEFAULT_FS="abfss://${azure_storage_container_name}@${azure_storage_account_name}.dfs.core.windows.net"
line too long (105 > 90)
http://gerrit.cloudera.org:8080/#/c/11630/1/fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
File fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java:
http://gerrit.cloudera.org:8080/#/c/11630/1/fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java@148
PS1, Line 148: !(fs instanceof AzureBlobFileSystem) && !(fs
instanceof SecureAzureBlobFileSystem) &&
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/common/skip.py
File tests/common/skip.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/common/skip.py@59
PS1, Line 59: class SkipIfABFS:
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/common/skip.py@63
PS1, Line 63:
flake8: E261 at least two spaces before inline comment
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/common/skip.py@63
PS1, Line 63: t
flake8: E501 line too long (96 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/common/skip.py@77
PS1, Line 77:
flake8: E261 at least two spaces before inline comment
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/common/skip.py@77
PS1, Line 77: u
flake8: E501 line too long (92 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/custom_cluster/test_hdfs_fd_caching.py@136
PS1, Line 136: s
flake8: E501 line too long (95 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/custom_cluster/test_hdfs_fd_caching.py@152
PS1, Line 152: s
flake8: E501 line too long (95 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/failure/test_failpoints.py
File tests/failure/test_failpoints.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/failure/test_failpoints.py@29
PS1, Line 29: I
flake8: E501 line too long (97 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/metadata/test_ddl.py@91
PS1, Line 91:
flake8: E261 at least two spaces before inline comment
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/query_test/test_hdfs_caching.py
File tests/query_test/test_hdfs_caching.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/query_test/test_hdfs_caching.py@28
PS1, Line 28:
flake8: E501 line too long (99 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/query_test/test_insert_parquet.py@32
PS1, Line 32: k
flake8: E501 line too long (99 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py
File tests/util/abfs_util.py:
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@23
PS1, Line 23: import os
flake8: F401 'os' imported but unused
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@29
PS1, Line 29: from tests.util.filesystem_utils import ADLS_CLIENT_ID,
ADLS_TENANT_ID, ADLS_CLIENT_SECRET
flake8: F401 'tests.util.filesystem_utils.ADLS_CLIENT_ID' imported but unused
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@29
PS1, Line 29: from tests.util.filesystem_utils import ADLS_CLIENT_ID,
ADLS_TENANT_ID, ADLS_CLIENT_SECRET
flake8: F401 'tests.util.filesystem_utils.ADLS_CLIENT_SECRET' imported but
unused
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@29
PS1, Line 29: from tests.util.filesystem_utils import ADLS_CLIENT_ID,
ADLS_TENANT_ID, ADLS_CLIENT_SECRET
flake8: F401 'tests.util.filesystem_utils.ADLS_TENANT_ID' imported but unused
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@31
PS1, Line 31: class ABFSClient(BaseFilesystem):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@70
PS1, Line 70:
flake8: E261 at least two spaces before inline comment
http://gerrit.cloudera.org:8080/#/c/11630/1/tests/util/abfs_util.py@82
PS1, Line 82: =
flake8: E225 missing whitespace around operator
--
To view, visit http://gerrit.cloudera.org:8080/11630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Ch
[Impala-ASF-CR] IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support.
mackror...@apache.org has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11630 Change subject: IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. .. IMPALA-7681. Add Azure Blob File System (ADLS Gen2) support. HADOOP-15407 adds a new FileSystem implementation called "ABFS" for the ADLS Gen2 service. It's in the hadoop-azure module as a replacement for WASB. Filesystem semantics should be the same, so skipped tests and other behavior changes have simply mirrored what is done for ADLS Gen1 by default. URI schemes are configured separately, however, so new functions are needed. This also allows behavior to be tweaked independently should any quirks be found in subsequent performance testing. Was not able to get a full test run completed due to snags in the data loading on non-HDFS storage. Consequently, abfs_util.py has not yet been through a full test run either. Patch was tested against TPC-DS with no apparent problems. Change-Id: I5120b071760e7655e78902dce8483f8f54de445d --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/scan-range.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_scanners.py M tests/stress/test_ddl_stress.py A tests/util/abfs_util.py M tests/util/filesystem_utils.py 45 files changed, 326 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/11630/1 -- To view, visit http://gerrit.cloudera.org:8080/11630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5120b071760e7655e78902dce8483f8f54de445d Gerrit-Change-Number: 11630 Gerrit-PatchSet: 1 Gerrit-Owner: mackror...@apache.org
