Fucun Chu has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17503 )
Change subject: IMPALA-10771: Add Tencent COS support ...................................................................... IMPALA-10771: Add Tencent COS support This patch adds support for COS(Cloud Object Storage). Using the hadoop-cos, the implementation is similar to other remote FileSystems. New flags for COS: - num_cos_io_threads: Number of COS I/O threads. Defaults to be 16. Follow-up: - Support for caching COS file handles will be addressed in IMPALA-10772. - test_concurrent_inserts and test_failing_inserts in test_acid_stress.py are skipped due to slow file listing on COS (IMPALA-10773). Tests: - Upload hdfs test data to a COS bucket. Modify all locations in HMS DB to point to the COS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: Idce135a7591d1b4c74425e365525be3086a39821 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/bin/run-all.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/authorization/test_ranger.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_hive_text_codec_interop.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_lineage.py M tests/custom_cluster/test_local_catalog.py M tests/custom_cluster/test_local_tz_conversion.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_metastore_service.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_query_retries.py M tests/custom_cluster/test_restart_services.py M tests/custom_cluster/test_topic_update_frequency.py M tests/data_errors/test_data_errors.py M tests/failure/test_failpoints.py M tests/metadata/test_catalogd_debug_actions.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_partition_metadata.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_acid.py M tests/query_test/test_date_queries.py M tests/query_test/test_hbase_queries.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_join_queries.py M tests/query_test/test_nested_types.py M tests/query_test/test_observability.py M tests/query_test/test_partitioning.py M tests/query_test/test_resource_limits.py M tests/query_test/test_scanners.py M tests/stress/test_acid_stress.py M tests/stress/test_ddl_stress.py M tests/util/filesystem_utils.py 60 files changed, 275 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/17503/5 -- To view, visit http://gerrit.cloudera.org:8080/17503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idce135a7591d1b4c74425e365525be3086a39821 Gerrit-Change-Number: 17503 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu <[email protected]> Gerrit-Reviewer: Fucun Chu <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
