[
https://issues.apache.org/jira/browse/IMPALA-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17887377#comment-17887377
]
Michael Smith commented on IMPALA-13422:
----------------------------------------
[~boroknagyz] might be fixing this with https://gerrit.cloudera.org/c/21882/.
> TestIcebergTable.test_load failed when running concurrently
> -----------------------------------------------------------
>
> Key: IMPALA-13422
> URL: https://issues.apache.org/jira/browse/IMPALA-13422
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> TestIcebergTable.test_load failed when two tests running concurrently in an
> exhaustive build:
> {code:java}
> 19:12:26 [gw2] FAILED
> query_test/test_iceberg.py::TestIcebergTable::test_load[protocol: beeswax |
> exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]
> 19:12:26 [gw3] FAILED
> query_test/test_iceberg.py::TestIcebergTable::test_load[protocol: beeswax |
> exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]{code}
> Stacktrace of both tests are the same:
> {code:python}
> query_test/test_iceberg.py:1300: in test_load
> "iceberg_mixed_file_format_test", "parquet")
> common/file_utils.py:62: in create_iceberg_table_from_directory
> check_call(['hdfs', 'dfs', '-put', '-d', local_dir, hdfs_dir])
> /data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/subprocess.py:190:
> in check_call
> raise CalledProcessError(retcode, cmd)
> E CalledProcessError: Command '['hdfs', 'dfs', '-put', '-d',
> '/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/testdata/data/iceberg_test/iceberg_mixed_file_format_test',
> '/test-warehouse/iceberg_mixed_file_format_test']' returned non-zero exit
> status 1{code}
> Stderr of the first test:
> {code}
> CREATE DATABASE `test_load_a61184e9`;
> -- 2024-10-03 19:12:05,971 INFO MainThread: Started query
> 0e431e374cd58716:0013fb8100000000
> -- 2024-10-03 19:12:06,022 INFO MainThread: Created database
> "test_load_a61184e9" for test ID
> "query_test/test_iceberg.py::TestIcebergTable::()::test_load[protocol:
> beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]"
> Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
> Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906113255_8d49367d-e338-4996-ade5-ee500a19c1d1-job_16619542960420_0003-1-00001.orc':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/d43cc1ea-096f-4594-9583-b1b27f8f0230-m0.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-5574591442446832859-1-055baf62-de6d-4583-bf21-f187f9482343.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-6603964444137547572-1-871d1473-8566-46c0-a530-a2256b3f396f.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v2.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v4.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v6.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v8.metadata.json':
> File exists{code}
> stderr of the second test:
> {code}
> CREATE DATABASE `test_load_53f920bb`;
> -- 2024-10-03 19:12:05,971 INFO MainThread: Started query
> 68460e4230189328:9f6261bc00000000
> -- 2024-10-03 19:12:06,022 INFO MainThread: Created database
> "test_load_53f920bb" for test ID
> "query_test/test_iceberg.py::TestIcebergTable::()::test_load[protocol:
> beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]"
> Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
> Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906114830_907f72c7-36ac-4135-8315-27ff880faff0-job_16619542960420_0004-1-00001.parquet':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906114900_9c1b7b46-5643-428f-a007-519c5500ed04-job_16619542960420_0004-1-00001.orc':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/055baf62-de6d-4583-bf21-f187f9482343-m0.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/871d1473-8566-46c0-a530-a2256b3f396f-m0.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/a0332d05-4c9c-4640-8656-69ba61b7859a-m0.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-1847986799759674135-1-a0332d05-4c9c-4640-8656-69ba61b7859a.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-5589333083859779254-1-d43cc1ea-096f-4594-9583-b1b27f8f0230.avro':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v1.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v3.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v5.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/v7.metadata.json':
> File exists
> put:
> `/test-warehouse/iceberg_mixed_file_format_test/metadata/version-hint.txt':
> File exists{code}
> It seems the failure is due to two runs of the test adding files to the same
> path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
