Quanlong Huang created IMPALA-13422:
---------------------------------------
Summary: TestIcebergTable.test_load failed when running
concurrently
Key: IMPALA-13422
URL: https://issues.apache.org/jira/browse/IMPALA-13422
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
TestIcebergTable.test_load failed when two tests running concurrently in an
exhaustive build:
{code:java}
19:12:26 [gw2] FAILED
query_test/test_iceberg.py::TestIcebergTable::test_load[protocol: beeswax |
exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
parquet/none]
19:12:26 [gw3] FAILED
query_test/test_iceberg.py::TestIcebergTable::test_load[protocol: beeswax |
exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error':
1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]{code}
Stacktrace of both tests are the same:
{code:python}
query_test/test_iceberg.py:1300: in test_load
"iceberg_mixed_file_format_test", "parquet")
common/file_utils.py:62: in create_iceberg_table_from_directory
check_call(['hdfs', 'dfs', '-put', '-d', local_dir, hdfs_dir])
/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/subprocess.py:190:
in check_call
raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command '['hdfs', 'dfs', '-put', '-d',
'/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/testdata/data/iceberg_test/iceberg_mixed_file_format_test',
'/test-warehouse/iceberg_mixed_file_format_test']' returned non-zero exit
status 1{code}
Stderr of the first test:
{code}
CREATE DATABASE `test_load_a61184e9`;
-- 2024-10-03 19:12:05,971 INFO MainThread: Started query
0e431e374cd58716:0013fb8100000000
-- 2024-10-03 19:12:06,022 INFO MainThread: Created database
"test_load_a61184e9" for test ID
"query_test/test_iceberg.py::TestIcebergTable::()::test_load[protocol: beeswax
| exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
parquet/none]"
Picked up JAVA_TOOL_OPTIONS:
-javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
Picked up JAVA_TOOL_OPTIONS:
-javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
put:
`/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906113255_8d49367d-e338-4996-ade5-ee500a19c1d1-job_16619542960420_0003-1-00001.orc':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/d43cc1ea-096f-4594-9583-b1b27f8f0230-m0.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-5574591442446832859-1-055baf62-de6d-4583-bf21-f187f9482343.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-6603964444137547572-1-871d1473-8566-46c0-a530-a2256b3f396f.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v2.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v4.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v6.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v8.metadata.json':
File exists{code}
stderr of the second test:
{code}
CREATE DATABASE `test_load_53f920bb`;
-- 2024-10-03 19:12:05,971 INFO MainThread: Started query
68460e4230189328:9f6261bc00000000
-- 2024-10-03 19:12:06,022 INFO MainThread: Created database
"test_load_53f920bb" for test ID
"query_test/test_iceberg.py::TestIcebergTable::()::test_load[protocol: beeswax
| exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error':
1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
Picked up JAVA_TOOL_OPTIONS:
-javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
Picked up JAVA_TOOL_OPTIONS:
-javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
put:
`/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906114830_907f72c7-36ac-4135-8315-27ff880faff0-job_16619542960420_0004-1-00001.parquet':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906114900_9c1b7b46-5643-428f-a007-519c5500ed04-job_16619542960420_0004-1-00001.orc':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/055baf62-de6d-4583-bf21-f187f9482343-m0.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/871d1473-8566-46c0-a530-a2256b3f396f-m0.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/a0332d05-4c9c-4640-8656-69ba61b7859a-m0.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-1847986799759674135-1-a0332d05-4c9c-4640-8656-69ba61b7859a.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-5589333083859779254-1-d43cc1ea-096f-4594-9583-b1b27f8f0230.avro':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v1.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v3.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v5.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/v7.metadata.json':
File exists
put:
`/test-warehouse/iceberg_mixed_file_format_test/metadata/version-hint.txt':
File exists{code}
It seems the failure is due to two runs of the test adding files to the same
path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
