[
https://issues.apache.org/jira/browse/IMPALA-7733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886546#comment-16886546
]
Michael Ho edited comment on IMPALA-7733 at 7/16/19 11:26 PM:
--------------------------------------------------------------
A recent instance when running
{{query_test/test_tpcds_queries.py::TestTpcdsInsert}}:
{noformat}
query_test/test_tpcds_queries.py:521: in test_tpcds_partitioned_insert
self.run_test_case('partitioned-insert', vector)
common/impala_test_suite.py:563: in run_test_case
result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:500: in __exec_in_impala
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:798: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:184: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:364: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:385: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:Error(s) moving partition files. First error (of 1) was:
Hdfs op (RENAME
s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq
TO
s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq)
failed, error was:
s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq
E Error(5): Input/output error
{noformat}
was (Author: kwho):
A recent instance when running
{{query_test/test_tpcds_queries.py::TestTpcdsInsert::()::test_tpcds_partitioned_insert}}:
{noformat}
query_test/test_tpcds_queries.py:521: in test_tpcds_partitioned_insert
self.run_test_case('partitioned-insert', vector)
common/impala_test_suite.py:563: in run_test_case
result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:500: in __exec_in_impala
result = self.__execute_query(target_impalad_client, query, user=user)
common/impala_test_suite.py:798: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:184: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:364: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:385: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:Error(s) moving partition files. First error (of 1) was:
Hdfs op (RENAME
s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq
TO
s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq)
failed, error was:
s3a://<redacted>/test-warehouse/tpcds_parquet.db/store_sales_insert/_impala_insert_staging/834b0c158076d9d0_015f77df00000000/.834b0c158076d9d0-015f77df00000004_337386663_dir/ss_sold_date_sk=2451539/834b0c158076d9d0-015f77df00000004_1260764580_data.0.parq
E Error(5): Input/output error
{noformat}
> TestInsertParquetQueries.test_insert_parquet is flaky in S3 due to rename
> -------------------------------------------------------------------------
>
> Key: IMPALA-7733
> URL: https://issues.apache.org/jira/browse/IMPALA-7733
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 3.1.0
> Reporter: Vuk Ercegovac
> Assignee: Tianyi Wang
> Priority: Blocker
> Labels: broken-build, flaky
>
> I see two examples in the past two months or so where this test fails due to
> a rename error on S3. The test's stacktrace looks like this:
> {noformat}
> query_test/test_insert_parquet.py:112: in test_insert_parquet
> self.run_test_case('insert_parquet', vector, unique_database,
> multiple_impalad=True)
> common/impala_test_suite.py:408: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:625: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:176: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:350: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:371: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Error(s) moving partition files. First error (of 1) was:
> Hdfs op (RENAME
> s3a://<removed>/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq
> TO
> s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq)
> failed, error was:
> s3a://<removed>/test-warehouse/test_insert_parquet_968f37fe.db/orders_insert_table/_impala_insert_staging/4e45cd68bcddd451_3c7156ed00000000/.4e45cd68bcddd451-3c7156ed00000002_803672621_dir/4e45cd68bcddd451-3c7156ed00000002_448261088_data.0.parq
> E Error(5): Input/output error{noformat}
> Since we know this happens once in a while, some ideas to deflake it:
> * retry
> * check for this specific issue... if we think its platform flakiness, then
> we should skip it.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
