[
https://issues.apache.org/jira/browse/IMPALA-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161705#comment-17161705
]
Fang-Yu Rao commented on IMPALA-9453:
-------------------------------------
We have observed the same failed test in a recent s3 build but impalad's do not
crash.
We did not see any error in {{load-test-warehouse-snapshot.log}}. But we did
see some s3-related errors in {{impalad_node1.ERROR}} which may result from
some end-to-end test.
{code:java}
AmazonS3Exception: The specified key does not exist. (Service: Amazon S3;
Status Code: 404; Error Code: NoSuchKey; Request ID: 50641CCF095FB531; S3
Extended Request ID:
87dbcAKzl1qztI6sofSaiZfLqy5Xk8yAHfdTSSTdGwhsGWjpwAc89G5ojpcVD/O5yZ4pskMMMpM=)java.io.FileNotFoundException:
re-open
s3a://impala-test-uswest2-1/test-warehouse/test_file_modifications_2d724663/data.csv
at 0 on
s3a://impala-test-uswest2-1/test-warehouse/test_file_modifications_2d724663/data.csv:
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not
exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request
ID: 50641CCF095FB531; S3 Extended Request ID:
87dbcAKzl1qztI6sofSaiZfLqy5Xk8yAHfdTSSTdGwhsGWjpwAc89G5ojpcVD/O5yZ4pskMMMpM=),
S3 Extended Request ID:
87dbcAKzl1qztI6sofSaiZfLqy5Xk8yAHfdTSSTdGwhsGWjpwAc89G5ojpcVD/O5yZ4pskMMMpM=:NoSuchKey
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:258)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:112)
at
org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:205)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:359)
at
org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:223)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:349)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:221)
at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:265)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464)
at
org.apache.hadoop.fs.s3a.S3AInputStream.readFully(S3AInputStream.java:731)
at
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key
does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey;
Request ID: 50641CCF095FB531; S3 Extended Request ID:
87dbcAKzl1qztI6sofSaiZfLqy5Xk8yAHfdTSSTdGwhsGWjpwAc89G5ojpcVD/O5yZ4pskMMMpM=),
S3 Extended Request ID:
87dbcAKzl1qztI6sofSaiZfLqy5Xk8yAHfdTSSTdGwhsGWjpwAc89G5ojpcVD/O5yZ4pskMMMpM=
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866)
at
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1467)
at
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$reopen$0(S3AInputStream.java:206)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
... 11 more
hdfsOpenFile(s3a://impala-test-uswest2-1/test-warehouse/test_file_modifications_2d724663/data.csv):
FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
error:
FileNotFoundException: Path
s3a://impala-test-uswest2-1/test-warehouse/test_file_modifications_2d724663/data.csv
is recorded as deleted by S3Guard at
2020-07-19T05:49:47.235Zjava.io.FileNotFoundException: Path
s3a://impala-test-uswest2-1/test-warehouse/test_file_modifications_2d724663/data.csv
is recorded as deleted by S3Guard at 2020-07-19T05:49:47.235Z
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2792)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:4614)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1068)
at org.apache.hadoop.fs.s3a.S3AFileSystem.open(S3AFileSystem.java:1046)
hdfsPread: FSDataInputStream#read error:
{code}
It seems that there may be something wrong with a database named
{{test_file_modifications_2d724663}}. We are not very sure whether it is
related to
[https://github.infra.cloudera.com/CDH/Impala/blob/cdpd-master/tests/query_test/test_hdfs_file_mods.py#L68-L130].
The log files of impalad are also provided for easy reference.
[^impalad_node1.ERROR] [^impalad_node2.ERROR] [^impalad.ERROR]
> S3 build failed with many strange symptoms
> ------------------------------------------
>
> Key: IMPALA-9453
> URL: https://issues.apache.org/jira/browse/IMPALA-9453
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 3.4.0
> Reporter: Tim Armstrong
> Assignee: Sahil Takiar
> Priority: Blocker
> Labels: broken-build, crash
> Attachments: impalad.ERROR, impalad_node1.ERROR, impalad_node2.ERROR
>
>
> There were a lot of incorrect results:
> {noformat}
> uery_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder) common/test_result_verifier.py:456: in
> verify_raw_results VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results E assert Comparing
> QueryTestResults (expected vs actual): E 7300 != 6990
> Stacktrace
> query_test/test_mt_dop.py:49: in test_mt_dop
> self.run_test_case('QueryTest/mt-dop', new_vector)
> common/impala_test_suite.py:690: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:523: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:456: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:278: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 7300 != 6990
> Standard Error
> ERROR:test_configuration:Comparing QueryTestResults (expected vs actual):
> 7300 != 6990
> {noformat}
> The impalads eventually crashed:
> {noformat}
> F0302 00:50:55.607841 483 parquet-page-reader.cc:67]
> e24eb0839fa75423:8ac0bf7300000002] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a impala::HdfsParquetScanner::GetNextInternal()
> @ 0x2c8403e impala::HdfsParquetScanner::ProcessSplit()
> @ 0x28b826b impala::HdfsScanNode::ProcessSplit()
> @ 0x28b7440 impala::HdfsScanNode::ScannerThread()
> @ 0x28b679d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x28b8d91
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x20aa1e9 boost::function0<>::operator()()
> @ 0x266a84a impala::Thread::SuperviseThread()
> @ 0x2672ace boost::_bi::list5<>::operator()<>()
> @ 0x26729f2 boost::_bi::bind_t<>::operator()()
> @ 0x26729b5 boost::detail::thread_data<>::run()
> @ 0x3e98b19 thread_proxy
> @ 0x7f8c2ccefe24 start_thread
> @ 0x7f8c2985b34c __clone
> {noformat}
> {noformat}
> F0302 00:50:41.466794 32643 parquet-page-reader.cc:67]
> dd48a46583bea9c8:e3be641000000002] Check failed: col_end <
> file_desc.file_length (7010 vs. 7010)
> *** Check failure stack trace: ***
> @ 0x4f7277c google::LogMessage::Fail()
> @ 0x4f74021 google::LogMessage::SendToLog()
> @ 0x4f72156 google::LogMessage::Flush()
> @ 0x4f7571d google::LogMessageFatal::~LogMessageFatal()
> @ 0x2e3a520 impala::ParquetPageReader::InitColumnChunk()
> @ 0x2e37dee impala::ParquetColumnChunkReader::InitColumnChunk()
> @ 0x2cd8000 impala::BaseScalarColumnReader::Reset()
> @ 0x2c91239 impala::HdfsParquetScanner::InitScalarColumns()
> @ 0x2c8775a impala::HdfsParquetScanner::NextRowGroup()
> @ 0x2c85c2a impala::HdfsParquetScanner::GetNextInternal()
> @ 0x2c8403e impala::HdfsParquetScanner::ProcessSplit()
> @ 0x28b826b impala::HdfsScanNode::ProcessSplit()
> @ 0x28b7440 impala::HdfsScanNode::ScannerThread()
> @ 0x28b679d
> _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
> @ 0x28b8d91
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x20aa1e9 boost::function0<>::operator()()
> @ 0x266a84a impala::Thread::SuperviseThread()
> @ 0x2672ace boost::_bi::list5<>::operator()<>()
> @ 0x26729f2 boost::_bi::bind_t<>::operator()()
> @ 0x26729b5 boost::detail::thread_data<>::run()
> @ 0x3e98b19 thread_proxy
> @ 0x7f16a5766e24 start_thread
> @ 0x7f16a22d234c __clone
> Wrote minidump to
> /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/ee_tests/minidumps/impalad/49436f97-da0e-47c9-76b372a5-6b3fb146.dmp
> {noformat}
> If it makes a differnence, this was using CDP, so that version of the S3
> connector.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
