Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11596 )
Change subject: IMPALA-7662: fix error race when scanner open fails ...................................................................... IMPALA-7662: fix error race when scanner open fails This is very similar to IMPALA-7335, except happens when 'progress_' is incremented in the call chain HdfsScanNode::ProcessSplit -> HdfsScanNodeBase::CreateAndOpenScanner() -> HdfsScanner::Close() The fix required restructuring the code so that SetDoneInternal() is called with the error *before* HdfsScanner::Close(). This required a refactoring because HdfsScanNodeBase doesn't actually know about SetDoneInternal(). My fix is to put the common logic between HdfsScanNode and HdfsScanNodeMt into a helper in HdfsScanNodeBase, then in HdfsScanNode, make sure to call SetDoneInternal() before closing the scanner. I also reworked HdfsScanNode::ProcessSplit() to handle error propagation internally. I think the joint responsibility between ProcessSplit() and its caller for handling errors made things harder than necessary. Testing: Added a debug action and test that reproduced the race before the fix. Change-Id: I45a61210ca7d057b048c77d9f2f2695ec450f19b Reviewed-on: http://gerrit.cloudera.org:8080/11596 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/exec-node.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-scanner.h M common/thrift/PlanNodes.thrift A testdata/workloads/functional-query/queries/QueryTest/parquet-error-propagation-race.test M tests/failure/test_failpoints.py M tests/query_test/test_scanners.py 12 files changed, 108 insertions(+), 39 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I45a61210ca7d057b048c77d9f2f2695ec450f19b Gerrit-Change-Number: 11596 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Pooja Nilangekar <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
