Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12097 )
Change subject: IMPALA-7980: Fix spinning because of buggy num_unqueued_files_. ...................................................................... Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/12097/5/be/src/exec/hdfs-scan-node-base.h File be/src/exec/hdfs-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/12097/5/be/src/exec/hdfs-scan-node-base.h@465 PS5, Line 465: remaining_scan_range_issuances_ > This feels a bit of a misnomer as parquet scanner will actually issue more Do you have any suggestions? Todd previously said: If we want to make this reflect its actual numeric value, maybe something like remaining_scan_range_addition_steps ? ie the Parquet scanner has one "scan range addition step" whereas the text ones have one such step per file? Or perhaps remaining_scan_range_issuances_? http://gerrit.cloudera.org:8080/#/c/12097/5/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/12097/5/be/src/exec/hdfs-scan-node.cc@78 PS5, Line 78: if (!(ranges_issued_barrier_.pending() != 0) && initial_ranges_issued_ && progress_.done()) { > line too long (95 > 90) Done http://gerrit.cloudera.org:8080/#/c/12097/5/be/src/exec/hdfs-scan-node.cc@79 PS5, Line 79: auto remaining = remaining_scan_range_issuances_.Load(); : if (remaining != 0) { : LOG(INFO) << "XXX Assertion Failed " << remaining << " " << GetStackTrace(); : } > DCHECK_EQ(remaining_scan_range_issuances_.Load(), 0) << GetStackTrace(); D'oh. I don't know how this one got by me. http://gerrit.cloudera.org:8080/#/c/12097/5/be/src/exec/hdfs-scan-node.cc@212 PS5, Line 212: thread_state_.Close(this); > I wonder if you can DCHECK remaining_scan_range_issuances_ to 0 after threa I'll try it. I think the limit reach case is handled by progress_.done() being false. -- To view, visit http://gerrit.cloudera.org:8080/12097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I133de13238d3d05c510e2ff771d48979125735b1 Gerrit-Change-Number: 12097 Gerrit-PatchSet: 5 Gerrit-Owner: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Tue, 29 Jan 2019 00:26:48 +0000 Gerrit-HasComments: Yes
