Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12097 )
Change subject: IMPALA-7980: Fix spinning threads because of buggy handling of num_unqueued_files_. ...................................................................... Patch Set 2: (10 comments) http://gerrit.cloudera.org:8080/#/c/12097/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12097/2//COMMIT_MSG@19 PS2, Line 19: set to 1 initially; decrement at exit points of : IssueInitialRanges that seems like the negation of what I'd expect given the name above? http://gerrit.cloudera.org:8080/#/c/12097/2//COMMIT_MSG@21 PS2, Line 21: base-sequence-scanner.cc. did you consider an approach like adding something like this to the top level base class: virtual bool AllScanRangesIssued() = 0; and then in parquet you can implement it as 'return initial_scan_ranges_issued_;' and then all the fancy counter stuff can go into base-sequence-scanner.cc? http://gerrit.cloudera.org:8080/#/c/12097/2//COMMIT_MSG@38 PS2, Line 38: ~30 threads to be spinning in the kernel. We were able to use perf to finda lot typo http://gerrit.cloudera.org:8080/#/c/12097/2//COMMIT_MSG@74 PS2, Line 74: a an typo http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.h File be/src/exec/hdfs-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.h@266 PS2, Line 266: Should Must? http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.h@277 PS2, Line 277: plugins impala has plugins? perhaps we can use [[deprecated]] or something here? ATTRIBUTE_DEPRECATED? (do we have that in impala's copy of gutil?) http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.h@287 PS2, Line 287: no_more_add_disk_io_ranges_possible_.Add(delta); can you DCHECK that the result doesn't go below zero? http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.h@471 PS2, Line 471: no_more_add_disk_io_ranges_possible_ > I feel like this should be called more_add_disk_io_ranges_possible_, since yea, I was also confused by the inverted naming here. If we want to make this reflect its actual numeric value, maybe something like remaining_scan_range_addition_steps ? ie the Parquet scanner has one "scan range addition step" whereas the text ones have one such step per file? Or perhaps remaining_scan_range_issuances_? http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.cc File be/src/exec/hdfs-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node-base.cc@453 PS2, Line 453: UpdateNoMoreAddDiskIoRangesBarrier(-1); is it important to not decrement this in the error cases below on line 471-494? ie why not just unconditionally decrement this, in the same way that you unconditionally set initial_ranges_issued_ = true on L448? http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node.h File be/src/exec/hdfs-scan-node.h: http://gerrit.cloudera.org:8080/#/c/12097/2/be/src/exec/hdfs-scan-node.h@139 PS2, Line 139: : /// Number of times scanner thread didn't find work to do. : RuntimeProfile::Counter* scanner_thread_workless_loops_counter_ = nullptr; hrm, I see the value of this for your test, but not entirely convinced it makes sense to keep around in profiles. Curious on others' opinions -- To view, visit http://gerrit.cloudera.org:8080/12097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I133de13238d3d05c510e2ff771d48979125735b1 Gerrit-Change-Number: 12097 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Philip Zeyliger <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Fri, 18 Jan 2019 19:31:18 +0000 Gerrit-HasComments: Yes
