Henry Robinson has submitted this change and it was merged. Change subject: IMPALA-4415: Fix unassigned scan range of size 1 ......................................................................
IMPALA-4415: Fix unassigned scan range of size 1 ComputeScanRangeAssignment() computes the per-node scan-range load as the total number of bytes to be scanned / number of nodes, and casts that to a float. However, that computation suffers from a precision issue where the average * num_nodes may be 1 byte less than the total. The scan range assignment loop continues until at least average * num_nodes bytes have been assigned. If the last scan range has only 1 byte, it will not be assigned (if it has 2 or more bytes, it will be considered for assignment before the loop exit condition is met). The fix is to make sure that all instances are assigned in the last iteration of the assignment loop, even if the per-node threshold is already met. Testing: No local repro was found - only S3 and LocalFS builds have it. A unit test requires a lot of infrastructure from simple-scheduler-util that doesn't exist yet (e.g. ComputeScanRangeAssignment() is not called). S3 and LocalFS full test builds are in progress. Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6 Reviewed-on: http://gerrit.cloudera.org:8080/4907 Tested-by: Internal Jenkins Reviewed-by: Henry Robinson <[email protected]> --- M be/src/scheduling/simple-scheduler.cc 1 file changed, 14 insertions(+), 7 deletions(-) Approvals: Henry Robinson: Looks good to me, approved Internal Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/4907 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id3af767ee9d121ca62ac383ef9e696a18dc903d6 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Henry Robinson <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Marcel Kornacker <[email protected]>
