[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15142 ) Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ This fixes a subtle memory managment issue where freeing of a buffer is delayed longer than it should be. This means that the full buffer pool reservation is not available for repartitioning, which can lead to crashes or hang for very specific queries. The fix is to transfer resources from output_unmatched_batch_ as soon as the last row from the batch is appended to the output batch. This bug would only be triggered by join modes that output unmatched rows from the right side (RIGHT OUTER JOIN, FULL OUTER JOIN, RIGHT ANTI JOIN) *and* have an empty probe side (otherwise unmatched rows are output by iterating over the hash table). Testing: Added DCHECKs to check that all resources are available before repartitioning. Added a regression test that triggered the bug. Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Reviewed-on: http://gerrit.cloudera.org:8080/15142 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-node.cc M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test 3 files changed, 22 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15142 ) Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 01 Feb 2020 02:25:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15142 ) Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5487/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 31 Jan 2020 21:33:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15142 ) Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 31 Jan 2020 21:33:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15142 ) Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 31 Jan 2020 21:31:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15142 ) Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5571/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 31 Jan 2020 17:45:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9349: free output unmatched batch buffers promptly in PHJ
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15142 Change subject: IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ .. IMPALA-9349: free output_unmatched_batch_ buffers promptly in PHJ This fixes a subtle memory managment issue where freeing of a buffer is delayed longer than it should be. This means that the full buffer pool reservation is not available for repartitioning, which can lead to crashes or hang for very specific queries. The fix is to transfer resources from output_unmatched_batch_ as soon as the last row from the batch is appended to the output batch. This bug would only be triggered by join modes that output unmatched rows from the right side (RIGHT OUTER JOIN, FULL OUTER JOIN, RIGHT ANTI JOIN) *and* have an empty probe side (otherwise unmatched rows are output by iterating over the hash table). Testing: Added DCHECKs to check that all resources are available before repartitioning. Added a regression test that triggered the bug. Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-node.cc M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test 3 files changed, 22 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/15142/1 -- To view, visit http://gerrit.cloudera.org:8080/15142 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie13b51d4d909afb0fe2e7b7dc00b085c51058fed Gerrit-Change-Number: 15142 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
