[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Internal Jenkins has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Internal Jenkins has submitted this change and it was merged. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. IMPALA-4076: Fix runtime filter sort compare method Fixed 2 isssues: - The getSelectivity() method sometimes returned NaN double values which could not be sorted properly. - The compare method for sorting runtime filters was swtiched to use the builtin Double comparison method. Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Reviewed-on: http://gerrit.cloudera.org:8080/4652 Reviewed-by: Alex Behm Tested-by: Internal Jenkins --- M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 2 files changed, 202 insertions(+), 3 deletions(-) Approvals: Internal Jenkins: Verified Alex Behm: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 5 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Internal Jenkins Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Alex Behm has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Taras Bobrovytsky has uploaded a new patch set (#4). Change subject: IMPALA-4076: Fix runtime filter sort compare method .. IMPALA-4076: Fix runtime filter sort compare method Fixed 2 isssues: - The getSelectivity() method sometimes returned NaN double values which could not be sorted properly. - The compare method for sorting runtime filters was swtiched to use the builtin Double comparison method. Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 --- M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 2 files changed, 202 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/4652/4 -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Taras Bobrovytsky has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/4652/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: Line 341: } > Sorry, I confused myself (and the join sides). You are right. Please leave Done http://gerrit.cloudera.org:8080/#/c/4652/2/testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test File testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test: Line 1123: # constructed by hand to trigger the issue with the sort compare method violating the > You are right, let's remove setting the query option in PlannerTest, sorry. Ok, removed. -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Alex Behm has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/4652/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: Line 341: } > The equation is trying to get the selectivity from cardinalities. Only doin Sorry, I confused myself (and the join sides). You are right. Please leave as is :) -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Alex Behm has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/4652/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: Line 341: } > We are checking the cardinality of the child left child (the potential dest The equation is trying to get the selectivity from cardinalities. Only doing a division is not sufficient because it doesn't correctly capture the 0 cardinality case. So the math is wrong, and we need an extra check to compensate. I maintain that the selectivity should be 0 if one side of an inner join has a 0 cardinality. Think about it this way: Returning -1 will discard the most selective runtime filters. http://gerrit.cloudera.org:8080/#/c/4652/2/testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test File testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test: Line 1123: # so the number of runtime filters has to be greater than 32 and they have to be in a > I don't think it would silently fail, because the plan would change. But ju You are right, let's remove setting the query option in PlannerTest, sorry. -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Taras Bobrovytsky has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/4652/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: Line 341: } > This doesn't seem quite right. A cardinality of 0 translates to a selectivi We are checking the cardinality of the child left child (the potential destination of the runtime filters). X/0 can never equal zero no matter what X is. http://gerrit.cloudera.org:8080/#/c/4652/2/testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test File testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test: Line 1121: # IMPALA-4076: This query was constructed by hand to trigger the issue with sorting of > Should also have brief description of the scenario being tested, something Done Line 1123: # so the number of runtime filters has to be greater than 32 and they have to be in a > We should set this 32 number a part of the planner test. Otherwise, if we c I don't think it would silently fail, because the plan would change. But just in case, I set the query option to 10. Line 1203: inner join small_two on (True) > you can remove the "on (True)" conditions Done -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Taras Bobrovytsky has uploaded a new patch set (#3). Change subject: IMPALA-4076: Fix runtime filter sort compare method .. IMPALA-4076: Fix runtime filter sort compare method Fixed 2 isssues: - The getSelectivity() method sometimes returned NaN double values which could not be sorted properly. - The compare method for sorting runtime filters was swtiched to use the builtin Double comparison method. Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 --- M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 3 files changed, 206 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/4652/3 -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 3 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Alex Behm has posted comments on this change. Change subject: IMPALA-4076: Fix runtime filter sort compare method .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/4652/2/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: Line 341: } This doesn't seem quite right. A cardinality of 0 translates to a selectivity of 0 (not -1) http://gerrit.cloudera.org:8080/#/c/4652/2/testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test File testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test: Line 1121: # IMPALA-4076: This query was constructed by hand to trigger the issue with sorting of Should also have brief description of the scenario being tested, something like: Test pruning the least selective runtime filters to obey QUERY_OPTION_NAME in the presence of zero-cardinality plan nodes. Line 1123: # so the number of runtime filters has to be greater than 32 and they have to be in a We should set this 32 number a part of the planner test. Otherwise, if we change default value of the query option, we may silently lose test coverage. Line 1203: inner join small_two on (True) you can remove the "on (True)" conditions -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Taras Bobrovytsky has uploaded a new patch set (#2). Change subject: IMPALA-4076: Fix runtime filter sort compare method .. IMPALA-4076: Fix runtime filter sort compare method Fixed 2 isssues: - The getSelectivity() method sometimes returned NaN double values which could not be sorted properly. - The compare method for sorting runtime filters was swtiched to use the builtin Double comparison method. Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 --- M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 2 files changed, 201 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/4652/2 -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 2 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm
[Impala-ASF-CR] IMPALA-4076: Fix runtime filter sort compare method
Taras Bobrovytsky has uploaded a new change for review. http://gerrit.cloudera.org:8080/4652 Change subject: IMPALA-4076: Fix runtime filter sort compare method .. IMPALA-4076: Fix runtime filter sort compare method Fixed 2 isssues: - The getSelectivity() method sometimes returned NaN double values which could not be sorted properly. - The compare method for sorting runtime filters was swtiched to use the builtin Double comparison method. Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 --- M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test 2 files changed, 202 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/4652/1 -- To view, visit http://gerrit.cloudera.org:8080/4652 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iad433f2ece423ea29e79e81b68fa53cb0af18378 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky