subject:"\[Impala\-ASF\-CR\] IMPALA\-7942 \(part 2\)\: Add query hints for predicate selectivities"

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-20 Thread Qifan Chen (Code Review)

Qifan Chen has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity.
For some predicates, this may lead to worse query plan due to CBO.

This patch adds a new query hint: 'SELECTIVITY' to help specify a
selectivity value for a predicate.

The parser will interpret expressions wrapped in () followed by a
C-style comment /*  */ as a predicate hint. The
predicate hint currently can be in the form of +SELECTIVITY(f) where
'f' is a positive floating point number, in the range of (0, 1], to
use as the selectivity for the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound predicate example:

  select col from t where (a=1 or b=2) /* +SELECTIVITY(0.5) */;

As a limitation of this path, the selectivity hints for 'AND' compound
predicates, either in the original SQL query or internally generated,
are ignored. We may supported this in the near future.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Reviewed-on: http://gerrit.cloudera.org:8080/18023
Tested-by: Impala Public Jenkins 
Reviewed-by: Qifan Chen 
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
10 files changed, 414 insertions(+), 5 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Qifan Chen: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 36
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-20 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 34: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 34
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 20 Apr 2023 10:13:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-19 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 34: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 34
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 19 Apr 2023 18:17:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-19 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 34:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12818/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 34
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 19 Apr 2023 13:23:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-19 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 34:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9234/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 34
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 19 Apr 2023 13:03:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-19 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 34:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@12
PS33, Line 12:  to help specify a
 : sele
> nit. remove and replace with "to help specify"
Done


http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@18
PS33, Line 18: ,
> , in the range of [0, 1],
Done


http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@29
PS33, Line 29: As a limitation of this path, the selectivity hints for 'AND' 
compound
 : predicates, either in the original SQL query or
> As a limitation of this path, the selectivity hints for 'AND' compound pred
Done


http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@31
PS33, Line 31: are ignored. We may supported this in the near future.
> May file a new JIRA and mention the JIRA number here.
I didn't create a new JIRA yet, so not JIRA number provided here.


http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
File fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@153
PS33, Line 153: // 'AND' compound predicates will be replaced by 
children in Expr#getConjuncts,
> nit be
Done


http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@154
PS33, Line 154: // so selectivity hint will be missing, we add a 
warning here.
> nit be
Done


http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@155
PS33, Line 155: analyzer.addWarning("Selectivity hints are ignored for 
'AND' compound "
> There is one case in CaseExpr.java where the translation to AND predicate t
Done



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 34
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 19 Apr 2023 13:02:31 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-19 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#34). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity.
For some predicates, this may lead to worse query plan due to CBO.

This patch adds a new query hint: 'SELECTIVITY' to help specify a
selectivity value for a predicate.

The parser will interpret expressions wrapped in () followed by a
C-style comment /*  */ as a predicate hint. The
predicate hint currently can be in the form of +SELECTIVITY(f) where
'f' is a positive floating point number, in the range of (0, 1], to
use as the selectivity for the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound predicate example:

  select col from t where (a=1 or b=2) /* +SELECTIVITY(0.5) */;

As a limitation of this path, the selectivity hints for 'AND' compound
predicates, either in the original SQL query or internally generated,
are ignored. We may supported this in the near future.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
10 files changed, 414 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/34
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 34
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-18 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 33:

(7 comments)

Looks very good!

http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@12
PS33, Line 12: , we can use this
 : hint
nit. remove and replace with "to help specify"


http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@18
PS33, Line 18:
, in the range of [0, 1],


http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@29
PS33, Line 29: But pay attention, selectivity hint is invalid for 'AND' compound
 : predicates and between predicates in this patch
As a limitation of this path, the selectivity hints for 'AND' compound 
predicates, either in the original SQL query or internally generated, are 
ignored.


http://gerrit.cloudera.org:8080/#/c/18023/33//COMMIT_MSG@31
PS33, Line 31: this in the near future.
May file a new JIRA and mention the JIRA number here.


http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
File fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@153
PS33, Line 153: // 'AND' compound predicates will replaced by children 
in Expr#getConjuncts, so
nit be


http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@154
PS33, Line 154: // selectivity hint will missing, we add a warning here.
nit be


http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@155
PS33, Line 155: analyzer.addWarning("Selectivity hint is invalid for 
'AND' compound predicates.");
> Currently, this warning will not appear for BETWEEN predicate, since I remo
There is one case in CaseExpr.java where the translation to AND predicate takes 
place.

I think we can change the warning message to something like below to cover all 
such cases.

"Selectivity hints are ignored for 'AND' compound predicates, either in the SQL 
query or internally generated."



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 33
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Tue, 18 Apr 2023 22:54:33 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-17 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 33:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
File fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@155
PS33, Line 155: analyzer.addWarning("Selectivity hint is invalid for 
'AND' compound predicates.");
> Will this appear for BETWEEN predicate also?
Currently, this warning will not appear for BETWEEN predicate, since I removed 
selectivity assgined in 'BetweenToCompoundRule.java' when transform 
BetweenPredicate to CompoundPredicate.

If we want to appear for BETWEEN predicate also, we need to add code in 
'BetweenToCompoundRule.java' like this:
CompoundPredicate result = new CompoundPredicate(compoundOperator, lower, 
upper);
result.setSelectivityHint(bp.getSelectivity());
return result;

How do you think?



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 33
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Tue, 18 Apr 2023 02:30:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-17 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 33:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
File fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/33/fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java@155
PS33, Line 155: analyzer.addWarning("Selectivity hint is invalid for 
'AND' compound predicates.");
Will this appear for BETWEEN predicate also?



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 33
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 17 Apr 2023 16:45:05 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-13 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 33: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 33
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 13 Apr 2023 19:05:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-13 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 32:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12789/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 32
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 13 Apr 2023 14:04:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-13 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 33:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9224/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 33
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 13 Apr 2023 13:45:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-13 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#32). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity.
For some predicates, this may lead to worse query plan due to CBO.

This patch adds a new query hint: 'SELECTIVITY', we can use this
hint to specify a selectivity value for a predicate.

The parser will interpret expressions wrapped in () followed by a
C-style comment /*  */ as a predicate hint. The
predicate hint currently can be in the form of +SELECTIVITY(f) where
'f' is a positive floating point number to use as the selectivity for
the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound predicate example:

  select col from t where (a=1 or b=2) /* +SELECTIVITY(0.5) */;

But pay attention, selectivity hint is invalid for 'AND' compound
predicates and between predicates in this patch. We may supported
this in the near future.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
10 files changed, 412 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/32
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 32
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 31: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9220/


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 31
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 12 Apr 2023 21:14:32 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 30:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12784/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 12 Apr 2023 16:19:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 31:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9220/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 31
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 12 Apr 2023 16:01:29 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-12 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 30:

Finally, I removed modification in 'Expr', and this patch does not supported 
selectivity hint for 'AND' compound predicates, maybe we can implement this in 
other patch.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 12 Apr 2023 16:00:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-12 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#30). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity.
For some predicates, this may lead to worse query plan due to CBO.

This patch adds a new query hint: 'SELECTIVITY', we can use this
hint to specify a selectivity value for a predicate.

The parser will interpret expressions wrapped in () followed by a
C-style comment /*  */ as a predicate hint. The
predicate hint currently can be in the form of +SELECTIVITY(f) where
'f' is a positive floating point number to use as the selectivity for
the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound predicate example:

  select col from t where (a=1 or b=2) /* +SELECTIVITY(0.5) */;

But pay attention, selectivity hint is invalid for 'AND' compound
predicates and between predicates in this patch. We may supported
this in the near future.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
10 files changed, 418 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/30
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 30
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-12 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
> Hi Kurt, I think you are right. I will discuss with other people to find a
Back to earlier suggestion, it's ok to add a boolean flag here to or else a new 
function to collect expressions. Just make separate calls for applying hints 
don't change what is logically collected in other cases. If you need to 
propagate anything through the expression hierarchy, a visitor function would 
be more appropriate as collector will collect into a flat vector.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 12 Apr 2023 14:31:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-11 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate
selectivities
..

Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

Option 2) sounds like a good starting point to address 1). But we need to find
out how the selectivity at AND predicate is computed from child(0) and
child(1). From that we may be able to back fit the selectivity hint from the
AND predicate down. This will be for the general cases where SELECT_HINT(AND)
!= -1.

For the special case: SELECT_HINT(AND) = -1, then just store child(0) and
child(1) in the list without changing their selectivity hint (even being -1) at
all.

--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Amogh Margoor
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiang Yang
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Tue, 11 Apr 2023 20:03:31 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-11 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. (
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate
selectivities
..

Patch Set 29:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
> Consider Expr.isTriviallyTrue() for example in the same file. That currentl
Hi Kurt, I think you are right. I will discuss with other people to find a
better way to solve this problem.

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
> The concern makes sense to me. Expr#getConjuncts() is widely used. Chaning
Hi Quanlong, you are right, this change only required by hints on compound
predicates. First version of this patch only supported single predicate. Qifan
suggested that we'd better support compound predicates as well which are often
used in prod env, so I try to add this in later change. I will discuss with
Qifan again, maybe we do not provide hint for AND compound predicates in this
patch.

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1067
PS29, Line 1067: // Selectivity hint cannot be set. If Predicate been set
selectivity hint, we return
: // itself directly, such as CompoundPredicate.
Otherwise, hint value will missing.
> This comment is not accurate and should be removed. The original comment f
Hi Qifan. What do you mean 'Note a Predicate contains the selectivityHint_ as
the new data member.'? And 'If the above is true, then we do not need
getLocalConjuncts() at all.'? I don't understand.
If we do not add hint check in 'getConjuncts()', hint for AND compound
predicates will invalid, but still valid for OR compound predicates. So maybe
we have these options:
1. Remove hint check directly, do not support hint for AND compound predicates,
and maybe give some warnings;
2. Transfer hint value to AND compound predicate's children, but how to decide
the hint value of each child is a problem.
Here is my opinion, how do you think?

--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-11 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate
selectivities
..

Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

It looks to me the original form of getConjuncts() should work fine even when
selectivity hints exist in the tree anchored at . This is because both
child(0) and child(1) should be a when is a
CompoundPredicate. Note a Predicate contains the selectivityHint_ as the new
data member.

If the above is true, then we do not need getLocalConjuncts() at all.

IMHO, when selectivity hints are supplied, we should let them flow to the rest
of the compilation phases as a single piece of data to represent the moment of
truth. Allowing two or more representations can introduce unnecessary
complexity in the down stream.

--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-10 Thread Quanlong Huang (Code Review)

Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate
selectivities
..

Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
> Consider Expr.isTriviallyTrue() for example in the same file. That currentl
The concern makes sense to me. Expr#getConjuncts() is widely used. Chaning its
behavior might unintentionally impact other optimizations, e.g. rewrite of
ExtractCommonConjunct or predicate pushdown. We need to carefully examine the
related code path.

Is this change only required by hints on compound predicates? If so, we can
split this patch into two and merge the support on single predicate hints first.

--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-10 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate
selectivities
..

Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
> Sorry I didn't understand what you mean, can you show me the detail modific
Consider Expr.isTriviallyTrue() for example in the same file. That currently
considers all of the Exprs returned by getConjuncts. Now if you have a hint, it
is going to not consider the child conjuncts. I don't know if that will be be
correct still. If nothing else it is hard to follow and a risk for future bugs.
There are multiple other caller in analyzer that I did not look at. My
suggestion would be to not change the existing getConjuncts() or any logic that
is calls. Instead, make a new function getLocalConjuncts and conditionally call
that from where you need to after determining that the conditional calls will
not adversely affect correctness or other optimizations.

--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Amogh Margoor
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Xiang Yang
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Mon, 10 Apr 2023 14:09:23 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
> This should be checked by the caller. I there is only one caller, maybe bet
Sorry I didn't understand what you mean, can you show me the detail 
modification? In this way, I can modify the code following your suggestion.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 07 Apr 2023 04:21:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/29/fe/src/main/java/org/apache/impala/analysis/Expr.java@1064
PS29, Line 1064: if (! predicateHintValid(this)) {
This should be checked by the caller. I there is only one caller, maybe better 
to have a boolean flag i.e. includeChildConjuncts.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 06 Apr 2023 21:12:16 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 06 Apr 2023 12:49:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12756/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 06 Apr 2023 08:09:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9205/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 06 Apr 2023 07:50:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 29:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/28/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/28/fe/src/main/java/org/apache/impala/analysis/Expr.java@1065
PS28, Line 1065:   list.addAll(getLocalConjuncts());
> This still has not addressed the concern that getConjuncts() which
 > is a generic method for collecting conjuncts and the behavior
 > should not change based on a hint. Probably best to add a new
 > function that returns only the local conjuncts and call that with
 > the appropriate hint checking logic inline in the caller so the
 > context and intent is all clear.

I try to modify the code, I'm not sure if this modification is what you 
mentioned above.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 06 Apr 2023 07:49:00 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-06 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#29). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity.
For some predicates, this may lead to worse query plan due to CBO.

This patch adds a new query hint: 'SELECTIVITY', we can use this
hint to specify a selectivity value for a predicate.

The parser will interpret expressions wrapped in () followed by a
C-style comment /*  */ as a predicate hint. The
predicate hint currently can be in the form of +SELECTIVITY(f) where
'f' is a positive floating point number to use as the selectivity for
the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound predicate example:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 455 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/29
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 29
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-05 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28:

Please wait to merge this until the concerns are resolved.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 05 Apr 2023 14:12:18 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-05 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18023/28/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/28/fe/src/main/java/org/apache/impala/analysis/Expr.java@1065
PS28, Line 1065: if (allowConjunctsFromChild(this)) {
This still has not addressed the concern that getConjuncts() which is a generic 
method for collecting conjuncts and the behavior should not change based on a 
hint. Probably best to add a new function that returns only the local conjuncts 
and call that with the appropriate hint checking logic inline in the caller so 
the context and intent is all clear.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 05 Apr 2023 14:11:21 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-04 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Tue, 04 Apr 2023 14:10:44 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-03 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 03 Apr 2023 13:00:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-03 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12734/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 03 Apr 2023 07:59:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-03 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9195/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 03 Apr 2023 07:41:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-03 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 28:

(8 comments)

Thanks for review, Qifan.

http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@9
PS27, Line 9: Currently, Impala only uses simple estimation to compute 
selectivity.
> nit. stop the sentence here (.)
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@10
PS27, Line 10: For some predicates, this may lead to
> nit.
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@11
PS27, Line 11:
> nit. The next sentence at line 13 mentions the specific work done in this p
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@13
PS27, Line 13: l
> nit. hint.
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@14
PS27, Line 14:
> nit. specify a selectivity value for a predicate.
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@17
PS27, Line 17: predicate hint curren
> nit. C-style comment /*  */
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@17
PS27, Line 17:  where
 : 'f' is a
> nit. can be in the form of
Done


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@25
PS27, Line 25: predicate example
> nit. predicate example
Done



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 03 Apr 2023 07:39:03 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-03 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#28). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity.
For some predicates, this may lead to worse query plan due to CBO.

This patch adds a new query hint: 'SELECTIVITY', we can use this
hint to specify a selectivity value for a predicate.

The parser will interpret expressions wrapped in () followed by a
C-style comment /*  */ as a predicate hint. The
predicate hint currently can be in the form of +SELECTIVITY(f) where
'f' is a positive floating point number to use as the selectivity for
the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound predicate example:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 446 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/28
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 28
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-04-02 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 27: Code-Review+1

(8 comments)

Looks great.  Just have some minor comments on the commit message.

http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@9
PS27, Line 9: Currently, Impala only uses simple estimation to compute 
selectivity
nit. stop the sentence here (.)


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@10
PS27, Line 10: for some predicates, and this may lead
nit.

For some predicates, this may lead to ...


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@11
PS27, Line 11: Hence, we add new hints to reduce such errors.
nit. The next sentence at line 13 mentions the specific work done in this 
patch. I think we can remove this sentence.


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@13
PS27, Line 13: s
nit. hint.


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@14
PS27, Line 14: original selectivity computing
nit. specify a selectivity value for a predicate.


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@17
PS27, Line 17: C-style /* comment */
nit. C-style comment /*  */


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@17
PS27, Line 17: currently
 : supports
nit. can be in the form of


http://gerrit.cloudera.org:8080/#/c/18023/27//COMMIT_MSG@25
PS27, Line 25: Predicate Example
nit. predicate example



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Sun, 02 Apr 2023 13:29:08 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-31 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 27: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 31 Mar 2023 13:02:50 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-31 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 26:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12724/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 31 Mar 2023 08:09:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-31 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 27:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9188/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 27
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 31 Mar 2023 07:55:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-31 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 26:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java@1068
PS23, Line 1068:   // and conjunct evaluation.  This is not optimal for 
jitted exprs because it
> THis is a generic function to collect conjuncts. The definition should not
I extract a method 'allowConjunctsFromChild' here.

We add a check for selectivity hint here is due to CompoundPredicate. If we 
don't check here, selectivity hint for AND CompoundPredicate will always 
missing, since conjuncts will be it's children, instead of itself.


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@70
PS23, Line 70: result.setSelectivityHint(bp.getSelectivity());
> What if the compound predicate is not a between predicate? Checks here seem
In my opinion, this rule is only used to transform BetweenPredicate to 
CompoundPredicate, so the input predicate must be BetweenPredicate. If current 
predicate is not BetweenPredicate, this rule will return directly in line 41.
So I think set new created CompoundPredicate's selectivity hint value from 
BetweenPredicate seems reasonable.

I don't understand 'What if the compound predicate is not a between predicate', 
can you explain this in more detail?



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 31 Mar 2023 07:49:01 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-31 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#26). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors.

This patch adds a new query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

The parser will interpret expressions wrapped in () followed by a
C-style /* comment */ as a predicate hint. The predicate hint currently
supports +SELECTIVITY(f) where 'f' is a positive floating point number
to use as the selectivity for the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound Predicate Example:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 446 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/26
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 26
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 23:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java@1068
PS23, Line 1068: && !((CompoundPredicate) 
this).selectivityValidHintSet()) {
> Why? This patch only add a selectivity hint check in original if-condition,
THis is a generic function to collect conjuncts. The definition should not 
change if a predicate hint is set. If you want to parameterize this function to 
behave differently, I suggest that you pass a flag in so that the semantics are 
clear from all of the callers.


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@70
PS23, Line 70: result.setSelectivityHint(bp.getSelectivity());
> This rule is to transform BetweenPredicate to CompoundPredicate, which is t
What if the compound predicate is not a between predicate? Checks here seem too 
broad to isolate that case.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 23
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 30 Mar 2023 20:01:54 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 25: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 30 Mar 2023 14:18:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 24:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12715/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 30 Mar 2023 09:16:43 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread wangsheng (Code Review)

wangsheng has removed Yifan Zhang from this change.  ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Removed reviewer Yifan Zhang.
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread wangsheng (Code Review)

wangsheng has removed Fucun Chu from this change.  ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Removed reviewer Fucun Chu.
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 25:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9185/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 25
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 30 Mar 2023 09:01:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#24). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors.

This patch adds a new query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

The parser will interpret expressions wrapped in () followed by a
C-style /* comment */ as a predicate hint. The predicate hint currently
supports +SELECTIVITY(f) where 'f' is a positive floating point number
to use as the selectivity for the preceding expression.

Single predicate example:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Compound Predicate Example:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 429 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/24
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 24
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-30 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 23:

(16 comments)

Hi Kurt Deschler and Xiang Yang, thanks for review. I already modify the patch 
as possible.

http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@11
PS23, Line 11: Hence, we add new hints to reduce such errors. Maybe in the 
future,
> No need to mention histograms here. Instead just say this is giving a qay t
Done


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@14
PS23, Line 14: another
> Don't say another unless you want to qualify that means
Done


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@15
PS23, Line 15: hint to original selectivity computing.
> Describe the syntax additions here at a high level. i.e. The parser will in
Done


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@17
PS23, Line 17: Format like this:
> Single predicate example:
Done


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@21
PS23, Line 21: Besides, this hint is also valid for compound predicate like 
this:
> Compound Predicate Example:
Done


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@25
PS23, Line 25: But pay attention, if we want to use 'SELECTIVITY' hint for 
predicate,
> State this first when you are describing the general syntax.
Done


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@26
PS23, Line 26: braket
> nit: brackets
Done


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java@1068
PS23, Line 1068: && !((CompoundPredicate) 
this).selectivityValidHintSet()) {
> Skipping child conjuncts here based on a hint does not seem legal
Why? This patch only add a selectivity hint check in original if-condition, 
this is illegal?
As you can see, the gerrit-verify-dryrun task success. I think this additional 
check seem ok.


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Predicate.java@38
PS23, Line 38:   protected double selectivityHint_;
 :   // true if the selectivity hint is set to a valid value.
 :   protected boolean hasValidSelectivityHint_;
> From my point of view, it's not necessary to use the redundant variable 'ha
I think you are right. I remove 'hasValidSelectivityHint_' in latest patch, 
only reserve 'selectivityHint_', and use 'hasValidSelectivityHint()' in each 
kinds of Predicate.


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Predicate.java@183
PS23, Line 183:   public boolean selectivityValidHintSet() {
> nit: rename to hasValidSelectivityHint() ?
Done


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@70
PS23, Line 70: result.setSelectivityHint(bp.getSelectivity());
> Probably better to no propagate the selectivity. If user want it to apply a
This rule is to transform BetweenPredicate to CompoundPredicate, which is 
transparent to the user. If we don't assign BetweenPredicate selectivity hint 
to new created CompoundPredicate, this hint value will missing due to this 
transformation, which means user set selectivity hint for BetweenPredicate are 
always invalid. I think this maybe inappropriate.


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5144
PS21, Line 5144: >
> nit: add space on both sides of the '>'. same the followings.
Done


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5144
PS21, Line 5144: >
> nit: add space on both sides of the '>'. same the followings.
Done


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5181
PS21, Line 5181: >
> nit: same as above.
Done


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5181
PS21, Line 5181: >
> nit: same as above.
Done

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-26 Thread Xiang Yang (Code Review)

Xiang Yang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 23:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@26
PS23, Line 26: braket
nit: brackets


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Predicate.java@38
PS23, Line 38:   protected double selectivityHint_;
 :   // true if the selectivity hint is set to a valid value.
 :   protected boolean hasValidSelectivityHint_;
>From my point of view, it's not necessary to use the redundant variable 
>'hasValidSelectivityHint_' to cache the result for a performance reason, and I 
>suggest to replace it with a just-in-time calculate function.


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Predicate.java@183
PS23, Line 183:   public boolean selectivityValidHintSet() {
nit: rename to hasValidSelectivityHint() ?


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5144
PS21, Line 5144: >
nit: add space on both sides of the '>'. same the followings.


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5181
PS21, Line 5181: >
nit: same as above.


http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test:

http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@161
PS21, Line 161: 
Does there need a compound 'OR' predicate without hint test case?



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 23
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Xiang Yang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Sun, 26 Mar 2023 12:05:43 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-24 Thread Kurt Deschler (Code Review)

Kurt Deschler has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 23:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@11
PS23, Line 11: Hence, we add new hints to reduce such errors. Maybe in the 
future,
No need to mention histograms here. Instead just say this is giving a qay to 
override estimates.


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@14
PS23, Line 14: another
Don't say another unless you want to qualify that means


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@15
PS23, Line 15: hint to original selectivity computing.
Describe the syntax additions here at a high level. i.e. The parser will 
interpret expressions wrapped in () followed by a C-style /* comment */ as a 
predicate hint. The predicate hint currently supports +SELECTIVITY(f) where 'f' 
is a positive floating point number to use as the selectivity for the preceding 
expression.


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@17
PS23, Line 17: Format like this:
Single predicate example:


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@21
PS23, Line 21: Besides, this hint is also valid for compound predicate like 
this:
Compound Predicate Example:


http://gerrit.cloudera.org:8080/#/c/18023/23//COMMIT_MSG@25
PS23, Line 25: But pay attention, if we want to use 'SELECTIVITY' hint for 
predicate,
State this first when you are describing the general syntax.


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/analysis/Expr.java@1068
PS23, Line 1068: && !((CompoundPredicate) 
this).selectivityValidHintSet()) {
Skipping child conjuncts here based on a hint does not seem legal


http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:

http://gerrit.cloudera.org:8080/#/c/18023/23/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@70
PS23, Line 70: result.setSelectivityHint(bp.getSelectivity());
Probably better to no propagate the selectivity. If user want it to apply 
across predicates, they should be able to wrap them in a () block.



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 23
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 24 Mar 2023 19:57:45 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-23 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 23: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 23
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 23 Mar 2023 21:00:18 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-23 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 22:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12682/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 22
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 23 Mar 2023 16:07:59 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-23 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 22:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/analysis/Predicate.java@171
PS21, Line 171:   public void setSelectivityHint(double selectivityHint) {
> Should we update hasValidSelectivityHint_ as well? Or is it enough to only
Done

Yes, we need to update 'hasValidSelectivityHint_' and 'selectivityHint_' at 
same time. So I update this boolean flag
according to input selectivityHint.


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@68
PS21, Line 68: // Selectivity hint value of this new CompoundPredicate not 
been set, so inherited
> Could you add a comment for why we always set the selectivity hint regradle
Done

A new created Predicate's selectivity hint is always false, unless we use hint 
in sql. So whether 'BetweenPredicate' been set selectivity hint in sql or not, 
we can use this to replace the new created.


http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test:

http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@130
PS21, Line 130: select * from tpch.lineitem
> missing the SELECT part here, which causes the test failure
Done



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 22
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 23 Mar 2023 15:47:35 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-23 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#22). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 427 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/22
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 22
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-23 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 23:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9173/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 23
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 23 Mar 2023 15:48:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-22 Thread Quanlong Huang (Code Review)

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 21:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@69
PS19, Line 69:
> I think selectivity hint is valid for Predicate, instead of Expr, so I impl
I see. This makes sense to me. Thanks for the explanation!


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/analysis/Predicate.java@171
PS21, Line 171: this.selectivityHint_ = selectivityHint;
Should we update hasValidSelectivityHint_ as well? Or is it enough to only set 
hasValidSelectivityHint_ in constructors and analyzeSelectivityHint()?


http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
File fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java:

http://gerrit.cloudera.org:8080/#/c/18023/21/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java@68
PS21, Line 68: result.setSelectivityHint(bp.getSelectivity());
Could you add a comment for why we always set the selectivity hint regradless 
whether the selectivity of 'bp' comes from hints? Maybe this is always better 
than what the planner will estimate?


http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test:

http://gerrit.cloudera.org:8080/#/c/18023/21/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@130
PS21, Line 130: where (l_shipdate BETWEEN '1998-09-01' AND '1998-09-03')/* 
+SELECTIVITY(0.5)*/
missing the SELECT part here, which causes the test failure



--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 23 Mar 2023 00:49:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-22 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 21: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9169/


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 22 Mar 2023 20:46:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-22 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 20:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12667/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 22 Mar 2023 15:55:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-22 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 20:

(25 comments)

Thanks for review, Quanlong, already modified related code according to your 
comments.

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/cup/sql-parser.cup@3407
PS19, Line 3407: // It's used to replace the selectivity estimated by the 
planner.
   : selectivity_hint ::=
> nit: I think we don't need to mention Impala since these are already Impala
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@37
PS19, Line 37: s no sense for a query.
> nit: "0 makes no sense for a query"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@39
PS19, Line 39: true if the selectivity hint is set to a valid value.
> nit: "true if the selectivity hint is set to a valid value"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@57
PS19, Line 57: hasValidSelectivityHint_ = other.hasValidSelectivityHint_;
> We should copy hasValidSelectivityHint_ as well.
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@69
PS19, Line 69:
> nit: Any reason we put this here instead of put it inside analyzeHints() ?
I think selectivity hint is valid for Predicate, instead of Expr, so I 
implements 'analyzeSelectivityHint()' in Predicate instead of Expr. But 
'analyzeHints()' is in Expr, so I didn't put it inside analyzeHints(). 
Otherwise, we need to move 'analyzeSelectivityHint()' to Expr


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@79
PS19, Line 79:
> nit: "larger"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@177
PS19, Line 177:   public boolean selectivityValidHintSet() {
> Can we return hasValidSelectivityHint_ directly? Also the method name can c
Done

Yes, of course. I forgot modify this method in previous patch.


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5163
PS19, Line 5163: "Syntax error in line 1");
> Let's also test some expressions like "1/3" and very long decimal values li
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
File fe/src/test/java/org/apache/impala/planner/PlannerTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@1380
PS19, Line 1380: Test SELECTIVITY hints
> nit: "new" will be stale. Might reword to "Test SELECTIVITY hints"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test:

http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@1
PS19, Line 1: # Table 'tpch.lineitem' has 6001215 rows, so the scan on it has 
cardinality as 6.00M.
> nit: "Table 'tpch.lineitem' has 6001215 rows, so the scan on it has cardina
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@2
PS19, Line 2: If
> nit: start a new sentence with "If the selectivity of the predicate is 0.1"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@4
PS19, Line 4: Planner assigns the default selectivity (0.1) to this predicate
> nit: "Planner assigns the default selectivity (0.1) to this predicate"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@15
PS19, Line 15: 98% of the
> nit: "98% of the values"
Done


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@50
PS19, Line 50: getStats().getNumNulls() / numRows
> I'm confused with this. Shouldn't the selectivity a value lower than 1? Thi
Done

This comment seems not correct, for IS NULL

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-22 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#20). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 418 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/20
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 20
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-22 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 21:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9169/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 21
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 22 Mar 2023 15:36:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-21 Thread Quanlong Huang (Code Review)

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 19:

(25 comments)

Thanks for your continous work on this, Sheng Wang! There are some merge 
conflicts. Could you rebase the patch to the latest master branch?

The only issue I found is we need to copy hasValidSelectivityHint_ in the 
constructor of Predicate. Also left some tiny comments.

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/cup/sql-parser.cup@3407
PS19, Line 3407: // If set, Impala will use this value to replace original 
selectiviy
   : // computed in sql analysis phase.
nit: I think we don't need to mention Impala since these are already Impala 
codes. We can make it shorter, e.g.

"It's used to replace the selectivity estimated by the planner."


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@37
PS19, Line 37: 0 is make no sense for a query
nit: "0 makes no sense for a query"


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@39
PS19, Line 39: true if selectivity predicate been set as a valid value
nit: "true if the selectivity hint is set to a valid value"


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@57
PS19, Line 57:   }
We should copy hasValidSelectivityHint_ as well.


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@69
PS19, Line 69: analyzeSelectivityHint(analyzer);
nit: Any reason we put this here instead of put it inside analyzeHints() ?


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@79
PS19, Line 79: bigger
nit: "larger"


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/main/java/org/apache/impala/analysis/Predicate.java@177
PS19, Line 177: return selectivityHint_ > 0 && selectivityHint_ <= 1.0;
Can we return hasValidSelectivityHint_ directly? Also the method name can 
change to hasValidSelectivityHint().


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5163
PS19, Line 5163: "Syntax error in line 1");
Let's also test some expressions like "1/3" and very long decimal values like 
"0.33".


http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
File fe/src/test/java/org/apache/impala/planner/PlannerTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/19/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@1380
PS19, Line 1380: Test new hint of 'SELECTIVITY'
nit: "new" will be stale. Might reword to "Test SELECTIVITY hints"


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test:

http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@1
PS19, Line 1: # Table 'tpch.lineitem' records count is 6001215, so cardinality 
is 6.00M,
nit: "Table 'tpch.lineitem' has 6001215 rows, so the scan on it has cardinality 
as 6.00M."


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@2
PS19, Line 2: if
nit: start a new sentence with "If the selectivity of the predicate is 0.1"


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@4
PS19, Line 4: This predicate cannot compute selectivity, default value is 0.1
nit: "Planner assigns the default selectivity (0.1) to this predicate"


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@15
PS19, Line 15: 98% values
nit: "98% of the values"


http://gerrit.cloudera.org:8080/#/c/18023/19/testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test@50
PS19, Line 50: numRows - getStats().getNumNulls()
I'm confused with this. Shouldn't the selectivity a value lower than 1? This 
seems the cardinality of the scan with "l_shipdate IS NOT NULL".

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-20 Thread Yifan Zhang (Code Review)

Yifan Zhang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 18: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Yifan Zhang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Tue, 21 Mar 2023 02:04:40 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-15 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 18: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 15 Mar 2023 07:53:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-14 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 18:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9144/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 18
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 15 Mar 2023 02:43:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-09 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 17:

All tests passed, here is the jenkins url:
https://jenkins.impala.io/job/pre-review-test/1519/


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 17
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Fri, 10 Mar 2023 07:02:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-09 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 17:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12597/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 17
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 09 Mar 2023 12:38:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-09 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#17). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 415 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/17
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 17
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-09 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 16:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12596/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 09 Mar 2023 09:57:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-09 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#16). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 415 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/16
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 16
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-06 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 15: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9111/


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 06 Mar 2023 21:16:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-06 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9111/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 06 Mar 2023 16:08:42 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-06 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 15: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 15
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 06 Mar 2023 16:08:42 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-06 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 14: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 14
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 06 Mar 2023 13:37:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-02 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 14:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12514/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 14
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 02 Mar 2023 13:18:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-03-02 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#14). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 385 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/14
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 14
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-28 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 13:

It seem that this patch lead to some test cases failed. I will fix as soon as 
possible!


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Wed, 01 Mar 2023 02:18:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-28 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 13: Code-Review+1

Thanks!


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Tue, 28 Feb 2023 14:32:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-24 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 13:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12444/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Sat, 25 Feb 2023 04:22:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-24 Thread wangsheng (Code Review)

wangsheng has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 13:

(29 comments)

> Can you please provide feedback on all comments?
 >
 > For example, for the  following comment, a feedback can be DONE, a
 > reply etc.
 >
 > Commit Message
 > Line 14:
 > TABLE_NUM_ROWS?
 >
 > The feedbacks are useful to judge the rework. Thanks!

Ok, Qifan, thanks for reply. I've already answer all comments.

http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/analysis/Predicate.java
File fe/src/main/java/org/apache/impala/analysis/Predicate.java:

http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/analysis/Predicate.java@78
PS9, Line 78: ti
> nit. should be a double value in (0, 1].
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:

http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/analysis/TableRef.java@547
PS9, Line 547:   return;
> The above hints are all limited to hdfs tables since they are related to hd
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/analysis/TableRef.java@554
PS9, Line 554: }
> nit. for this HDFS table reference. On paper, one can assign different # ro
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/analysis/TableRef.java@555
PS9, Line 555:
> It seems we need to handle NumberFormatException thrown from this method. F
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/planner/ScanNode.java
File fe/src/main/java/org/apache/impala/planner/ScanNode.java:

http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/planner/ScanNode.java@81
PS9, Line 81: H
> nit. "."
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/main/java/org/apache/impala/planner/ScanNode.java@82
PS9, Line 82: ng tableNumRowsHint_ = -1;
:
> I wonder if this is still correct. I thought the hint can be used to overri
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:

http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5053
PS9, Line 5053:  void testCreatePartitio
> This error may not be user friendly for super long SQL query.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5056
PS9, Line 5056: "PARTITIONED BY SPEC (BUC
> same comment as above.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5060
PS9, Line 5060:  "STORED AS ICEBERG" + tb
> same comment as above.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5064
PS9, Line 5064:  int,
> nit. may add a qualifier "non-HDFS" before 'table' to make it clear.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5071
PS9, Line 5071: "PARTITIONED BY SPEC (BU
> See the comment on TABLE_NUM_ROWS for "syntax error in line 1".
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5074
PS9, Line 5074: "PARTITIONED BY SPEC (TRUNCATE(0, p1), DAY(p2)) STORED 
AS ICEBERG" +
> Isn't this supported now in patch set 9?
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5085
PS9, Line 5085:  Legal hint return correct
> is this a mistake? 0.1 is a perfect selectivity value.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5087
PS9, Line 5087: zesOk("select * from tp
> Same as above. 0 is okay.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5093
PS9, Line 5093:
> should be [0,1], right?
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5098
PS9, Line 5098: gal
> See the comment about. Should be allowed.
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@5102
PS9, Line 5102:
  : // Multiple illegal hints wil
> From the updated parser, it seems this hint should be accepted in that the
Done


http://gerrit.cloudera.org:8080/#/c/18023/9/testdata/workloads/functional-planner/queries/PlannerTest/hdfs-cardinality-hint.test
File

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-24 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#13). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 375 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/13
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 13
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-20 Thread Qifan Chen (Code Review)

Qifan Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 12:

Can you please provide feedback on all comments?

For example, for the  following comment, a feedback can be DONE, a reply etc.

Commit Message
Line 14:
TABLE_NUM_ROWS?

The feedbacks are useful to judge the rework. Thanks!


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Mon, 20 Feb 2023 13:22:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-15 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 12:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/12386/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Thu, 16 Feb 2023 06:43:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-15 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#12). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 334 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/12
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 12
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-11 Thread wangsheng (Code Review)

wangsheng has uploaded a new patch set (#10). ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..

IMPALA-7942 (part 2): Add query hints for predicate selectivities

Currently, Impala only uses simple estimation to compute selectivity
for some predicates, and this may lead to worse query plan due to CBO.
Hence, we add new hints to reduce such errors. Maybe in the future,
we can use histograms to get more precise query plan.

This patch adds another query hints: 'SELECTIVITY', we can use this
hint to original selectivity computing.

Format like this:

  select col from t where (a=1) /* +SELECTIVITY(0.5) */;

Besides, this hint is also valid for compound predicate like this:

  select col from t where (a=1 and b=2) /* +SELECTIVITY(0.5) */;

But pay attention, if we want to use 'SELECTIVITY' hint for predicate,
we need to wrap the predicate by braket, even for single binary
predicate.

Testing:
- Added new fe tests in 'PlannerTest'
- Added new fe tests in 'AnalyzeStmtsTest' for negative cases

Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Predicate.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-selectivity-hint.test
12 files changed, 334 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/18023/10
--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 10
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng

[Impala-ASF-CR] IMPALA-7942 (part 2): Add query hints for predicate selectivities

2023-02-11 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/18023 )

Change subject: IMPALA-7942 (part 2): Add query hints for predicate 
selectivities
..


Patch Set 10:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/12356/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/18023
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2776b9bbd878b8a21d9c866b400140a454f59e1b
Gerrit-Change-Number: 18023
Gerrit-PatchSet: 10
Gerrit-Owner: wangsheng 
Gerrit-Reviewer: Amogh Margoor 
Gerrit-Reviewer: Fucun Chu 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Qifan Chen 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Reviewer: wangsheng 
Gerrit-Comment-Date: Sat, 11 Feb 2023 09:15:15 +
Gerrit-HasComments: No

92 matches

Mail list logo