[
https://issues.apache.org/jira/browse/HIVE-25938?focusedWorklogId=725092&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725092
]
ASF GitHub Bot logged work on HIVE-25938:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Feb/22 12:20
Start Date: 11/Feb/22 12:20
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3011:
URL: https://github.com/apache/hive/pull/3011#discussion_r804583395
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -2026,6 +2038,10 @@ protected RelNode
applyMaterializedViewRewriting(RelOptPlanner planner, RelNode
// Optimize plan
if (!ruleExclusionRegex.isEmpty()) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("The CBO rules matching the following regex are excluded
from planning: {}",
+ ruleExclusionRegex);
+ }
Review comment:
I think these logs are slightly redundant if
`logger.CalcitePlanner.level` is set to DEBUG
(https://github.com/apache/calcite/blob/812e3e98eae518cf85cd1b6b7f055fb96784a423/core/src/main/java/org/apache/calcite/plan/AbstractRelOptPlanner.java#L316).
This is a bit higher level so not exactly the same thing so don't feel
strongly about having it or not.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -2450,6 +2466,10 @@ protected RelNode executeProgram(RelNode basePlan,
HepProgram program,
}
if (!ruleExclusionRegex.isEmpty()) {
+ if (LOG.isDebugEnabled()) {
Review comment:
Idem as above
##########
File path: ql/src/test/results/clientpositive/llap/excluded_rule_explain.q.out
##########
@@ -0,0 +1,112 @@
+PREHOOK: query: EXPLAIN CBO SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN CBO SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+CBO PLAN:
+HiveProject(_o__c0=[1])
+ HiveTableScan(table=[[_dummy_database, _dummy_table]],
table:alias=[_dummy_table])
+
+PREHOOK: query: EXPLAIN FORMATTED CBO SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN FORMATTED CBO SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n
\"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n
],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n
\"rowCount\": 1.0,\n \"avgRowSize\": 129.0,\n \"rowType\": [\n
{\n \"type\": \"BIGINT\",\n \"nullable\": true,\n
\"name\": \"BLOCK__OFFSET__INSIDE__FILE\"\n },\n {\n
\"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"INPUT__FILE__NAME\"\n },\n {\n
\"fields\": [\n {\n \"type\": \"BIGINT\",\n
\"nullable\": true,\n \"name\": \"writeid\"\n
},\n {\n \"type\": \"INTEGER\",\n
\"nullable\": true,\n \"name\": \"bucketid\"\n },\n
{\n \"type\": \"BIGINT\",\n \"nullable\":
true,\n \"name\": \"rowid\"\n }\n ],\n
\"name\": \"ROW__ID\"\n },\n {\n \"type\":
\"BOOLEAN\",\n \"nullable\": true,\n \"name\":
\"ROW__IS__DELETED\"\n }\n ]\n },\n {\n \"id\": \"1\",\n
\"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject\",\n
\"fields\": [\n \"_o__c0\"\n ],\n \"exprs\": [\n {\n
\"literal\": 1,\n \"type\": {\n \"type\":
\"INTEGER\",\n \"nullable\": false\n }\n }\n
],\n \"rowCount\": 1.0\n }\n ]\n}"}
+PREHOOK: query: EXPLAIN SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+STAGE DEPENDENCIES:
+ Stage-0 is a root stage
+
+STAGE PLANS:
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ TableScan
+ alias: _dummy_table
+ Row Limit Per Split: 1
+ Select Operator
+ expressions: 1 (type: int)
+ outputColumnNames: _col0
+ ListSink
+
+PREHOOK: query: EXPLAIN FORMATTED SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN FORMATTED SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n
\"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n
],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n
\"rowCount\": 1.0,\n \"avgRowSize\": 0.0,\n \"rowType\": [\n
{\n \"type\": \"BIGINT\",\n \"nullable\": true,\n
\"name\": \"BLOCK__OFFSET__INSIDE__FILE\"\n },\n {\n
\"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"INPUT__FILE__NAME\"\n },\n {\n
\"fields\": [\n {\n \"type\": \"BIGINT\",\n
\"nullable\": true,\n \"name\": \"writeid\"\n
},\n {\n \"type\": \"INTEGER\",\n
\"nullable\": true,\n \"name\": \"bucketid\"\n },\n
{\n \"type\": \"BIGINT\",\n \"nullable\":
true,\n \"name\": \"rowid\"\n }\n ],\n
\"name\": \"ROW__ID\"\n },\n {\n \"type\":
\"BOOLEAN\",\n \"nullable\": true,\n \"name\":
\"ROW__IS__DELETED\"\n }\n ]\n },\n {\n \"id\": \"1\",\n
\"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject\",\n
\"fields\": [\n \"_o__c0\"\n ],\n \"exprs\": [\n {\n
\"literal\": 1,\n \"type\": {\n \"type\":
\"INTEGER\",\n \"nullable\": false\n }\n }\n
],\n \"rowCount\": 1.0\n }\n ]\n}","optimizedSQL":"SELECT 1 AS
`_o__c0`\nFROM `_dummy_database`.`_dummy_table`","cboInfo":"Plan optimized by
CBO.","STAGE DEPENDENCIES":{"Stage-0":{"ROOT STAGE":"TRUE"}},"STAGE
PLANS":{"Stage-0":{"Fetch Operator":{"limit:":"-1","Processor
Tree:":{"TableScan":{"alias:":"_dummy_table","database:":"_dummy_database","Row
Limit Per
Split:":"1","table:":"_dummy_table","isTempTable:":"false","OperatorId:":"TS_0","children":{"Select
Operator":{"expressions:":"1 (type:
int)","columnExprMap:":{"_col0":"1"},"outputColumnNames:":["_col0"],"OperatorId:":"SEL_1","children":{"ListSink":{"OperatorId:":"LIST_SINK_3"}}}}}}}}}}
+PREHOOK: query: EXPLAIN CBO SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN CBO SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+Excluded rules: HiveJoinPushTransitivePredicatesRule|HivePreFilteringRule
+
+CBO PLAN:
+HiveProject(_o__c0=[1])
+ HiveTableScan(table=[[_dummy_database, _dummy_table]],
table:alias=[_dummy_table])
+
+PREHOOK: query: EXPLAIN FORMATTED CBO SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN FORMATTED CBO SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+{"CBOPlan":"{\n \"rels\": [\n {\n \"id\": \"0\",\n \"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n
\"table\": [\n \"_dummy_database\",\n \"_dummy_table\"\n
],\n \"table:alias\": \"_dummy_table\",\n \"inputs\": [],\n
\"rowCount\": 1.0,\n \"avgRowSize\": 129.0,\n \"rowType\": [\n
{\n \"type\": \"BIGINT\",\n \"nullable\": true,\n
\"name\": \"BLOCK__OFFSET__INSIDE__FILE\"\n },\n {\n
\"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"INPUT__FILE__NAME\"\n },\n {\n
\"fields\": [\n {\n \"type\": \"BIGINT\",\n
\"nullable\": true,\n \"name\": \"writeid\"\n
},\n {\n \"type\": \"INTEGER\",\n
\"nullable\": true,\n \"name\": \"bucketid\"\n },\n
{\n \"type\": \"BIGINT\",\n \"nullable\":
true,\n \"name\": \"rowid\"\n }\n ],\n
\"name\": \"ROW__ID\"\n },\n {\n \"type\":
\"BOOLEAN\",\n \"nullable\": true,\n \"name\":
\"ROW__IS__DELETED\"\n }\n ]\n },\n {\n \"id\": \"1\",\n
\"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject\",\n
\"fields\": [\n \"_o__c0\"\n ],\n \"exprs\": [\n {\n
\"literal\": 1,\n \"type\": {\n \"type\":
\"INTEGER\",\n \"nullable\": false\n }\n }\n
],\n \"rowCount\": 1.0\n }\n ]\n}","cboInfo":"Excluded rules:
HiveJoinPushTransitivePredicatesRule|HivePreFilteringRule"}
+PREHOOK: query: EXPLAIN SELECT 1
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+POSTHOOK: query: EXPLAIN SELECT 1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+#### A masked pattern was here ####
+Excluded rules: HiveJoinPushTransitivePredicatesRule|HivePreFilteringRule
Review comment:
Shouldn't I see somewhere the "Plan optimized by CBO." statement? I
think it appears in some other `.q.out` files.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -2026,6 +2038,10 @@ protected RelNode
applyMaterializedViewRewriting(RelOptPlanner planner, RelNode
// Optimize plan
if (!ruleExclusionRegex.isEmpty()) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("The CBO rules matching the following regex are excluded
from planning: {}",
+ ruleExclusionRegex);
+ }
Review comment:
Also I don't think you need the `LOG.isDebugEnabled()` guard. There
doesn't to seem any performance critical operation below and I know there was
some effort to eliminate this pattern from the project (if possible).
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
##########
@@ -64,6 +64,7 @@
import org.apache.hadoop.hive.ql.optimizer.physical.StageIDsRearranger;
import org.apache.hadoop.hive.ql.parse.ASTNode;
import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer;
+import org.apache.hadoop.hive.ql.parse.CalcitePlanner;
Review comment:
Minor but I don't think its worth coupling this with CalcitePlanner for
just a prefix that we don't really care if it ends up being different.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##########
@@ -575,7 +578,12 @@ Operator genOPTree(ASTNode ast, PlannerContext plannerCtx)
throws SemanticExcept
getQB().getParseInfo().setHintList(oldHints);
}
LOG.info("CBO Succeeded; optimized logical plan.");
- this.ctx.setCboInfo("Plan optimized by CBO.");
+
+ String cboInfo = "Plan optimized by CBO.";
+ if (!ruleExclusionRegex.isEmpty()) {
+ cboInfo = cboInfo + (" " + EXCLUDED_RULES_PREFIX +
ruleExclusionRegex);
+ }
+ this.ctx.setCboInfo(cboInfo);
Review comment:
Refactor the duplicate fragment somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 725092)
Time Spent: 20m (was: 10m)
> Print excluded rules from CBO
> -----------------------------
>
> Key: HIVE-25938
> URL: https://issues.apache.org/jira/browse/HIVE-25938
> Project: Hive
> Issue Type: Improvement
> Components: CBO
> Affects Versions: 4.0.0
> Reporter: Alessandro Solimando
> Assignee: Alessandro Solimando
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HIVE-25880 introduced a configuration parameter for excluding CBO rules based
> on a regex on their description.
> Calcite logs when a rule is excluded (see
> [AbstractRelOptPlanner.java#L316|https://github.com/apache/calcite/blob/e42b85a45bd16dd58db1546736e653deda5463fe/core/src/main/java/org/apache/calcite/plan/AbstractRelOptPlanner.java#L316]
> and
> [VolcanoRuleCall.java#L169|https://github.com/apache/calcite/blob/e42b85a45bd16dd58db1546736e653deda5463fe/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoRuleCall.java#L169]).
> To ease investigations, this should be complemented by printing the regex
> used (if not blank) in Hive DEBUG logs, and the same in CBO information
> (i.e., EXPLAIN CBO's output).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
