[jira] [Work logged] (HIVE-24434) Filter out materialized views for rewriting if plan pattern is not allowed
[ https://issues.apache.org/jira/browse/HIVE-24434?focusedWorklogId=530761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530761 ] ASF GitHub Bot logged work on HIVE-24434: - Author: ASF GitHub Bot Created on: 04/Jan/21 16:48 Start Date: 04/Jan/21 16:48 Worklog Time Spent: 10m Work Description: kasakrisz merged pull request #1782: URL: https://github.com/apache/hive/pull/1782 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 530761) Time Spent: 1h (was: 50m) > Filter out materialized views for rewriting if plan pattern is not allowed > -- > > Key: HIVE-24434 > URL: https://issues.apache.org/jira/browse/HIVE-24434 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Some materialized views are not enabled for Calcite based rewriting. Rules > for validating materialized views are implemented by HIVE-20748. > Since text based materialized view query rewrite doesn't have such > limitations some logic must be implemented to flag materialized view whether > they are enabled to text based rewrite only or both. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24434) Filter out materialized views for rewriting if plan pattern is not allowed
[
https://issues.apache.org/jira/browse/HIVE-24434?focusedWorklogId=530505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530505
]
ASF GitHub Bot logged work on HIVE-24434:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 03:53
Start Date: 04/Jan/21 03:53
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1782:
URL: https://github.com/apache/hive/pull/1782#discussion_r551107074
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1831,7 +1832,8 @@ public RelOptMaterialization
getMaterializedViewForRebuild(String dbName, String
private List getValidMaterializedViews(List
materializedViewTables,
List tablesUsed, boolean forceMVContentsUpToDate, boolean
expandGroupingSets,
- HiveTxnManager txnMgr) throws HiveException {
+ HiveTxnManager txnMgr,
EnumSet
scope)
Review comment:
We already have a `Materialization` class so renamed this one to
`HiveRelOptMaterialization` which fixed this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530505)
Time Spent: 50m (was: 40m)
> Filter out materialized views for rewriting if plan pattern is not allowed
> --
>
> Key: HIVE-24434
> URL: https://issues.apache.org/jira/browse/HIVE-24434
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some materialized views are not enabled for Calcite based rewriting. Rules
> for validating materialized views are implemented by HIVE-20748.
> Since text based materialized view query rewrite doesn't have such
> limitations some logic must be implemented to flag materialized view whether
> they are enabled to text based rewrite only or both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24434) Filter out materialized views for rewriting if plan pattern is not allowed
[
https://issues.apache.org/jira/browse/HIVE-24434?focusedWorklogId=530504=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530504
]
ASF GitHub Bot logged work on HIVE-24434:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 03:52
Start Date: 04/Jan/21 03:52
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1782:
URL: https://github.com/apache/hive/pull/1782#discussion_r551106837
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Materialization.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.metadata;
+
+import org.apache.calcite.plan.RelOptMaterialization;
+
+import java.util.EnumSet;
+
+import static org.apache.commons.collections.CollectionUtils.intersection;
+
+/**
+ * Wrapper class of {@link RelOptMaterialization} and corresponding flags.
+ */
+public class Materialization {
Review comment:
Changed this to extend `RelOptMaterialization`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530504)
Time Spent: 40m (was: 0.5h)
> Filter out materialized views for rewriting if plan pattern is not allowed
> --
>
> Key: HIVE-24434
> URL: https://issues.apache.org/jira/browse/HIVE-24434
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Some materialized views are not enabled for Calcite based rewriting. Rules
> for validating materialized views are implemented by HIVE-20748.
> Since text based materialized view query rewrite doesn't have such
> limitations some logic must be implemented to flag materialized view whether
> they are enabled to text based rewrite only or both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24434) Filter out materialized views for rewriting if plan pattern is not allowed
[
https://issues.apache.org/jira/browse/HIVE-24434?focusedWorklogId=530503=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-530503
]
ASF GitHub Bot logged work on HIVE-24434:
-
Author: ASF GitHub Bot
Created on: 04/Jan/21 03:51
Start Date: 04/Jan/21 03:51
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1782:
URL: https://github.com/apache/hive/pull/1782#discussion_r551106677
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CBOPlan.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import org.apache.calcite.rel.RelNode;
+
+/**
+ * Wrapper of Calcite plan.
+ */
+public class CBOPlan {
+ private final RelNode plan;
+ private final String invalidAutomaticRewritingMaterializationReason;
+
+ public CBOPlan(RelNode plan, String
invalidAutomaticRewritingMaterializationReason) {
+this.plan = plan;
+this.invalidAutomaticRewritingMaterializationReason =
invalidAutomaticRewritingMaterializationReason;
+ }
+
+ /**
+ * Root node of plan.
+ * @return Root {@link RelNode}
+ */
+ public RelNode getPlan() {
+return plan;
+ }
+
+ /**
+ * Returns an error message if this plan can not be a definition of a
Materialized view which is an input of
+ * Calcite based materialized view query rewrite.
+ * null or empty string otherwise.
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 530503)
Time Spent: 0.5h (was: 20m)
> Filter out materialized views for rewriting if plan pattern is not allowed
> --
>
> Key: HIVE-24434
> URL: https://issues.apache.org/jira/browse/HIVE-24434
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 4.0.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Some materialized views are not enabled for Calcite based rewriting. Rules
> for validating materialized views are implemented by HIVE-20748.
> Since text based materialized view query rewrite doesn't have such
> limitations some logic must be implemented to flag materialized view whether
> they are enabled to text based rewrite only or both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24434) Filter out materialized views for rewriting if plan pattern is not allowed
[
https://issues.apache.org/jira/browse/HIVE-24434?focusedWorklogId=527846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-527846
]
ASF GitHub Bot logged work on HIVE-24434:
-
Author: ASF GitHub Bot
Created on: 23/Dec/20 21:08
Start Date: 23/Dec/20 21:08
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1782:
URL: https://github.com/apache/hive/pull/1782#discussion_r548025308
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1831,7 +1832,8 @@ public RelOptMaterialization
getMaterializedViewForRebuild(String dbName, String
private List getValidMaterializedViews(List
materializedViewTables,
List tablesUsed, boolean forceMVContentsUpToDate, boolean
expandGroupingSets,
- HiveTxnManager txnMgr) throws HiveException {
+ HiveTxnManager txnMgr,
EnumSet
scope)
Review comment:
nit. Import to avoid using fully-qualified class name here?
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Materialization.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.metadata;
+
+import org.apache.calcite.plan.RelOptMaterialization;
+
+import java.util.EnumSet;
+
+import static org.apache.commons.collections.CollectionUtils.intersection;
+
+/**
+ * Wrapper class of {@link RelOptMaterialization} and corresponding flags.
+ */
+public class Materialization {
Review comment:
Should this class extend `RelOptMaterialization` rather than wrapping
it? I think it would make sense since we may extend it with other properties in
the future and may be convenient to be accessible anywhere in Calcite code.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CBOPlan.java
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.parse;
+
+import org.apache.calcite.rel.RelNode;
+
+/**
+ * Wrapper of Calcite plan.
+ */
+public class CBOPlan {
+ private final RelNode plan;
+ private final String invalidAutomaticRewritingMaterializationReason;
+
+ public CBOPlan(RelNode plan, String
invalidAutomaticRewritingMaterializationReason) {
+this.plan = plan;
+this.invalidAutomaticRewritingMaterializationReason =
invalidAutomaticRewritingMaterializationReason;
+ }
+
+ /**
+ * Root node of plan.
+ * @return Root {@link RelNode}
+ */
+ public RelNode getPlan() {
+return plan;
+ }
+
+ /**
+ * Returns an error message if this plan can not be a definition of a
Materialized view which is an input of
+ * Calcite based materialized view query rewrite.
+ * null or empty string otherwise.
Review comment:
nit. null -> Null
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 527846)
Time Spent: 20m (was: 10m)
> Filter out materialized views for rewriting if plan pattern is not allowed
> --
>
>
[jira] [Work logged] (HIVE-24434) Filter out materialized views for rewriting if plan pattern is not allowed
[ https://issues.apache.org/jira/browse/HIVE-24434?focusedWorklogId=524367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-524367 ] ASF GitHub Bot logged work on HIVE-24434: - Author: ASF GitHub Bot Created on: 15/Dec/20 10:58 Start Date: 15/Dec/20 10:58 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request #1782: URL: https://github.com/apache/hive/pull/1782 ### What changes were proposed in this pull request? 1. Store the scope of each materialized view in the cache/registry. It specifies whether the MV can be used in both Calcite and Sql text based query rewrites or Sql text based only. 2. Scope value is calculated when the MV definition is parsed during adding the MV to the Registry. ### Why are the changes needed? Number of Materialized views added to the Calcite based rewrite planner affects query compilation performance. Too much view definition increases the planning time without any benefit. There are plan patterns which are not supported by the planner. MV having such patterns in its definition can be filtered out before even added to the planner. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` mvn test -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_rewrite_11.q -pl itests/qtest -Pitests ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 524367) Remaining Estimate: 0h Time Spent: 10m > Filter out materialized views for rewriting if plan pattern is not allowed > -- > > Key: HIVE-24434 > URL: https://issues.apache.org/jira/browse/HIVE-24434 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 4.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Some materialized views are not enabled for Calcite based rewriting. Rules > for validating materialized views are implemented by HIVE-20748. > Since text based materialized view query rewrite doesn't have such > limitations some logic must be implemented to flag materialized view whether > they are enabled to text based rewrite only or both. -- This message was sent by Atlassian Jira (v8.3.4#803005)
