[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[ https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=467881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-467881 ] ASF GitHub Bot logged work on HIVE-23963: - Author: ASF GitHub Bot Created on: 07/Aug/20 15:09 Start Date: 07/Aug/20 15:09 Worklog Time Spent: 10m Work Description: jcamachor merged pull request #1357: URL: https://github.com/apache/hive/pull/1357 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 467881) Time Spent: 1h 10m (was: 1h) > UnsupportedOperationException in queries 74 and 84 while applying > HiveCardinalityPreservingJoinRule > --- > > Key: HIVE-23963 > URL: https://issues.apache.org/jira/browse/HIVE-23963 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The following TPC-DS queries: > * cbo_query74.q > * cbo_query84.q > * query74.q > * query84.q > fail on the metastore with the partitioned TPC-DS 30TB dataset. > The stacktraces for cbo_query74 and cbo_query84 show that the problem > originates while applying HiveCardinalityPreservingJoinRule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[
https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=467771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-467771
]
ASF GitHub Bot logged work on HIVE-23963:
-
Author: ASF GitHub Bot
Created on: 07/Aug/20 08:47
Start Date: 07/Aug/20 08:47
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1357:
URL: https://github.com/apache/hive/pull/1357#discussion_r466908433
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
##
@@ -81,8 +84,16 @@ public RelDistribution apply(TargetMapping mapping) {
return this;
}
List newKeys = new ArrayList<>(keys.size());
+
+// Instead of using a HashMap for lookup
newKeys.add(mapping.getTargetOpt(key)); should be called but not all the
+// mapping supports that. See HIVE-23963. Replace this when this is fixed
in calcite.
Review comment:
Created: https://issues.apache.org/jira/browse/CALCITE-4166
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 467771)
Time Spent: 1h (was: 50m)
> UnsupportedOperationException in queries 74 and 84 while applying
> HiveCardinalityPreservingJoinRule
> ---
>
> Key: HIVE-23963
> URL: https://issues.apache.org/jira/browse/HIVE-23963
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The following TPC-DS queries:
> * cbo_query74.q
> * cbo_query84.q
> * query74.q
> * query84.q
> fail on the metastore with the partitioned TPC-DS 30TB dataset.
> The stacktraces for cbo_query74 and cbo_query84 show that the problem
> originates while applying HiveCardinalityPreservingJoinRule.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[
https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=467449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-467449
]
ASF GitHub Bot logged work on HIVE-23963:
-
Author: ASF GitHub Bot
Created on: 06/Aug/20 17:33
Start Date: 06/Aug/20 17:33
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1357:
URL: https://github.com/apache/hive/pull/1357#discussion_r466575229
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
##
@@ -81,8 +84,16 @@ public RelDistribution apply(TargetMapping mapping) {
return this;
}
List newKeys = new ArrayList<>(keys.size());
+
+// Instead of using a HashMap for lookup
newKeys.add(mapping.getTargetOpt(key)); should be called but not all the
+// mapping supports that. See HIVE-23963. Replace this when this is fixed
in calcite.
Review comment:
@kasakrisz , please create the Calcite JIRA so it is easier to track.
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 467449)
Time Spent: 50m (was: 40m)
> UnsupportedOperationException in queries 74 and 84 while applying
> HiveCardinalityPreservingJoinRule
> ---
>
> Key: HIVE-23963
> URL: https://issues.apache.org/jira/browse/HIVE-23963
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The following TPC-DS queries:
> * cbo_query74.q
> * cbo_query84.q
> * query74.q
> * query84.q
> fail on the metastore with the partitioned TPC-DS 30TB dataset.
> The stacktraces for cbo_query74 and cbo_query84 show that the problem
> originates while applying HiveCardinalityPreservingJoinRule.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[
https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=467196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-467196
]
ASF GitHub Bot logged work on HIVE-23963:
-
Author: ASF GitHub Bot
Created on: 06/Aug/20 09:56
Start Date: 06/Aug/20 09:56
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1357:
URL: https://github.com/apache/hive/pull/1357#discussion_r466296959
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
##
@@ -81,8 +84,16 @@ public RelDistribution apply(TargetMapping mapping) {
return this;
}
List newKeys = new ArrayList<>(keys.size());
+
+// Instead of using a HashMap for lookup
newKeys.add(mapping.getTargetOpt(key)); should be called but not all the
+// mapping supports that. See HIVE-23963. Replace this when this is fixed
in calcite.
Review comment:
If it is meant to be fixed in Calcite then we should create a JIRA and
add an entry in `org.apache.hadoop.hive.ql.optimizer.calcite.Bug`. We could
even skip the JIRA creation and move this comment in `Bug` as
`CALCITE-X_fixed` or something similar.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 467196)
Time Spent: 40m (was: 0.5h)
> UnsupportedOperationException in queries 74 and 84 while applying
> HiveCardinalityPreservingJoinRule
> ---
>
> Key: HIVE-23963
> URL: https://issues.apache.org/jira/browse/HIVE-23963
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The following TPC-DS queries:
> * cbo_query74.q
> * cbo_query84.q
> * query74.q
> * query84.q
> fail on the metastore with the partitioned TPC-DS 30TB dataset.
> The stacktraces for cbo_query74 and cbo_query84 show that the problem
> originates while applying HiveCardinalityPreservingJoinRule.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[
https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=467067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-467067
]
ASF GitHub Bot logged work on HIVE-23963:
-
Author: ASF GitHub Bot
Created on: 06/Aug/20 03:56
Start Date: 06/Aug/20 03:56
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1357:
URL: https://github.com/apache/hive/pull/1357#discussion_r466130752
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
##
@@ -82,7 +83,14 @@ public RelDistribution apply(TargetMapping mapping) {
}
List newKeys = new ArrayList<>(keys.size());
for (Integer key : keys) {
- newKeys.add(mapping.getTargetOpt(key));
+ // Instead of this inner for loop
newKeys.add(mapping.getTargetOpt(key)); should be called but not all the
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 467067)
Time Spent: 0.5h (was: 20m)
> UnsupportedOperationException in queries 74 and 84 while applying
> HiveCardinalityPreservingJoinRule
> ---
>
> Key: HIVE-23963
> URL: https://issues.apache.org/jira/browse/HIVE-23963
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The following TPC-DS queries:
> * cbo_query74.q
> * cbo_query84.q
> * query74.q
> * query84.q
> fail on the metastore with the partitioned TPC-DS 30TB dataset.
> The stacktraces for cbo_query74 and cbo_query84 show that the problem
> originates while applying HiveCardinalityPreservingJoinRule.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[
https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=466327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-466327
]
ASF GitHub Bot logged work on HIVE-23963:
-
Author: ASF GitHub Bot
Created on: 04/Aug/20 16:57
Start Date: 04/Aug/20 16:57
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1357:
URL: https://github.com/apache/hive/pull/1357#discussion_r465194632
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelDistribution.java
##
@@ -82,7 +83,14 @@ public RelDistribution apply(TargetMapping mapping) {
}
List newKeys = new ArrayList<>(keys.size());
for (Integer key : keys) {
- newKeys.add(mapping.getTargetOpt(key));
+ // Instead of this inner for loop
newKeys.add(mapping.getTargetOpt(key)); should be called but not all the
Review comment:
nit. Can we add the mapping to a map and do the lookup, instead of
having a nested loop?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 466327)
Time Spent: 20m (was: 10m)
> UnsupportedOperationException in queries 74 and 84 while applying
> HiveCardinalityPreservingJoinRule
> ---
>
> Key: HIVE-23963
> URL: https://issues.apache.org/jira/browse/HIVE-23963
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Stamatis Zampetakis
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The following TPC-DS queries:
> * cbo_query74.q
> * cbo_query84.q
> * query74.q
> * query84.q
> fail on the metastore with the partitioned TPC-DS 30TB dataset.
> The stacktraces for cbo_query74 and cbo_query84 show that the problem
> originates while applying HiveCardinalityPreservingJoinRule.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23963) UnsupportedOperationException in queries 74 and 84 while applying HiveCardinalityPreservingJoinRule
[ https://issues.apache.org/jira/browse/HIVE-23963?focusedWorklogId=466047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-466047 ] ASF GitHub Bot logged work on HIVE-23963: - Author: ASF GitHub Bot Created on: 04/Aug/20 06:19 Start Date: 04/Aug/20 06:19 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request #1357: URL: https://github.com/apache/hive/pull/1357 ### What changes were proposed in this pull request? Avoid `UnsupportedOperationException` when calculating cost using `HiveOnTezCostModel`. https://issues.apache.org/jira/browse/HIVE-23963?focusedCommentId=17169985=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17169985 ### Why are the changes needed? See above section. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Run query 74 and 84 using a 30tb metastore dump: ``` mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestTezPerfDBCliDriver -Dqfile=query74.q,query84.q -Dtest.metastore.db=postgres.tpcds -pl itests/qtest -Pitests ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 466047) Remaining Estimate: 0h Time Spent: 10m > UnsupportedOperationException in queries 74 and 84 while applying > HiveCardinalityPreservingJoinRule > --- > > Key: HIVE-23963 > URL: https://issues.apache.org/jira/browse/HIVE-23963 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Stamatis Zampetakis >Assignee: Krisztian Kasa >Priority: Major > Attachments: cbo_query74_stacktrace.txt, cbo_query84_stacktrace.txt > > Time Spent: 10m > Remaining Estimate: 0h > > The following TPC-DS queries: > * cbo_query74.q > * cbo_query84.q > * query74.q > * query84.q > fail on the metastore with the partitioned TPC-DS 30TB dataset. > The stacktraces for cbo_query74 and cbo_query84 show that the problem > originates while applying HiveCardinalityPreservingJoinRule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
