[
https://issues.apache.org/jira/browse/HIVE-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603738#comment-13603738
]
Phabricator commented on HIVE-4041:
-----------------------------------
hbutani has commented on the revision "HIVE-4041 [jira] Support multiple
partitionings in a single Query".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java:212
Yes the translation code could do with your review.
We were not paying much attention to optimization at the time we wrote it.
So the TableFuncDef holds onto ShapeDetails(input, output..)
The Shape class has Serde props that we use to reconstruct the OIs during
runtime.
This happens in PTFTranslator. Read the translate for WindowingSpec method
(line 138) in PTFTranslator.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkDeDuplication.java:415
Just added a Jira 4186 for this.
ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java:482 yes I
agree. I just don't want to make these changes in this Jira. Want to only add
multi partition support here.
ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingComponentizer.java:38 I
will add more comments. So there is:
- 1 PTFOp
- It can contain one or more PTF invocations.
- When the PTF is WindowTableFunc, it can contain 1 or more UDAFs; they have
the same partitioning.
- During translation we create a WindowingSpec for each destination with
Windowing(over clauses).
- Here we then componentize the single WindowingSpec into multiple
WindowingSpecs based on the partitioning.
ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingComponentizer.java:85
Yes the having is to be removed. Haven't gotten around to it. Again didn't want
to make this change in this Jira. It is on my todo.
ql/src/test/queries/clientpositive/windowing_multipartitioning.q:21 I added
all the multipartition tests from the spreadsheet except the ones that have no
order. Those I will add once we resolve how to handle no order.
REVISION DETAIL
https://reviews.facebook.net/D9381
To: JIRA, ashutoshc, hbutani
> Support multiple partitionings in a single Query
> ------------------------------------------------
>
> Key: HIVE-4041
> URL: https://issues.apache.org/jira/browse/HIVE-4041
> Project: Hive
> Issue Type: Bug
> Components: PTF-Windowing
> Reporter: Harish Butani
> Assignee: Harish Butani
> Attachments: HIVE-4041.D9381.1.patch, WindowingComponentization.pdf
>
>
> Currently we disallow queries if the partition specifications of all Wdw fns
> are not the same. We can relax this by generating multiple PTFOps based on
> the unique partitionings in a Query. For partitionings that only differ in
> sort, we can introduce a sort step in between PTFOps, which can happen in the
> same Reduce task.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
