[
https://issues.apache.org/jira/browse/HIVE-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749572#comment-13749572
]
Hudson commented on HIVE-4963:
------------------------------
FAILURE: Integrated in Hive-trunk-hadoop1-ptest #137 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/137/])
HIVE-4963 : Support in memory PTF partitions (Harish Butani via Ashutosh
Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1517236)
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPartition.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPersistence.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLeadLag.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NPath.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java
*
/hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestPTFRowContainer.java
* /hive/trunk/ql/src/test/queries/clientpositive/ptf_reuse_memstore.q
*
/hive/trunk/ql/src/test/queries/clientpositive/windowing_adjust_rowcontainer_sz.q
* /hive/trunk/ql/src/test/results/clientpositive/ptf_reuse_memstore.q.out
*
/hive/trunk/ql/src/test/results/clientpositive/windowing_adjust_rowcontainer_sz.q.out
> Support in memory PTF partitions
> --------------------------------
>
> Key: HIVE-4963
> URL: https://issues.apache.org/jira/browse/HIVE-4963
> Project: Hive
> Issue Type: New Feature
> Components: PTF-Windowing
> Reporter: Harish Butani
> Assignee: Harish Butani
> Fix For: 0.12.0
>
> Attachments: HIVE-4963.D11955.1.patch, HIVE-4963.D12279.1.patch,
> HIVE-4963.D12279.2.patch, HIVE-4963.D12279.3.patch, PTFRowContainer.patch
>
>
> PTF partitions apply the defensive mode of assuming that partitions will not
> fit in memory. Because of this there is a significant deserialization
> overhead when accessing elements.
> Allow the user to specify that there is enough memory to hold partitions
> through a 'hive.ptf.partition.fits.in.mem' option.
> Savings depends on partition size and in case of windowing the number of
> UDAFs and the window ranges. For eg for the following (admittedly extreme)
> case the PTFOperator exec times went from 39 secs to 8 secs.
>
> {noformat}
> select t, s, i, b, f, d,
> min(t) over(partition by 1 rows between unbounded preceding and current row),
> min(s) over(partition by 1 rows between unbounded preceding and current row),
> min(i) over(partition by 1 rows between unbounded preceding and current row),
> min(b) over(partition by 1 rows between unbounded preceding and current row)
> from over10k
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
