[
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974978#comment-15974978
]
Harish Jaiprakash commented on TEZ-3680:
----------------------------------------
Without the getSpillPathDetails move to callInternal, the move into constructor
changes are extra. The older code was simpler, that it happens right there. Is
the plan to move it later? Else does it make sense to revert those changes to
the earlier version?
> Optimizations to UnorderedPartitionedKVWriter
> ---------------------------------------------
>
> Key: TEZ-3680
> URL: https://issues.apache.org/jira/browse/TEZ-3680
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch,
> TEZ-3680.3.patch, TEZ-3680.4.patch, TEZ-3680.5.patch, TEZ-3680.6.patch
>
>
> 1. Consider increasing the number of threads in spill executor.
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to
> configure the buffer size. If smaller buffer sizes are provided, there is a
> chance of getting frequent spills; currently the spill executor operates in
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying
> progress came up. This may not be common in regular tez jobs. But in
> processes like LLAP (hive based), it is possible to get into such situations.
> I will attach the profiler snapshot showing this. It would be good to
> update/notify less frequently.
> 3. Optimize mergeAll().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
