[ 
https://issues.apache.org/jira/browse/TEZ-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976246#comment-15976246
 ] 

Rajesh Balamohan commented on TEZ-3680:
---------------------------------------

TestTaskSchedulerManager failure reported is due to timeout issue. Will commit 
shortly .7 patch to master. 

> Optimizations to UnorderedPartitionedKVWriter
> ---------------------------------------------
>
>                 Key: TEZ-3680
>                 URL: https://issues.apache.org/jira/browse/TEZ-3680
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch, 
> TEZ-3680.3.patch, TEZ-3680.4.patch, TEZ-3680.5.patch, TEZ-3680.6.patch, 
> TEZ-3680.7.patch
>
>
> 1. Consider increasing the number of threads in spill executor. 
> {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}} can be used to 
> configure the buffer size. If smaller buffer sizes are provided, there is a 
> chance of getting frequent spills; currently the spill executor operates in 
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying 
> progress came up. This may not be common in regular tez jobs. But in 
> processes like LLAP (hive based), it is possible to get into such situations. 
> I will attach the profiler snapshot showing this. It would be good to 
> update/notify less frequently.
> 3. Optimize mergeAll().



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to