Rajesh Balamohan commented on TEZ-3680:

TestTaskSchedulerManager failure reported is due to timeout issue. Will commit 
shortly .7 patch to master. 

> Optimizations to UnorderedPartitionedKVWriter
> ---------------------------------------------
>                 Key: TEZ-3680
>                 URL: https://issues.apache.org/jira/browse/TEZ-3680
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: profiler.png, TEZ-3680.1.patch, TEZ-3680.2.patch, 
> TEZ-3680.3.patch, TEZ-3680.4.patch, TEZ-3680.5.patch, TEZ-3680.6.patch, 
> TEZ-3680.7.patch
> 1. Consider increasing the number of threads in spill executor. 
> configure the buffer size. If smaller buffer sizes are provided, there is a 
> chance of getting frequent spills; currently the spill executor operates in 
> single threaded mode.
> 2. During profiling, things like incrementing the counters, notifying 
> progress came up. This may not be common in regular tez jobs. But in 
> processes like LLAP (hive based), it is possible to get into such situations. 
> I will attach the profiler snapshot showing this. It would be good to 
> update/notify less frequently.
> 3. Optimize mergeAll().

This message was sent by Atlassian JIRA

Reply via email to