[ https://issues.apache.org/jira/browse/TEZ-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175736#comment-17175736 ]
Ashutosh Chauhan commented on TEZ-4213: --------------------------------------- +1 > Bound appContext executor capacity using a configurable property > ---------------------------------------------------------------- > > Key: TEZ-4213 > URL: https://issues.apache.org/jira/browse/TEZ-4213 > Project: Apache Tez > Issue Type: Bug > Reporter: Panagiotis Garefalakis > Assignee: Panagiotis Garefalakis > Priority: Major > Attachments: TEZ-4213.01.patch, TEZ-4213.02.patch, TEZ-4213.03.patch, > TEZ-4213.04.patch, TEZ-4213.05.patch, TEZ-4213.06.patch, TEZ-4213.07.patch > > Time Spent: 2h > Remaining Estimate: 0h > > After TEZ-4170 was merged, appContext executor pool is also used by the > RootInputInitializerManager to speed up SplitGeneration. > However, this executor pool currently has not capacity limit > https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L624 > The problem the occurs when generating splits for larger inputs (thousands or > more) is that it can could result to > {color:red}java.lang.OutOfMemoryError{color} > that is also reproducible with a test case. > https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L130 > To avoid such errors, I propose to limit the capacity of this pool to a > configurable value that can be for example the number of physical cores by > default. -- This message was sent by Atlassian Jira (v8.3.4#803005)