[
https://issues.apache.org/jira/browse/TEZ-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923583#comment-15923583
]
Jonathan Eagles commented on TEZ-1526:
--------------------------------------
[~sseth], I think I may have overstated what reality is. The AM Processing
overhead for large numbers of tasks is dramatically reduced. This is mainly due
to the vertex start with creates all TaskImpl (and therefore TezTaskID). Below
illustrates the overhead in the AM both at initialization and runtime. 1 map
feeds 1,000,001 reduce tasks (that auto-reduce down to 1 task).
{noformat}
time HADOOP_CLASSPATH="$TEZ_CONF_DIR:$TEZ_HOME/*:$TEZ_HOME/lib/*" yarn jar
$TEZ_HOME/tez-examples-*.jar orderedwordcount
-Dtez.runtime.optimize.local.fetch=false
-Dtez.shuffle-vertex-manager.enable.auto-parallel=true
-Dtez.am.resource.memory.mb=4096 hamlet.txt owc2/ 1000001
before patch 11m 45s (9m 45s for DAGAppMaster DAG initialization and 2m runtime)
with patch 0m 37s
{noformat}
Beyond this the references held is exactly inline with the previous design. Is
there anything else we should address in this patch?
FYI the TestCartesianProductVertexManagerUnpartitioned change was just a hidden
bug that was uncovered with v6 of the patch
> LoadingCache for TezTaskID slow for large jobs
> ----------------------------------------------
>
> Key: TEZ-1526
> URL: https://issues.apache.org/jira/browse/TEZ-1526
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Labels: performance
> Attachments: 100000-TezTaskIDs.patch, hamlet.txt, TEZ-1526.3.patch,
> TEZ-1526.4.patch, TEZ-1526.5.patch, TEZ-1526.6.patch, TEZ-1526.7.patch,
> TEZ-1526.8.patch, TEZ-1526.memory.test.patch, TEZ-1526-v1.patch,
> TEZ-1526-v2.patch
>
>
> Using the LoadingCache with default builder settings. 100,000 TezTaskIDs are
> created in 10 seconds on my setup. With a LoadingCache initialCapacity of
> 10,000 they are created in 300 ms. With no LoadingCache, they are created in
> 10 ms. A test case in attached to illustrate the condition I would like to be
> sped up.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
