[
https://issues.apache.org/jira/browse/TEZ-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15921387#comment-15921387
]
Jonathan Eagles commented on TEZ-1526:
--------------------------------------
Failure in TezFaultTolerance in due to TEZ-2686.
[~sseth] pieced together a version of the patch that I think addresses you
comments regarding synchronization since weak hash does a nearly synchronous
API it mostly work, but at scale it will eventually fail. Performance numbers
speak for themselves. For job with large number of Tasks, this significantly
reduces startup time in the DAG as a nice side effect. There are jobs that were
impossible to run before that are now possible.
{noformat:title=Impossible Job with 400,003 tasks 1G AM heap}
time HADOOP_CLASSPATH="$TEZ_CONF_DIR:$TEZ_HOME/*:$TEZ_HOME/lib/*" yarn jar
$TEZ_HOME/tez-examples-*.jar orderedwordcount -Dtez.runtime.sorter.class=LEGACY
-Dtez.runtime.optimize.local.fetch=false
-Dtez.shuffle-vertex-manager.enable.auto-parallel=true hamlet.txt output/ 400001
0m 19s
{noformat}
> LoadingCache for TezTaskID slow for large jobs
> ----------------------------------------------
>
> Key: TEZ-1526
> URL: https://issues.apache.org/jira/browse/TEZ-1526
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Labels: performance
> Attachments: 100000-TezTaskIDs.patch, TEZ-1526.3.patch,
> TEZ-1526.4.patch, TEZ-1526.5.patch, TEZ-1526.6.patch, TEZ-1526.7.patch,
> TEZ-1526.8.patch, TEZ-1526.memory.test.patch, TEZ-1526-v1.patch,
> TEZ-1526-v2.patch
>
>
> Using the LoadingCache with default builder settings. 100,000 TezTaskIDs are
> created in 10 seconds on my setup. With a LoadingCache initialCapacity of
> 10,000 they are created in 300 ms. With no LoadingCache, they are created in
> 10 ms. A test case in attached to illustrate the condition I would like to be
> sped up.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
