[
https://issues.apache.org/jira/browse/TEZ-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4381:
------------------------------
Description:
TestSecureShuffle is very slow. We need to experiment with it.
There is already an optimization like:
{code}
conf.setInt(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT,
2);
{code}
Maybe it is worth checking if lowering the number of failed attempts can help.
In case of shuffle failures, sooner or later the reducer and mapper task is
retried, so lowering:
{code}
TezConfiguration.TEZ_AM_TASK_MAX_FAILED_ATTEMPTS
{code}
might help, theoretically, as we want the DAG to fail as fast as possible.
> Speed up TestSecureShuffle
> --------------------------
>
> Key: TEZ-4381
> URL: https://issues.apache.org/jira/browse/TEZ-4381
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: László Bodor
> Priority: Major
>
> TestSecureShuffle is very slow. We need to experiment with it.
> There is already an optimization like:
> {code}
>
> conf.setInt(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT,
> 2);
> {code}
> Maybe it is worth checking if lowering the number of failed attempts can
> help. In case of shuffle failures, sooner or later the reducer and mapper
> task is retried, so lowering:
> {code}
> TezConfiguration.TEZ_AM_TASK_MAX_FAILED_ATTEMPTS
> {code}
> might help, theoretically, as we want the DAG to fail as fast as possible.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
