[
https://issues.apache.org/jira/browse/TEZ-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16555900#comment-16555900
]
Kuhu Shukla commented on TEZ-3952:
----------------------------------
Thanks [~nishantdash] for the patch!
The approach is all good and the tests pass for me.. Some minor comments.
In {{TezConfiguration.java}} :
New keys should use dots instead of underscores to follow the norm from other
config keys.
{code:java}
public static final String TEZ_AM_MINIMUM_ALLOWED_SPECULATIVE_TASKS =
TEZ_AM_PREFIX + "minimum_allowed_speculative_tasks";
public static final int TEZ_AM_MINIMUM_ALLOWED_SPECULATIVE_TASKS_DEFAULT = 10;
{code}
should be
{code:java}
public static final String TEZ_AM_MINIMUM_ALLOWED_SPECULATIVE_TASKS =
TEZ_AM_PREFIX + "minimum.allowed.speculative.tasks";
public static final int TEZ_AM_MINIMUM_ALLOWED_SPECULATIVE_TASKS_DEFAULT = 10;
{code}
This needs to apply to all the newly added keys.
The value defaults follow Mapreduce values in Hadoop so that is good.
In {{LegacySepeculator.java}}:
Rename
{code:java}
private final long SOONEST_RETRY_AFTER_NO_SPECULATE;
private final long SOONEST_RETRY_AFTER_SPECULATE;
private final double PROPORTION_RUNNING_TASKS_SPECULATABLE;
private final double PROPORTION_TOTAL_TASKS_SPECULATABLE;
private final int MINIMUM_ALLOWED_SPECULATIVE_TASKS;
private static final int VERTEX_SIZE_THRESHOLD_FOR_TIMEOUT_SPECULATION = 1;
{code}
to non-all-caps variable names like other fields for example :
{{soonestRetryAfterNoSpeculate}} etc. as they are not static finals anymore.
In {{TestSpeculation.java}}:
Nit: Add a newline at the beginning of the test body at line 153.
> Allow Tez task speculation to grant greater customization of certain
> parameters
> -------------------------------------------------------------------------------
>
> Key: TEZ-3952
> URL: https://issues.apache.org/jira/browse/TEZ-3952
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Nishant Dash
> Assignee: Nishant Dash
> Priority: Major
> Attachments: TEZ-3952.001.patch, TEZ-3952.002.patch,
> TEZ-3952.003.patch
>
>
> Many of the settings for Tez task speculation are hardcoded and should
> instead be configurable. For example, there's no equivalent config settings
> for the following MapReduce settings:
> - mapreduce.job.speculative.speculative-cap-running-tasks
> - mapreduce.job.speculative.retry-after-no-speculate
> - mapreduce.job.speculative.retry-after-speculate
> - mapreduce.job.speculative.minimum-allowed-tasks
> - mapreduce.job.speculative.speculative-cap-total-tasks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
