Xingjun Hao created YARN-11054:
----------------------------------
Summary: Alleviate LocalJobRunnerMetricName Conflicts
Key: YARN-11054
URL: https://issues.apache.org/jira/browse/YARN-11054
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 3.3.1
Reporter: Xingjun Hao
Fix For: 3.3.1
In some scenarios, Sqoop will use LocalJobRuner (YarnLocal mode) to run a lot
of jobs, assuming 2 million jobs have been run, LocalJobRunner MetricName
generated by nextInt function is in the range of (0, 2147483647),
Then the probability of conflict is about 2000/2147483647 = 1/1000, which means
that an average of 1 task will fail for every 1000 jobs run.
If LocalJobRunner MetricName is generated by nextLong() whose range is (0,
9223372036854775807), considering that Long's range is 1 billion times that of
Int, the probability of a new MetricName conflicts is also reduced by one
trillionth times.
the probability of conflict also goes to 1/500000000 from 1/1000. ( under the
situation that there are about 200million jobs have been run)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
