[
https://issues.apache.org/jira/browse/HADOOP-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HADOOP-3116:
--------------------------------
Description: The array, in JobInProgress, that holds the references to TIPs
is required now only to serve the clients (web UI via JSPs and JobClient APIs,
that traverse the array and create the output). The array can now be removed
since HADOOP-2119 introduces some datastructures for running/non-running TIPs
and we could probably add some more if required (e.g., for completed TIPs).
That way, we will gain when we have large jobs (many tasks) in the sense that
we don't have to go through the big array every time a client request is made.
We could instead iterate over the datastructures. Also, we should do these
traversals without locking the JobTracker to avoid cases where long traversal
leads to loss of heartbeats, etc. Some staleness at the client side (in favor
of improving the JobTracker's performance) is probably okay. (was: The array
that holds the references to TIPs is required now only to serve the clients
(web UI via JSPs and JobClient APIs, that traverse the array and create the
output). The array can now be removed since HADOOP-2119 introduces some
datastructures for running/non-running TIPs and we could probably add some more
if required (e.g., for completed TIPs). That way, we will gain when we have
large jobs (many tasks) in the sense that we don't have to go through the big
array every time a client request is made. We could instead iterate over the
datastructures. Also, we should do these traversals without locking the
JobTracker to avoid cases where long traversal leads to loss of heartbeats,
etc. Some staleness at the client side (in favor of improving the JobTracker's
performance) is probably okay.)
Summary: Investigate whether the array in the JobInProgress that holds
TIP references can be removed (was: Investigate whether the array that holds
TIP references can be removed)
> Investigate whether the array in the JobInProgress that holds TIP references
> can be removed
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-3116
> URL: https://issues.apache.org/jira/browse/HADOOP-3116
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Devaraj Das
> Fix For: 0.18.0
>
>
> The array, in JobInProgress, that holds the references to TIPs is required
> now only to serve the clients (web UI via JSPs and JobClient APIs, that
> traverse the array and create the output). The array can now be removed since
> HADOOP-2119 introduces some datastructures for running/non-running TIPs and
> we could probably add some more if required (e.g., for completed TIPs). That
> way, we will gain when we have large jobs (many tasks) in the sense that we
> don't have to go through the big array every time a client request is made.
> We could instead iterate over the datastructures. Also, we should do these
> traversals without locking the JobTracker to avoid cases where long traversal
> leads to loss of heartbeats, etc. Some staleness at the client side (in favor
> of improving the JobTracker's performance) is probably okay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
