[
https://issues.apache.org/jira/browse/HADOOP-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665694#action_12665694
]
Vivek Ratan commented on HADOOP-4413:
-------------------------------------
bq. Vivek, the patch doesn't apply cleanly to trunk anymore. Can you please
regenerate a new patch ?
I've been working on a new patch. Should be out soon.
bq. Can you explain why you need the scheduler object in the instrumentation
classes ?
You're right - it is not used currently, but it could very well be in the
future. It seemed fair that the instrumentation class for the Scheduler have
access to the Scheduler object, as the former decides what to log or capture.
The same model is followed by JobTrackerInstrumentation and
TaskTrackerInstrmentation, and I was being consistent.
bq. For many of the APIs defined, it seems to make sense to include some more
information like which job and which task are affected.
bq. I did not see events related to job lifecycle - like when it was submitted,
initialized, scheduled, completed etc
Remember that there is a JobTrackerInstrumentation class that captures job
related information - jobs added/removed, tasks launched, etc. This is
information independent of schedulers. CapacitySchedulerInstrumentation is
instrumenting just the capacity scheduler, so I've included events that mimic
the scheduler's logic. I didn't see it very useful to include job-specific
information in the scheduler events, especially as that is captured by
JobTrackerInstrumentation , but we can add that in later if we feel it's
important for our analysis.
bq. At the same time, I don't see how the lookingFor*Task events are useful.
Part of the Capacity Scheduler logic is to determine whether to assign a TT a
map or reduce task, if a TT can accept both. The lookingFor*Task events capture
this logic decision.
bq. For creating an instance of the CapacitySchedulerInstrumentation, you can
use the ReflectionUtils API. ..
I'm following the same logic that is used to create the other instrumentation
classes - JobTrackerInstrumentation and TaskTrackerInstrmentation.
bq. The API toFullPropertyName is made private, but is being used by
TestQueueCapacities. Either the code must be duplicated in the test method, or
it should be left package private.
Yes, I caught this yesterday too. Not sure why I made the change, but it's not
needed. I'll remove it.
bq. Like Mac indicated above, it would be good if the event log was more
formatted than it is now.
Agreed. Am looking into that.
> Capacity Scheduler to provide a scheduler history log to record actions taken
> and why
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-4413
> URL: https://issues.apache.org/jira/browse/HADOOP-4413
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Mac Yang
> Attachments: 4413.1.patch
>
>
> It would be very useful if the capacity scheduler can provide a log that
> record the decisions made and actions taken by the scheduler.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
