[
https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002707#comment-14002707
]
Karthik Kambatla edited comment on YARN-1474 at 5/20/14 2:01 AM:
-----------------------------------------------------------------
Thanks [~ozawa] for your patience with the reviews. I guess we can leave
setRMContext as is. And, let us handle the incompatible change to reinitialize
in a separate JIRA.
On an HA cluster, I noticed that the scheduler threads (FS - updateThread,
continuousSchedulingThread; CS - asyncSchedulerThread) start on init() itself.
Ideally, the threads should start only on start(). I guess we should adopt a
modified version of your earlier patch:
# From {{reinitialize()}}, move the part corresponding to {{if (!initialized)}}
to serviceInit.
# Don't call {{reinitialize()}} in serviceInit or serviceStart.
# For the individual threads in the schedulers, init them in serviceInit, but
call thread.start() in serviceStart()
# serviceStop() for FS looks good. We should fix the serviceStop() for CS.
Other comments:
# In TestFairScheduler, the following is not required.
{code}
// To initialize scheduler
scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFairSchedulerEventLog, the following is not required. In this case and
the above, some tests might require calling resourceManager.startt().
{code}
scheduler.serviceInit(conf);
scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFifoScheduler, we don't need the following:
{code}
scheduler.setRMContext(rm.getRMContext());
{code}
# TestFSLeafQueue doesn't need this either:
{code}
scheduler.serviceInit(conf);
scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestLeafQueue, we should call cs.init() instead of cs.serviceInit(). Also,
in any other places.
# In TestQueueParsing, you might need to call capacityScheduler.init() in
addition to or instead of
{code}
capacityScheduler.reinitialize(conf, null);
{code}
# In TestRMContainerAllocator, we might have to call init() instead of
reinitialize().
# In TestRMWebApp, we should call init() instead of reinitialize()
In general, in the tests,
# If there is an RM / Mock RM involved, we don't have to call setRMContext and
reinitialize as long as RM#init is called.
# If there is no RM / Mock RM, we should call a setRMContext followed by init
on the scheduler. Subsequent, calls should remain reinitialize
was (Author: kkambatl):
Thanks [~ozawa] for your patience with the reviews. I guess we can leave
setRMContext as is. And, let us handle the incompatible change to reinitialize
in a separate JIRA.
On an HA cluster, I noticed that the scheduler threads (FS - updateThread,
continuousSchedulingThread; CS - asyncSchedulerThread) start on init() itself.
Ideally, the threads should start only on start(). I guess we should adopt a
modified version of your earlier patch:
# From {{reinitialize()}}, move the part corresponding to {{if (!initialized)}}
to serviceInit.
# Don't call {{reinitialize()}} in serviceInit or serviceStart.
# For the individual threads in the schedulers, init them in serviceInit, but
call thread.start() in serviceStart()
# serviceStop() for FS looks good. We should fix the serviceStop() for CS.
# In TestFairScheduler, the following is not required.
{code}
// To initialize scheduler
scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFairSchedulerEventLog, the following is not required. In this case and
the above, some tests might require calling resourceManager.startt().
{code}
scheduler.serviceInit(conf);
scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFifoScheduler, we don't need the following:
{code}
scheduler.setRMContext(rm.getRMContext());
{code}
# TestFSLeafQueue doesn't need this either:
{code}
scheduler.serviceInit(conf);
scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestLeafQueue, we should call cs.init() instead of cs.serviceInit(). Also,
in any other places.
# In TestQueueParsing, you might need to call capacityScheduler.init() in
addition to or instead of
{code}
capacityScheduler.reinitialize(conf, null);
{code}
# In TestRMContainerAllocator, we might have to call init() instead of
reinitialize().
# In TestRMWebApp, we should call init() instead of reinitialize()
In general, in the tests,
# If there is an RM / Mock RM involved, we don't have to call setRMContext and
reinitialize as long as RM#init is called.
# If there is no RM / Mock RM, we should call a setRMContext followed by init
on the scheduler. Subsequent, calls should remain reinitialize
> Make schedulers services
> ------------------------
>
> Key: YARN-1474
> URL: https://issues.apache.org/jira/browse/YARN-1474
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Affects Versions: 2.3.0, 2.4.0
> Reporter: Sandy Ryza
> Assignee: Tsuyoshi OZAWA
> Attachments: YARN-1474.1.patch, YARN-1474.10.patch,
> YARN-1474.11.patch, YARN-1474.12.patch, YARN-1474.13.patch,
> YARN-1474.14.patch, YARN-1474.2.patch, YARN-1474.3.patch, YARN-1474.4.patch,
> YARN-1474.5.patch, YARN-1474.6.patch, YARN-1474.7.patch, YARN-1474.8.patch,
> YARN-1474.9.patch
>
>
> Schedulers currently have a reinitialize but no start and stop. Fitting them
> into the YARN service model would make things more coherent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
