[
https://issues.apache.org/jira/browse/YARN-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259975#comment-16259975
]
Zian Chen commented on YARN-6124:
---------------------------------
Thank you [~eepayne] for your comments. for AdminService#refreshQueues, the
first comment, I totally agree with your suggestion since it calls all of the
schedulers, not just capacity scheduler. I'll update the comments according to
your suggestions.
For the reason why conf.size() is been called in AdminService#refreshQueues,
the reason exactly comes from the exception which you have observed in your
newest comment. Let me explain it in details,
I agree with your comments that
SchedulingMonitorManager#updateSchedulingMonitors will call
Configuration#getBoolean which will call Configuration#getProps. However, if
we look into the logic inside CapacityScheduler#reinitialize after applying
YARN-6124.3.patch, we need to make sure we change Queue configuration without
throwing any exception then we can enable preemption related configuration by
calling super.reinitialize(which will call
schedulingMonitorManager#updateSchedulingMonitors in the end), but when we
change the Queue configuration, we will call csConfProvider#loadConfiguration,
however at this point, if we don't call Configuration#getProps by any chance,
the configuration will not really load into memory, meanwhile the input stream
we specified for reading the Configuration inside AdminService#refreshQueues is
already closed, which leads to the refreshQueues:
com.ctc.wstx.exc.WstxIOException: Stream closed exception occurs.
Because of this situation, we need to invoke Configuration#getProps right after
we call Configuration#getConfiguration in order to load all the properties we
need before the input stream close. Wangda and I also think there should be a
better way to achieve this, so we went through all the method which invokes
Configuration#getProps but we only find conf.size() which can be used for our
purpose. That's why conf.size() is been called here.
Hope this can explain the reason we call conf.size() here, we'll appreciate
your help if you could find a better way to avoid the WstxIOException and load
the configuration. Thank you so much for your help!
> Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin
> -refreshQueues
> -----------------------------------------------------------------------------------------
>
> Key: YARN-6124
> URL: https://issues.apache.org/jira/browse/YARN-6124
> Project: Hadoop YARN
> Issue Type: Task
> Reporter: Wangda Tan
> Assignee: Zian Chen
> Attachments: YARN-6124.wip.1.patch, YARN-6124.wip.2.patch,
> YARN-6124.wip.3.patch
>
>
> Now enabled / disable / update SchedulingEditPolicy config requires restart
> RM. This is inconvenient when admin wants to make changes to
> SchedulingEditPolicies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
