[
https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286567#comment-17286567
]
Qi Zhu commented on YARN-10623:
-------------------------------
Thanks [~gandras] for your reply.
Our production cluster, have more than 25000 nodes with many clusters, in every
cluster, our users will change the queue configuration very frequently.
We use FairScheduler, it can refresh queue automatically by
AllocationFileLoaderService reload by thread:
{code:java}
if (this.allocFile != null) {
this.fs = allocFile.getFileSystem(conf);
reloadThread = new Thread(() -> {
while (running) {
try {
synchronized (this) {
reloadListener.onCheck();
}
long time = clock.getTime();
long lastModified =
fs.getFileStatus(allocFile).getModificationTime();
if (lastModified > lastSuccessfulReload &&
time > lastModified + ALLOC_RELOAD_WAIT_MS) {
try {
reloadAllocations();
} catch (Exception ex) {
if (!lastReloadAttemptFailed) {
LOG.error("Failed to reload fair scheduler config file - " +
"will use existing allocations.", ex);
}
lastReloadAttemptFailed = true;
}
} else if (lastModified == 0l) {
if (!lastReloadAttemptFailed) {
LOG.warn("Failed to reload fair scheduler config file because" +
" last modified returned 0. File exists: "
+ fs.exists(allocFile));
}
lastReloadAttemptFailed = true;
}
} catch (IOException e) {
LOG.error("Exception while loading allocation file: " + e);
}
try {
Thread.sleep(reloadIntervalMs);
} catch (InterruptedException ex) {
LOG.info(
"Interrupted while waiting to reload alloc configuration");
}
}
});
reloadThread.setName("AllocationFileReloader");
reloadThread.setDaemon(true);
}
{code}
But if we change to CapacityScheduler, we should manage every use update queue
related conf, refresh by admin, this is very needed for us to realize
consistent with FairScheduler automatically refresh queue, and i think this is
a good improvement for our users.
What do you think about this?
Thanks.
> Capacity scheduler should support refresh queue automatically by a thread
> policy.
> ---------------------------------------------------------------------------------
>
> Key: YARN-10623
> URL: https://issues.apache.org/jira/browse/YARN-10623
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
> Reporter: Qi Zhu
> Assignee: Qi Zhu
> Priority: Major
> Attachments: YARN-10623.001.patch
>
>
> In fair scheduler, it is supported that refresh queue related conf
> automatically by a thread to reload, but in capacity scheduler we only
> support to refresh queue related changes by refreshQueues, it is needed for
> our cluster to realize queue manage.
> cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras] [~bteke] [~shuzirra]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
