[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277186#comment-17277186
]
Andras Gyori commented on YARN-10532:
-------------------------------------
Thank you [~zhuqi] for the new patch. My additions are:
* use the getLastSubmittedTimestamp instead of the explicit member variable
because of the locking mechanism
* AutoCreatedQueueDeletionEvent
** Please add license header
** The following line is redundant in AutoCreatedQueueDeletionPolicy:
{code:java}
assert null == scheduler : "Unexpected duplicate call to init";
{code}
instanceof CapacityScheduler covers this case
*
** monitoringInterval should only be a value of the
QUEUE_AUTO_REMOVAL_MONITORING_INTERVAL, I am not quite sure about the other
values there
** the getter methods should be marked visiblefortesting only
** there are unused setters/getters there
** reinitalize should be removed, I dont think we will use it anywhere
** getPolicyName should return the stringified name of the class, not a
hardcoded value
** please rethink method names, for example initQueues makes no sense in this
revision
* CapacityScheduler
** the removeQueue could be rewritten to:
{code:java}
writeLock.lock();
try {
LOG.info("Removing queue: " + queue.getQueuePath());
if (!(queue instanceof AbstractCSQueue)) {
throw new SchedulerDynamicEditException(
"The queue that we are asked " + "to remove (" +
queue.getQueuePath()
+ ") is not a DynamicQueue");
}
if (!((AbstractCSQueue) queue).isEligibleForAutoDeletion()) {
LOG.warn("Queue " + queue.getQueuePath() +
" is marked for deletion, but not eligible for deletion");
return;
}
ParentQueue parentQueue = (ParentQueue)queue.getParent();
if (parentQueue != null) {
((ParentQueue) queue.getParent()).removeChildQueue(queue);
} else {
throw new SchedulerDynamicEditException(
"The queue " + queue.getQueuePath()
+ " can't removed normally.");
}
if (parentQueue.childQueues.contains(queue) ||
queueManager.getQueue(queue.getQueuePath()) != null) {
throw new SchedulerDynamicEditException(
"The queue " + queue.getQueuePath()
+ " has not been removed normally.");
}
LOG.info("Removed queue: " + queue.getQueuePath());
} finally {
writeLock.unlock();
}
{code}
** If a queue is not eligible for auto deletion, we should not throw an error,
just a warning (this is a plausible scenario in case of a race condition)
* queue-auto-removal-monitoring-interval is redundant with
queue-expiration-time, but is a misleading name. Perhaps the
queue-expiration-time is the better one, this should be preferred.
* LeafQueue
** signalToSubmitToQueue should be renamed to signalSubmission
** I agree that the number of all applications should be taken into
consideration in isEligibleForAutoDeletion
* Please add more test cases and test the logic of the policy itself
> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is
> not being used
> --------------------------------------------------------------------------------------------
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Qi Zhu
> Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch,
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch,
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch,
> YARN-10532.009.patch, YARN-10532.010.patch
>
>
> It's better if we can delete auto-created queues when they are not in use for
> a period of time (like 5 mins). It will be helpful when we have a large
> number of auto-created queues (e.g. from 500 users), but only a small subset
> of queues are actively used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
