[
https://issues.apache.org/jira/browse/YARN-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415500#comment-16415500
]
Weiwei Yang commented on YARN-7497:
-----------------------------------
Hi [~yangjiandan]
Thanks for the patch, this is a big alternative of ZK implementation when
dealing with a lot of queues. Some comments about the patch:
# My major concern over the patch is that it doesn't use WAL fashion to
prevent potential inconsistent in-memory/persisted conf. I suggest to implement
the log/retrieve logic to make sure it works that way. Something like
\{{ZKConfigurationStore}}.
# I feel it is better to call {{FSSchedulerConfigurationStore }}instead of
\{{HDFSSchedulerConfigurationStore}} as this is should work as long as on any
HFS.
Some other misc comments:
*Conf*
# Need to add some doc in \{{yarn-site.xml}}, property
\{{yarn.scheduler.configuration.store.class}}; also the added configuration
files, make sure \{{TestYarnConfiguration}} passes.
*TestMutableCSConfigurationProvider*
# writeConf: please use try-finally to make sure both fileSystem and
outputStream are closed
*HDFSSchedulerConfigurationStore*
# line 52: typo HFDS
# line 63, 74: pendingId not used
# line 68: typo configFilePathFileter -> configFilePathFilter
# line 78: better to add a null check before using \{{path}}
# line 86: I don't think we need to throw a RuntimeException here, can we
replace it with IOException? So CapacityScheduler can handle this during
\{{initScheduler}}
# line 102, 145: {{configInputStream}} is not closed
# line 205: move \{{outputStream}} to finally statement
Hope it makes sense. Thanks
> Add HDFSSchedulerConfigurationStore for RM HA
> ---------------------------------------------
>
> Key: YARN-7497
> URL: https://issues.apache.org/jira/browse/YARN-7497
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Reporter: Jiandan Yang
> Assignee: Jiandan Yang
> Priority: Major
> Attachments: YARN-7497.001.patch, YARN-7497.002.patch,
> YARN-7497.003.patch, YARN-7497.004.patch, YARN-7497.005.patch,
> YARN-7497.006.patch, YARN-7497.007.patch, YARN-7497.008.patch
>
>
> YARN-5947 add LeveldbConfigurationStore using Leveldb as backing store, but
> it does not support Yarn RM HA.
> YARN-6840 supports RM HA, but too many scheduler configurations may exceed
> znode limit, for example 10 thousand queues.
> HDFSSchedulerConfigurationStore store conf file in HDFS, when RM failover,
> new active RM can load scheduler configuration from HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
