SteNicholas commented on code in PR #2100:
URL:
https://github.com/apache/incubator-celeborn/pull/2100#discussion_r1393587784
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4053,4 +4056,21 @@ object CelebornConf extends Logging {
.doc("Kerberos keytab file path for HDFS storage connection.")
.stringConf
.createOptional
+
+ val DYNAMIC_CONFIG_STORE_BACKEND: ConfigEntry[String] =
+ buildConf("celeborn.dynamicConfig.store.backend")
+ .categories("master", "worker")
+ .doc("Store backend for dynamic config, NONE means disable dynamic
config store")
+ .version("0.3.2")
+ .stringConf
+ .checkValues(Set("FS", "NONE"))
+ .createWithDefault("NONE")
+
+ val DYNAMIC_CONFIG_REFRESH_TIME: ConfigEntry[Long] =
+ buildConf("celeborn.dynamicConfig.refresh.time")
+ .categories("master", "worker")
+ .version("0.3.2")
+ .doc("For refreshing the related dynamic config periodically")
Review Comment:
```suggestion
.doc("The time interval for refreshing the corresponding dynamic
config periodically")
```
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/ConfigService.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+public interface ConfigService {
Review Comment:
Could this add some comments for interface itself and methods?
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4053,4 +4056,21 @@ object CelebornConf extends Logging {
.doc("Kerberos keytab file path for HDFS storage connection.")
.stringConf
.createOptional
+
+ val DYNAMIC_CONFIG_STORE_BACKEND: ConfigEntry[String] =
Review Comment:
Adds the below comments before this config to provider better readability.
```
// //////////////////////////////////////////////////////
// Dynamic Config //
// //////////////////////////////////////////////////////
```
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/DynamicConfigServiceFactory.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+import org.apache.celeborn.common.CelebornConf;
+
+public class DynamicConfigServiceFactory {
+
+ public static ConfigService getConfigService(CelebornConf celebornConf) {
Review Comment:
Could this support other implementation of `ConfigService`? IMO, this could
use SPI mechanism to load the service implementation for better extensibility.
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/DynamicConfigServiceFactory.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+import org.apache.celeborn.common.CelebornConf;
+
+public class DynamicConfigServiceFactory {
+
+ public static ConfigService getConfigService(CelebornConf celebornConf) {
Review Comment:
Could this support other implementation of the `ConfigService`? IMO, this
could use SPI mechanism to laod the service implementation for better plugable
extensibility.
##########
service/src/test/scala/org/apache/celeborn/server/common/service/config/ConfigServiceSuiteJ.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+import org.apache.celeborn.common.CelebornConf;
+import org.apache.celeborn.server.common.service.config.ConfigService;
+import org.apache.celeborn.server.common.service.config.DynamicConfig;
+import
org.apache.celeborn.server.common.service.config.DynamicConfig.ConfigType;
+import org.apache.celeborn.server.common.service.config.FsConfigServiceImpl;
+import org.apache.celeborn.server.common.service.config.SystemConfig;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class ConfigServiceSuiteJ {
+
+ @Test
+ public void testFsConfig() {
Review Comment:
Could this test case verify whether the tenant-level config overrides the
system-level config?
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/DynamicConfig.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+import org.apache.celeborn.common.internal.config.ConfigEntry;
+import org.apache.celeborn.common.util.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Dynamic configuration is a type of configuration that can be changed at
runtime as needed. It can be used at system level/tenant level.
+ * When applying dynamic configuration, the priority order is as follows:
tenant level overrides system level,
+ * which in turn overrides static configuration(CelebornConf). This means that
if a configuration is defined at the tenant level,
+ * it will be used instead of the system level or static
configuration(CelebornConf). If the tenant-level configuration is missing,
+ * the system-level configuration will be used. If the system-level
configuration is also missing, CelebornConf
Review Comment:
```suggestion
* the system-level configuration will be used. If the system-level
configuration is also missed, CelebornConf
```
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4053,4 +4056,21 @@ object CelebornConf extends Logging {
.doc("Kerberos keytab file path for HDFS storage connection.")
.stringConf
.createOptional
+
+ val DYNAMIC_CONFIG_STORE_BACKEND: ConfigEntry[String] =
+ buildConf("celeborn.dynamicConfig.store.backend")
+ .categories("master", "worker")
+ .doc("Store backend for dynamic config, NONE means disable dynamic
config store")
Review Comment:
```suggestion
.doc("Store backend for dynamic config, NONE means disabling dynamic
config")
```
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/FsConfigServiceImpl.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+import org.apache.celeborn.common.util.ThreadUtils;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+
+import org.apache.celeborn.common.CelebornConf;
+
+public class FsConfigServiceImpl implements ConfigService {
+ private static final Logger LOG =
LoggerFactory.getLogger(FsConfigServiceImpl.class);
+ private CelebornConf celebornConf;
+ private final AtomicReference<SystemConfig> systemConfigAtomicReference =
new AtomicReference<>();
+ private final AtomicReference<Map<String, TenantConfig>>
tenantConfigAtomicReference = new AtomicReference<>(new HashMap<>());
+ private static final String CONF_TENANT_ID = "tenantId";
+ private static final String CONF_LEVEL = "level";
+ private static final String CONF_CONFIG = "config";
+
+ private final ScheduledExecutorService configRefreshService =
Review Comment:
Does the `configRefreshService` need to close?
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/ConfigService.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+public interface ConfigService {
+
+ SystemConfig getSystemConfig();
+
+ TenantConfig getRawTenantConfig(String tenantId);
+
+ default DynamicConfig getTenantConfig(String tenantId) {
+ TenantConfig tenantConfig = getRawTenantConfig(tenantId);
+ if (tenantConfig == null || tenantConfig.getConfigs().isEmpty()) {
+ return getSystemConfig();
+ } else {
+ return tenantConfig;
+ }
+ }
+
+ void refreshAllCache();
+
Review Comment:
Does the `ConfigService` need to introduce `close` method to close the
resource of the service?
##########
service/src/main/scala/org/apache/celeborn/server/common/service/config/DynamicConfig.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.server.common.service.config;
+
+import org.apache.celeborn.common.internal.config.ConfigEntry;
+import org.apache.celeborn.common.util.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Dynamic configuration is a type of configuration that can be changed at
runtime as needed. It can be used at system level/tenant level.
+ * When applying dynamic configuration, the priority order is as follows:
tenant level overrides system level,
+ * which in turn overrides static configuration(CelebornConf). This means that
if a configuration is defined at the tenant level,
+ * it will be used instead of the system level or static
configuration(CelebornConf). If the tenant-level configuration is missing,
Review Comment:
```suggestion
* it will be used instead of the system-level or static
configuration({@link CelebornConf}). If the tenant-level configuration is
missing,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
