kfaraz commented on code in PR #20063:
URL: https://github.com/apache/druid/pull/20063#discussion_r3820893177
##########
server/src/main/java/org/apache/druid/metadata/SQLMetadataRuleManager.java:
##########
@@ -336,7 +321,9 @@ public boolean overrideRule(final String dataSource, final
List<Rule> newRules,
final String ruleString;
try {
ruleString = jsonMapper.writeValueAsString(newRules);
- if
(ruleString.equals(jsonMapper.writeValueAsString(rules.get().get(dataSource))))
{
+ // Deliberately a nullable lookup: an unknown datasource must not
compare equal to an
+ // empty rule list, otherwise setting empty rules on it would silently
be a no-op.
Review Comment:
Does it make a difference?
A datasource having no entry in the rule map (i.e. an unknown datasource)
has the same effect as a datasource explicitly having an empty list in the rule
map. The effective rules in both the cases would be the cluster level rules.
##########
server/src/main/java/org/apache/druid/server/coordinator/rules/RetentionRulesSnapshot.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.coordinator.rules;
+
+import com.google.common.collect.Maps;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Immutable snapshot of the retention rules of every datasource.
+ */
+public class RetentionRulesSnapshot
+{
+ private static final RetentionRulesSnapshot EMPTY = new
RetentionRulesSnapshot(Map.of(), List.of());
+
+ /**
+ * Override rules of each datasource, not including {@link
#clusterDefaultRules}.
+ */
+ private final Map<String, List<Rule>> datasourceToRules;
+ /**
+ * Override rules of each datasource, already concatenated with {@link
#clusterDefaultRules}.
+ * Contains an entry only for datasources that have override rules, so that
datasources
+ * without overrides can share the {@link #clusterDefaultRules} instance.
+ */
+ private final Map<String, List<Rule>> datasourceToRulesWithDefault;
+ private final List<Rule> clusterDefaultRules;
+
+ public static RetentionRulesSnapshot empty()
+ {
+ return EMPTY;
+ }
+
+ /**
+ * @param datasourceToRules Rules configured for each datasource, not
including the
+ * cluster defaults.
+ * @param clusterDefaultRules Rules configured for the default datasource.
These apply
+ * to every datasource after its own rules.
+ */
+ public RetentionRulesSnapshot(Map<String, List<Rule>> datasourceToRules,
List<Rule> clusterDefaultRules)
Review Comment:
Since this will only be instantiated in `SQLMetadataRuleManager`, I wonder
if we should remove the 2nd argument `clusterDefaultRules` since those rules
are already present inside the 1st argument, `datasourceToRules`.
Instead of `clusterDefaultRules`, we can pass the `defaultDatasourceName`
which can be used to extract the cluster default rules from the 1st argument.
It helps this class avoid having to reconcile any discrepancy between the
1st and 2nd args.
##########
server/src/main/java/org/apache/druid/server/coordinator/rules/RetentionRulesSnapshot.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.coordinator.rules;
+
+import com.google.common.collect.Maps;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Immutable snapshot of the retention rules of every datasource.
+ */
+public class RetentionRulesSnapshot
+{
+ private static final RetentionRulesSnapshot EMPTY = new
RetentionRulesSnapshot(Map.of(), List.of());
+
+ /**
+ * Override rules of each datasource, not including {@link
#clusterDefaultRules}.
+ */
+ private final Map<String, List<Rule>> datasourceToRules;
+ /**
+ * Override rules of each datasource, already concatenated with {@link
#clusterDefaultRules}.
+ * Contains an entry only for datasources that have override rules, so that
datasources
+ * without overrides can share the {@link #clusterDefaultRules} instance.
+ */
+ private final Map<String, List<Rule>> datasourceToRulesWithDefault;
+ private final List<Rule> clusterDefaultRules;
+
+ public static RetentionRulesSnapshot empty()
+ {
+ return EMPTY;
+ }
+
+ /**
+ * @param datasourceToRules Rules configured for each datasource, not
including the
+ * cluster defaults.
+ * @param clusterDefaultRules Rules configured for the default datasource.
These apply
+ * to every datasource after its own rules.
+ */
+ public RetentionRulesSnapshot(Map<String, List<Rule>> datasourceToRules,
List<Rule> clusterDefaultRules)
+ {
+ this.clusterDefaultRules = List.copyOf(clusterDefaultRules);
+
+ // Copy the rule lists as well as the map spine, so that a caller still
holding one of
+ // the source lists cannot mutate this snapshot.
+ final Map<String, List<Rule>> rules =
Maps.newHashMapWithExpectedSize(datasourceToRules.size());
+ final Map<String, List<Rule>> rulesWithDefault =
Maps.newHashMapWithExpectedSize(datasourceToRules.size());
+ datasourceToRules.forEach((datasource, overrideRules) -> {
+ rules.put(datasource, List.copyOf(overrideRules));
+ if (!overrideRules.isEmpty()) {
+ final List<Rule> combinedRules = new ArrayList<>(overrideRules.size()
+ this.clusterDefaultRules.size());
+ combinedRules.addAll(overrideRules);
+ combinedRules.addAll(this.clusterDefaultRules);
+ rulesWithDefault.put(datasource,
Collections.unmodifiableList(combinedRules));
+ }
+ });
+ this.datasourceToRules = Map.copyOf(rules);
+ this.datasourceToRulesWithDefault = Map.copyOf(rulesWithDefault);
+ }
+
+ /**
+ * Return all rules that exist in the cluster.
+ */
+ public Map<String, List<Rule>> getAllRules()
+ {
+ return datasourceToRules;
+ }
+
+ /**
+ * Override rules configured for this datasource, excluding the cluster
defaults.
+ * <p>
+ * No cluster defaults are appended, so a datasource with no overrides
returns an empty
+ * list. Use {@link #getRulesWithDefault} to get the rules that actually
apply to its segments.
+ */
+ public List<Rule> getRules(String datasource)
Review Comment:
Nit: We can rename these 2 methods to `getOverrideRules` and
`getApplicableRules`/`getEffectiveRules` for clarity.
##########
server/src/main/java/org/apache/druid/server/coordinator/rules/RetentionRulesSnapshot.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.coordinator.rules;
+
+import com.google.common.collect.Maps;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Immutable snapshot of the retention rules of every datasource.
+ */
+public class RetentionRulesSnapshot
+{
+ private static final RetentionRulesSnapshot EMPTY = new
RetentionRulesSnapshot(Map.of(), List.of());
+
+ /**
+ * Override rules of each datasource, not including {@link
#clusterDefaultRules}.
Review Comment:
This map does seem to include the cluster defaults as well. Are we filtering
them out at any point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
