clintropolis commented on code in PR #19571: URL: https://github.com/apache/druid/pull/19571#discussion_r3399475548
########## processing/src/main/java/org/apache/druid/timeline/partition/StreamRangeShardSpec.java: ########## @@ -0,0 +1,164 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.timeline.partition; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.Range; +import com.google.common.collect.RangeSet; + +import javax.annotation.Nullable; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * A {@link NumberedShardSpec} that additionally declares, per dimension, the set of values a streaming segment + * contains ({@link #partitionDimensionValues}), letting the broker prune segments whose values cannot match a query filter + * before compaction. A dimension absent from {@link #partitionDimensionValues} is not pruned on. + */ +public class StreamRangeShardSpec extends NumberedShardSpec Review Comment: is 'range' the right way to refer to this instead of like StreamValuesShardSpec or... something? (idk, naming is hard) I guess I would expect range to contain just like a min/max value for the partition column, but this is full value lists ########## indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java: ########## @@ -693,6 +735,27 @@ public void run() ); if (addResult.isOk()) { + // Accumulate observed dimension values per segment for StreamRangeShardSpec at publish time. + if (!partitionDimensions.isEmpty()) { + final SegmentId segmentId = addResult.getSegmentIdentifier().asSegmentId(); + final Map<String, Set<String>> segValues = observedPartitionDimValuesBySegment + .computeIfAbsent(segmentId, k -> new ConcurrentHashMap<>()); + for (String dim : partitionDimensions) { + final Set<String> dimSet = segValues.computeIfAbsent( + dim, + k -> Collections.synchronizedSet(new HashSet<>()) + ); + // Empty getDimension result means a null/missing value; record null so IS NULL is not pruned + // (distinct from "", which getDimension returns as ["" ]). + final List<String> dimValues = row.getDimension(dim); + if (dimValues == null || dimValues.isEmpty()) { + dimSet.add(null); + } else { + dimSet.addAll(dimValues); + } + } + } Review Comment: it would be cool if there was an interface to abstract this 'do something to collect information for the shard spec' operation (thinking about other useful kinds of shard-specs like this, such as more classic ranges where we just collect min/max values, or if we could build bloom filters or something for higher cardinality columns). I guess this would need streaming partition spec to either be an interface or have a 'processor' thingy that could provide the appropriate interface implementation. This comment is not necessarily a blocker for this PR, just thinking how we could make something like this a bit more flexible/less specific for future improvements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
