cecemei commented on code in PR #18968:
URL: https://github.com/apache/druid/pull/18968#discussion_r2771999278
##########
server/src/main/java/org/apache/druid/server/compaction/CompactionStatusTracker.java:
##########
@@ -80,10 +80,7 @@ public Set<String> getSubmittedTaskIds()
* This method assumes that the given candidate is eligible for compaction
* based on the current compaction config/supervisor of the datasource.
*/
- public CompactionStatus computeCompactionStatus(
- CompactionCandidate candidate,
- CompactionCandidateSearchPolicy searchPolicy
- )
+ public CompactionStatus computeCompactionStatus(CompactionCandidate
candidate)
Review Comment:
not sure i understand, dont we still track pending/running/completed status?
##########
server/src/main/java/org/apache/druid/server/compaction/CompactionEligibility.java:
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.compaction;
+
+import org.apache.druid.java.util.common.StringUtils;
+
+import java.util.Objects;
+
+/**
+ * Describes the eligibility of an interval for compaction.
+ */
+public class CompactionEligibility
Review Comment:
added a new evaluate static method
##########
server/src/main/java/org/apache/druid/client/indexing/ClientCompactionIntervalSpec.java:
##########
@@ -38,18 +42,37 @@ public class ClientCompactionIntervalSpec
private final Interval interval;
@Nullable
+ private final List<SegmentDescriptor> uncompactedSegments;
+ @Nullable
private final String sha256OfSortedSegmentIds;
@JsonCreator
public ClientCompactionIntervalSpec(
@JsonProperty("interval") Interval interval,
+ @JsonProperty("uncompactedSegments") @Nullable List<SegmentDescriptor>
uncompactedSegments,
Review Comment:
SpecificSegmentsSpec is not supported by msq, and it somewhat felt a bit
deprecated to me, maybe because of the segment lock stuff. i kinda like to
specify an interval since it gives some certainty, also wonder maybe we should
check non-null for interval here? i didnt see any instance with null interval.
##########
server/src/main/java/org/apache/druid/server/compaction/MostFragmentedIntervalFirstPolicy.java:
##########
@@ -47,13 +47,16 @@ public class MostFragmentedIntervalFirstPolicy extends
BaseCandidateSearchPolicy
private final int minUncompactedCount;
private final HumanReadableBytes minUncompactedBytes;
private final HumanReadableBytes maxAverageUncompactedBytesPerSegment;
+ private final double incrementalCompactionUncompactedBytesRatioThreshold;
@JsonCreator
public MostFragmentedIntervalFirstPolicy(
@JsonProperty("minUncompactedCount") @Nullable Integer
minUncompactedCount,
@JsonProperty("minUncompactedBytes") @Nullable HumanReadableBytes
minUncompactedBytes,
@JsonProperty("maxAverageUncompactedBytesPerSegment") @Nullable
HumanReadableBytes maxAverageUncompactedBytesPerSegment,
+ @JsonProperty("incrementalCompactionUncompactedBytesRatioThreshold")
@Nullable
Review Comment:
there's another pr that'd almost ready, i could also move this incremental
part to a separate pr if you feel strong about this need to be separate from
eligibility refactor.
##########
server/src/main/java/org/apache/druid/server/compaction/DataSourceCompactibleSegmentIterator.java:
##########
@@ -329,17 +331,17 @@ private void
findAndEnqueueSegmentsToCompact(CompactibleSegmentIterator compacti
continue;
}
- final CompactionCandidate candidates =
CompactionCandidate.from(segments, config.getSegmentGranularity());
- final CompactionStatus compactionStatus =
CompactionStatus.compute(candidates, config, fingerprintMapper);
- final CompactionCandidate candidatesWithStatus =
candidates.withCurrentStatus(compactionStatus);
+ final CompactionCandidate candidatesWithStatus =
Review Comment:
hmm still thinking about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
