abhishekrb19 commented on code in PR #15738:
URL: https://github.com/apache/druid/pull/15738#discussion_r1467089141
##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataQuery.java:
##########
@@ -173,6 +174,49 @@ public CloseableIterator<DataSegment>
retrieveUnusedSegments(
);
}
+ /**
+ * Retrieves segments for a given datasource that are marked unused and that
are *fully contained by* any interval
Review Comment:
nit: link up the other related function to highlight the difference:
```suggestion
* Similar to {@link #retrieveUnusedSegments}, but also retrieves
associated metadata for the segments for a given datasource that are marked
unused and that are *fully contained by* any interval
```
##########
server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java:
##########
@@ -3343,6 +3503,64 @@ private ImmutableList<DataSegment>
retrieveUnusedSegments(
);
}
+ private ImmutableList<DataSegmentPlus> retrieveUnusedSegmentDtos(
Review Comment:
```suggestion
private ImmutableList<DataSegmentPlus> retrieveUnusedSegmentsPlus(
```
##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataManager.java:
##########
@@ -976,7 +977,7 @@ public Optional<Iterable<DataSegment>>
iterateAllUsedNonOvershadowedSegmentsForD
* Returns an iterable.
*/
@Override
- public Iterable<DataSegment> iterateAllUnusedSegmentsForDatasource(
+ public Iterable<DataSegmentPlus> iterateAllUnusedSegmentsForDatasource(
Review Comment:
Update javadoc to reflect the change?
"Retrieves segments and its associated metadata..."
##########
server/src/test/java/org/apache/druid/server/http/MetadataResourceTest.java:
##########
@@ -74,6 +75,10 @@ public class MetadataResourceTest
.withNumPartitions(NUM_PARTITIONS)
.eachOfSizeInMb(500)
.toArray(new DataSegment[0]);
+
+ private final List<DataSegmentPlus> segmentDtos = Arrays.stream(segments)
Review Comment:
Same here in this class
##########
server/src/test/java/org/apache/druid/server/http/DataSegmentPlusTest.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.InjectableValues;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.fasterxml.jackson.databind.ser.std.ToStringSerializer;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import nl.jqno.equalsverifier.EqualsVerifier;
+import org.apache.druid.TestObjectMapper;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.indexer.partitions.HashedPartitionsSpec;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.jackson.JacksonUtils;
+import org.apache.druid.timeline.CompactionState;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedShardSpec;
+import org.joda.time.DateTime;
+import org.joda.time.Interval;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+public class DataSegmentPlusTest
+{
+ private static final ObjectMapper MAPPER = new TestObjectMapper();
Review Comment:
You could just do that comes with the serializer for `DateTime` (we should
probably just add it in `TestOjbectMapper` later):
```suggestion
MAPPER = new DefaultObjectMapper();
```
##########
server/src/test/java/org/apache/druid/server/http/DataSegmentPlusTest.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.InjectableValues;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.fasterxml.jackson.databind.ser.std.ToStringSerializer;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import nl.jqno.equalsverifier.EqualsVerifier;
+import org.apache.druid.TestObjectMapper;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.indexer.partitions.HashedPartitionsSpec;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.jackson.JacksonUtils;
+import org.apache.druid.timeline.CompactionState;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedShardSpec;
+import org.joda.time.DateTime;
+import org.joda.time.Interval;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+public class DataSegmentPlusTest
+{
+ private static final ObjectMapper MAPPER = new TestObjectMapper();
+ private static final int TEST_VERSION = 0x9;
+
+ @Before
+ public void setUp()
+ {
+ InjectableValues.Std injectableValues = new InjectableValues.Std();
+ injectableValues.addValue(DataSegment.PruneSpecsHolder.class,
DataSegment.PruneSpecsHolder.DEFAULT);
+ MAPPER.setInjectableValues(injectableValues);
+ MAPPER.registerModule(
+ new SimpleModule()
+ .addDeserializer(DateTime.class, new DateTimeDeserializer())
+ .addSerializer(DateTime.class, ToStringSerializer.instance)
+ );
+ }
+ @Test
+ public void testEquals()
+ {
+ EqualsVerifier.forClass(DataSegmentPlus.class)
+ .withNonnullFields("dataSegment", "createdDate")
+ .usingGetClass()
+ .verify();
+ }
+
+ @Test
+ public void testSerde() throws JsonProcessingException
+ {
+ final Interval interval = Intervals.of("2011-10-01/2011-10-02");
+ final ImmutableMap<String, Object> loadSpec = ImmutableMap.of("something",
"or_other");
+
+ String createdDateStr = "2024-01-20T00:00:00.701Z";
+ String usedStatusLastUpdatedDateStr = "2024-01-20T01:00:00.701Z";
+ DateTime createdDate = DateTimes.of(createdDateStr);
+ DateTime usedStatusLastUpdatedDate =
DateTimes.of(usedStatusLastUpdatedDateStr);
+ DataSegmentPlus segmentPlus = new DataSegmentPlus(
+ new DataSegment(
+ "something",
+ interval,
+ "1",
+ loadSpec,
+ Arrays.asList("dim1", "dim2"),
+ Arrays.asList("met1", "met2"),
+ new NumberedShardSpec(3, 0),
+ new CompactionState(
+ new HashedPartitionsSpec(100000, null,
ImmutableList.of("dim1")),
+ new DimensionsSpec(
+ DimensionsSpec.getDefaultSchemas(ImmutableList.of("dim1",
"bar", "foo"))
+ ),
+ ImmutableList.of(ImmutableMap.of("type", "count", "name",
"count")),
+ ImmutableMap.of("filter", ImmutableMap.of("type", "selector",
"dimension", "dim1", "value", "foo")),
+ ImmutableMap.of(),
+ ImmutableMap.of()
+ ),
+ TEST_VERSION,
+ 1
+ ),
+ createdDate,
+ usedStatusLastUpdatedDate
+ );
+
+ final Map<String, Object> objectMap = MAPPER.readValue(
+ MAPPER.writeValueAsString(segmentPlus),
+ JacksonUtils.TYPE_REFERENCE_MAP_STRING_OBJECT
+ );
+
+ Assert.assertEquals(3, objectMap.size());
+ final Map<String, Object> segmentObjectMap = MAPPER.readValue(
+ MAPPER.writeValueAsString(segmentPlus.getDataSegment()),
+ JacksonUtils.TYPE_REFERENCE_MAP_STRING_OBJECT
+ );
+
+ // verify dataSegment
+ Assert.assertEquals(11, segmentObjectMap.size());
+ Assert.assertEquals("something", segmentObjectMap.get("dataSource"));
+ Assert.assertEquals(interval.toString(), segmentObjectMap.get("interval"));
+ Assert.assertEquals("1", segmentObjectMap.get("version"));
+ Assert.assertEquals(loadSpec, segmentObjectMap.get("loadSpec"));
+ Assert.assertEquals("dim1,dim2", segmentObjectMap.get("dimensions"));
+ Assert.assertEquals("met1,met2", segmentObjectMap.get("metrics"));
+ Assert.assertEquals(ImmutableMap.of("type", "numbered", "partitionNum", 3,
"partitions", 0), segmentObjectMap.get("shardSpec"));
+ Assert.assertEquals(TEST_VERSION, segmentObjectMap.get("binaryVersion"));
+ Assert.assertEquals(1, segmentObjectMap.get("size"));
+ Assert.assertEquals(6, ((Map)
segmentObjectMap.get("lastCompactionState")).size());
+
+ // verify extra metadata
+ Assert.assertEquals(createdDateStr, objectMap.get("createdDate"));
+ Assert.assertEquals(usedStatusLastUpdatedDateStr,
objectMap.get("usedStatusLastUpdatedDate"));
+
+ DataSegmentPlus deserializedSegmentPlus =
MAPPER.readValue(MAPPER.writeValueAsString(segmentPlus), DataSegmentPlus.class);
+
+ // verify dataSegment
+ Assert.assertEquals(segmentPlus.getDataSegment().getDataSource(),
deserializedSegmentPlus.getDataSegment().getDataSource());
+ Assert.assertEquals(segmentPlus.getDataSegment().getInterval(),
deserializedSegmentPlus.getDataSegment().getInterval());
+ Assert.assertEquals(segmentPlus.getDataSegment().getVersion(),
deserializedSegmentPlus.getDataSegment().getVersion());
+ Assert.assertEquals(segmentPlus.getDataSegment().getLoadSpec(),
deserializedSegmentPlus.getDataSegment().getLoadSpec());
+ Assert.assertEquals(segmentPlus.getDataSegment().getDimensions(),
deserializedSegmentPlus.getDataSegment().getDimensions());
+ Assert.assertEquals(segmentPlus.getDataSegment().getMetrics(),
deserializedSegmentPlus.getDataSegment().getMetrics());
+ Assert.assertEquals(segmentPlus.getDataSegment().getShardSpec(),
deserializedSegmentPlus.getDataSegment().getShardSpec());
+ Assert.assertEquals(segmentPlus.getDataSegment().getSize(),
deserializedSegmentPlus.getDataSegment().getSize());
+ Assert.assertEquals(segmentPlus.getDataSegment().getId(),
deserializedSegmentPlus.getDataSegment().getId());
+ Assert.assertEquals(segmentPlus.getDataSegment().getLastCompactionState(),
deserializedSegmentPlus.getDataSegment().getLastCompactionState());
+
+ // verify extra metadata
+ Assert.assertEquals(segmentPlus.getCreatedDate(),
deserializedSegmentPlus.getCreatedDate());
+ Assert.assertEquals(segmentPlus.getUsedStatusLastUpdatedDate(),
deserializedSegmentPlus.getUsedStatusLastUpdatedDate());
+ }
+
+ // Copied from org.apache.druid.jackson.JodaStuff
+ private static class DateTimeDeserializer extends StdDeserializer<DateTime>
Review Comment:
and remove this?
##########
server/src/test/java/org/apache/druid/server/http/DataSegmentPlusTest.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.JsonToken;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.InjectableValues;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.fasterxml.jackson.databind.ser.std.ToStringSerializer;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import nl.jqno.equalsverifier.EqualsVerifier;
+import org.apache.druid.TestObjectMapper;
+import org.apache.druid.data.input.impl.DimensionsSpec;
+import org.apache.druid.indexer.partitions.HashedPartitionsSpec;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.jackson.JacksonUtils;
+import org.apache.druid.timeline.CompactionState;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NumberedShardSpec;
+import org.joda.time.DateTime;
+import org.joda.time.Interval;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+
+public class DataSegmentPlusTest
+{
+ private static final ObjectMapper MAPPER = new TestObjectMapper();
+ private static final int TEST_VERSION = 0x9;
+
+ @Before
+ public void setUp()
Review Comment:
Thanks for adding this test!
##########
server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataQuery.java:
##########
@@ -455,12 +547,73 @@ private UnmodifiableIterator<DataSegment>
retrieveSegmentsInIntervalsBatch(
@Nullable final SortOrder sortOrder,
@Nullable final DateTime maxUsedStatusLastUpdatedTime
)
+ {
+ final Query<Map<String, Object>> sql = buildSegmentQuery(
+ dataSource,
+ intervals,
+ matchMode,
+ used,
+ limit,
+ lastSegmentId,
+ sortOrder,
+ maxUsedStatusLastUpdatedTime,
+ false
+ );
+
+ final ResultIterator<DataSegment> resultIterator =
getDataSegmentResultIterator(sql);
+
+ return filterDataSegmentIteratorByInterval(resultIterator, intervals,
matchMode);
+ }
+
+ private UnmodifiableIterator<DataSegmentPlus>
retrieveSegmentsPlusInIntervalsBatch(
+ final String dataSource,
+ final Collection<Interval> intervals,
+ final IntervalMode matchMode,
+ final boolean used,
+ @Nullable final Integer limit,
+ @Nullable final String lastSegmentId,
+ @Nullable final SortOrder sortOrder,
+ @Nullable final DateTime maxUsedStatusLastUpdatedTime
+ )
+ {
+
+ final Query<Map<String, Object>> sql = buildSegmentQuery(
+ dataSource,
+ intervals,
+ matchMode,
+ used,
+ limit,
+ lastSegmentId,
+ sortOrder,
+ maxUsedStatusLastUpdatedTime,
+ true
+ );
+
+ final ResultIterator<DataSegmentPlus> resultIterator =
getDataSegmentPlusResultIterator(sql);
+
+ return filterDataSegmentPlusIteratorByInterval(resultIterator, intervals,
matchMode);
+ }
+
+ private Query<Map<String, Object>> buildSegmentQuery(
Review Comment:
nit:
```suggestion
private Query<Map<String, Object>> buildSegmentsTableQuery(
```
##########
server/src/main/java/org/apache/druid/server/http/DataSegmentPlus.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.guice.annotations.UnstableApi;
+import org.apache.druid.metadata.MetadataStorageTablesConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.DateTime;
+
+import javax.annotation.Nullable;
+import java.util.Objects;
+
+/**
+ * Encapsulates additional metadata about a {@link DataSegment}
+ * </p>
+ * createdDate: The time when the segment was created
+ * usedStatusLastUpdatedDate: The time when the segment's used status was last
updated
+ *
+ * The class closesly resembles the row structure of the {@link
MetadataStorageTablesConfig#getSegmentsTable()}
+ */
Review Comment:
```suggestion
/**
* Encapsulates a {@link DataSegment} and additional metadata about it:
* <li> {@code createdDate}: The time when the segment was created </li>
* <li> {@code usedStatusLastUpdatedDate}: The time when the segments used
status was last updated </li>
*
* <p>
* The class closesly resembles the row structure of the {@link
MetadataStorageTablesConfig#getSegmentsTable()}
* </p>
*/
```
##########
server/src/main/java/org/apache/druid/server/http/DataSegmentPlus.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.guice.annotations.UnstableApi;
+import org.apache.druid.metadata.MetadataStorageTablesConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.joda.time.DateTime;
+
+import javax.annotation.Nullable;
+import java.util.Objects;
+
+/**
+ * Encapsulates additional metadata about a {@link DataSegment}
+ * </p>
+ * createdDate: The time when the segment was created
+ * usedStatusLastUpdatedDate: The time when the segment's used status was last
updated
+ *
+ * The class closesly resembles the row structure of the {@link
MetadataStorageTablesConfig#getSegmentsTable()}
+ */
+@UnstableApi
Review Comment:
Curious why this class is annotated `UnstableApi`? Adding more fields to
this class should be both forward and backward compatible, right?
##########
server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java:
##########
@@ -1268,13 +1283,24 @@ public void
testRetrieveUnusedSegmentsUsingNoIntervalsNoLimitAndNoLastSegmentId(
);
Assert.assertEquals(segments.size(), actualUnusedSegments.size());
Assert.assertTrue(segments.containsAll(actualUnusedSegments));
+
+ final ImmutableList<DataSegmentPlus> actualUnusedSegmentDtos =
retrieveUnusedSegmentDtos(
Review Comment:
Rename all the old references in this file from `retrieveUnusedSegmentDtos`
-> `retrieveUnusedSegmentsPlus`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
