jon-wei commented on a change in pull request #7490: Add reload by interval API
URL: https://github.com/apache/incubator-druid/pull/7490#discussion_r277906688
##########
File path:
server/src/main/java/org/apache/druid/metadata/SQLMetadataSegmentManager.java
##########
@@ -219,83 +218,144 @@ public void stop()
}
}
- @Override
- public boolean enableDataSource(final String dataSource)
+ private VersionedIntervalTimeline<String, DataSegment>
getVersionedIntervalTimeline(final String dataSource, final Interval interval)
{
- try {
- final IDBI dbi = connector.getDBI();
- VersionedIntervalTimeline<String, DataSegment> segmentTimeline =
connector.inReadOnlyTransaction(
- (handle, status) -> VersionedIntervalTimeline.forSegments(
- Iterators.transform(
- handle
- .createQuery(
- StringUtils.format(
- "SELECT payload FROM %s WHERE dataSource =
:dataSource",
- getSegmentsTable()
- )
- )
- .setFetchSize(connector.getStreamingFetchSize())
- .bind("dataSource", dataSource)
- .map(ByteArrayMapper.FIRST)
- .iterator(),
- payload -> {
- try {
- return jsonMapper.readValue(payload, DataSegment.class);
- }
- catch (IOException e) {
- throw new RuntimeException(e);
- }
+ return connector.inReadOnlyTransaction(
+ (handle, status) -> VersionedIntervalTimeline.forSegments(
+ Iterators.transform(
+ handle
+ .createQuery(
+ StringUtils.format(
+ "SELECT payload FROM %1$s WHERE dataSource =
:dataSource AND start >= :start AND %2$send%2$s <= :end",
Review comment:
For this query, you'd want to retrieve segments that have an interval that
overlaps with the provided interval, vs. being contained by the provided
interval.
Suppose I have the following 3 segments:
- 2011-04-04/2011-04-09, version 2
- 2011-04-09/2011-04-15, version 2
- 2011-04-08/2011-04-12, version 1
The last segment with version 1 is partially overshadowed by each individual
segment with version 2, and fully overshadowed by the set with version 2.
Now suppose I call markUsed on the interval 2011-04-07/2011-04-13. With the
contains check, neither of the version 2 segments would be added to the
timeline, and the code would fail to detect that the version 1 segment is
really overshadowed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
