gianm commented on a change in pull request #7537: Avoid file rename when
pushing segments with HDFSDataPusher
URL: https://github.com/apache/incubator-druid/pull/7537#discussion_r278817563
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/plumber/RealtimePlumber.java
##########
@@ -454,7 +454,7 @@ public void doRun()
DataSegment segment = dataSegmentPusher.push(
mergedFile,
sink.getSegment().withDimensions(IndexMerger.getMergedDimensionsFromQueryableIndexes(indexes)),
- false
+ true
Review comment:
Why'd you change this to `true`? It will involve creating extra segments on
deep storage if people have replicas with their realtime plumbers.
I thought about https://github.com/apache/incubator-druid/pull/7558 while
reviewing this line, because I was about to say on a review here that
BaseAppenderatorDriver is able to safely delete the unpublished replicas'
pushed segments. I was going to say that unfortunately the technique is not
applicable here, because it relies on seeing a publish failure, but without
transactional metadata checks _all_ replicas will have their publishes succeed.
(RealtimePlumber doesn't do transactional metadata checks.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
