capistrant commented on a change in pull request #10676:
URL: https://github.com/apache/druid/pull/10676#discussion_r555388018
##########
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java
##########
@@ -576,6 +584,64 @@ static Granularity
findGranularityFromSegments(List<DataSegment> segments)
}
}
+ /**
+ * Wait for segments to become available on the cluster. If waitTimeout is
reached, giveup on waiting. This is a
+ * QoS method that can be used to make Batch Ingest tasks wait to finish
until their ingested data is available on
+ * the cluster. Doing so gives an end user assurance that a Successful task
status means their data is available
+ * for querying.
+ *
+ * @param toolbox {@link TaskToolbox} object with for assisting with task
work.
+ * @param segmentsToWaitFor {@link List} of segments to wait for
availability.
+ * @param waitTimeout Millis to wait before giving up
+ * @return True if all segments became available, otherwise False.
+ */
+ protected boolean waitForSegmentAvailability(
Review comment:
My initial thoughts:
* I agree that the code is working to achieve the same goal. so re-use would
be nice
* IndexTask appears to be straightforward if we were to take this approach.
* Parallel Indexing and Hadoop Indexing do not seem as straightforward the
way I understand things
* parallel: we'd like to do this handoff wait from the supervisor, IMO.
However there is no existing utilization of Appenderator or AppendoratorDriver
at this level of parallel indexing.
* hadoop: Hadoop does not use Appendorator or AppendoratorDriver so we
would have to bolt that on with an Appendorator impl that is pretty much only
there for show since hadoop indexing does not have any use other than the fact
that it would be needed for this implementation (as far as I can tell)
I'm sure I could make it work eventually. But I wonder if the jumping
through hoops to get the task code ready to be compatible for using the
registerHandoff in BaseAppenderatorDriver is worth it both in terms of effort
and code legibility.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
