jihoonson commented on issue #7048: Make IngestSegmentFirehoseFactory splittable for parallel ingestion URL: https://github.com/apache/incubator-druid/pull/7048#issuecomment-462634766 > OK. And I guess it doesn't really matter how segments are assigned between tasks — all the locking and segment allocation works no matter how much overlap there is, right? Would you tell me more about what kind of overlap you're thinking? I think it's fine if each task processes a disjoint subset of input segments. > Since this will require the subtasks to understand how to have specific segments named rather than just intervals, perhaps the task itself should be extended to allow you to specify a list of segments as an alternative to an interval in the first place? I think we can extend `IngestSegmentFirehoseFactory` to accept a list of segments. In `IngestSegmentFirehoseFactory`, I think the `interval` should be empty if `segments` are specified. This `segments` should also be integrated with the `intervals` of `GranularitySpec`. If `intervals` are given, the task should read only the segments overlapping with the intervals.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
