suneet-s commented on a change in pull request #10264:
URL: https://github.com/apache/druid/pull/10264#discussion_r471539146
##########
File path:
server/src/main/java/org/apache/druid/segment/realtime/firehose/CombiningFirehoseFactory.java
##########
@@ -33,11 +36,12 @@
import java.io.IOException;
import java.util.Iterator;
import java.util.List;
+import java.util.stream.Stream;
/**
* Creates firehose that combines data from different Firehoses. Useful for
ingesting data from multiple sources.
*/
-public class CombiningFirehoseFactory implements
FirehoseFactory<InputRowParser>
+public class CombiningFirehoseFactory implements
FiniteFirehoseFactory<InputRowParser, List<FirehoseFactory>>
Review comment:
I looked for implementations of `FirehoseFactory` that do not also
implement `FiniteFirehoseFactory` and I found a few... eg.
`ClippedFirehoseFactory`, `FixedCountFirehoseFactory`, etc.
I think the underlying problem is in `IndexIOConfig#getNonNullInputSource`
`IndexIOConfig` accepts a FirehoseFactory, but
[getNonNullInputSource](https://github.com/apache/druid/blob/f6594fff608d4b2e071c7bdd6d86d7f87398ce4f/indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java#L1135)
expects a `FiniteFirehoseFactory`.
I wonder if we could fix this by creating an adapter that translates any
FirehoseFactory into a non-splittable FiniteFirehoseFactory (ie. the same
functionality you've implemented in this class).
I understand that the FirehoseFactories are deprecated, so I'm open to other
ways of fixing the broken quickstart material as well - can we update the spec
used so that the combining firehose factory works without a code change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
