You should just be able to read from the source directly - the DirectRunner can read from both Bounded and Unbounded sources without any additional configuration.
On Tue, Feb 7, 2017 at 6:16 PM, Wyatt Frelot <[email protected]> wrote: > Good evening all, > > I am working on project whose goal is to show the strength of beam and > build a "pipeline" that ingests both bounded and unbounded data. > > I use NiFi to both "batch up" and stream the same dataset. I am running > this using the Direct Runner > > Bounded: I can setup the pipeline to read and process the files. Works as > suspected > > Unbounded: *Doesn't work*, but I am certain it's because I don't > understand how the pipeline needs to be created to create the PCollection. > I can't find any good examples > > #*I KNOW THIS IS WRONG* > > > > > > > *StreamingOptions streamingOptions =PipelineOptions.class.;Pipeline stream = > Pipeline.create(streamingOptions);stream.apply("WordStream", Read.from(new > UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<>())) > .apply("ExtraLinesStream",ParDo.of(new AnomalyAlertDetector())) > .apply("WriteToAccumuloStream", ParDo.of(new WriteToAccumuloStream()));* > > > >
