Hello! After some experimentation I was able to make it half-broken instead of completely broken:
- I am using Combine.globally(new Combiner()).asSingletonView() to reduce PCollection into a Map - I made the main input to have smaller windows, and the PCollectionView window to be GlobalWindows with triggers This way I am able to write *some* of the data from my unbound source, but it seems that the PCollectionView has schemas missing, and my app eventually fails anyway with NPE. It looks like the Map is being re-created and destroyed every time the window triggers. In general, is there any example of reducing an unbound PCollection into a PCollectionView<Map> *without* windows? Such that the single instance of Map is being used for the entire lifetime of the app, and is not destroyed & re-created on window trigger? Thanks in advance. Best Regards, Pavel Solomin Tel: +351 962 950 692 | Skype: pavel_solomin | Linkedin <https://www.linkedin.com/in/pavelsolomin> On Tue, 5 Oct 2021 at 16:20, Pavel Solomin <[email protected]> wrote: > Hello! > > I am developing a pipeline which uses FileIO.<>writeDynamic() to write > Avro files. Each destination has its own schema, and I am trying to create > a PCollectionView of schemas to be used in Writer. > > The pipeline works fine if I don't apply window, but in case of unbound > source, I have to apply windowing before writing and before building > PCollectionView. > > As per the doc > https://beam.apache.org/documentation/programming-guide/#Side-inputs-and-windowing > : > > "If the main input and side inputs have identical windows, the projection > provides the exact corresponding window" > > But it doesn't seem to work exactly like this. With windowed collection, > as soon as I apply > > .apply(Window.into(FixedWindows.of(Duration.standardSeconds(5)))) > > I start getting empty PCollectionView when Writer tries to process its > elements. I am almost sure I am not setting up the window or view > correctly, in a way it works for unbounded collections, can anyone help me? > I've attached a small java project which demonstrates the issue (writer > without window works, but writer with window throws NPE, due to > PCollectionView being empty). > > Thanks! > > Best Regards, > Pavel Solomin > > Tel: +351 962 950 692 | Skype: pavel_solomin | Linkedin > <https://www.linkedin.com/in/pavelsolomin> > > > >
