Hello! I am developing a pipeline which uses FileIO.<>writeDynamic() to write Avro files. Each destination has its own schema, and I am trying to create a PCollectionView of schemas to be used in Writer.
The pipeline works fine if I don't apply window, but in case of unbound source, I have to apply windowing before writing and before building PCollectionView. As per the doc https://beam.apache.org/documentation/programming-guide/#Side-inputs-and-windowing : "If the main input and side inputs have identical windows, the projection provides the exact corresponding window" But it doesn't seem to work exactly like this. With windowed collection, as soon as I apply .apply(Window.into(FixedWindows.of(Duration.standardSeconds(5)))) I start getting empty PCollectionView when Writer tries to process its elements. I am almost sure I am not setting up the window or view correctly, in a way it works for unbounded collections, can anyone help me? I've attached a small java project which demonstrates the issue (writer without window works, but writer with window throws NPE, due to PCollectionView being empty). Thanks! Best Regards, Pavel Solomin Tel: +351 962 950 692 | Skype: pavel_solomin | Linkedin <https://www.linkedin.com/in/pavelsolomin>
<<attachment: file-sink-with-window.zip>>
