Hi beam team, I have been working on a data pipeline which consumes data from an SFTP location and the data is passed through multiple transforms and finally published to a kafka topic. During this we need to maintain the state of the file, so we write to the database during the pipeline. We came across that JDBCIo write is only considered as a sink and it returns a PDone. But for our requirement we need to pass down the elements that were persisted.
Using writeVoid is also not useful since this is an unbounded stream and we cannot use Wait.On with that without using windows and triggers. Read file --> write the file state in db --> enrich data --> publish to kafka --> update file state The above is the basic requirement from the pipeline. Any suggestions? Thank you
