[
https://issues.apache.org/jira/browse/BEAM-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17483467#comment-17483467
]
Wei Fong Hsia commented on BEAM-13298:
--------------------------------------
[~aromanenko] - continuing our conversation here.
Apologies I didn't really update the tickets or what not.
I've been trying to do this - I have tons of rough work - I can push them up
into the fork I have in my repo if you'd like to take a look.
The methodology I took (with a lot of guidance from the devs) is to create a
new class, WriteRecordsWithOutput - essentially a complete copy of WriteRecords.
Then alter WriteRecords, so that it delegates to WriteRecordsWithOutput.
This is similar to how AvroIO has it built.
Then create another class to handle the WriteResults (similar to that of
BigQueryIO's WriteResult) and output that as an output of the new class,
WriteRecordsWithOutput.
WriteResults can disregard the output and continue to simply return the PDone
as it does today.
> Add "withResults()" for KafkaIO.write
> -------------------------------------
>
> Key: BEAM-13298
> URL: https://issues.apache.org/jira/browse/BEAM-13298
> Project: Beam
> Issue Type: Sub-task
> Components: io-ideas, io-java-kafka
> Environment: Production
> Reporter: Ranjan Dahal
> Assignee: Alexey Romanenko
> Priority: P2
> Labels: KafkaIO
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> I am looking at use case where we have to wait until the Kafka Write
> operation is completed before we can move forward with another transform.
> Currently, JdbcIO support withResults() which waits for the previous
> transform to complete as part of Wait.on(Signal) and moves on to the next.
> Similarly, it would be very beneficial to have this capability on KafkaIO
> (and others like PubSubIO, BigQueryIO etc).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
