Hi Rion, For now, it’s only possible to have a PCollection of Void as a result of JdbcIO's write using “withResults()” [1]. This is mostly used along with “Wait” PTransform in case if you need to continue to execute downstream the transforms only once Jdbc writes are finished (see an example in Javadoc [1]).
I’m not sure if it will work for your case, since you need to check the results of writes. Though, as a workaround, I guess you can do write, wait for finishing and then read it back to check the results. In the same time, since we already had the similar requests in the past [3], I think we can have an extended version of “withResults()", like “withWriteResults()” that will return a PCollection of statements results (like we do, for example, for BigtableIO [2]). I’d be happy to guide/assist you (if needed) if you decide to work on this feature [3]. Regards, Alexey [1] https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/io/jdbc/JdbcIO.Write.html#withResults-- [2] https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.Write.html#withWriteResults-- [3] https://issues.apache.org/jira/browse/BEAM-11873 > On 18 Mar 2021, at 03:26, Rion Williams <[email protected]> wrote: > > Hey all, > > Is there any mechanism in place (or out of the box) to support reading the > results of statements executed by the JdbcSink or would I need to implement > my own to support it? > > The problem that I'm trying to solve relates to observability (i.e. metrics) > and incrementing specific counters based on the response from a given > statement executing. One example might be if I need to upsert 40 widgets that > are coming in, although some may be the same widget, I only want to increment > my metric if the widget didn't already exist. > > Thanks, > > Rion
