This will eventually be a great addition to aid usability -- thanks for starting to think through and address, John!
On Fri, Nov 10, 2023, 10:54 AM Danny McCormick via dev <dev@beam.apache.org> wrote: > Thanks - the general ideas seem solid, I added some questions/comments as > well. > > On Fri, Nov 10, 2023 at 1:32 PM Robert Bradshaw via dev < > dev@beam.apache.org> wrote: > >> Thanks. I added some comments to the doc and open PR. >> >> On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev <dev@beam.apache.org> >> wrote: >> > >> > Hi All, >> > >> > I've written up a design for adding DLQs to existing Beam IOs. It's >> been through a round of reviews with some Dataflow folks at Google, but I'd >> appreciate any comments the rest of Beam have around how to refine the >> design. >> > >> > TL;DR: Make it easy for a user to configure IOs to route bad data to an >> alternate sink instead of crashing the pipeline or having the record be >> retried indefinitely. >> > >> > >> https://docs.google.com/document/d/1NGeCk6tOqF-TiGEAV7ixd_vhIiWz9sHPlCa1P_77Ajs/edit?usp=sharing >> > >> > Thanks! >> > >> > John >> >
