Re: Go IOs ?
On Mon, Oct 28, 2019 at 9:15 AM Robert Burke wrote: > There are IOs that work [1], but they haven't been vetted for production > use (performance, overhead etc) just yet. As such, I don't recommend > putting them on the site at this time. Of course, folks a free to discuss > and disagree with me on that :). > Makes sense. > > The Go SDK doesn't yet have a way of scaling transforms without putting in > a reshards/GBK, which single threads reads and slows things down > dramatically. The portable API to do so is usually called Splittable DoFn, > and is presently being worked on for the Go SDK [2]. > I think scaling through re-shards should be enough for many cases. We have some transforms in Java/Python, that depend on reshards for scaling and successfully used by my folks in production. For example, https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L224 https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L424 It would be good if we can create JIRAs to capture work needed to make existing Go IO connectors production ready. For example, (1) Add reshards to parallelize reading. (2) Add performance tests to detect anomalies Thanks, Cham > A slightly faster path to performant IOs is to leverage the Cross Language > Transform work so the Go SDK can use the existing Java and Python IOs. IIRC > the trick would be to integrate the client side work to communicate with > the expansion server that Python uses to make that work. > +1 > > [1] https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io > [2] https://s.apache.org/beam-go-sdf > > > On Sun, Oct 27, 2019, 10:10 AM Austin Bennett > wrote: > >> Hi All, >> >> https://beam.apache.org/documentation/io/built-in/ >> >> Only includes Ios for Java and Python. I have heard talk of a Go SDK :-) >> Are there working IOs? If yes, should they also be on that page? >> >> Thanks, >> Austin >> >> >>
Re: Go IOs ?
There are IOs that work [1], but they haven't been vetted for production use (performance, overhead etc) just yet. As such, I don't recommend putting them on the site at this time. Of course, folks a free to discuss and disagree with me on that :). The Go SDK doesn't yet have a way of scaling transforms without putting in a reshards/GBK, which single threads reads and slows things down dramatically. The portable API to do so is usually called Splittable DoFn, and is presently being worked on for the Go SDK [2]. A slightly faster path to performant IOs is to leverage the Cross Language Transform work so the Go SDK can use the existing Java and Python IOs. IIRC the trick would be to integrate the client side work to communicate with the expansion server that Python uses to make that work. [1] https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io [2] https://s.apache.org/beam-go-sdf On Sun, Oct 27, 2019, 10:10 AM Austin Bennett wrote: > Hi All, > > https://beam.apache.org/documentation/io/built-in/ > > Only includes Ios for Java and Python. I have heard talk of a Go SDK :-) > Are there working IOs? If yes, should they also be on that page? > > Thanks, > Austin > > >
Re: Go IOs ?
Seems like we have several file-based IO connectors and datastore here: https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io We should definitely add these to the list. I guess bulk of IOs will be supported through cross-language transforms framework when we have that for Go SDK. Thanks, Cham On Sun, Oct 27, 2019 at 10:10 AM Austin Bennett wrote: > Hi All, > > https://beam.apache.org/documentation/io/built-in/ > > Only includes Ios for Java and Python. I have heard talk of a Go SDK :-) > Are there working IOs? If yes, should they also be on that page? > > Thanks, > Austin > > >
Go IOs ?
Hi All, https://beam.apache.org/documentation/io/built-in/ Only includes Ios for Java and Python. I have heard talk of a Go SDK :-) Are there working IOs? If yes, should they also be on that page? Thanks, Austin