Thanks for drafting this Damon! I left some comments on the doc. It's really cool that users can go to one source with a specified file format (json, avro, xml, csv, parquet) and retrieve the relevant file writing PTransform. I also like how the same configuration can be re-used for different file formats. Looking forward to seeing its implementation :)
Ahmed On Mon, Nov 14, 2022 at 11:53 AM Damon Douglas <douglas.da...@gmail.com> wrote: > Hello Everyone, > > I hope you are doing well. The following design document proposes, via a > single configuration, a producer of a Beam File writing transform > supporting multiple formats. > bit.ly/fileioschematransformwriteprovider > > For those new to Beam and Schema, I've added a final section of > suggested pre-requisite reading. It's important that everyone can > participate in the conversation at any level of experience, even if this is > the first day learning Beam. *Please feel invited to let me know > anything that isn't clear so this document can strive to include everyone.* > > *My personal thoughts on the proposal's value* > > I've witnessed many smart people and teams argue and divide over the > subject of a programming language. Beam multi-language support allows us > to join transforms written in various languages, currently Java, Python, > Go, and experimentally TypeScript into a single unified pipeline. It's > Beam's schema and processing of these objects, called Rows, that allow this > unification possible. The aforementioned proposal continues this vision > for producing file and object system sinks via a single language agnostic > configuration and supporting provider. > > Ada Lovelace dreamed of a machine that processed objects instead of just > numbers, so that they might produce music and the human things of life. > Through Beam schema awareness, let us live Ada's dream and join our > multiple languages so that we may end our strife and produce the valuable > stuff of life. > > Sincerely, > > Damon >
