Hello Everyone, I hope you are doing well. The following design document proposes, via a single configuration, a producer of a Beam File writing transform supporting multiple formats. bit.ly/fileioschematransformwriteprovider
For those new to Beam and Schema, I've added a final section of suggested pre-requisite reading. It's important that everyone can participate in the conversation at any level of experience, even if this is the first day learning Beam. *Please feel invited to let me know anything that isn't clear so this document can strive to include everyone.* *My personal thoughts on the proposal's value* I've witnessed many smart people and teams argue and divide over the subject of a programming language. Beam multi-language support allows us to join transforms written in various languages, currently Java, Python, Go, and experimentally TypeScript into a single unified pipeline. It's Beam's schema and processing of these objects, called Rows, that allow this unification possible. The aforementioned proposal continues this vision for producing file and object system sinks via a single language agnostic configuration and supporting provider. Ada Lovelace dreamed of a machine that processed objects instead of just numbers, so that they might produce music and the human things of life. Through Beam schema awareness, let us live Ada's dream and join our multiple languages so that we may end our strife and produce the valuable stuff of life. Sincerely, Damon
