Seems useful. Nice to see SQL's capabilities expanding to more data sources. Commented on the doc. This also may be a useful example for adding more NoSQL data sources. And for non-SQL schema-driven transforms it is helpful.
Kenn On Wed, Jan 8, 2020 at 11:28 AM Kirill Kozlov <[email protected]> wrote: > Hello everyone! > > I have written up a proposal [1] for a DataStore SQL connector. I would > love to hear comments and suggestions from the Beam dev community! > > A quick summary: > DataStore [2] is a NoSQL database with a dynamic schema, where entities > (documents) are stored in Kinds (databases). Each entity has a key (unique > identifier) [3], which consists of a partition id and a path (can be used > to link to other entities). > Proposal is to implement *PTransforms* to perform conversion between > DataStore data types and Beam types: *EntityToRow* and *RowToEntity*. > PTransforms can be used independently or via *SQL Table* (which will use > them implicitly). > SQL Table should allow users to specify a row schema field name to store > the key in. > > [1] > https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?usp=sharing > [2] https://cloud.google.com/datastore/ > [3] > https://cloud.google.com/datastore/docs/concepts/entities#kinds_and_identifiers >
