Hello everyone! I have written up a proposal [1] for a DataStore SQL connector. I would love to hear comments and suggestions from the Beam dev community!
A quick summary: DataStore [2] is a NoSQL database with a dynamic schema, where entities (documents) are stored in Kinds (databases). Each entity has a key (unique identifier) [3], which consists of a partition id and a path (can be used to link to other entities). Proposal is to implement *PTransforms* to perform conversion between DataStore data types and Beam types: *EntityToRow* and *RowToEntity*. PTransforms can be used independently or via *SQL Table* (which will use them implicitly). SQL Table should allow users to specify a row schema field name to store the key in. [1] https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?usp=sharing [2] https://cloud.google.com/datastore/ [3] https://cloud.google.com/datastore/docs/concepts/entities#kinds_and_identifiers
