On 08/15/2016 09:01 AM, [email protected] wrote:
I would be interested in contributing towards implementing a dialect for BigQuery. Is there a minimal dialect implementation that I could use as a guide on how to start doing this for BigQuery?
I took a peek at BigQuery and the first issue is that it does not seem to have a pep-249-style (https://www.python.org/dev/peps/pep-0249/) DBAPI available, looking at google's docs I just see this: https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/ . So the first step would likely be creating some kind of DBAPI-like facade that approximates the connection / cursor / execute semantics of pep 249. That would make construction of a SQLAlchemy dialect more straightforward. For reflection it looks like the APIs at https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/python/latest/bigquery_v2.tables.html would handle that.
BigQuery does not seem to use SQL for inserts/updates/deletes and instead has direct API commands. It still may be possible to approximate these with a SQLAlchemy dialect, if the SQL compiler here returned special command tokens (or just python functions) rather than strings which were then interpreted by the dialects execution mechanics to route those statements into the specific API calls, rather than SQL-strings (see the CALCHIPAN example below for one way to do this).
I think a SQLAlchemy dialect here is likely feasible but it would be obviously extremely limited in its behaviors. To start this, I'd first start with the README for writing dialects:
https://bitbucket.org/zzzeek/sqlalchemy/src/94a95b3e8fa0e6c8f9201f85a5f119cd424d72ac/README.dialects.rst?fileviewer=file-view-defaultthat will point you to a very undeveloped dialect for MS Access as the "example", but that example is only to illustrate file layout, not implementation. For implementation, I'd first familiarize roughly with a "normal" dialect, e.g. look in https://bitbucket.org/zzzeek/sqlalchemy/src/94a95b3e8fa0e6c8f9201f85a5f119cd424d72ac/lib/sqlalchemy/dialects/?at=master and then check out sqlite, mysql, and postgresql a little bit.
Then, because you're writing a very "alternative-" style dialect, maybe look at some of the more weird ones externally, which are listed at docs.sqlalchemy.org/en/latest/dialects/index.html#external-dialects. In particular you can check out my own CALCHIPAN dialect https://bitbucket.org/zzzeek/calchipan/ which is a dialect for Pandas dataframes - the Google BigQuery API here would be a little bit like this in some ways (though not as complicated). In particular with CALCHIPAN you can see how the SQL compiler generates callable functions, rather than strings, in order to invoke Pandas APIs in response to SQLAlchemy Core elements, rather than sending SQL strings to a database. It also makes a "fake" pep-249 DBAPI so you can see how that looks too.
I think this is doable so let me know how it goes!
Thanks Raul -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected] <mailto:[email protected]>. To post to this group, send email to [email protected] <mailto:[email protected]>. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
-- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
