GitHub user ChadRosseau edited a discussion: Add support for Cloudflare D1 via external DB-API + SQLAlchemy dialect
## Add support for Cloudflare D1 via external DB-API + SQLAlchemy dialect ### Problem Superset requires a Python DB-API driver and a SQLAlchemy dialect in order to connect to different database engines. Cloudflare D1 (a serverless SQLite-compatible database built on Workers) isn’t currently supported. This makes it impossible for teams using D1 as their backend to connect it as a datasource in Superset. Ourselves and many of our peers are avid users of the Cloudflare stack, including D1, and would welcome a Superset integration with D1. ### Proposal Our team* would like to build an external Python package that adds: * A DB-API 2.0 compliant driver for D1, wrapping its HTTP API * A SQLAlchemy dialect built on top of that driver * The database-specific connector code required to enable extra features, as per [this page](https://preset.io/blog/building-database-connector/) As per the [Superset 'Connecting to Databases' documentation](https://superset.apache.org/docs/configuration/databases/), these are the two crucial components in a database integration. Users would then be able to `pip install` the driver and set up connections in Superset using a connection string such as: ``` d1://<api_token>@<account_id>/<database_name> ``` This approach keeps changes outside of Superset core. Superset will discover the dialect if installed. The only possible core addition later would be adding d1-specific instructions to superset documentation, such as in `docs/docs/configuration/databases.mdx`. ### Limitations We’re Aware Of D1 is not a traditional socket-based database, so some DB-API features can’t be fully supported. For example: * `commit()` will effectively be a no-op, since D1 autocommits. * `rollback()` cannot be supported and would raise `NotImplementedError`. We believe these tradeoffs are acceptable since they align with how D1 itself works. ### Questions for Maintainers 1. While we've looked through current issues and discussions and not found anything, has there been any previous work on Cloudflare D1 integration? 2. Is the plan to implement this as an external package (and not in core) the right path? 3. Would Superset be open to listing D1 as a supported engine in the docs once the package is stable? 4. Are there any dialect features beyond reflection and query execution that Superset expects at minimum? Thank you in advance for your ideas and feedback. *cc: @danielalyoshin, @jang-35, @murphylee10, @ThunderRoar GitHub link: https://github.com/apache/superset/discussions/35273 ---- This is an automatically sent email for notifications@superset.apache.org. To unsubscribe, please send an email to: notifications-unsubscr...@superset.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
