rochdikhalid opened a new issue, #25342: URL: https://github.com/apache/superset/issues/25342
## [SIP] Seamless Integration of Azure Blob Storage for Superset Import and Export ### Introduction The driving force behind this SIP is to streamline our DevOps processes by enhancing Apache Superset with seamless integration with Azure Blob Storage. The purpose of this integration is to facilitate the transition of Superset assets between different database environments such as testing and production. ### Background Picture this scenario: we have a database configured for staging, and another database tailored for production. Our workflow involves ensuring the utmost consistency between these environments. When utilizing Superset in our test server, it naturally connects to the test database, enabling effective data exploration and dashboard creation. However, when we seek to replicate this in our production environment, we're faced with the daunting task of recreating all the dashboards from beginning. Instead of this time-consuming process, Superset provides a solution that could automatically generates all assets including dashboards, charts, datasets, etc. The solution is to simply import these assets into the production environment without creating dashboards from the ground up. Unfortunately, while Superset existing import and export functionality are valuable, it lacks cloud-based storage options. It confines users to their local storage and references the source database in the generated ZIP file after exporting assets. This means that the generated ZIP file requires manual adjustments when moving from the staging database to the production database. We should edit the database name, and corresponding connecting string within the ZIP file before we import it in our production server. ### Motivation The motivation behind this SIP is to add a seamless integration with Azure Blob Storage, enabling users to easily export and import assets to/from this popular cloud-based storage solution. In addition to the Azure Blob Storage integration, one specific challenge we aim to address is to simplify the import/export process by providing a user-friendly mechanism that allows users to modify the source database information (e.g. name, connection string) within the ZIP file. This feature will not just save time but also guarantee a more seamless transition when moving Superset assets across various database environments. ### Proposed Change #### Export to Azure Blob Storage Create a new API endpoint `/export-to-azure` that enables users to export assets to Azure Blob Storage. The endpoint will generate a ZIP file containing the assets and upload it to a specified Azure Blob Storage Container. #### Import from Azure Blob Storage Create a new API endpoint `/import-from-azure` that allows users to import assets from a ZIP file stored in Azure Blob Storage. This endpoint will download the ZIP file, extract its contents, and perform asset import operations. #### User Interface Changes Minimal UI changes needed to provide users the options to trigger export/import to/from Azure Blob Storage. When users click on the 'Export' option, we can introduce a new popup window that provides both the existing 'Export' choice and an additional option for 'Azure Blob Storage.' The same approach can be implemented for the 'Import' functionality as well, with the ability for users to select the target database environment directly during the import process, eliminating the need for manual adjustments after the export. ### New or Changed Public Interfaces #### New API Endpoints * `/export-to-azure/`: Exports Superset assets to Azure Blob Storage. * `/import-from-azure`: Imports Superset assets from Azure Blob Storage. #### User Interface * A new popup window that provides users the options to import/export from/to Azure Blob Storage with the option to pick the desired database when importing data. ### New dependencies The feature will require basically the `azure-storage-blob` package to interact with Azure Blob Storage. This package is actively maintained and available on PyPI. ### Migration Plan and Compatibility The feature will not require a database migration since it does not introduce any changes to the database schema. It will coexist with the current import/export functionality, allowing for a smooth transition. Users have the freedom to opt for either the new Azure Blob Storage integration or stick with the existing method. ### Rejected Alternatives #### Alternative 1 One alternative was to depend on third-party plugins for Azure Blob Storage integration. However, this approach could cause some compatibility issues. #### Alternative 2 Another option was to require users to manually integrate Superset with Azure Blob Storage using custom code. This approach could lead to inconsistent implementations, and it would less user-friendly compared to the proposed option. The proposed change is the preferred solution as it provides a seamless way to import /export assets to/from Azure Blob Storage within the Superset ecosystem, and it introduces a user-friendly modification when migrating Superset assets between different database environments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
