Ralkion opened a new pull request, #27594: URL: https://github.com/apache/superset/pull/27594
# SUMMARY We are adding the implementation and usage of the AWS S3 upload feature in Apache Superset. This feature allows users to send reports directly to an AWS S3 bucket, and is configured to be an additional delivery option (like email & slack). When writing files to S3 buckets, there are three methods available for authentication: 1. Taking credentials from the user in the frontend (via a modal) per report 2. Taking credentials from the superset_config.py file 3. Utilizing an AWS IAM role associated with the ECS/EC2 role Methods 1 & 2 require an AWS IAM User, while method 3 makes use of the executing IAM Role for the running system. # TESTING INSTRUCTIONS ## Installation and Setup ### Pre-requisites * Ensure Superset is installed and running. * Create an AWS IAM User and an access key / secret key for that user. * Create an S3 bucket with a bucket policy that grants access to the user above, and an aws role of your choice. ### Configuration of feature ##### Enable the feature User should update the superset_config.py (docker/pythonpath_dev/superset_config.py) file with following: Ensure that the reports feature is enabled, and enable the feature with the `FEATURE_FLAGS` in superset_config.py. ``` FEATURE_FLAGS["ALERT_REPORTS"] = True FEATURE_FLAGS["ENABLE_AWS"] = True ALERT_REPORTS_NOTIFICATION_DRY_RUN = False ``` Note: If the User doesn’t enable the feature flag of ENABLE_AWS, but has ALERT_REPORTS set to True then alerts & reports will still work as per normal (Email & Slack delivery options). The AWS S3 option will be unavailable. Once enabled, when creating a report, the user will be able to deploy the result to an S3 bucket. They will be able to specify where/how the file should be written through the web interface.   ##### Method 1: Frontend User Credentials Using this method allows each report to be delivered to different buckets, and to use a different set of credentials. In our use case, this was important as we may be delivering reports directly to different customer S3 buckets. When accessing the feature in the frontend, users will be prompted to enter their AWS credentials through the existing modal. Users can provide their IAM User's Access Key and the Secret Access Key into the user interface itself. When generating the report, these values will be used to write the file to S3.  ##### Method 2: Taking credentials from the superset_config.py file The report may instead be configured to use a default set of credentials as specified in the config file. This use case allows for a more streamlined method for sending reports, always using the same credentials. This can be used primarily when a company is delivering reports to their own S3 bucket, and could be done with an on-premise setup. Add the following code in superset_config.py. ``` # AWS Credentials AWS_ACCESS_KEY = '####' AWS_SECRET_KEY = '####' ```  ##### Method 3: AWS IAM Role Finally, the report may instead be configured to use the running system's IAM role. This use case follows AWS best practices in granting access to a resource through a role instead of directly through a user. Similar to method 2, this method will always use the same credentials - but this is no longer changeable through the config file. Although this method would most commonly be used by services running in AWS (EC2 or ECS), it could also be run by properly configuring the AWS CLI (which I will not get into here).  ### Usage Once setup has been done, the AWS S3 option can be used for sending reports/alert directly to the S3 bucket from either the settings menu, or when editing a dashboard or chart. #### Navigate to settings, then on Alerts & Reports. When creating a report directly, one can choose to deploy the report to an S3 bucket, and may also choose which authentication method they would like to use when delivering the report. * Create a new report * In the Notification Method select S3. * Then select one of the available S3 method options: AWS_S3_credentials, AWS_S3_pyconfig, or AWS_S3_IAM. * Provide the bucket name to which the report will be delivered. * if using AWS_S3_credentials, then fill in the required aws credentials.  #### From the dashboard / chart All three S3 method options are available when viewing a dashboard or chart page. * Create a new dashboard or chart, or view an existing one * Using the dashboard or chart's menu, select Set up a S3 report and use any of the S3 method options available. * Observe that the dashboard or chart's report can be seen in the Alerts & Reports page.   ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [x] Required feature flags: `ALERT_REPORTS`, (new) `ENABLE_AWS` - [x] Changes UI - [x] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [x] Migration is atomic, supports rollback & is backwards-compatible - [x] Confirm DB migration upgrade and downgrade tested - [x] Runtime estimates and downtime expectations provided - [x] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
