william-fundrecs commented on issue #31482: URL: https://github.com/apache/superset/issues/31482#issuecomment-2551949222
Agree, but setting output location for query runs is not default functionality, it has to be configured at the DB level/workgroup setting in AWS. I think this would be outside the remit of Superset. The plugin would need to be able to query Athena API to check this is set or we would need to check a checkbox/set field value in UI at DB connection level when adding connection. > OutputLocation The location in Amazon S3 where your query and calculation results are stored, such as s3://path/to/query/bucket/. To run the query, you must specify the query results location using one of the ways: either for individual queries using either this setting (client-side), or in the workgroup, using [WorkGroupConfiguration](https://docs.aws.amazon.com/athena/latest/APIReference/API_WorkGroupConfiguration.html). If none of them is set, Athena issues an error that no output location is provided. If workgroup settings override client-side settings, then the query uses the settings specified for the workgroup. See [WorkGroupConfiguration:EnforceWorkGroupConfiguration](https://docs.aws.amazon.com/athena/latest/APIReference/API_WorkGroupConfiguration.html#athena-Type-WorkGroupConfiguration-EnforceWorkGroupConfiguration). As it is, once Athena output location has been set, queries against the Athena tables will automatically persist the results in desired format (CSV, Avro, Parquet, ORC, JSON, delimited text) https://docs.aws.amazon.com/athena/latest/ug/creating-databases-prerequisites.html https://aws.amazon.com/about-aws/whats-new/2021/08/athena-can-write-query-results-parquet-avro-orc-json-formats/ it makes sense to use what is already available and download the file that's already been created in S3, rather than get the raw results, process to dataframe, write to file format and return the file through API. Whether this PR should be expanded to download the other available file types if different format has been set is a good question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org For additional commands, e-mail: notifications-h...@superset.apache.org
