[GitHub] [airflow] mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom
mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom URL: https://github.com/apache/airflow/pull/5276#issuecomment-503492742 @andreitsibets I actually wrote a whole [blogpost](https://medium.com/slido-dev-blog/automate-executing-aws-athena-queries-and-moving-the-results-around-s3-with-airflow-dd3603dc611d?source=friends_link=eb8c182412464c74019aacb07074ba92) on how to do what you asked in the comment above. It is kind of a walkthrough so it may be helpful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom
mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom URL: https://github.com/apache/airflow/pull/5276#issuecomment-503160808 @andreitsibets here is how I currently do it: https://gist.github.com/mrshu/e44b559d39b8c9464d524b655588ef0d This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom
mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom URL: https://github.com/apache/airflow/pull/5276#issuecomment-502347671 @feluelle @bryanyang0528 @mik-laj Would you mind taking another look? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom
mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom URL: https://github.com/apache/airflow/pull/5276#issuecomment-500108040 @feluelle Sorry it took me this long to get back to this. I've added a test as you suggested and also refactored the `AWSAthenaOperator`'s test suite a bit: - the Athena query ID is now stored in a separate constant, so that it can be easily compared in tests - the Athena query ID is now a random UUID rather than `'1234'` - the test suite now contains a simple DAG, which is necessary for testing XComs If you would like to try the changes out yourself, it should be as easy as running ```bash ./run_unit_tests.sh tests.contrib.operators.test_aws_athena_operator:TestAWSAthenaOperator -s --logging-level=DEBUG ``` Please do let me know if there is anything else I can do to get this to "mergeable" state. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom
mrshu commented on issue #5276: [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom URL: https://github.com/apache/airflow/pull/5276#issuecomment-495365289 Sure @feluelle, would you mind giving me some pointers as to what would be the best way of doing that? I can see that `AWSAthenaOperator` has its own test suite where everything is mocked while `xcom_pull` is only being tested in `s3_to_sftp`, `ssh` and `sftp` operators which use a different structure. Do you think it would make sense to just create a similar test function in the already existing file? Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
