ChirangaL opened a new issue, #26511: URL: https://github.com/apache/airflow/issues/26511
### Apache Airflow version 2.4.0 ### What happened We are using cross account AWS connection to invoke a Glue job using GlueOperator. When the verbose flag is set we expected the cloudwatch logs to be retrieved from the AWS account where glue job was executed and displayed in Airflow logs. We constantly receive below error on Airflow logs rather than getting cloudwatch logs `Polling for AWS Glue Job <Job name> current run state with status RUNNING No new Glue driver logs found. This might be because there are no new logs, or might be an error. If the error persists, check the CloudWatch dashboard at: https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home` ### What you think should happen instead When AWS GlueOperator is called with a aws_conn_id parameter and verbose flag we expect 1. Glue job to be executed in the aws account as per aws_conn_id 2. Cloudwatch logs to be relieved from aws account as per aws_conn_id and displayed in Airflow logs ### How to reproduce 1. submit a glue job in a different AWS account using GlueOperator as below `with DAG( dag_id="sample_dag", description="Sample DAG testing", schedule_interval=None, start_date=datetime(2022, 9, 1), catchup=False, ) as dag: submit_glue_job = SeekGlueJobOperator( aws_conn_id="different_aws_account", task_id="submit_glue_job", job_name=job_name, wait_for_completion=True, retry_limit=1, script_location=None, iam_role_name="iam-role-for-glue-job-invocation", script_args={}, run_job_kwargs={"NumberOfWorkers": 2, 'WorkerType': "Standard"}, verbose=True, region_name="ap-southeast-1" ) ` 2. Check Airflow logs to see if cloudwatch logs are retrieved and displayed ### Operating System Mac 12.6 ### Versions of Apache Airflow Providers _No response_ ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
