virendhar-aws opened a new issue, #23900:
URL: https://github.com/apache/airflow/issues/23900
### Description
When GlueJobs are run directly using AwsGlueJobOperator, a feature to relay
Glue Job's to Airflow log. Currently - I have to make couple of hops to access
Glue Job logs.
### Use case/motivation
Following is the excerpts from Ariflow logs - we see from Glue Job execution,
> [2021-02-17 17:33:56,869] {{taskinstance.py:901}} INFO - Executing
<Task(AwsGlueJobOperator): sample-glue-job> on 2021-02-17T17:33:51.077578+00:00
> [2021-02-17 17:33:56,944] {{standard_task_runner.py:54}} INFO - Started
process 7401 to run task
> [2021-02-17 17:33:56,973] {{standard_task_runner.py:77}} INFO - Running:
['airflow', 'run', 'glue-job-trigger-v2', 'sample-glue-job',
'2021-02-17T17:33:51.077578+00:00', '--job_id', '1761', '--pool',
'default_pool', '--raw', '-sd',
'DAGS_FOLDER/dags/trigger_glue_v2/sample_dag_v2.py', '--cfg_path',
'/tmp/tmp_or8jt2e']
> [2021-02-17 17:33:57,071] {{standard_task_runner.py:78}} INFO - Job 1761:
Subtask sample-glue-job
> [2021-02-17 17:33:57,293] {{logging_mixin.py:112}} INFO - Running %s on
host %s <TaskInstance: glue-job-trigger-v2.sample-glue-job
2021-02-17T17:33:51.077578+00:00 [running]> ip-10-192-21-133.ec2.internal
> [2021-02-17 17:33:57,795] {{glue.py:114}} INFO - Initializing AWS Glue
Job: sample-glue-job
> [2021-02-17 17:33:57,906] {{logging_mixin.py:112}} INFO - [2021-02-17
17:33:57,905] {{base_aws.py:357}} INFO - Airflow Connection:
aws_conn_id=dlp-aws-conn-trigger-glue-crawler-dev
> [2021-02-17 17:33:58,038] {{logging_mixin.py:112}} INFO - [2021-02-17
17:33:58,038] {{base_aws.py:380}} WARNING - Unable to use Airflow Connection
for credentials.
> [2021-02-17 17:33:58,156] {{logging_mixin.py:112}} INFO - [2021-02-17
17:33:58,156] {{base_aws.py:381}} INFO - Fallback on boto3 credential strategy
> [2021-02-17 17:33:58,223] {{logging_mixin.py:112}} INFO - [2021-02-17
17:33:58,223] {{base_aws.py:386}} INFO - Creating session using boto3
credential strategy region_name=None
> [2021-02-17 17:33:58,473] {{logging_mixin.py:112}} INFO - [2021-02-17
17:33:58,473] {{glue.py:165}} INFO - Job Already exist. Returning Name of the
job
> [2021-02-17 17:33:58,735] {{logging_mixin.py:112}} INFO - [2021-02-17
17:33:58,735] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:04,930] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:04,930] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:11,148] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:11,147] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:17,339] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:17,339] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:23,526] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:23,526] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:29,684] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:29,684] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:35,833] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:35,833] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:41,958] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:41,958] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:48,171] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:48,171] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:34:54,333] {{logging_mixin.py:112}} INFO - [2021-02-17
17:34:54,332] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:35:00,488] {{logging_mixin.py:112}} INFO - [2021-02-17
17:35:00,488] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job
current run state with status RUNNING
> [2021-02-17 17:35:06,652] {{logging_mixin.py:112}} INFO - [2021-02-17
17:35:06,651] {{glue.py:145}} INFO - Exiting Job
jr_0eb8c810fa49c81fc086a3d4b27ecc20299b7a36c25de2f6d19edbc7105e234b Run State:
SUCCEEDED
> [2021-02-17 17:35:06,754] {{glue.py:121}} INFO - AWS Glue Job:
sample-glue-job status: SUCCEEDED. Run Id:
jr_0eb8c810fa49c81fc086a3d4b27ecc20299b7a36c25de2f6d19edbc7105e234b
> [2021-02-17 17:35:06,886] {{taskinstance.py:1070}} INFO - Marking task as
SUCCESS.dag_id=glue-job-trigger-v2, task_id=sample-glue-job,
execution_date=20210217T173351, start_date=20210217T173356,
end_date=20210217T173506
> [2021-02-17 17:35:11,688] {{logging_mixin.py:112}} INFO - [2021-02-17
17:35:11,688] {{local_task_job.py:102}} INFO - Task exited with return code 0
>
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
