deepakjindal90 commented on issue #8087:
URL: https://github.com/apache/airflow/issues/8087#issuecomment-841718405
I am using 1.10.12 and facing same issue due to which my GlueJob called two
times, below are the highlighted log where glue job called two times.
**Scenario** : I have used one dummy operator and success of that task i
have called glue job using 'on_success_callback' callback function
**Note** : This is coming intermittently, not everytime.
[2021-05-15 03:30:10,863] {{taskinstance.py:901}} INFO - Executing
<Task(DummyOperator): insert_run_control> on 2021-05-14T03:30:00+00:00
[2021-05-15 03:30:10,926] {{standard_task_runner.py:54}} INFO - Started
process 6044 to run task
[2021-05-15 03:30:10,943] {{standard_task_runner.py:77}} INFO - Running:
['airflow', 'run', 'dev.udh.loyalty.mk_premier_qual_rule',
'insert_run_control', '2021-05-14T03:30:00+00:00', '--job_id', '8489',
'--pool', 'default_pool', '--raw', '-sd',
'DAGS_FOLDER/commercial/loyalty/mk_premier_qual_rule.py', '--cfg_path',
'/tmp/tmptc3np1i2']
[2021-05-15 03:30:11,063] {{standard_task_runner.py:78}} INFO - Job 8489:
Subtask insert_run_control
[2021-05-15 03:30:11,335] {{logging_mixin.py:112}} INFO - Running %s on host
%s <TaskInstance: dev.udh.loyalty.mk_premier_qual_rule.insert_run_control
2021-05-14T03:30:00+00:00 [running]> ip-10-36-70-203.ec2.internal
[2021-05-15 03:30:11,524] {{logging_mixin.py:112}} INFO - **Task
insert_run_control success :**
[2021-05-15 03:30:11,585] {{logging_mixin.py:112}} INFO - **Updating audit
operator infor table...**
[2021-05-15 03:30:11,654] {{glue.py:119}} INFO - **Initializing AWS Glue
Job: cbs-udh-audit-runner**
[2021-05-15 03:30:30,330] {{glue.py:126}} INFO - AWS Glue Job:
cbs-udh-audit-runner status: SUCCEEDED. Run Id:
jr_61870a034ff896de378199a6148aef24943beb53c42bfbf9f8ecc835f11db975
[2021-05-15 03:30:30,427] {{taskinstance.py:1070}} INFO - Marking task as
SUCCESS.dag_id=dev.udh.loyalty.mk_premier_qual_rule,
task_id=insert_run_control, execution_date=20210514T033000,
start_date=20210515T033010, end_date=20210515T033030
**[2021-05-15 03:30:30,512] {{logging_mixin.py:112}} INFO - [2021-05-15
03:30:30,511] {{local_task_job.py:159}} WARNING - State of this instance has
been externally set to success. Taking the poison pill.**
[2021-05-15 03:30:30,619] {{logging_mixin.py:112}} INFO - **Task
insert_run_control success :**
[2021-05-15 03:30:30,680] {{logging_mixin.py:112}} INFO - **Updating audit
operator infor table...**
[2021-05-15 03:30:30,760] {{glue.py:119}} INFO - **Initializing AWS Glue
Job: cbs-udh-audit-runner**
[2021-05-15 03:30:49,408] {{glue.py:126}} INFO - AWS Glue Job:
cbs-udh-audit-runner status: SUCCEEDED. Run Id:
jr_97de1ed79f3cee852c33bec0cae589c23d5cdb61f3c7262db525acd8f038e3ac
[2021-05-15 03:30:49,495] {{helpers.py:325}} INFO - Sending Signals.SIGTERM
to GPID 6044
[2021-05-15 03:30:49,544] {{helpers.py:291}} INFO - Process
psutil.Process(pid=6044, status='terminated', exitcode=0, started='03:30:10')
(6044) terminated with exit code 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
