michaelgaotile opened a new issue #22291:
URL: https://github.com/apache/airflow/issues/22291
### Apache Airflow version
2.2.2
### What happened
This is on Amazon MWAA Airflow
Tasks are scheduled for @daily 0:00 UTC,
Sometimes the task will fail right away, logs show no reason. The second
attempt will kick off at past 14:00 UTC, a 14 hour delay.
Log 1:
*** Reading remote log from Cloudwatch log_group: xxxxx
log_stream:xxxxxx/2022-03-14T00_00_00+00_00/1.log.
[2022-03-15 00:05:03,281] {{logging_mixin.py:104}} INFO - [2022-03-15
00:05:03,281] {{base_aws.py:368}} INFO - Airflow Connection: xxxxx
[2022-03-15 00:05:03,308] {{logging_mixin.py:104}} INFO - [2022-03-15
00:05:03,308] {{base_aws.py:179}} INFO - No credentials retrieved from
Connection
[2022-03-15 00:05:03,308] {{logging_mixin.py:104}} INFO - [2022-03-15
00:05:03,308] {{base_aws.py:87}} INFO - Creating session with
aws_access_key_id=None region_name=xx
[2022-03-15 00:05:03,481] {{logging_mixin.py:104}} INFO - [2022-03-15
00:05:03,481] {{base_aws.py:157}} INFO - role_arn is None
[2022-03-15 00:05:03,848] {{taskinstance.py:877}} INFO - Dependencies all
met for <TaskInstance: xxxxxxxxx[queued]>
[2022-03-15 00:05:04,011] {{taskinstance.py:877}} INFO - Dependencies all
met for <TaskInstance: x2022-03-14T00:00:00+00:00 xxxxxxxxxxx[queued]>
[2022-03-15 00:05:04,011] {{taskinstance.py:1068}} INFO -
--------------------------------------------------------------------------------
[2022-03-15 00:05:04,012] {{taskinstance.py:1069}} INFO - Starting attempt 1
of 4
[2022-03-15 00:05:04,012] {{taskinstance.py:1070}} INFO -
--------------------------------------------------------------------------------
[2022-03-15 00:05:04,100] {{taskinstance.py:1089}} INFO - Executing
<Task(DatabricksRunNowOperator): xxxxxxxxx> on 2022-03-14T00:00:00+00:00
[2022-03-15 00:05:04,152] {{standard_task_runner.py:52}} INFO - Started
process 82 to run task
Log 2:
*** Reading remote log from Cloudwatch log_group:xxxxxxxxlog_stream:
xxxxxxxxt/2022-03-14T00_00_00+00_00/2.log.
[2022-03-15 14:12:00,700] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:00,699] {{base_aws.py:368}} INFO - Airflow Connection:
aws_conn_id=aws_default
[2022-03-15 14:12:00,709] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:00,709] {{base_aws.py:179}} INFO - No credentials retrieved from
Connection
[2022-03-15 14:12:00,709] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:00,709] {{base_aws.py:87}} INFO - Creating session with
aws_access_key_id=None region_name=xxxx
[2022-03-15 14:12:00,725] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:00,725] {{base_aws.py:157}} INFO - role_arn is None
[2022-03-15 14:12:00,851] {{taskinstance.py:877}} INFO - Dependencies all
met for <TaskInstance: xxxxxxxxxxxxx 2022-03-14T00:00:00+00:00 [queued]>
[2022-03-15 14:12:00,871] {{taskinstance.py:877}} INFO - Dependencies all
met for <TaskInstance: xxxxxxxxxxx2022-03-14T00:00:00+00:00 [queued]>
[2022-03-15 14:12:00,871] {{taskinstance.py:1068}} INFO -
--------------------------------------------------------------------------------
[2022-03-15 14:12:00,871] {{taskinstance.py:1069}} INFO - Starting attempt 2
of 4
[2022-03-15 14:12:00,872] {{taskinstance.py:1070}} INFO -
--------------------------------------------------------------------------------
[2022-03-15 14:12:00,883] {{taskinstance.py:1089}} INFO - Executing
<Task(DatabricksRunNowOperator): xxx> on 2022-03-14T00:00:00+00:00
[2022-03-15 14:12:00,886] {{standard_task_runner.py:52}} INFO - Started
process 16758 to run task
[2022-03-15 14:12:00,980] {{logging_mixin.py:104}} INFO - Running
<TaskInstance: xxxxxxxxxxxxxxxx 2022-03-14T00:00:00+00:00 [running]> on host
xxxxxxx
[2022-03-15 14:12:01,041] {{taskinstance.py:1283}} INFO - Exporting the
following env vars:
AIRFLOW_CTX_DAG_OWNER=xxxxxxx
AIRFLOW_CTX_DAG_ID=xxxxxxxxx
AIRFLOW_CTX_TASK_ID=xxxxxxxxxxx
AIRFLOW_CTX_EXECUTION_DATE=2022-03-14T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-03-14T00:00:00+00:00
[2022-03-15 14:12:01,058] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:01,058] {{base.py:78}} INFO - Using connection to: id: databricks.
Host:xxxxxxxxx, Port: None, Schema: , Login: , Password:
[2022-03-15 14:12:01,059] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:01,059] {{databricks.py:170}} INFO - Using token auth.
[2022-03-15 14:12:01,241] {{databricks.py:73}} INFO - Run submitted with
run_id: 5177061
[2022-03-15 14:12:01,242] {{logging_mixin.py:104}} INFO - [2022-03-15
14:12:01,242] {{databricks.py:170}} INFO - Using token auth.
[2022-03-15 14:12:01,439] {{databricks.py:78}} INFO - View run status, Spark
UI, and logs at https:/xxxxxxxxxxxx
### What you think should happen instead
I expect the task to start at the correct time, and in the event of failure,
to restart at the specified restart interval, not 14 hours later.
### How to reproduce
Seems to happen randomly across different jobs.
### Operating System
MWAA
### Versions of Apache Airflow Providers
_No response_
### Deployment
MWAA
### Deployment details
mw1.medium
max worker count = 20
min worker count = 1
scheduler count = 4
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
