karjaljo commented on issue #55489:
URL: https://github.com/apache/airflow/issues/55489#issuecomment-3990641833
Was not sure if it's better to open a new thread so I posted here first even
though this is closed.
My use case is as follows. I have a set of files coming in at random times
within 5min intervals. Each of these files have the timestamp of the 5min end
time for instance data_file_202603031205.h5. The first dag makes sure that
enough data has arrived and then creates a dataset event. Next DAG is triggered
and it does some additional processing for the files with the same timestamp.
Now this worked perfectly in airflow 2.11, but now that I try to upgrade to 3.1
this whole processing chain fails due to change in the logical_date value
missing from the dataset/asset triggered dagrun. Is there any way to get around
this?
How do I run this kind of task in airflow 3.1 in a dag that is triggered by
an asset event ?
poll_for_network_status = FileSensor(
task_id="wait_for_network_status",
fs_conn_id="fs_tutka",
filepath=f"/path/to/file/{{{{ logical_date.strftime('%Y%m%d%H%M')
}}}}_data_file.h”5,
deferrable=True,
poke_interval=10,
timeout=60 * 5,
mode="reschedule",
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
