fpopic opened a new issue, #36629:
URL: https://github.com/apache/airflow/issues/36629
### Apache Airflow Provider(s)
sftp
### Versions of Apache Airflow Providers
apache-airflow-providers-sftp==4.4.0
### Apache Airflow version
2.6.3
### Operating System
Ubuntu 20.04.6 LTS
### Deployment
Virtualenv installation
### Deployment details
_No response_
### What happened
I tried to use parameter `newer_than` of type `datetime` with jinja
expression for logical execution date `{{ ds }}` or timestamp `{{ ts }}` but
couldn't find a single way how to please the `SFTPSensor.newer_than` checks.
```py
import datetime
import pendulum
from airflow import models
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.operators.empty import EmptyOperator
with models.DAG(
"dag_example_sftp_sensor_newer_than_example",
schedule_interval="@once",
start_date=datetime.datetime(2024, 1, 1)
) as dag:
start_dag = EmptyOperator(task_id="start_dag")
end_dag = EmptyOperator(task_id="end_dag")
wait_for_sftp_file = SFTPSensor(
task_id=f"wait_for_sftp_file",
sftp_conn_id="sftp_conn_id",
path=f"some-other-jinja-expression-depending-on-airflow-ds-{{ds}}",
newer_than=pendulum.from_format('{{ ds }}', fmt="YYYY-MM-DD")
)
start_dag >> wait_for_sftp_file >> end_dag
```
I get
```
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/dagbag.py",
line 346, in parse
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/home/airflow/gcs/dags/dag_example_sftp_sensor_newer_than_example.py", line
XXX, in <module>
newer_than=pendulum.from_format('{{ ds }}', fmt="YYYY-MM-DD")
File "/opt/python3.8/lib/python3.8/site-packages/pendulum/__init__.py",
line 259, in from_format
parts = _formatter.parse(string, fmt, now(), locale=locale)
File
"/opt/python3.8/lib/python3.8/site-packages/pendulum/formatting/formatter.py",
line 413, in parse
raise ValueError("String does not match format {}".format(fmt))
ValueError: String does not match format YYYY-MM-DD
```
In case I hard-code the value (not using jinja) to something like
```py
newer_than=pendulum.from_format('2024-01-01', fmt="YYYY-MM-DD")
```
everything works.
### What you think should happen instead
Parameter `newer_than` should be working with jinja templates {{ts}} or
{{ds}}.
### How to reproduce
```shell
airflow tasks test dag_example_sftp_sensor_newer_than_example
wait_for_sftp_file 2024-01-01
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
