jscheffl commented on PR #38015:
URL: https://github.com/apache/airflow/pull/38015#issuecomment-2041348474
Uh, oh, late to the party. I noticed this change which - after some reading
- really looks understandable. But when testing 2.9.0RC2 I had a lot of broken
DAGs and it felt to be like a breaking change.
Therefore I raised a PR to add a newsfragment such that we have less support
issues and user compliants. Still I see it a bit problematic... in our cases
the DAGs just broke because we have strong mypy/pylint checks and want to
ensure we have typed code...
I our case we had a DAG and wanted to have the DAG params being typed.
Imageine something like
```
with ... as dag:
@task
def get_file_list(params):
"""Prepares the file list to be re-computed based on the passed
parameters."""
return params["files"]
@task(retries=2)
def recompute_batch(file) -> None:
... # do something with the file
recompute_batch.expand(file=get_file_list())
```
... so here the `get_file_list(params)` is not having types defined which is
not nice...
Then changing this to `get_file_list(params: ParamsDict)` is typed - but to
add the task in the DAG we need to call the function `get_file_list()` in the
code for the expansion. Required argument then is missing.
So we had the option to use either `get_file_list(params: ParamsDict|None)`
which then requires a check against `None` in the code in order to satisfy mypy.
That is why in many DAGs we use the signature like `def
get_file_list(params: ParamsDict = dag.params) -> List[str]:`.
With this change being now on the main line. What is the recommended way for
key context variables and parallel type checks and demand for typed context?
The full example we use typed today, broken in 2.9.0rc2
```
with ... as dag:
@task
def get_file_list(params: ParamsDict = dag.params) -> List[str]:
"""Prepares the file list to be re-computed based on the passed
parameters."""
result: List[str] = params["files"]
return result
@task(retries=2)
def recompute_batch(file: str) -> None:
...
recompute_batch.expand(file=get_file_list())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
