Lee-W commented on issue #43176:
URL: https://github.com/apache/airflow/issues/43176#issuecomment-3455535232
> Did you ask for a list to make this more general? What I had in mind
applies only to the output of a task, when the xcom key is not provided (and
thus set to return_value) - in other words this check should detect templates
of the form
I'm asking for a list of fields we need to check to make it more specific.
Not all string in `{{ ... }}` format should be checked. We need a concrete idea
on what to be checked.
> Expensive = takes either absolutely or relatively long to load. The main
example that comes to mind is pandas. However, I think this is something that's
more reasonable to live inside a unit test (like the dagbag one), or a
threshold used by the processor to emit warnings. Re generalization - it's
simpler to define what is allowed (e.g. airflow or standard library imports)
rather than what isn't - but this approach will flag much of the existing code.
Yep, but we'll have to define a concrete list. Maybe worth create another
issue to get this list. And if `airflow....DAG` or `airflow...dag` is imported
and these kind of operations exists, we raise a warning. But we need a concrete
list of "these kinds of operation.
> Taskflow is fine and context manager is fine too. But perhaps dag =
DAG(....) should be discouraged.
Totally agree, this one shouldn't be hard.
> Can be checked by a unit test (dagbag).
Yep, then this would be out of Ruff's capability.
> Potentially possible via a dagbag test, assuming it processes dag policies
(might require dev effort to support).
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
