ramitkataria commented on PR #71312:
URL: https://github.com/apache/airflow/pull/71312#issuecomment-5344926277
Yes, that makes sense to me, and I think it covers everything this PR does:
the chunked-upload case would be `{"scope": {"dag_id": ..., "task_id": ...,
"hook": ".*S3Hook"}, "controls": {"exclude_datasets": [...]}}`, and
`exclude_hook_lineage_hooks` becomes hook-only scope with `[".*"]`. I was wrong
that hook can't be a scope: it can't join the up-front per-task resolution, but
I could resolve dag/task/operator eagerly as now, carry any surviving
hook-scoped rules into the collector read, and match each asset's `context`
class there. Does that implementation shape sound reasonable, or did you have
something else in mind?
I can reword the PR to this but I was wondering what your thoughts are on
these before I do it:
1. Should hook scope be limited to per-asset controls, i.e. reject rules
combining it with `emit` etc. during validation (like the existing operator +
`emit_dag_events` check)? And would you want `hook_lineage: false` allowed at
hook scope as a nicer spelling of "drop everything from this hook"?
2. What should `exclude_datasets` patterns match against? Hook assets have
Airflow URIs pre-translation while operator lineage is OL datasets (namespace +
name), so should I match the translated OL identity everywhere so one pattern
behaves the same across sources?
3. Should it also filter manually annotated inlets/outlets, or only
extractor and hook lineage?
4. For tier resolution, is replace-rather-than-merge (most specific
matching rule wins per asset, so a task rule can narrow or clear a global list)
the behaviour you'd expect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
