ramitkataria commented on issue #63371:
URL: https://github.com/apache/airflow/issues/63371#issuecomment-6511227850
Hi @SamWheating, a quick note on how this ended up being solved, in case
it's useful: instead of a switch on each hook, the OpenLineage provider's
`emission_policy` can now drop datasets by pattern, optionally only for one
hook (#71312, available from the next OpenLineage provider release). For the
chunked-upload example:
```ini
[openlineage]
emission_policy = [{"scope": {"dag_id": "my_dag", "task_id": "upload",
"hook": "airflow.providers.amazon.aws.hooks.s3.S3Hook"}, "controls":
{"exclude_datasets": ["s3://some_bucket/some_prefix/file_.*"]}}]
```
To report one prefix-level asset of your own instead, set `"hook_lineage":
false` for the task and declare the asset in `outlets`. Since Airflow 3.2 you
can also turn hook lineage off everywhere with `[lineage]
max_assets_per_collector = 0`. Details are in the [emission policy
docs](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/emission_policy.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
