kacpermuda commented on PR #71312: URL: https://github.com/apache/airflow/pull/71312#issuecomment-5239160815
I like the idea of widening the support of emission policy, that's for sure. Few thoughts: 1. Not sure what should be a scope and what can be a control from this PR. Hook class may be the new scope (where we already have exact and regex modes), not control. If mixed with dag_id, or dag_id and task_id, it can provide the granularity that is needed (disable hook level lineage inside dag or just for specific task but just from specific hook). > not narrowed down to hook-level lineage, but operator lineage too 3. Not sure if there is a need to drop only some datasets (regex based) only when they come from hook level lineage, and not if they come from operator methods. We do have some [filters](https://openlineage.io/docs/client/python/configuration#filters) on python client level, but they are looking purely at job name now and are dropping events. If we wanted to, we can extend them to drop datasets that do not match some conditions - that would be a wide dataset filter. If we need to differentiate between hll and operator datasets, we might need to do it in controls, but then again, we might just use hook as scope and then exclude_datasets as control, that can be used both on operator and hook level? Thinking out loud here, not sure how to make it less confusing for people. > going even forward with it, maybe it could be just generalizable "preprocessor" for this data? This could take a form of generalized filters (python function) that are being passed context and lineage and return lineage output. 4. I like the idea of generic preprocessors, but I know that people rarely want to write them and we already have a transform transport that can be used to achieve whatever custom logic they have, so I don't think we need to provide more ways of customizing it. If you need more customization than the emission_policy provides, use transform transport and write your own transformer - would be my advice. > more options than just regex 5. Not sure if this is needed now, for scope we have exact and regex, and for more custom stuff you can always use transform transport I think. @ramitkataria , have you considered some alternatives when it comes to what is control and what is scope here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
