anandhimurali commented on code in PR #38982:
URL: https://github.com/apache/airflow/pull/38982#discussion_r1602602899
##########
airflow/providers/openlineage/conf.py:
##########
@@ -65,6 +65,19 @@ def custom_extractors() -> set[str]:
return set(extractor.strip() for extractor in option.split(";") if
extractor.strip())
+@cache
+def custom_facet_functions() -> set[str]:
+ """[openlineage] custom_facet_functions."""
+ option = conf.get(_CONFIG_SECTION, "custom_facet_functions", fallback="")
+ if not option:
+ option = os.getenv("OPENLINEAGE_CUSTOM_FACET_FUNCTIONS", "")
Review Comment:
>I think good question is whether we can do it without global configs? I
think this might be good to be configurable on task/dag/global level, rather
than fixed for whole Airflow instance. It goes well with our effort to have
lineage be incrementally enabled.
@mobuchowski , User can choose to `return None` in the custom facet
function, if they don't want to add the custom facet for certain tasks/dags.
Thinking if the complexity is going to increase if we handle the
task/dag/global level settings at config level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
