mobuchowski commented on code in PR #47500:
URL: https://github.com/apache/airflow/pull/47500#discussion_r1985315425
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/operators/dbt.py:
##########
@@ -286,6 +286,10 @@ def get_openlineage_facets_on_complete(self,
task_instance) -> OperatorLineage:
if isinstance(self.run_id, int) and self.wait_for_termination is True:
return
generate_openlineage_events_from_dbt_cloud_run(operator=self,
task_instance=task_instance)
+ self.log.info(
+ "Extraction of OpenLineage events from DBT will be skipped: "
+ "`run_id` is not set OR `self.wait_for_termination` is False. "
Review Comment:
If you're logging this, you might as well display which one of those is true
##########
providers/dbt/cloud/src/airflow/providers/dbt/cloud/utils/openlineage.py:
##########
@@ -110,8 +114,15 @@ def generate_openlineage_events_from_dbt_cloud_run(
# catalog is available only if docs are generated
catalog = None
- with suppress(Exception):
+ try:
+ log.debug("Retrieving information about catalog artifact from DBT.")
catalog = operator.hook.get_job_run_artifact(operator.run_id,
path="catalog.json").json()["data"]
+ except Exception: # type: ignore
+ log.info(
+ "Openlineage could not find DBT catalog artifact, usually
available when docs are generated."
+ "Proceeding with metadata extraction. "
+ "If you see error logs above about `HTTP error: Not Found` it's
safe to ignore them."
+ )
Review Comment:
If that's true and the feature is optional, we should not log those errors
and maybe display one collective error message at the end if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
