mobuchowski commented on code in PR #74776:
URL: https://github.com/apache/airflow/pull/74776#discussion_r4281405098
##########
providers/openlineage/src/airflow/providers/openlineage/utils/utils.py:
##########
@@ -2242,8 +2242,9 @@ def get_filtered_unknown_operator_keys(operator:
BaseOperator) -> dict:
def should_use_external_connection(hook) -> bool:
- # If we're at Airflow 2.10, the execution is process-isolated, so we can
safely run those again.
- return True
+ from airflow.providers.openlineage import conf
+
+ return conf.sql_use_external_connection()
Review Comment:
Changing should_use_external_connection() does not cover all SQL lineage
extraction paths. For example, MySQLToGCSOperator calls
get_openlineage_facets_with_sql() through common.compat. With a current
OpenLineage provider, that resolves to the implementation in
openlineage/sqlparser.py, which defaults to use_connection=True and never
checks should_use_external_connection().
The check in common.compat only runs in its fallback implementation for
older OpenLineage versions. Consequently, MySQLToGCSOperator can still query
information_schema when sql_use_external_connection=False.
Please make the shared helper in openlineage/sqlparser.py honor the setting
too, while preserving use_connection=False when explicitly passed by a caller.
##########
providers/common/sql/src/airflow/providers/common/sql/operators/sql.py:
##########
@@ -321,7 +321,7 @@ def get_openlineage_facets_on_start(self) ->
OperatorLineage | None:
hook=hook,
database_info=database_info,
database=self.database,
- sqlalchemy_engine=hook.get_sqlalchemy_engine(),
+ sqlalchemy_engine=hook.get_sqlalchemy_engine() if
use_external_connection else None,
Review Comment:
When the setting is false and the operator uses PostgresHook or MsSqlHook,
extraction still calls get_openlineage_default_schema() before reaching this
guard. Those implementations execute SELECT CURRENT_SCHEMA; and SELECT
SCHEMA_NAME();, respectively, opening an external database connection. A slow
or unavailable database can therefore still block extraction or raise before
table-only lineage is generated. The disabled path also needs to avoid
database-backed default-schema discovery or provide an offline fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
