amoghrajesh commented on code in PR #69477:
URL: https://github.com/apache/airflow/pull/69477#discussion_r3549268541
##########
providers/snowflake/src/airflow/providers/snowflake/operators/snowflake.py:
##########
@@ -284,7 +307,7 @@ def __init__(
self.query_ids: list[str] = []
-class SnowflakeSqlApiOperator(SQLExecuteQueryOperator):
+class SnowflakeSqlApiOperator(ResumableJobMixin, SQLExecuteQueryOperator):
Review Comment:
Good question @kacpermuda!
For a genuinely single attempt happy path (no retry at all), there is no
difference. `submit_job` always runs, which calls `self._hook.execute_query()`
exactly as before, populating `hook.query_ids` the same way.
`get_db_hook()`/`self._hook` is the same cached instance throughout & nothing
about that changes.
Although thanks to your comment, I went into digging into it further though,
I found a real gap adjacent to what you asked - on a **reconnect** or
**already-succeeded** retry (`task_state_store` already has a stored handle
from a prior attempt and this does not require an actual crash, any retry after
a first attempt got far enough qualifies), `submit_job`/`execute_query` never
runs on that attempt's fresh hook instance & the mixin skips straight to
`poll_until_complete`/`get_job_result` using the externally supplied handle.
That left `hook.query_ids` (populated by `execute_query`) at its empty default,
even though the *operator's* `self.query_ids` was correctly set. Since
`get_openlineage_database_specific_lineage` reads `self.query_ids` on the hook,
that meant a reconnect or already succeeded retry would have silently produced
no OL lineage for that attempt.
I fixed that too in 0fc8ca6097e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
