wannieman98 opened a new pull request, #76998:
URL: https://github.com/apache/airflow/pull/76998
`GCSToBigQueryOperator` and `BigQueryToGCSOperator` document
`(<project>.|<project>:)<dataset>.<table>` as the format of
`destination_project_dataset_table` / `source_project_dataset_table`, and
`execute()` resolves both spellings with `BigQueryHook.split_tablename`.
`get_openlineage_facets_on_complete` does not: it passes the raw string to
`Client.get_table`, which only understands the dotted form.
`my-project:my_dataset.my_table` is read as table `my_table` in a dataset named
`my-project:my_dataset` in the client's default project, so the lookup fails.
The OpenLineage provider catches that exception, discards the operator-level
lineage and falls back to hook-level lineage. The task itself succeeds, but for
every task that uses the colon form the COMPLETE event has:
* no `externalQuery` run facet, so the BigQuery job id is lost;
* no schema, documentation or column-lineage facets on the datasets;
* the raw wildcard path as the GCS dataset name, instead of the directory
name plus a `symlink` facet.
The same task with `my-project.my_dataset.my_table` emits all of them. The
only trace is a warning from the OpenLineage extractor (`OpenLineage method
failed to extract data from Operator with the following exception`).
Both methods now resolve the table the same way the job configuration does
and give `Client.get_table` an explicit `TableReference`, as #66962 did for
`BigQueryStreamingBufferEmptySensor`.
One side effect to review: a two-part `dataset.table` name is now looked up
in the connection's project, which is where the load or extract job reads or
writes it, instead of in `project_id` (the project the job runs in). The two
differ only when `project_id` is set to a project other than the connection's;
in that case the old lookup pointed at a table the job never touched.
The existing tests did not catch this because they mock the whole hook, so
`get_table` accepted any argument and `split_tablename` returned a fixed tuple.
The new tests run the real `split_tablename` and assert on the reference that
reaches `get_table`.
Reproduce (any Airflow 3 deployment with the OpenLineage provider enabled):
```python
GCSToBigQueryOperator(
task_id="load_colon",
bucket="my-bucket",
source_objects=["folder/file_*.csv"],
destination_project_dataset_table="my-project:my_dataset.my_table",
write_disposition="WRITE_TRUNCATE",
)
```
The load job succeeds; the task's OpenLineage COMPLETE event has no
`externalQuery` facet. Change the colon to a dot and the facet is there.
Checks run locally so far (plain `pytest` in a virtualenv against this
branch, without the `tests_common` plugin):
* The two new parametrized tests (6 cases) pass with the change and fail
without it.
* All `openlineage` tests in `test_gcs_to_bigquery.py` and
`test_bigquery_to_gcs.py`: 26 passed.
* `ruff format --check` and `ruff check` on the four changed files: clean.
Not run yet: `prek` hooks, mypy and `breeze` tests. The PR stays in draft
until they are.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Fable 5.2) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
