abhijeets25012-tech opened a new pull request, #76971: URL: https://github.com/apache/airflow/pull/76971
Fixes #76880. `get_files_to_move()` with `replace=False` compared two different kinds of name. `files_to_move` holds names relative to `s3_prefix` (`a.txt`), while `WasbHook.get_blobs_list_recursive()` returns full blob names (`out/a.txt`). The subtraction never matched, so existing destination blobs went into the transfer list again. On the default `upload_blob(overwrite=False)` that ends in a `ResourceExistsError`, which is what the reporter hit against Azurite. The filter now compares the blob name `move_file()` is about to write, built through `_create_key()`, against what the container holds. That covers the documented behaviour for `blob_prefix`, and it also tightens the `blob_name` path, which used to compare only the last path segment and could skip a file because a different blob happened to share its basename. The old tests passed because they mocked the listing with relative names, which no real hook returns. Those fixtures now use full blob names, and two tests cover the reported case: one on `get_files_to_move()` and one on `execute()` that asserts `load_file` is not called. closes: #76880 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Qoder following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. --- ##### Testing - `uv run --project providers/microsoft/azure pytest providers/microsoft/azure/tests/unit/microsoft/azure/transfers/test_s3_to_wasb.py -q` gives 29 passed. - I reverted only the operator change and re-ran the affected tests to make sure they can fail: 4 failed, including both new ones. - `ruff check` and `ruff format --check` are clean on both files, and full `pre-commit` passed on the commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
