sohailshk opened a new pull request, #77003:
URL: https://github.com/apache/airflow/pull/77003
`S3Hook.parse_s3_url` split `s3://` URLs with `urlsplit`, which treats
everything after a `?` as a query string and drops it from the path. A key such
as `dir/report?.csv` was parsed as `dir/report`, so `check_for_key`, `get_key`,
`S3KeySensor` and the other methods that accept a full `s3://bucket/key` URL
looked up the wrong object. `?` is also the single-character glob in
`S3KeySensor(wildcard_match=True)`, so `s3://bucket/data/file?.csv` was
truncated to `data/file`. `#` was handled earlier with `allow_fragments=False`
(#41796); `?` was missed.
The bucket and key are now split on the first `/` after the `//`, which
keeps the key as written. Like `urlsplit`, the split still removes tabs and
newlines first, and leading slashes are still stripped (`s3://bucket//a` gives
`a`, as before). The `https://` branches were not affected.
Side effect: `convert_asset_to_openlineage` (`assets/s3.py`) passes Asset
URIs to `parse_s3_url`, so for an S3 Asset URI that has a query string the
OpenLineage dataset name now keeps the `?...` part instead of dropping it.
`transfers/sftp_to_s3.py` and `s3_to_sftp.py` use a bare `urlsplit(...).path`
and are left for a follow-up.
Tests: three new `test_parse_s3_url` cases (`?` in the key, trailing `?`,
`?` with `=`), a case that a trailing newline is still dropped, and a moto test
that `check_for_key("s3://bucket/dir/report?.csv")` finds the object. The `?`
cases and the moto test fail on main and pass here (4 failed -> 17 passed with
`-k "parse or question_mark"`); the newline case passes on both. `test_s3.py`
for hooks: 158 passed. 16 `test_unify_and_provide_bucket_name_combination`
cases error at setup in my environment (`No module named 'airflow_shared'`), on
main too.
I did not run it against real S3.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
The implementation and tests were written with AI assistance.
Generated-by: AI coding assistant following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
