goanpeca opened a new pull request, #69190:
URL: https://github.com/apache/airflow/pull/69190
Add a docs recipe and a system-test example DAG showing how to reuse the
existing Amazon provider S3 integration against any S3-compatible object
storage, using Backblaze B2 as a worked example.
The Amazon provider already talks to S3-compatible endpoints: pointing an
`aws` connection at a custom `endpoint_url` makes the S3 task handler and
`ObjectStoragePath("s3://...")` write to that endpoint instead of AWS S3. That
capability is not documented from the "bring your own S3-compatible bucket"
angle, so users rediscover the connection extras (`endpoint_url`,
`region_name`, path-style addressing) by trial and error. This change writes
that path down, with Backblaze B2 as one concrete provider so the steps are
copy-pasteable.
The change is intentionally small and additive: it adds documentation and
one example, and touches no provider or core code.
What is included:
- `providers/amazon/docs/logging/b2-remote-logging.rst`: a recipe that maps
the S3-compatible credentials onto the AWS connection fields, configures
`[logging]` remote logging through that connection, verifies the logs land in
the bucket, and reuses the same connection for `ObjectStoragePath` DAG data. It
cross-references the existing `write-logs-amazon-s3` handler doc rather than
restating it, and it is picked up automatically by the globbed toctree in
`providers/amazon/docs/logging/index.rst` (no manual index edit needed).
-
`providers/amazon/tests/system/amazon/aws/example_backblaze_b2_object_storage.py`:
an input to transform to output example DAG built on `ObjectStoragePath`,
sitting alongside the other `example_s3*.py` system tests and wired through the
standard `get_test_run(dag)` harness. Bucket and connection id are read from
the environment so the example carries no secrets.
Scope notes for reviewers:
- No new operator, hook, bundle, or `provider.yaml` entry: this documents
existing behavior, so there is nothing new for the registry to pick up.
- The framing is generic S3-compatible object storage. AWS S3 is the
baseline; Backblaze B2 is used only as one concrete endpoint to make the steps
runnable.
- No newsfragment: per the contributor docs, `providers/` changelogs are
regenerated from `git log` by the release managers and do not consume per-PR
newsfragments.
Testing:
- Built the provider docs locally and confirmed the new page renders and
appears under the Amazon provider "Logging for Tasks" section, with the
cross-reference to the S3 task handler resolving.
- The example DAG is a system test (network access to a real bucket required
to execute against live B2) and follows the existing `example_s3*.py`
structure; the module imports and parses cleanly and exposes `test_run` via
`get_test_run(dag)`.
- Ran the repo static checks (ruff / ruff-format) on the new Python file.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Code, Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
