damccorm opened a new pull request, #41268: URL: https://github.com/apache/beam/pull/41268
Note: this PR was auto-generated by tooling. Before moving it out of draft, the author has done a full review of the PR, specifically evaluating it on the following criteria: - The PR is an appropriate fix for the issue (e.g. we're not changing a test to avoid a bug) - The PR is a simple fix to the issue (e.g. we're not adding infra when we could just modify a test) - The PR is not covering for other problems (e.g. if we're bumping timeouts, it is not because we're covering for a performance problem) Full details of the methodology of generation and review can be found here - [Apache Beam Python SDK Performance PRs Methodology & Branches Impact Ranking](https://docs.google.com/document/d/1Edrmzp-qFHYs2j7dg9uFyV_jEMgq4QKt_JQDuuk-lmI/edit?usp=drivesdk) If you have concerns with the PR or with the underlying methodology used to generate this, please comment below - they will be addressed by the author not the tooling :) **Summary of changes:** Optimizes `FileSystems.get_filesystem` and `FileSystems.get_scheme` in `sdks/python/apache_beam/io/filesystems.py`: 1. **Cache `scheme -> [FileSystem]` lookup table in `FileSystems.get_filesystem`:** - Previously, every `FileSystems.get_filesystem(path)` call traversed `FileSystem.get_all_subclasses()` and built `[fs for fs in FileSystem.get_all_subclasses() if fs.scheme() == path_scheme]`. - Caches `_systems_by_scheme` invalidated only if `FileSystem.__subclasses__()` changes. 2. **Fast-path local paths (`"://" not in path`) and use `match_result.group("scheme")` instead of `match_result.groupdict()["scheme"]` in `FileSystems.get_scheme`.** **Dataflow benchmarks showing improvement:** - Before the change: [`2026-11-12_12_17_53-9264017659679288896`](https://console.cloud.google.com/dataflow/jobs/us-central1/2026-11-12_12_17_53-9264017659679288896?project=apache-beam-testing) - After the change: [`2026-11-12_12_17_52-9971694672267197161`](https://console.cloud.google.com/dataflow/jobs/us-central1/2026-11-12_12_17_52-9971694672267197161?project=apache-beam-testing) - Summary of perf improvement: In a single-worker (`n1-standard-8`, `--autoscaling_algorithm=NONE --num_workers=1 --max_num_workers=1`) Cloud Dataflow pipeline running `30,000,000` GCS URIs ยท `ParDo(FileSystems.get_filesystem & get_scheme)`, built-in transform stage execution time improved from **`594,683 ms` -> `350,531 ms` (`1.70x` faster, `-41.1%`)** (total single-worker job vCPU time: `1595s` -> `1322s`). **Additional benchmarks** - **Strict Single-PR Isolated Microbenchmark (`upstream/master` vs `upstream/master` + this PR alone):** [`bench_46_filesystems_scheme_cache` (`c2feb961eef`)](https://github.com/damccorm/beam/blob/c2feb961eef2e7d2410825aab9537c62a95e6790/sdks/python/apache_beam/testing/benchmarks/python_sdk_perf/microbenchmarks.py#L643-L649) (on branch [`perf/py-benchmarks-suite`](https://github.com/damccorm/beam/tree/perf/py-benchmarks-suite)) โ 200,000 `FileSystems.get_filesystem` lookups on `gs://` and local paths improved from **`1,011.64 ms` -> `351.00 ms` (`2.88x` faster, `-65.3%`, `-660.64 ms`)** in strict isolation. - **Per-Change Attribution (Most to Least Impact, 200,000 pairs of calls in strict isolation):** 1. *Most impact (`96.1%` of savings, `-635.14 ms`):* Caching `_systems_by_scheme` in `FileSystems.get_filesystem` โ **`1,011.64 ms` -> `351.00 ms` (`2.88x` faster)**. 2. *Least impact (`3.9%` of savings, `-25.50 ms`):* Fast-pathing `"://" not in path` and `match_result.group("scheme")` in `FileSystems.get_scheme` โ **`140.26 ms` -> `114.76 ms` (`1.22x` faster, `-18.2%`)**. - **On-Worker Cloud Dataflow `n2-standard-4` Benchmark (`p28_ms:24_filesystems_scheme_cache_200k` & `subbench_ms:06_io_filesystems_and_avro_ms`):** On a Cloud Dataflow `n2-standard-4` worker, 200,000 `FileSystems.get_filesystem` lookups improved from **`1,828 ms` -> `574 ms` (`3.18x` faster, `-68.6%`)**, and `subbench_ms:06` improved from **`661 ms` -> `301 ms` (`2.20x` faster)**. - **End-to-End Cloud Dataflow Benchmark (`io_coders_transforms`):** Included in [`run_io_coders_transforms_pipeline` (`c2feb961eef`)](https://github.com/damccorm/beam/blob/c2feb961eef2e7d2410825aab9537c62a95e6790/sdks/python/apache_beam/testing/benchmarks/python_sdk_perf/dataflow_e2e_benchmarks.py#L305-L335) (3,000,000 records on `n2-standard-4`), which improved worker stage execution time by **1.48x (`-32.4%`, `49.2s` -> `33.3s`)**. ### Testing - Verified with `pytest sdks/python/apache_beam/io/filesystems_test.py`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
