anishmehta24 opened a new pull request, #40165: URL: https://github.com/apache/beam/pull/40165
Re-lands #38749 (reverted in #38901) so the Python SDK honors the `disableCounterMetrics`, `disableStringSetMetrics` and `disableBoundedTrieMetrics` experiments the way the Java SDK already does, letting high-throughput jobs stop reporting metric kinds that put pressure on the metrics backend. **Why the first attempt broke, and what is different here.** #38749 replaced `DelegatingCounter.inc` (and the `add`s on `DelegatingStringSet` / `DelegatingBoundedTrie`) — which are `MetricUpdater` *instance attributes* with signature `__call__(value=_DEFAULT)` — with plain methods `inc(self, n=1)`. That broke callers using `counter.inc(value=...)` ([thread](https://github.com/apache/beam/pull/38749#discussion_r3369938771)) and anything relying on those attributes being `MetricUpdater` instances (pickling via `MetricUpdater.__reduce__`, `isinstance` checks). This PR leaves the metric objects untouched: - `MetricUpdater.__call__` (`metrics/execution.py`) drops the update when its cell type is in a module-level `_DISABLED_CELL_TYPES` set. With no experiment set this is a truthiness check on an empty set, so the hot path is unchanged; with one set it is a single set lookup. Declared in `execution.pxd` for the Cython build. - `MetricsFlag` (`metrics/metric.py`) mirrors the Java `Metrics.MetricsFlag`: `set_default_pipeline_options(options)` maps the three experiments onto `CounterCell` / `StringSetCell` / `BoundedTrieCell`, logs which kinds are disabled, and — as in Java — the first call wins so user code on a worker cannot change what the harness started with. `reset()` exists for tests. - Hooked from `Pipeline.__init__` (so the DirectRunner honors it) and `sdk_worker_main.create_harness` (portable workers), next to the existing `FileSystems.set_options` calls, same as #38749. Because the gate is evaluated per call rather than captured at construction, it also covers metric objects created before the options were known (module-level counters, DoFns constructed before `Pipeline(...)`) and objects unpickled on the worker. **Tests** (`metric_test.py::MetricsFlagTest`, 9 cases): experiments → flags for each kind and combined; first-call-wins; a regression test that `inc()`, `inc(4)`, `inc(value=5)`, `dec()`, `add(value=...)` all still work and `counter.inc` is still a `MetricUpdater`; each disabled kind is a no-op for objects created before and after the flag while other kinds keep reporting; process-wide counters; unpickled metrics; and an end-to-end `TestPipeline` with `--experiments=disableCounterMetrics` asserting the counter is absent from results while the distribution is intact. Without the implementation the suite fails. Run locally on Windows (pure-Python build): `pytest apache_beam/metrics/ pipeline_test.py runners/worker/sdk_worker_main_test.py runners/worker/statesampler_test.py` → 109 passed (one pre-existing `pipeline_test` failure from a dill version mismatch in my env), `ruff check`, `yapf`, `pylint` clean on touched files, `mypy` clean for `metrics/execution.py` and `metrics/metric.py`. Prepared with AI assistance (Claude Code) and human-verified. fixes #38746 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead. - [x] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
