nbali opened a new pull request, #40390:
URL: https://github.com/apache/beam/pull/40390
Addresses #27170 (alleviates it, does not fix it).
## What
Adds `RpcQosOptions.Builder.withRampupThrottlingDisabled()` to FirestoreIO,
so a pipeline can opt out of the 500/50/5 write ramp-up.
```java
FirestoreIO.v1().write().batchWrite()
.withRpcQosOptions(RpcQosOptions.newBuilder().withRampupThrottlingDisabled().build())
.build();
```
With the option set, `RpcQosImpl.RpcWriteAttemptImpl`:
- skips the `WriteRampUp` check in `awaitSafeToProceed`, so writes no longer
sleep waiting for ramp-up budget;
- leaves the ramp-up budget out of the `min(...)` in `newFlushBuffer`, so
batch sizes come only from the latency-based `WriteBatcher` and `batchMaxCount`.
Adaptive throttling (`AdaptiveThrottler`) and retry backoff still apply, so
writes still back off when Firestore returns errors.
The default is unchanged (ramp-up enabled). `isShouldThrottleRampup()` is
included in `equals`/`hashCode`/`toString`/display data, and survives
`toBuilder()`.
## Why
#27170 describes how the ramp-up makes `batchWrite()` very slow for small
jobs. Each `RpcQos` instance (one per DoFn instance) starts with a budget of
`500 / hintMaxNumWorkers` writes per second. With the default hint of 500 that
is 1 write per second. The budget grows by 50% every 5 minutes, so one instance
stays at about 1 write per second for the first 10–15 minutes. The budget also
caps the batch size, so these are mostly 1-write requests.
The existing workaround, `withHintMaxNumWorkers(...)`, still applies a
ramp-up. This option turns it off for cases where the 500/50/5 guidance isn't
needed: for example, writes to a collection that is already serving traffic, or
documents with well-distributed IDs.
## Where it comes from
This is the Firestore equivalent of an option DatastoreIO already has.
`DatastoreV1.Write` (and `DeleteEntity`, `DeleteKey` and their `...WithSummary`
variants) expose `withRampupThrottlingDisabled()`. When set,
`DatastoreV1.Mutate.expand` doesn't add `RampupThrottlingFn`, the DoFn that
applies the same `500 / hintNumWorkers * 1.5^((minutes - 5) / 5)` budget, to
the write pipeline. The name is the same, so users of both connectors find it
in the same place. For Firestore it lives on `RpcQosOptions`, next to
`withHintMaxNumWorkers`, because that is where the Firestore ramp-up is
configured.
## Why this alleviates rather than fixes #27170
The default behavior is unchanged: a pipeline that doesn't set this option
(or the hint) still starts at about 1 write per second per instance. This PR
gives users a supported way out. A behavior change for the default, such as
growing the budget faster while Firestore reports no errors, is left for a
follow-up.
## Testing
- `RpcQosOptionsTest`: the default keeps the ramp-up enabled; the disabled
option survives `toBuilder()` and affects equality; new display-data key.
- `RpcQosTest`:
- with ramp-up disabled and `hintMaxNumWorkers = 10000`, the first batch
is sized by the batcher (500), not by the ramp-up budget (1);
- repeated 500-write requests within the same second don't sleep.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] ~Update `CHANGES.md` with noteworthy changes.~
- [ ] ~If this contribution is large, please file an Apache [Individual
Contributor License Agreement]~(https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more
tips on [how to make review process
smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit
[https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more
information about GitHub Actions CI or the [workflows
README](https://github.com/apache/beam/blob/master/.github/workflows/README.md)
for a list of workflows and how to trigger them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
