nbali commented on issue #27170: URL: https://github.com/apache/beam/issues/27170#issuecomment-5961038697
Two PRs that alleviate this (they don't fix it): both copy existing DatastoreIO behavior into FirestoreIO. - #40390 adds `RpcQosOptions.Builder.withRampupThrottlingDisabled()`, mirroring `DatastoreV1.Write.withRampupThrottlingDisabled()`. With it, writes no longer wait for the 500/50/5 ramp-up budget and batches are no longer capped by it. Error-based throttling and retry backoff still apply. - #40391 makes the ramp-up grow from one pipeline-wide start time (a singleton side input), mirroring `DatastoreV1.Mutate` / `RampupThrottlingFn`, instead of from each DoFn instance's first write. DoFn instances created later (autoscaling, retried bundles) no longer start over at 1 write/s. It's skipped when `--updateCompatibilityVersion` is below 2.78.0. Neither changes the default: without the opt-out or `withHintMaxNumWorkers`, a small job still starts at `500 / 500 = 1` write/s and grows 50% every 5 minutes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
