[
https://issues.apache.org/jira/browse/BEAM-12272?focusedWorklogId=603321&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603321
]
ASF GitHub Bot logged work on BEAM-12272:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 27/May/21 23:13
Start Date: 27/May/21 23:13
Worklog Time Spent: 10m
Work Description: danthev commented on a change in pull request #14723:
URL: https://github.com/apache/beam/pull/14723#discussion_r641030864
##########
File path: sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
##########
@@ -276,15 +277,33 @@ class _Mutate(PTransform):
Only idempotent Datastore mutation operations (upsert and delete) are
supported, as the commits are retried when failures occur.
"""
- def __init__(self, mutate_fn):
+
+ # Default hint for the expected number of workers in the ramp-up throttling
+ # step for write or delete operations.
+ _DEFAULT_HINT_NUM_WORKERS = 500
Review comment:
Bumping this because the edit to my previous comment is easy to miss.
The gist is that reporting throttling-msecs does not break autoscaling as far
as I can tell.
I did most of my testing with the Java implementation. I just noticed that
not only did I miss adding the throttling counter in the Python implementation,
but it appears Datastore never had one for the adaptive throttler either. Does
the metric work the same in Python, can I just add a counter called
`throttling-msecs`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 603321)
Time Spent: 3h 10m (was: 3h)
> Python - Backport FirestoreIO connector's ramp-up to DatastoreIO connector
> --------------------------------------------------------------------------
>
> Key: BEAM-12272
> URL: https://issues.apache.org/jira/browse/BEAM-12272
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
> Reporter: Daniel Thevessen
> Assignee: Daniel Thevessen
> Priority: P2
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> The FirestoreIO connector (BEAM-8376) for the Java SDK is currently out for
> review.
> Some of the features for the Firestore connector are new relative to the
> Datastore connector, so the Firestore team would like to add them there as
> well. Most notably the Firestore connector has a gradual ramp-up feature that
> slowly increases throughput in line with Datastore/Firestore best practices.
> Some minor adjustments to constants for batch sizing and backoff may also be
> worth looking at.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
