[
https://issues.apache.org/jira/browse/BEAM-12260?focusedWorklogId=612175&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612175
]
ASF GitHub Bot logged work on BEAM-12260:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Jun/21 21:29
Start Date: 18/Jun/21 21:29
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#14713:
URL: https://github.com/apache/beam/pull/14713#discussion_r654669657
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
##########
@@ -225,7 +233,7 @@
* number of entities per request may be lower when we flush for the end of
a bundle or if we hit
* {@link DatastoreV1.DATASTORE_BATCH_UPDATE_BYTES_LIMIT}.
*/
- @VisibleForTesting static final int DATASTORE_BATCH_UPDATE_ENTITIES_MIN = 10;
+ @VisibleForTesting static final int DATASTORE_BATCH_UPDATE_ENTITIES_MIN = 5;
Review comment:
Ditto. Note that this could potentially double the number of RPCs to
Datastore.
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
##########
@@ -212,7 +220,7 @@
* <p>Testing has found that a batch of 200 entities will generally finish
within the timeout even
* in adverse conditions.
*/
- @VisibleForTesting static final int DATASTORE_BATCH_UPDATE_ENTITIES_START =
200;
+ @VisibleForTesting static final int DATASTORE_BATCH_UPDATE_ENTITIES_START =
50;
Review comment:
Why was this updated ? If 50 is a better number could you also update
the comment please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 612175)
Time Spent: 1h 50m (was: 1h 40m)
> Java - Backport FirestoreIO connector's ramp-up to DatastoreIO connector
> ------------------------------------------------------------------------
>
> Key: BEAM-12260
> URL: https://issues.apache.org/jira/browse/BEAM-12260
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Daniel Thevessen
> Assignee: Daniel Thevessen
> Priority: P2
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The FirestoreIO connector
> ([BEAM-8376|https://issues.apache.org/jira/browse/BEAM-8376]) for the Java
> SDK is currently out for review.
> Some of the features for the Firestore connector are new relative to the
> Datastore connector, so the Firestore team would like to add them there as
> well. Most notably the Firestore connector has a gradual ramp-up feature that
> slowly increases throughput in line with Datastore/Firestore best practices.
> Some minor adjustments to constants for batch sizing and backoff may also be
> worth looking at.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
