[
https://issues.apache.org/jira/browse/BEAM-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319340#comment-16319340
]
Anna Smith commented on BEAM-3311:
----------------------------------
Thanks, [~sduskis]! The approaches suggested seem a bit more complicated than
writing a new builder method, especially since we have a need to retain
windowing information.
An implementation for bulk writes using a new builder method for a PTransform
of an Iterable is suggested here:
https://github.com/andrisnoko/beam/blob/bulk-write-needs-fix-autovalue/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
If this seems like a valid approach, we can add tests and remove merge
conflicts.
> Extend BigTableIO to write Iterable of KV
> ------------------------------------------
>
> Key: BEAM-3311
> URL: https://issues.apache.org/jira/browse/BEAM-3311
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-gcp
> Affects Versions: 2.2.0
> Reporter: Anna Smith
> Assignee: Solomon Duskis
>
> The motivation is to achieve qps as advertised in BigTable in Dataflow
> streaming mode (ex: 300k qps for 30 node cluster). Currently we aren't
> seeing this as the bundle size is small in streaming mode and the requests
> are overwhelmed by AuthentiationHeader. For example, in order to achieve qps
> advertised each payload is recommended to be ~1KB but without batching each
> payload is 7KB, the majority of which is the authentication header.
> Currently BigTableIO supports DoFn<KV<ByteString, Iterable<Mutation>>,...>
> where batching is done per Bundle on flush in finishBundle. We would like to
> be able to manually batch using a DoFn<Iterable<KV<ByteString,
> Iterable<Mutation>>>,...> so we can get around the small Bundle size in
> streaming. We have seen some improvements in qps to BigTable when running
> with Dataflow using this approach.
> Initial thoughts on implementation would be to extend Write in order to have
> a BulkWrite of Iterable<KV<ByteString, Iterable<Mutation>>>.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
