[
https://issues.apache.org/jira/browse/BEAM-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333422#comment-16333422
]
Knut Olav Loite commented on BEAM-3506:
---------------------------------------
Example project for the feature can be found here:
[https://github.com/olavloite/spanner-beam-example]
The example class WordCount writes the rows using the original write feature of
JdbcIO and is very slow on Google Cloud Spanner. The example class
WordCountWriteIterable uses the new feature and is significantly faster. The
example groups the rows by the first letter of the word and writes each of
these as one batch.
> JdbcIO: Support writing iterables (i.e. collections) of rows instead of only
> single rows
> ----------------------------------------------------------------------------------------
>
> Key: BEAM-3506
> URL: https://issues.apache.org/jira/browse/BEAM-3506
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-extensions
> Affects Versions: 2.3.0
> Reporter: Knut Olav Loite
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Labels: JdbcIO, jdbc
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> The current JdbcIO write interface expects a PCollection<T> where T is the
> row to be written. Each instance of T is then added to a batch and written to
> the database. The user has little control over how many rows will be added to
> one batch. If JdbcIO would also support writing a PCollection<Iterable<T>>
> the user would have more control over the number of rows in one batch.
> Especially when writing to cloud databases, such as Google Cloud Spanner, the
> batching of multiple rows together is important for performance.
> I already have a solution locally and I will submit a pull request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
