[
https://issues.apache.org/jira/browse/BEAM-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333409#comment-16333409
]
Knut Olav Loite commented on BEAM-3506:
---------------------------------------
The current batch size option (actually, currently a fixed value, but that
would also be easy to fix) would in this case rather be used as a maximum value
for how many writes to put into one batch. The basic idea is that you can have
different batch sizes per collection, but also that the user has more control
over the batch sizes by creating the collections based on the data. I'm working
on a simple example project that I will post as well to illustrate the working
of it.
I'm basically trying to fix the problem that is also reported in this
Stackoverflow question:
[https://stackoverflow.com/questions/47714879/google-dataflow-apache-beam-jdbcio-bulk-insert-into-mysql-database]
> JdbcIO: Support writing iterables (i.e. collections) of rows instead of only
> single rows
> ----------------------------------------------------------------------------------------
>
> Key: BEAM-3506
> URL: https://issues.apache.org/jira/browse/BEAM-3506
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-extensions
> Affects Versions: 2.3.0
> Reporter: Knut Olav Loite
> Assignee: Jean-Baptiste Onofré
> Priority: Minor
> Labels: JdbcIO, jdbc
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> The current JdbcIO write interface expects a PCollection<T> where T is the
> row to be written. Each instance of T is then added to a batch and written to
> the database. The user has little control over how many rows will be added to
> one batch. If JdbcIO would also support writing a PCollection<Iterable<T>>
> the user would have more control over the number of rows in one batch.
> Especially when writing to cloud databases, such as Google Cloud Spanner, the
> batching of multiple rows together is important for performance.
> I already have a solution locally and I will submit a pull request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
