[
https://issues.apache.org/jira/browse/BEAM-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16838451#comment-16838451
]
Brachi Packter commented on BEAM-7230:
--------------------------------------
You close the data source on @TearDown
{code:java}
@Teardown
public void teardown() throws Exception {
connection.close();
if (dataSource instanceof AutoCloseable) {
((AutoCloseable) dataSource).close();
}
}
{code}
So. If I want one Data source for any worker, shared between ant DoFn threads,
you must not close it up.
my code:
{code:java}
.apply(JdbcIO.<ValueInSingleWindow<Row>>write()
.withDataSourceProviderFn((SerializableFunction<Void, DataSource>) input1 -> {
ConnectionPoolProvider.init(dataBaseName, userName, project, region,
instanceName); return ConnectionPoolProvider.getInstance().getDataSource(); })
.withStatement(sqlInsertCommand)
.withPreparedStatementSetter((JdbcIO.PreparedStatementSetter<ValueInSingleWindow<Row>>)
this::prepareStatement ));
{code}
This leads to this exception:
{code:java}
Caused by: java.sql.SQLException: HikariDataSource HikariDataSource
(HikariPool-1) has been closed.
com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:96)
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteVoid$WriteFn.startBundle(JdbcIO.java:990)
{code}
> Using JdbcIO creates huge amount of connections
> -----------------------------------------------
>
> Key: BEAM-7230
> URL: https://issues.apache.org/jira/browse/BEAM-7230
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.11.0
> Reporter: Brachi Packter
> Assignee: Ismaël Mejía
> Priority: Major
>
> I want to write form DataFlow to GCP cloud SQL, I'm using connection pool,
> and still I see huge amount of connections in GCP SQL (4k while I set
> connection pool to 300), and most of them in sleep.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
