Dear all,
It happened again on Friday morning:
You can see a baseline in the connection amount from the 16th to the 18th.
Looking at the pg_stat_activity, all connections are used, even when the
pipeline is not used at 100 % (my use case is processing data from a
platform which is not used a
My mistake - using @Teardown in this way is a good approach. It may not be
executed sometimes, but like Reuven says it means the process died.
Kenn
On Thu, Jan 17, 2019 at 9:31 AM Jean-Baptiste Onofré
wrote:
> Hi,
>
> I don't think we have connection leak in normal behavior.
>
> The actual SQL
Hi,
I don't think we have connection leak in normal behavior.
The actual SQL statement is executed in @FinishBundle, where the
connection is closed.
The processElement adds record to process.
Does it mean that an Exception occurs in the batch addition ?
Regards
JB
On 17/01/2019 12:41, Alexey
But I think it will be called unless the process crashes though.
On Thu, Jan 17, 2019, 7:05 AM Ismaël Mejía Couldn't this be related also to the fact that @Teardown is
> best-effort in Dataflow?
>
> On Thu, Jan 17, 2019 at 12:41 PM Alexey Romanenko
> wrote:
> >
> > Kenn,
> >
> > I’m not sure
Couldn't this be related also to the fact that @Teardown is
best-effort in Dataflow?
On Thu, Jan 17, 2019 at 12:41 PM Alexey Romanenko
wrote:
>
> Kenn,
>
> I’m not sure that we have a connection leak in JdbcIO since new connection is
> being obtained from an instance of javax.sql.DataSource
Kenn,
I’m not sure that we have a connection leak in JdbcIO since new connection is
being obtained from an instance of javax.sql.DataSource (created in @Setup) and
which is org.apache.commons.dbcp2.BasicDataSource by default. BasicDataSource
uses connection pool and closes all idle connections
Hi Kenneth,
Thank you for your reply. I find out that the leak was coming somehow
from my individual DoFns. I replaced all the connections by a connection
pooling and I haven't seen connection leaks since. I will keep
monitoring the pipeline state and if I see new leaks, I would
investigate
Hi Jonathan,
JdbcIO.write() just invokes this DoFn:
https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L765
It establishes a connection in @StartBundle and then in @FinishBundle it
commits a batch and closes the connection. If an
Hello !
My question is maybe mainly GCP-oriented, so I apologize if it is not
fully related to the Beam community.
We have a streaming pipeline running on Dataflow which writes data to a
PostgreSQL instance hosted on Cloud SQL. This database is suffering from
connection leak spikes on a
