Every works uses exactly 1 connection, so in order to increase the
number of connections you must indeed increase the worker parallelism.
On 2/19/2021 6:51 AM, Li Peng wrote:
Hey folks,
I'm trying to use flink to write high throughput incoming data to a
SQL db using the JDBC Connector as described here:
https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/jdbc.html>
However, after enabling this, my data consumption rate slowed down to
a crawl. After doing some digging, it seemed like the number of
connections to the db was very low, almost 1 connection per worker
(there are 9 workers). Which seemed like the culprit if Flink was
blocking and waiting for a single db connection to do all this work?
Is there a way to tell the JDBC connector to use more db connections?
Or do I need to specifically increase the parallelism of the connector
to beyond the default?
And if I need to increase the parallelism, currently my setup is
having 9 workers, 1 task slot for each (this cluster is dedicated to
one job). That means I have to increase the number of task slots
before increasing parallelism, right?
My flink version is 1.10.1 and my jdbc connection
is flink-connector-jdbc_2.11:1.11.0.
Thanks!
Li
