[
https://issues.apache.org/jira/browse/FLINK-9794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539465#comment-16539465
]
ASF GitHub Bot commented on FLINK-9794:
---------------------------------------
GitHub user jrthe42 opened a pull request:
https://github.com/apache/flink/pull/6301
[FLINK-9794] [jdbc] JDBCOutputFormat does not consider idle connection and
multithreads synchronization
## What is the purpose of the change
This pull request fix bugs in original implementation of
`JDBCOutputFormat`, which does not consider idle connection and multithreads
synchronization .
- The Connection was established when JDBCOutputFormat is opened, and will
be used all the time. But if this connection lies idle for a long time, the
database will force close the connection, thus errors may occur.
- The flush() method is called when batchCount exceeds the threshold, but
it is also called while snapshotting state. So two threads may modify upload
and batchCount, but without synchronization.
## Brief change log
- Using a Timer to test the jdbc connection periodically and keep it alive
- Add synchronization for batch operation
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jrthe42/flink fix-jdbcoutputformat
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6301.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6301
----
commit d0c34811e3a30ebc97fd784ebf2fdd3f2085358b
Author: jrthe42 <jrthe42@...>
Date: 2018-07-11T01:50:29Z
Using a Timer to test the jdbc connection periodically and keep it alive
If jdbc connction lies idle for a long time, the database will force close
the connetion. Keep this connection valid using a timer.
commit d08938350bb3d78a324562e14d24a0892a052b5f
Author: jrthe42 <jrthe42@...>
Date: 2018-07-11T01:58:40Z
[FLINK-9794] JDBCOutputFormat does not consider idle connection and
multithreads synchronization
----
> JDBCOutputFormat does not consider idle connection and multithreads
> synchronization
> -----------------------------------------------------------------------------------
>
> Key: FLINK-9794
> URL: https://issues.apache.org/jira/browse/FLINK-9794
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Affects Versions: 1.4.0, 1.5.0
> Reporter: wangsan
> Assignee: vinoyang
> Priority: Major
> Labels: pull-request-available
>
> Current implementation of JDBCOutputFormat has two potential problems:
> 1. The Connection was established when JDBCOutputFormat is opened, and will
> be used all the time. But if this connection lies idle for a long time, the
> database will force close the connection, thus errors may occur.
> 2. The flush() method is called when batchCount exceeds the threshold, but it
> is also called while snapshotting state. So two threads may modify upload and
> batchCount, but without synchronization.
> We need fix these two problems to make JDBCOutputFormat more reliable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
