Re: Confusions About JDBCOutputFormat

2018-07-11 Thread Hequn Cheng
Cool. I will take a look. Thanks. On Wed, Jul 11, 2018 at 7:08 PM, wangsan wrote: > Well, I see. If the connection is established when writing data into DB, > we need to cache received rows since last write. > > IMO, maybe we do not need to open connections repeatedly or introduce > connection

Re: Confusions About JDBCOutputFormat

2018-07-11 Thread wangsan
Well, I see. If the connection is established when writing data into DB, we need to cache received rows since last write. IMO, maybe we do not need to open connections repeatedly or introduce connection pools. Test and refresh the connection periodically can simply solve this problem. I’ve

Re: Confusions About JDBCOutputFormat

2018-07-11 Thread Hequn Cheng
Hi wangsan, What I mean is establishing a connection each time write data into JDBC, i.e. establish a connection in flush() function. I think this will make sure the connection is ok. What do you think? On Wed, Jul 11, 2018 at 12:12 AM, wangsan wrote: > Hi Hequn, > > Establishing a connection

Re: Confusions About JDBCOutputFormat

2018-07-10 Thread wangsan
Hi Hequn, Establishing a connection for each batch write may also have idle connection problem, since we are not sure when the connection will be closed. We call flush() method when a batch is finished or snapshot state, but what if the snapshot is not enabled and the batch size not reached

Re: Confusions About JDBCOutputFormat

2018-07-10 Thread Hequn Cheng
Hi wangsan, I agree with you. It would be kind of you to open a jira to check the problem. For the first problem, I think we need to establish connection each time execute batch write. And, it is better to get the connection from a connection pool. For the second problem, to avoid multithread