[
https://issues.apache.org/jira/browse/FLINK-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980801#comment-16980801
]
hailong wang commented on FLINK-14902:
--------------------------------------
Thanks [~jark], [~lzljs3620320]. Here's what I think:
1、We should has a connection pool which can manage the connection. For creating
connections frequently will reduce the performance. At the same time, unlimited
connection is dangerous for DB.
As for the connection pool, I think we can use HikariCP which has good
performance and also used in hive. The connection size can be configured by
users.
2、As for how to launch multi-thread. We can have a FixedThreadPool which
thread is fix and workpool is infinite, For we can't create thread unlimitedly.
Inner thread, getConnection and lookup DB.
3、When datas come faster than DB query. When time exceed the AsynOperater
timeout time, AsynOperater will throw error by default. When time within in
it, We can use batch query which sql likes ' where (condition1) or (condition2)
……' which will reduce IO operation。But, the resultSet are the collection of
condition1, condition2……，So we should split it in memory.
4、We can use guava cache like jdbclookupfunction to have a memory cache.
[~jark], [~lzljs3620320] Do you think that's reasonable? Thank you for your any
advice.
Best.
Wang
> JDBCTableSource support AsyncLookupFunction
> -------------------------------------------
>
> Key: FLINK-14902
> URL: https://issues.apache.org/jira/browse/FLINK-14902
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
> Affects Versions: 1.9.0
> Reporter: hailong wang
> Assignee: hailong wang
> Priority: Major
> Fix For: 1.10.0
>
>
> JDBCTableSource support AsyncLookupFunction
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
