[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517280#comment-17517280 ] Martijn Visser commented on FLINK-25409: [~straw] Sorry for the late reply. I would reply to the latest email on the Dev mailing list, where you summarize the results of the discussion and mention that if there are no other remarks in a couple of days, you will open a PR to implement it. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512210#comment-17512210 ] Yuan Zhu commented on FLINK-25409: -- Hi, [~martijnvisser], [~jingzhang], [~wenlong.lwl]. I have proposed a discussion in dev mail, but there are few replies. Here is the discussion: [https://lists.apache.org/thread/kkoh569f5vrvw7spjtv4kxm98w7835sw|http://example.com/] How can I proceed this topic? Or I just submit a pr directly? > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493621#comment-17493621 ] Yuan Zhu commented on FLINK-25409: -- Hi, all. Sorry for my late reply. As discussion before, I make a brief design: [https://docs.google.com/document/d/1L2eo7VABZBdRxoRP_wPvVwuvTZOV9qrN9gEQxjhSJOc/edit?usp=sharing] > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464326#comment-17464326 ] Jing Zhang commented on FLINK-25409: [~straw]+1 on your proposal to introduce a common abstract class for those who has cache implementations. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463882#comment-17463882 ] Martijn Visser commented on FLINK-25409: [~wenlong.lwl] Yes, agreed, we're on the same page :) > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463861#comment-17463861 ] Wenlong Lyu commented on FLINK-25409: - hi, [~MartijnVisser], you are right about the sharing. However there is not lookup abstraction for DataStream API/connector, we need to use FlatMapFunction or AsyncFunction directly. LookupSource would be translate to a TableFunction instead. If you means that we need to introduce such support, It is another story IMO. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463847#comment-17463847 ] Martijn Visser commented on FLINK-25409: [~wenlong.lwl] I do think it is relevant. A connector should be available for both DataStream and Table/SQL users and share as much code as possible. There are of course specific implementation details for either DataStream and Table/SQL, but those should hopefully just be a thin layer that's connecting interfaces together. That also means that we should have a generic implementation in the Source API to enable lookups for connectors for which that's relevant, such as JDBC or HBase etc. Metrics could then come out of the box instead of a connector by connector implementation. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463826#comment-17463826 ] Wenlong Lyu commented on FLINK-25409: - hi, [~MartijnVisser] this ticket may be not relevant to the new source/sink API and FLIP-33, lookup source is a SQL concept. +1 for common LookupFunctionWithCache. It can help unify the caching config of different connectors and reduce duplicate work on connector development. In addition, we can add one more metric to measure lookup latency per record. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463700#comment-17463700 ] Martijn Visser commented on FLINK-25409: The JDBC connector still needs to be ported to the new Source API (FLIP-27 https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface) and Sink API (FLIP-143 https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API). Those interfaces also have implications to the metrics, as explained on FLIP-33 https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics and I can imagine refactoring the connector to these interfaces also has implications on this ticket. WDYT? > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463553#comment-17463553 ] Yuan Zhu commented on FLINK-25409: -- In 1.14.2, neglecting some private LookupFunctions outside of connectors: ||Cache Type||Legacy||Not Legacy|| |Guava cache|JdbcLookupFunction|JdbcRowDataLookupFunction| |Java map|CsvLookupFunction| | |Java map| |FileSystemLookupFunction| |Guava cache | |HBaseRowDataLookupFunction| They all have cache inside. We can extract a common abstract class to init cache metric if enabled and some useful metrics. The implementation logic can be left by child class. If some LookupSource without cache in connectors external repo won't to use metric indeed, they can just implement TableFunction like before or implement the newly abstract LookupFunction without init metric implementation. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463541#comment-17463541 ] Jing Zhang commented on FLINK-25409: [~straw]Thanks for reporting this issue. Could you list the look up table source which use cache, and only add cache metrics in those connectors? > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
