Hi Yuan,
Thanks for raising this discussion! I believe this will be very helpful for
lookup table developers, and standardizing metrics would be essential for
users to tuning their systems.
Here’s some thoughts in my mind:
1. It looks like “identifier” and “type” are options of cache instead of
metrics. I think they are finalized once the cache is created so maybe it’s not
quite helpful to report them to the metric system.
2. Some metrics can be aggregated simply in metric systems, like loadCount =
loadSuccessCount + loadExceptionCount, so maybe we can just keep fundamental
metrics (like loadSuccessCount and loadExceptionCount) to avoid redundancy.
3. About the interface of CacheMetricGroup I think it would be easier for cache
implementers to use if we expose wrapped function instead of let users provide
gauges directly. For example:
public interface CacheMetricGroup extends MetricGroup {
// Mark a cache hit
public void markCacheHit();
// Mark a cache miss
public void recordCacheMiss();
...
}
You can check SourceReaderMetricGroup[1] and its implementation[2] as a
reference.
Hope these would be helpful!
Best regards,
Qingsheng Ren
[1]
https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/SourceReaderMetricGroup.java
[2]
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/InternalSourceReaderMetricGroup.java
> On Mar 7, 2022, at 16:00, [email protected] wrote:
>
> Hi devs,
>
>
> I would like to propose a discussion thread about abstraction of Cache
> LookupFunction with metrics for cache in connectors to make cache out of box
> for connector developers. There are multiple LookupFunction implementations
> in individual connectors [1][2][3][4] so far.
> At the same time, users can monitor cache in LookupFunction by adding uniform
> cache metrics to optimize tasks or troubleshoot.
>
>
> I have posted an issue about this, see
> <https://issues.apache.org/jira/browse/FLINK-25409>, and made a brief design
> <https://docs.google.com/document/d/1L2eo7VABZBdRxoRP_wPvVwuvTZOV9qrN9gEQxjhSJOc/edit?usp=sharing>.
>
>
> Looking forward to your feedback, thanks.
>
>
> Best regards,
> Yuan
>
>
>
>
> [1]
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcRowDataLookupFunction.java
> [2]
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/FileSystemLookupFunction.java
> [3]
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/source/HBaseRowDataLookupFunction.java
> [4]
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hbase-2.2/src/main/java/org/apache/flink/connector/hbase2/source/HBaseRowDataAsyncLookupFunction.java
