[jira] [Commented] (SPARK-23714) Add metrics for cached KafkaConsumer
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585587#comment-16585587 ] Jungtaek Lim commented on SPARK-23714: -- [~yuzhih...@gmail.com] Maybe we can just apply Apache Commons Pool (SPARK-25251) and leverage metrics. [PR|https://github.com/apache/spark/pull/22138] for SPARK-25251 is also ready to review. > Add metrics for cached KafkaConsumer > > > Key: SPARK-23714 > URL: https://issues.apache.org/jira/browse/SPARK-23714 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Ted Yu >Priority: Major > > SPARK-23623 added KafkaDataConsumer to avoid concurrent use of cached > KafkaConsumer. > This JIRA is to add metrics for measuring the operations of the cache so that > users can gain insight into the caching solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23714) Add metrics for cached KafkaConsumer
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416138#comment-16416138 ] Ted Yu commented on SPARK-23714: [~tdas]: What do you think ? Thanks > Add metrics for cached KafkaConsumer > > > Key: SPARK-23714 > URL: https://issues.apache.org/jira/browse/SPARK-23714 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Ted Yu >Priority: Major > > SPARK-23623 added KafkaDataConsumer to avoid concurrent use of cached > KafkaConsumer. > This JIRA is to add metrics for measuring the operations of the cache so that > users can gain insight into the caching solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23714) Add metrics for cached KafkaConsumer
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403792#comment-16403792 ] Ted Yu commented on SPARK-23714: Ryan reminded me of the possibility of using the existing metrics system. I think that implies passing SparkEnv to KafkaDataConsumer > Add metrics for cached KafkaConsumer > > > Key: SPARK-23714 > URL: https://issues.apache.org/jira/browse/SPARK-23714 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Ted Yu >Priority: Major > > SPARK-23623 added KafkaDataConsumer to avoid concurrent use of cached > KafkaConsumer. > This JIRA is to add metrics for measuring the operations of the cache so that > users can gain insight into the caching solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23714) Add metrics for cached KafkaConsumer
[ https://issues.apache.org/jira/browse/SPARK-23714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403112#comment-16403112 ] Ted Yu commented on SPARK-23714: [~tdas] made the following suggestion: apache commons pool does expose metrics on idle/active counts. See https://commons.apache.org/proper/commons-pool/apidocs/org/apache/commons/pool2/impl/BaseGenericObjectPool.html > Add metrics for cached KafkaConsumer > > > Key: SPARK-23714 > URL: https://issues.apache.org/jira/browse/SPARK-23714 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Ted Yu >Priority: Major > > SPARK-23623 added KafkaDataConsumer to avoid concurrent use of cached > KafkaConsumer. > This JIRA is to add metrics for measuring the operations of the cache so that > users can gain insight into the caching solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
