[
https://issues.apache.org/jira/browse/KAFKA-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu updated KAFKA-8258:
--------------------------------
Description:
We noticed that the Spark's Kafka connector outputs a lot of following verbose
logs:
{code:java}
19/04/18 04:31:06 INFO Fetcher: [Consumer clientId=consumer-2, groupId=...]
Resetting offset for partition ... to offset ....
{code}
It comes from
[https://github.com/hachikuji/kafka/blob/76c796ca128c3c97231f3ebda994a07bb06b26aa/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L567]
This log was added in [https://github.com/apache/kafka/pull/4557]
In Spark, we use `seekToEnd` to discover latest offsets of a topic. If there
are thousands of partitions in this topic, it will output thousands of INFO
logs every call
Is it intentional? If not, can we change it to DEBUG?
was:
We noticed that the Spark's Kafka connector outputs a lot of following verbose
logs:
{code}
19/04/18 04:31:06 INFO Fetcher: [Consumer clientId=consumer-2, groupId=...]
Resetting offset for partition ... to offset ....
{code}
It comes from
https://github.com/hachikuji/kafka/blob/76c796ca128c3c97231f3ebda994a07bb06b26aa/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L567
This log was added in https://github.com/apache/kafka/pull/4557
In Spark, we use `seekToEnd` to discover latest offsets of a topic. If there
are thousands of partitions in this topic, it will output thousands of INFO
logs.
Is it intentional? If not, can we change it to DEBUG?
> Verbose logs in org.apache.kafka.clients.consumer.internals.Fetcher
> -------------------------------------------------------------------
>
> Key: KAFKA-8258
> URL: https://issues.apache.org/jira/browse/KAFKA-8258
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 1.1.0
> Reporter: Shixiong Zhu
> Priority: Major
>
> We noticed that the Spark's Kafka connector outputs a lot of following
> verbose logs:
> {code:java}
> 19/04/18 04:31:06 INFO Fetcher: [Consumer clientId=consumer-2, groupId=...]
> Resetting offset for partition ... to offset ....
> {code}
> It comes from
> [https://github.com/hachikuji/kafka/blob/76c796ca128c3c97231f3ebda994a07bb06b26aa/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L567]
> This log was added in [https://github.com/apache/kafka/pull/4557]
> In Spark, we use `seekToEnd` to discover latest offsets of a topic. If there
> are thousands of partitions in this topic, it will output thousands of INFO
> logs every call
> Is it intentional? If not, can we change it to DEBUG?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
