[jira] [Commented] (KAFKA-3816) Provide more context in Kafka Connect log messages using MDC
[
https://issues.apache.org/jira/browse/KAFKA-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886183#comment-16886183
]
Robin Moffatt commented on KAFKA-3816:
--
For completeness, the MDC parameter added in Apache Kafka 2.3 is
*{{connector.context}}*, as described in
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-449%3A+Add+connector+contexts+to+Connect+worker+logs].
The above references to connector.class / connector.name / connector.scope /
connector.task are outdated.
> Provide more context in Kafka Connect log messages using MDC
>
>
> Key: KAFKA-3816
> URL: https://issues.apache.org/jira/browse/KAFKA-3816
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
>Assignee: Randall Hauch
>Priority: Critical
> Fix For: 2.3.0
>
>
> Currently it is relatively difficult to correlate individual log messages
> with the various threads and activities that are going on within a Kafka
> Connect worker, let along a cluster of workers. Log messages should provide
> more context to make it easier and to allow log scraping tools to coalesce
> related log messages.
> One simple way to do this is by using _mapped diagnostic contexts_, or MDC.
> This is supported by the SLF4J API, and by the Logback and Log4J logging
> frameworks.
> Basically, the framework would be changed so that each thread is configured
> with one or more MDC parameters using the
> {{org.slf4j.MDC.put(String,String)}} method in SLF4J. Once that thread is
> configured, all log messages made using that thread have that context. The
> logs can then be configured to use those parameters.
> It would be ideal to define a convention for connectors and the Kafka Connect
> framework. A single set of MDC parameters means that the logging framework
> can use the specific parameters on its message formats.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (KAFKA-3816) Provide more context in Kafka Connect log messages using MDC
[
https://issues.apache.org/jira/browse/KAFKA-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841234#comment-16841234
]
ASF GitHub Bot commented on KAFKA-3816:
---
guozhangwang commented on pull request #5743: KAFKA-3816: Add MDC logging to
Connect runtime
URL: https://github.com/apache/kafka/pull/5743
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide more context in Kafka Connect log messages using MDC
>
>
> Key: KAFKA-3816
> URL: https://issues.apache.org/jira/browse/KAFKA-3816
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
>Priority: Critical
>
> Currently it is relatively difficult to correlate individual log messages
> with the various threads and activities that are going on within a Kafka
> Connect worker, let along a cluster of workers. Log messages should provide
> more context to make it easier and to allow log scraping tools to coalesce
> related log messages.
> One simple way to do this is by using _mapped diagnostic contexts_, or MDC.
> This is supported by the SLF4J API, and by the Logback and Log4J logging
> frameworks.
> Basically, the framework would be changed so that each thread is configured
> with one or more MDC parameters using the
> {{org.slf4j.MDC.put(String,String)}} method in SLF4J. Once that thread is
> configured, all log messages made using that thread have that context. The
> logs can then be configured to use those parameters.
> It would be ideal to define a convention for connectors and the Kafka Connect
> framework. A single set of MDC parameters means that the logging framework
> can use the specific parameters on its message formats.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3816) Provide more context in Kafka Connect log messages using MDC
[
https://issues.apache.org/jira/browse/KAFKA-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645101#comment-16645101
]
Randall Hauch commented on KAFKA-3816:
--
The above pull request makes very minimal changes to Connect to use four MDC
parameters:
# {{connector.class}} - the shortened name of the connector class (e.g.,
"MySink" for a class named "MySinkConnector")
# {{connector.name}} - the name of the connector
# {{connector.scope}} - the scope within a task, with values like "{{worker}}"
within the worker thread, "{{task}}" within the task worker (and task
implementation), and "{{offsets}}" in the scheduled thread that commits offsets
for source connectors.
# {{connector.task}} - the task number within a task worker thread, or unset
elsewhere
These must be explicitly used within the {{connect-log4j.properties}} file, and
are not enabled by default to maintain backward compatibility. If that file is
modified to use:
{noformat}
log4j.appender.stdout.layout.ConversionPattern=[%d] %p
%X{connector.name}|%X{connector.scope}%X{connector.task} %m (%c:%L)%n
{noformat}
the resulting log messages will include include the connector name, the task
number, and the scope (this does not use the connector class name, which seems
verbose). Here's an example of some log statements using the above Log4J line
format, where the connector name is "local-file-source", the task number is
"0", and the scope is either "task", "worker", or "offsets" (depending upon
where the log messages are called). When these parameters are not set (in other
contexts), the log messages show a blank value for them.
{noformat}
[2018-10-04 17:49:37,068] INFO | Kafka version : 2.1.0-SNAPSHOT
(org.apache.kafka.common.utils.AppInfoParser:109)
[2018-10-04 17:49:37,068] INFO | Kafka commitId : 00a7bd8b636c184e
(org.apache.kafka.common.utils.AppInfoParser:110)
[2018-10-04 17:49:37,222] INFO | Kafka cluster ID: bpHT_pLdRR-Y7HKEaEdHFA
(org.apache.kafka.connect.util.ConnectUtils:59)
[2018-10-04 17:49:37,239] INFO | Logging initialized @2093ms to
org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:193)
[2018-10-04 17:49:37,282] INFO | Added connector for http://:8083
(org.apache.kafka.connect.runtime.rest.RestServer:119)
[2018-10-04 17:49:37,302] INFO | Advertised URI: http://10.0.1.6:8083/
(org.apache.kafka.connect.runtime.rest.RestServer:267)
...
[2018-10-04 17:49:38,087] INFO local-file-source|connector Creating connector
local-file-source of type FileStreamSource
(org.apache.kafka.connect.runtime.Worker:238)
[2018-10-04 17:49:38,090] INFO local-file-source|connector Instantiated
connector local-file-source with version 2.1.0-SNAPSHOT of type class
org.apache.kafka.connect.file.FileStreamSourceConnector
(org.apache.kafka.connect.runtime.Worker:241)
...
[2018-10-04 17:49:38,093] INFO local-file-source|task0 Creating task
local-file-source-0 (org.apache.kafka.connect.runtime.Worker:404)
...
[2018-10-04 17:49:38,097] INFO local-file-source|task0 Instantiated task
local-file-source-0 with version 2.1.0-SNAPSHOT of type
org.apache.kafka.connect.file.FileStreamSourceTask
(org.apache.kafka.connect.runtime.Worker:419)
[2018-10-04 17:49:38,097] INFO local-file-source|task0 JsonConverterConfig
values:
converter.type = key
schemas.cache.size = 1000
schemas.enable = true
(org.apache.kafka.connect.json.JsonConverterConfig:279)
[2018-10-04 17:49:38,098] INFO local-file-source|task0 Set up the key converter
class org.apache.kafka.connect.json.JsonConverter for task local-file-source-0
using the worker config (org.apache.kafka.connect.runtime.Worker:442)
[2018-10-04 17:49:38,098] INFO local-file-source|task0 JsonConverterConfig
values:
converter.type = value
schemas.cache.size = 1000
schemas.enable = true
(org.apache.kafka.connect.json.JsonConverterConfig:279)
[2018-10-04 17:49:38,098] INFO local-file-source|task0 Set up the value
converter class org.apache.kafka.connect.json.JsonConverter for task
local-file-source-0 using the worker config
(org.apache.kafka.connect.runtime.Worker:448)
[2018-10-04 17:49:38,098] INFO local-file-source|task0 Set up the header
converter class org.apache.kafka.connect.storage.SimpleHeaderConverter for task
local-file-source-0 using the worker config
(org.apache.kafka.connect.runtime.Worker:454)
[2018-10-04 17:49:38,109] INFO local-file-source|task0 ProducerConfig values:
acks = all
batch.size = 16384
bootstrap.servers = [localhost:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 54
delivery.timeout.ms = 2147483647
enable.idempotence = false
interceptor.classes = []
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
linger.ms = 0
max.block.ms = 9223372036854775807
[jira] [Commented] (KAFKA-3816) Provide more context in Kafka Connect log messages using MDC
[
https://issues.apache.org/jira/browse/KAFKA-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638999#comment-16638999
]
ASF GitHub Bot commented on KAFKA-3816:
---
rhauch opened a new pull request #5743: KAFKA-3816: Add MDC logging to Connect
runtime [WIP]
URL: https://github.com/apache/kafka/pull/5743
**Work in Progress: do not merge**
Added LoggingContext as a simple mechanism to set and unset Mapped
Diagnostic Contexts (MDC) in the loggers to provide for each thread useful
parameters that can be used within the logging configuration. MDC avoids having
to modify lots of log statements, since the parameters are available to all log
statements issued by the thread, no matter what class makes those calls.
The design intentionally minimizes the number of changes to any existing
classes, and does not use Java 8 features so it can be easily backported if
desired.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Provide more context in Kafka Connect log messages using MDC
>
>
> Key: KAFKA-3816
> URL: https://issues.apache.org/jira/browse/KAFKA-3816
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
>Priority: Critical
>
> Currently it is relatively difficult to correlate individual log messages
> with the various threads and activities that are going on within a Kafka
> Connect worker, let along a cluster of workers. Log messages should provide
> more context to make it easier and to allow log scraping tools to coalesce
> related log messages.
> One simple way to do this is by using _mapped diagnostic contexts_, or MDC.
> This is supported by the SLF4J API, and by the Logback and Log4J logging
> frameworks.
> Basically, the framework would be changed so that each thread is configured
> with one or more MDC parameters using the
> {{org.slf4j.MDC.put(String,String)}} method in SLF4J. Once that thread is
> configured, all log messages made using that thread have that context. The
> logs can then be configured to use those parameters.
> It would be ideal to define a convention for connectors and the Kafka Connect
> framework. A single set of MDC parameters means that the logging framework
> can use the specific parameters on its message formats.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
