[ https://issues.apache.org/jira/browse/KAFKA-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659769#comment-16659769 ]
Mr Kafka edited comment on KAFKA-7510 at 10/22/18 10:33 PM: ------------------------------------------------------------ [~mjsax] {quote}Why does it not contradict KAFKA-6538? It is about adding key/value in human readable form to exception messages that end up in the logs. While this ticket is about removing key/value data from the logs. What do you mean by "it only affects it's implementation"? {quote} You can still add key/vaue/headers or as much useful data as you want to the log messages, those log messages will be under TRACE level and a user has to explicitly turn TRACE level on to see the extra information. Happy to create a PR to move out key/value logging from RecordCollectorImpl to trace as it directly blocks me and I'm likely to do this in a private fork regardless until it is in an official release. Not so interested on creating a KIP and long discussions on moving sensitive log data to a different log level which is as simple as adding a *log.trace(...)* below the *log.error(...)* statements. was (Author: mrkafka): [~mjsax] {quote} Why does it not contradict KAFKA-6538? It is about adding key/value in human readable form to exception messages that end up in the logs. While this ticket is about removing key/value data from the logs. What do you mean by "it only affects it's implementation"? {quote} You can still add key/vaue/headers or as much useful data as you want to the log messages, those log messages will be under TRACE level and a user has to explicitly turn TRACE level on to see the extra information. Happy to create a PR to move out key/value logging from RecordCollectorImpl to trace as it directly blocks me and I'm likely to do this in a private fork regardless until it is done. Not so interested on creating a KIP and long discussions on moving sensitive log data to a different log level which is as simple as adding a *log.trace(...)* below the *log.error(...)* statements. > KStreams RecordCollectorImpl leaks data to logs on error > -------------------------------------------------------- > > Key: KAFKA-7510 > URL: https://issues.apache.org/jira/browse/KAFKA-7510 > Project: Kafka > Issue Type: Improvement > Components: streams > Reporter: Mr Kafka > Priority: Major > Labels: user-experience > > org.apache.kafka.streams.processor.internals.RecordCollectorImpl leaks data > on error as it dumps the *value* / message payload to the logs. > This is problematic as it may contain personally identifiable information > (pii) or other secret information to plain text log files which can then be > propagated to other log systems i.e Splunk. > I suggest the *key*, and *value* fields be moved to debug level as it is > useful for some people while error level contains the *errorMessage, > timestamp, topic* and *stackTrace*. > {code:java} > private <K, V> void recordSendError( > final K key, > final V value, > final Long timestamp, > final String topic, > final Exception exception > ) { > String errorLogMessage = LOG_MESSAGE; > String errorMessage = EXCEPTION_MESSAGE; > if (exception instanceof RetriableException) { > errorLogMessage += PARAMETER_HINT; > errorMessage += PARAMETER_HINT; > } > log.error(errorLogMessage, key, value, timestamp, topic, > exception.toString()); > sendException = new StreamsException( > String.format( > errorMessage, > logPrefix, > "an error caught", > key, > value, > timestamp, > topic, > exception.toString() > ), > exception); > }{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)