[ https://issues.apache.org/jira/browse/KAFKA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049845#comment-15049845 ]
ASF GitHub Bot commented on KAFKA-2927: --------------------------------------- GitHub user granders opened a pull request: https://github.com/apache/kafka/pull/657 KAFKA-2927: reduce system test storage footprint Split kafka logging into two levels - DEBUG and INFO, and do not collect DEBUG by default. You can merge this pull request into a Git repository by running: $ git pull https://github.com/confluentinc/kafka KAFKA-2927-reduce-log-footprint Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/657.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #657 ---- commit 0dc3a1a367083f57f3cb6d8e1cd82571598d7108 Author: Geoff Anderson <ge...@confluent.io> Date: 2015-12-10T01:09:59Z Split kafka logging into two levels - DEBUG and INFO, and do not collect DEBUG by default ---- > System tests: reduce storage footprint of collected logs > -------------------------------------------------------- > > Key: KAFKA-2927 > URL: https://issues.apache.org/jira/browse/KAFKA-2927 > Project: Kafka > Issue Type: Bug > Reporter: Geoff Anderson > Assignee: Geoff Anderson > > Looking at recent night test runs (testing.confluent.io/kafka), the storage > requirements for log output from the various services has increased > significantly, up to 7-10G for a single test run, up from hundreds of MB > Current breakdown: > 23M Benchmark > 3.2M ClientCompatibilityTest > 613M ConnectDistributedTest > 1.1M ConnectRestApiTest > 1.5M ConnectStandaloneFileTest > 2.0M ConsoleConsumerTest > 440K KafkaVersionTest > 744K Log4jAppenderTest > 49M QuotaTest > 3.0G ReplicationTest > 1.2G TestMirrorMakerService > 185M TestUpgrade > 372K TestVerifiableProducer > 2.3G VerifiableConsumerTest > The biggest contributors in these test suites: > ReplicationTest: > verifiable_producer.log (currently TRACE level) > VerifiableConsumerTest: > kafka server.log > TestMirrorMakerService: > verifiable_producer.log > ConnectDistributedTest: > kafka server.log > The worst offenders are therefore > verifiable_producer.log which is logging at TRACE level, and kafka server.log > which is logging at debug level > One solution is to: > 1) Update the log4j configs to log separately to both an INFO level file, and > another file for DEBUG at least for the worst offenders. > 2) Don't collect these DEBUG (and below) logs by default; only mark for > collection during failure -- This message was sent by Atlassian JIRA (v6.3.4#6332)