[jira] [Commented] (KAFKA-2927) System tests: reduce storage footprint of collected logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051987#comment-15051987
 ] 

ASF GitHub Bot commented on KAFKA-2927:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/657


> System tests: reduce storage footprint of collected logs
> 
>
> Key: KAFKA-2927
> URL: https://issues.apache.org/jira/browse/KAFKA-2927
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.9.1.0
>
>
> Looking at recent night test runs (testing.confluent.io/kafka), the storage 
> requirements for log output from the various services has increased 
> significantly, up to 7-10G for a single test run, up from hundreds of MB
> Current breakdown:
> 23M   Benchmark
> 3.2M  ClientCompatibilityTest
> 613M  ConnectDistributedTest
> 1.1M  ConnectRestApiTest
> 1.5M  ConnectStandaloneFileTest
> 2.0M  ConsoleConsumerTest
> 440K  KafkaVersionTest
> 744K  Log4jAppenderTest
> 49M   QuotaTest
> 3.0G  ReplicationTest
> 1.2G  TestMirrorMakerService
> 185M  TestUpgrade
> 372K  TestVerifiableProducer
> 2.3G  VerifiableConsumerTest
> The biggest contributors in these test suites:
> ReplicationTest:
> verifiable_producer.log (currently TRACE level)
> VerifiableConsumerTest:
> kafka server.log
> TestMirrorMakerService:
> verifiable_producer.log
> ConnectDistributedTest:
> kafka server.log
> The worst offenders are therefore 
> verifiable_producer.log which is logging at TRACE level, and kafka server.log 
> which is logging at debug level
> One solution is to:
> 1) Update the log4j configs to log separately to both an INFO level file, and 
> another file for DEBUG at least for the worst offenders.
> 2) Don't collect these DEBUG (and below) logs by default; only mark for 
> collection during failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2927) System tests: reduce storage footprint of collected logs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049845#comment-15049845
 ] 

ASF GitHub Bot commented on KAFKA-2927:
---

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/657

KAFKA-2927: reduce system test storage footprint

Split kafka logging into two levels - DEBUG and INFO, and do not collect 
DEBUG by default.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
KAFKA-2927-reduce-log-footprint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/657.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #657


commit 0dc3a1a367083f57f3cb6d8e1cd82571598d7108
Author: Geoff Anderson 
Date:   2015-12-10T01:09:59Z

Split kafka logging into two levels - DEBUG and INFO, and do not collect 
DEBUG by default




> System tests: reduce storage footprint of collected logs
> 
>
> Key: KAFKA-2927
> URL: https://issues.apache.org/jira/browse/KAFKA-2927
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>
> Looking at recent night test runs (testing.confluent.io/kafka), the storage 
> requirements for log output from the various services has increased 
> significantly, up to 7-10G for a single test run, up from hundreds of MB
> Current breakdown:
> 23M   Benchmark
> 3.2M  ClientCompatibilityTest
> 613M  ConnectDistributedTest
> 1.1M  ConnectRestApiTest
> 1.5M  ConnectStandaloneFileTest
> 2.0M  ConsoleConsumerTest
> 440K  KafkaVersionTest
> 744K  Log4jAppenderTest
> 49M   QuotaTest
> 3.0G  ReplicationTest
> 1.2G  TestMirrorMakerService
> 185M  TestUpgrade
> 372K  TestVerifiableProducer
> 2.3G  VerifiableConsumerTest
> The biggest contributors in these test suites:
> ReplicationTest:
> verifiable_producer.log (currently TRACE level)
> VerifiableConsumerTest:
> kafka server.log
> TestMirrorMakerService:
> verifiable_producer.log
> ConnectDistributedTest:
> kafka server.log
> The worst offenders are therefore 
> verifiable_producer.log which is logging at TRACE level, and kafka server.log 
> which is logging at debug level
> One solution is to:
> 1) Update the log4j configs to log separately to both an INFO level file, and 
> another file for DEBUG at least for the worst offenders.
> 2) Don't collect these DEBUG (and below) logs by default; only mark for 
> collection during failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)