GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/17873
[SPARK-20614][PROJECT INFRA] Use the same log4j configuration with Jenkins
in AppVeyor
## What changes were proposed in this pull request?
Currently, there are flooding logs in AppVeyor (in the console). This has
been fine because we can download all the logs. However, (given my observations
so far), logs are truncated when there are too many. It has been grown recently
and it started to get truncated. For example, see
https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/build/1209-master
Even after the log is downloaded, it looks truncated as below:
```
[00:44:21] 17/05/04 18:56:18 INFO TaskSetManager: Finished task 197.0 in
stage 601.0 (TID 9211) in 0 ms on localhost (executor driver) (194/200)
[00:44:21] 17/05/04 18:56:18 INFO Executor: Running task 199.0 in stage
601.0 (TID 9213)
[00:44:21] 17/05/04 18:56:18 INFO Executor: Finished task 198.0 in stage
601.0 (TID 9212). 2473 bytes result sent to driver
...
```
Probably, it looks better to use the same log4j configuration that we are
using for SparkR tests in Jenkins -
https://github.com/apache/spark/blob/master/R/run-tests.sh#L26:
```
# Set everything to be logged to the file target/unit-tests.log
log4j.rootCategory=INFO, file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.append=true
log4j.appender.file.file=R/target/unit-tests.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss.SSS} %t
%p %c{1}: %m%n
# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN
org.eclipse.jetty.LEVEL=WARN
```
## How was this patch tested?
Manually tested with @spark-test account -
https://ci.appveyor.com/project/spark-test/spark/build/672-r-log4j
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark appveyor-reduce-logs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17873.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17873
----
commit 4d4e6f02ee9270eef2646c404cd46a097e517203
Author: hyukjinkwon <[email protected]>
Date: 2017-05-05T13:38:32Z
Use the same log4j configuration with Jenkins in AppVeyor
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
