GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/13918
[SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J for WRITE
operations
## What changes were proposed in this pull request?
[SPARK-8118](https://github.com/apache/spark/pull/8196) implements
redirecting Parquet JUL logger via SLF4J, but it is currently applied only when
READ operations occurs. If users use only WRITE operations, there occurs many
Parquet logs.
This PR makes the redirection work on WRITE operations, too.
**Before**
```
scala>
spark.range(10).write.format("parquet").mode("overwrite").save("/tmp/p")
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Jun 26, 2016 9:04:38 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig:
Compression: SNAPPY
... about 70 lines Parquet Log ...
scala>
spark.range(10).write.format("parquet").mode("overwrite").save("/tmp/p")
... about 70 lines Parquet Log ...
```
**After**
```
scala>
spark.range(10).write.format("parquet").mode("overwrite").save("/tmp/p")
[Stage 0:> (0 + 8)
/ 8]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
scala>
spark.range(10).write.format("parquet").mode("overwrite").save("/tmp/p")
```
This PR also removes a legacy try-catch block related to
[SPARK-9974](https://issues.apache.org/jira/browse/SPARK-9974) and fixes some
typos.
## How was this patch tested?
Manual.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-16221
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13918.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13918
----
commit a643114e71e8d5054910a60b79ef1d1d392da452
Author: Dongjoon Hyun <[email protected]>
Date: 2016-06-27T04:36:39Z
[SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J for WRITE
operations
Remove a legacy try-catch block and fix some typos. (Bug)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
