Steve Scaffidi commented on SPARK-8118:

For anybody who might find this helpful, I found a workaround for hive, 
documented here: 
(also posted to SO here: https://stackoverflow.com/a/45572400/14186)

And there's a workaround for spark that's similar here: 

> Turn off noisy log output produced by Parquet 1.7.0
> ---------------------------------------------------
>                 Key: SPARK-8118
>                 URL: https://issues.apache.org/jira/browse/SPARK-8118
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 1.4.1, 1.5.0
>            Reporter: Cheng Lian
>            Assignee: Cheng Lian
>            Priority: Minor
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust 
> {{ParquetRelation.enableLogForwarding}} accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet 
> logs via SLF4J, so that we can handle them consistently. In general these 
> logs are very useful. Esp. when used to diagnosing Parquet memory issue and 
> filter push-down.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to