GitHub user jimfcarroll opened a pull request:
https://github.com/apache/spark/pull/3271
[SPARK-4412][SQL] Fix Spark's control of Parquet logging.
The Spark ParquetRelation.scala code makes the assumption that the
parquet.Log class has already been loaded. If
ParquetRelation.enableLogForwarding executes prior to the parquet.Log class
being loaded then the code in enableLogForwarding has no affect.
ParquetRelation.scala attempts to override the parquet logger but, at least
currently (and if your application simply reads a parquet file before it does
anything else with Parquet), the parquet.Log class hasn't been loaded yet.
Therefore the code in ParquetRelation.enableLogForwarding has no affect. If you
look at the code in parquet.Log there's a static initializer that needs to be
called prior to enableLogForwarding or whatever enableLogForwarding does gets
undone by this static initializer.
The "fix" would be to force the static initializer to get called in
parquet.Log as part of enableForwardLogging.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jimfcarroll/spark parquet-logging
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3271.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3271
----
commit 37bdff7794463dd0570989d8fc0cb426cdbc92f4
Author: Jim Carroll <[email protected]>
Date: 2014-11-14T21:03:32Z
Fix Spark's control of Parquet logging.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
