Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/195#discussion_r10903544
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala ---
@@ -72,16 +73,43 @@ case class ParquetRelation(val tableName: String, val
path: String) extends Base
/** Output **/
override val output = attributes
+ /** Name (dummy value) */
+ // TODO: rethink whether ParquetRelation should inherit from BaseRelation
+ // (currently required to re-use HiveStrategies but should be removed)
+ override def tableName = "parquet"
+
// Parquet files have no concepts of keys, therefore no Partitioner
// Note: we could allow Block level access; needs to be thought through
override def isPartitioned = false
}
object ParquetRelation {
+ // change this to enable/disable Parquet logging
+ var DEBUG: Boolean = false
+
+ // TODO: consider redirecting Parquet's log output to log4j logger and
+ // using config file for log settings
+ def setParquetLogLevel() {
+ val level: Level = if (DEBUG) Level.FINEST else Level.WARNING
--- End diff --
Hmmm, yeah if the are logging statements in the critical path of producing
tuples then this is probably something we want to avoid. Otherwise the
performance penalty probably doesn't matter. Can you confirm if that is the
case.
If logback is sufficiently close to log4j then we might be consider
switching. An alternative would be to just leave the logging coming out of
JUL. Any thoughts @pwendell ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
