Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1764#discussion_r15743336
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala ---
@@ -353,15 +356,14 @@ private[parquet] object ParquetTypesConverter extends
Logging {
* in the parent directory. If so, this is used. Else we read the actual
footer at the given
* location.
* @param origPath The path at which we expect one (or more) Parquet
files.
- * @param configuration The Hadoop configuration to use.
+ * @param conf The Hadoop configuration to use.
* @return The `ParquetMetadata` containing among other things the
schema.
*/
- def readMetaData(origPath: Path, configuration: Option[Configuration]):
ParquetMetadata = {
+ def readMetaData(origPath: Path, conf: Configuration): ParquetMetadata =
{
if (origPath == null) {
throw new IllegalArgumentException("Unable to read Parquet metadata:
path is null")
}
val job = new Job()
- val conf = configuration.getOrElse(ContextUtil.getConfiguration(job))
--- End diff --
I wanted to get rid of the optional configuration, but perhaps I should put
this back. Making a new Job and then asking for a Configuration doesn't seem
like it'd be more useful than just constructing a new Configuration, though, at
least in terms of the properties set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
