Github user brkyvz commented on a diff in the pull request:
    --- Diff: 
    @@ -132,13 +132,24 @@ case class DataSource(
           new InMemoryFileIndex(sparkSession, globbedPaths, options, None)
    +    val dataSchema = userSpecifiedSchema.orElse {
    +      format.inferSchema(
    +        sparkSession,
    +        caseInsensitiveOptions,
    +        tempFileIndex.allFiles())
    +    }.getOrElse {
    +      throw new AnalysisException(
    +        s"Unable to infer schema for $format. It must be specified 
    +    }
         val partitionSchema = if (partitionColumns.isEmpty && 
catalogTable.isEmpty) {
           // Try to infer partitioning, because no DataSource in the read path 
provides the partitioning
           // columns properly unless it is a Hive DataSource
           val resolved = { partitionField =>
             val equality = sparkSession.sessionState.conf.resolver
    -        // SPARK-18510: try to get schema from userSpecifiedSchema, 
otherwise fallback to inferred
    -        userSpecifiedSchema.flatMap(_.find(f => equality(,
    +        // SPARK-18510: try to get partition schema from data schema, 
otherwise fallback to inferred
    +        dataSchema.find(f => equality(,
    --- End diff --
    this change is not necessary

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to