[
https://issues.apache.org/jira/browse/SPARK-48308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-48308.
---------------------------------
> Unify getting data schema without partition columns in FileSourceStrategy
> -------------------------------------------------------------------------
>
> Key: SPARK-48308
> URL: https://issues.apache.org/jira/browse/SPARK-48308
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: Johan Lasperas
> Assignee: Johan Lasperas
> Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.5.2, 4.0.0
>
>
> In
> [FileSourceStrategy,|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L191]
> the schema of the data excluding partition columns is computed 2 times in a
> slightly different way:
>
> {code:java}
> val dataColumnsWithoutPartitionCols =
> dataColumns.filterNot(partitionSet.contains) {code}
> vs
> {code:java}
> val readDataColumns = dataColumns
> .filterNot(partitionColumns.contains) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
