[
https://issues.apache.org/jira/browse/SPARK-27136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-27136:
-----------------------------------
Description:
The data source option check_files_exist is introduced in In
https://github.com/apache/spark/pull/23383 when the file source V2 framework is
implemented. In the PR, FileIndex was created as a member of FileTable, so that
we could implement partition pruning like 0f9fcab in the future. At that time
FileIndexes will always be created for file writes, so we needed the option to
decide whether to check file existence.
After https://github.com/apache/spark/pull/23774, the option is not needed
anymore. This PR is to clean the option.
was:
The data source option check_files_exist is introduced in In
https://github.com/apache/spark/pull/23383 when the file source V2 framework is
implemented. In the PR, FileIndex was created as a member of FileTable, so that
we could implement partition pruning like 0f9fcab in the future. For file
writes, we needed the option to decide whether to check file existence.
After https://github.com/apache/spark/pull/23774, the option is not needed
anymore. This PR is to clean the option.
> Remove data source option check_files_exist
> -------------------------------------------
>
> Key: SPARK-27136
> URL: https://issues.apache.org/jira/browse/SPARK-27136
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Gengliang Wang
> Priority: Major
>
> The data source option check_files_exist is introduced in In
> https://github.com/apache/spark/pull/23383 when the file source V2 framework
> is implemented. In the PR, FileIndex was created as a member of FileTable, so
> that we could implement partition pruning like 0f9fcab in the future. At that
> time FileIndexes will always be created for file writes, so we needed the
> option to decide whether to check file existence.
> After https://github.com/apache/spark/pull/23774, the option is not needed
> anymore. This PR is to clean the option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
