Yang Jie created SPARK-56172:
--------------------------------
Summary: Remove V2 file write guards, unify schema validation, and
support custom partition locations
Key: SPARK-56172
URL: https://issues.apache.org/jira/browse/SPARK-56172
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.2.0
Reporter: Yang Jie
Remove guards that block `FileDataSourceV2` from the V2 write path:
# `DataSourceV2Utils.getTableProvider` filter controlled by feature flag,
enabling catalog-managed tables to use V2.
# `FileTable.schema` removes over-aggressive column name duplication check
(moved to `FileWrite.validateInputs` for write path only) and skips
`supportsDataType` for partition columns on the read path.
# Add `customPartitionLocations` support:
`FileTable.getCustomPartitionLocations` mirrors V1's
`InsertIntoHadoopFsRelationCommand`, comparing catalog partition locations
against default paths. Plumb through all format Write/Table classes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
