[
https://issues.apache.org/jira/browse/SPARK-59342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Lasperas updated SPARK-59342:
-----------------------------------
Description:
Allow connectors configuring schema alignment behavior under
`spark.sql.storeAssignmentPolicy` = `ANSI`: accept implicit casts at analysis,
and rely on execution time checks to catch malformed values or overflows
The proposal is to extend the DSv2 interface to allow configuring this behavior
per table:
{code:java}
public interface ConfigurableSchemaAlignment extends Table {
SchemaAlignmentConfig schemaAlignmentConfig();
}
public interface SchemaAlignmentConfig {
default AnsiStoreAssignmentCastCheck ansiStoreAssignmentCastCheck() {
// AT_ANALYSIS or AT_RUNTIME.
return AnsiStoreAssignmentCastCheck.AT_ANALYSIS;
}
} {code}
was:
Allow connectors configuring following schema alignment behavior:
* allow writes under spark.sql.storeAssignmentPolicy=LEGACY
* accept implicit casts at analysis, and rely on execution time checks to
catch malformed values or overflows
The proposal is to extend the DSv2 interface to allow configuring this behavior
per table:
{code:java}
public interface Table {
...
default SchemaAlignmentConfig schemaAlignmentConfig() {
return SchemaAlignmentConfig.DEFAULT; }
}
public interface SchemaAlignmentConfig {
default boolean allowLegacyStoreAssignmentPolicy() {
return false;
}
default boolean deferCastValidationToRuntime() {
return false;
}
}
{code}
> Allow configuring schema alignment behavior for DSv2 writes
> -----------------------------------------------------------
>
> Key: SPARK-59342
> URL: https://issues.apache.org/jira/browse/SPARK-59342
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Johan Lasperas
> Priority: Major
> Labels: pull-request-available
>
> Allow connectors configuring schema alignment behavior under
> `spark.sql.storeAssignmentPolicy` = `ANSI`: accept implicit casts at
> analysis, and rely on execution time checks to catch malformed values or
> overflows
> The proposal is to extend the DSv2 interface to allow configuring this
> behavior per table:
> {code:java}
> public interface ConfigurableSchemaAlignment extends Table {
> SchemaAlignmentConfig schemaAlignmentConfig();
> }
> public interface SchemaAlignmentConfig {
> default AnsiStoreAssignmentCastCheck ansiStoreAssignmentCastCheck() {
> // AT_ANALYSIS or AT_RUNTIME.
> return AnsiStoreAssignmentCastCheck.AT_ANALYSIS;
> }
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
