[
https://issues.apache.org/jira/browse/SPARK-56975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
You Zhou updated SPARK-56975:
-----------------------------
Description:
{{DataStreamReader.table()}} accepts a user-specified schema without complaint
and then silently ignores it:
{{spark.readStream}}
{{.schema(new StructType().add("a", IntegerType))}}
{{.table("some_table") // no error; the schema has no effect}}
User-specified schema is not a meaningful input to {{.table()}} — catalog
tables declare their own schema, and {{TableCatalog.loadTable(Identifier)}} has
no parameter to receive a user schema, so even if Spark wanted to forward one
it couldn't. The user's {{.schema(...)}} call is therefore always a
misconfiguration.
The rest of {{DataStreamReader}} already surfaces this kind of misconfiguration
as a clear error:
- {{.load()}} goes through {{{}DataSourceV2Utils.getTableFromProvider{}}},
which throws {{_LEGACY_ERROR_TEMP_2242}} ("{{{}<provider>{}}} source does not
support user-specified schema") when the provider does not implement
{{{}supportsExternalMetadata(){}}}.
- {{.changes()}} explicitly calls {{assertNoSpecifiedSchema("changes")}} and
throws {{_LEGACY_ERROR_TEMP_1189}} ("User specified schema not supported with
{{{}changes{}}}.").
{{.table()}} is the odd one out: same invalid configuration, no error. Users
can write {{{}readStream.schema(s).table(name){}}}, see a working query, and
reasonably assume {{s}} had an effect — when in fact the resulting stream uses
the catalog schema and {{s}} was dropped. Surfacing this as a clear error
aligns {{.table()}} with the existing behavior of {{.load()}} and
{{.changes().}}
was:
{{DataStreamReader.table() }}accepts a user-specified schema without complaint
and then silently ignores it:
{{spark.readStream}}
{{.schema(new StructType().add("a", IntegerType))}}
{{.table("some_table") // no error; the schema has no effect}}
User-specified schema is not a meaningful input to {{.table()}} — catalog
tables declare their own schema, and {{TableCatalog.loadTable(Identifier)}} has
no parameter to receive a user schema, so even if Spark wanted to forward one
it couldn't. The user's {{.schema(...)}} call is therefore always a
misconfiguration.
The rest of {{DataStreamReader}} already surfaces this kind of misconfiguration
as a clear error:
- {{.load()}} goes through {{{}DataSourceV2Utils.getTableFromProvider{}}},
which throws {{_LEGACY_ERROR_TEMP_2242}} ("{{{}<provider>{}}} source does not
support user-specified schema") when the provider does not implement
{{{}supportsExternalMetadata(){}}}.
- {{.changes()}} explicitly calls {{assertNoSpecifiedSchema("changes")}} and
throws {{_LEGACY_ERROR_TEMP_1189}} ("User specified schema not supported with
{{{}changes{}}}.").
{{.table()}} is the odd one out: same invalid configuration, no error. Users
can write {{{}readStream.schema(s).table(name){}}}, see a working query, and
reasonably assume {{s}} had an effect — when in fact the resulting stream uses
the catalog schema and {{s}} was dropped. Surfacing this as a clear error
aligns {{.table()}} with the existing behavior of {{.load()}} and
{{.changes().}}
> DataStreamReader.table() should reject user-specified schema instead of
> silently ignoring it
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-56975
> URL: https://issues.apache.org/jira/browse/SPARK-56975
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: You Zhou
> Priority: Major
> Fix For: 4.2.0
>
>
> {{DataStreamReader.table()}} accepts a user-specified schema without
> complaint and then silently ignores it:
> {{spark.readStream}}
> {{.schema(new StructType().add("a", IntegerType))}}
> {{.table("some_table") // no error; the schema has no effect}}
> User-specified schema is not a meaningful input to {{.table()}} — catalog
> tables declare their own schema, and {{TableCatalog.loadTable(Identifier)}}
> has no parameter to receive a user schema, so even if Spark wanted to forward
> one it couldn't. The user's {{.schema(...)}} call is therefore always a
> misconfiguration.
> The rest of {{DataStreamReader}} already surfaces this kind of
> misconfiguration as a clear error:
> - {{.load()}} goes through {{{}DataSourceV2Utils.getTableFromProvider{}}},
> which throws {{_LEGACY_ERROR_TEMP_2242}} ("{{{}<provider>{}}} source does not
> support user-specified schema") when the provider does not implement
> {{{}supportsExternalMetadata(){}}}.
> - {{.changes()}} explicitly calls {{assertNoSpecifiedSchema("changes")}} and
> throws {{_LEGACY_ERROR_TEMP_1189}} ("User specified schema not supported with
> {{{}changes{}}}.").
> {{.table()}} is the odd one out: same invalid configuration, no error. Users
> can write {{{}readStream.schema(s).table(name){}}}, see a working query, and
> reasonably assume {{s}} had an effect — when in fact the resulting stream
> uses the catalog schema and {{s}} was dropped. Surfacing this as a clear
> error aligns {{.table()}} with the existing behavior of {{.load()}} and
> {{.changes().}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
