[
https://issues.apache.org/jira/browse/SPARK-56975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
You Zhou updated SPARK-56975:
-----------------------------
Description:
`DataStreamReader.table()` accepts a user-specified schema without complaint
and then silently ignores it:
```scala
spark.readStream
.schema(new StructType().add("a", IntegerType))
.table("some_table") // no error; the schema has no effect
```
User-specified schema is not a meaningful input to `.table()` — catalog tables
declare their own schema, and `TableCatalog.loadTable(Identifier)` has no
parameter to receive a user schema, so even if Spark wanted to forward one it
couldn't. The user's `.schema(...)` call is therefore always a misconfiguration.
The rest of `DataStreamReader` already surfaces this kind of misconfiguration
as a clear error:
- `.load()` goes through `DataSourceV2Utils.getTableFromProvider`, which throws
`_LEGACY_ERROR_TEMP_2242` ("`<provider>` source does not support user-specified
schema") when the provider does not implement `supportsExternalMetadata()`.
- `.changes()` explicitly calls `assertNoSpecifiedSchema("changes")` and throws
`_LEGACY_ERROR_TEMP_1189` ("User specified schema not supported with
`changes`.").
`.table()` is the odd one out: same invalid configuration, no error. Users can
write `readStream.schema(s).table(name)`, see a working query, and reasonably
assume `s` had an effect — when in fact the resulting stream uses the catalog
schema and `s` was dropped. Surfacing this as a clear error aligns `.table()`
with the existing behavior of `.load()` and `.changes()`.
was:
`DataStreamReader.table()` accepts a user-specified schema without complaint
and then silently ignores it:
```scala
spark.readStream
.schema(new StructType().add("a", IntegerType))
.table("some_table") // no error; the schema has no effect
```
User-specified schema is not a meaningful input to `.table()` — catalog
tables declare their own schema, and `TableCatalog.loadTable(Identifier)`
has no parameter to receive a user schema, so even if Spark wanted to
forward one it couldn't. The user's `.schema(...)` call is therefore
always a misconfiguration.
The rest of `DataStreamReader` already surfaces this kind of misconfiguration
as a clear error:
- `.load()` goes through `DataSourceV2Utils.getTableFromProvider`, which
throws `_LEGACY_ERROR_TEMP_2242` ("`<provider>` source does not support
user-specified schema") when the provider does not implement
`supportsExternalMetadata()`.
- `.changes()` explicitly calls `assertNoSpecifiedSchema("changes")` and
throws `_LEGACY_ERROR_TEMP_1189` ("User specified schema not supported
with `changes`.").
`.table()` is the odd one out: same invalid configuration, no error.
Users can write `readStream.schema(s).table(name)`, see a working query,
and reasonably assume `s` had an effect — when in fact the resulting
stream uses the catalog schema and `s` was dropped. Surfacing this as a
clear error aligns `.table()` with the existing behavior of `.load()` and
`.changes()`.
> DataStreamReader.table() should reject user-specified schema instead of
> silently ignoring it
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-56975
> URL: https://issues.apache.org/jira/browse/SPARK-56975
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: You Zhou
> Priority: Major
> Fix For: 4.2.0
>
>
> `DataStreamReader.table()` accepts a user-specified schema without complaint
> and then silently ignores it:
> ```scala
> spark.readStream
> .schema(new StructType().add("a", IntegerType))
> .table("some_table") // no error; the schema has no effect
> ```
> User-specified schema is not a meaningful input to `.table()` — catalog
> tables declare their own schema, and `TableCatalog.loadTable(Identifier)` has
> no parameter to receive a user schema, so even if Spark wanted to forward one
> it couldn't. The user's `.schema(...)` call is therefore always a
> misconfiguration.
> The rest of `DataStreamReader` already surfaces this kind of misconfiguration
> as a clear error:
> - `.load()` goes through `DataSourceV2Utils.getTableFromProvider`, which
> throws `_LEGACY_ERROR_TEMP_2242` ("`<provider>` source does not support
> user-specified schema") when the provider does not implement
> `supportsExternalMetadata()`.
> - `.changes()` explicitly calls `assertNoSpecifiedSchema("changes")` and
> throws `_LEGACY_ERROR_TEMP_1189` ("User specified schema not supported with
> `changes`.").
> `.table()` is the odd one out: same invalid configuration, no error. Users
> can write `readStream.schema(s).table(name)`, see a working query, and
> reasonably assume `s` had an effect — when in fact the resulting stream uses
> the catalog schema and `s` was dropped. Surfacing this as a clear error
> aligns `.table()` with the existing behavior of `.load()` and `.changes()`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
