yyanyy opened a new pull request, #57946:
URL: https://github.com/apache/spark/pull/57946
> [!NOTE]
> This is a stacked follow-up to #57865, which intentionally leaves
streaming unchanged.
> The first commit in this draft is the current #57865 change and will
disappear from the diff
> after that PR merges and this branch is rebased.
### What changes were proposed in this pull request?
This PR forwards user-provided options and required write privileges for
catalog-backed DataSource
V2 streaming writes through `DataStreamWriter.toTable`.
It:
* loads an existing V2 streaming target with the user option bag and
output-mode-specific write
privileges:
* append and update: `INSERT`;
* complete: `INSERT` and `DELETE`;
* stores typed write privileges and options in the transactional streaming
write-target reference
so each micro-batch re-resolution uses the same context;
* retains the same options on the target `DataSourceV2Relation`, allowing
them to flow through
`V2Writes` into `LogicalWriteInfo`; and
* adds independent genuine V2 streaming-table tests for initial target
loading, the actual
`StreamingWrite`, and transactional reloads across micro-batches.
The separate `SupportsCatalogOptions` `.format(...).start()` gap is out of
scope, as are batch/read
behavior, V1-specific paths, table creation, and new continuous-transaction
machinery.
### Why are the changes needed?
Before this change, `DataStreamWriter.toTable` loaded an existing
catalog-backed V2 target without
the user's options or required write privileges. Transactional streaming
re-resolution also used
the single-argument `loadTable` API, dropping both values again for every
micro-batch. In addition,
the streaming target relation did not retain the option bag used by the
command.
Catalogs that use options or `TableContext` privileges to select, construct,
or authorize a write
target therefore received incomplete information on these streaming paths.
### Does this PR introduce _any_ user-facing change?
Yes. Existing catalog-backed DataSource V2 streaming writes started with
`DataStreamWriter.toTable` now pass user options and the correct output-mode
privileges to target
loads and transactional reloads. The same option bag is retained for the
actual V2 streaming
write. There is no API change.
### How was this patch tested?
The complete affected suites were run with Java 17:
* `DataSourceV2OptionSuite`: 37 tests passed;
* `StreamingTransactionSuite`: 9 tests passed.
All 46 tests passed with no failures, exclusions, or filters.
`sql/Test/compile`, `sql/scalastyle`,
and `sql/Test/scalastyle` also passed.
The new tests use `InMemoryBaseTable` streaming tables and independently
assert the target-load
options/privileges, the options captured by the actual `StreamingWrite`'s
`LogicalWriteInfo`, and
the options/typed privileges observed by two transactional micro-batch
reloads.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex CLI 0.145.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
