siying opened a new pull request, #59935:
URL: https://github.com/apache/spark/pull/59935
### What changes were proposed in this pull request?
This PR relands #59665, which was reverted by commit
`6cd6698511134afe5f90694ae8e290235f6b8108` after post-merge CI failures.
It adds `StreamingSQLQueryTestSuite`, a `SQLQueryTestSuite` subclass that
lets SQL golden tests
exercise Structured Streaming flows. The test harness:
- parses `CREATE FLOW` statements and retains their streaming logical plans;
- recognizes test-only, case-insensitive `REFRESH FLOW` statements;
- runs the saved flow with `Trigger.AvailableNow` into an existing table and
reuses its checkpoint
across refreshes; and
- normalizes event-time watermark UUIDs in golden output.
It adds golden coverage for writing to a table, stream-static joins,
streaming unions, windowed
aggregations with watermarks, and multiple chained flows.
This reland also fixes the post-merge failure by accepting both
`InsertIntoStatement` and
`UnresolvedInsert` flow operations, and both `UnresolvedRelation` and
`UnresolvedInsertTarget`
targets. This matches the dual-shape handling used by pipeline SQL
registration after #59750.
### Why are the changes needed?
SQL golden tests cannot normally execute a query containing streaming
sources because batch
execution rejects its streaming logical plan. Running the saved `CREATE
FLOW` query through
`writeStream` lets SQL tests validate Structured Streaming behavior and
inspect the resulting table
with regular SQL.
The original PR was tested on a base from before #59750. That parser
follow-up restored
`InsertIntoStatement` as the parsed plan for static INSERT targets. The
original change later
merged cleanly at the text level, but its harness still matched only
`UnresolvedInsert`, causing 12
`StreamingSQLQueryTestSuite` cases to fail in post-merge CI. Supporting both
parser plan shapes
fixes that semantic integration issue.
### Does this PR introduce _any_ user-facing change?
No. This only changes the SQL test harness and its test coverage.
### How was this patch tested?
The original PR passed these focused tests before the parser follow-up was
merged:
```
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt \
'sql/testOnly org.apache.spark.sql.StreamingSQLQueryTestSuite -- -z "test
triggered flow"'
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt \
'sql/testOnly org.apache.spark.sql.StreamingSQLQueryTestSuite -- -z
"streaming/available-now"'
```
The second command runs five execution golden tests and their five analyzer
tests.
For this reland, the focused parser test was attempted locally, but SBT
could not resolve the Maven
repository hosts while downloading the newly required build dependencies, so
the build stopped
before project loading. The compatibility path is exercised by `test
triggered flow parsing` and
all ten AvailableNow execution/analyzer golden tests in PR CI.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
