siying commented on PR #59960: URL: https://github.com/apache/spark/pull/59960#issuecomment-6591765329
Relationship to existing batch SQL golden tests: These streaming files do **not** copy any batch input data or batch golden output verbatim. They use dedicated Parquet source/sink tables, `CREATE FLOW` / `REFRESH FLOW`, and (where relevant) an `event_time` watermark. For negative tests, the input rows are only minimal analysis/execution scaffolding: batch SQL produces rows, while the streaming test intentionally asserts a streaming-specific error. The parts that match existing batch coverage are the core SQL operator forms: | Streaming file | Closest batch SQL coverage | What matches | What differs | |---|---|---|---| | `available-now-cte.sql` | `cte.sql` | Non-recursive `WITH ... SELECT` shape | Tables, rows, flow wrapper, and output rows | | `available-now-rollup.sql`, `available-now-cube.sql` | `group-analytics.sql` | One- and two-column `ROLLUP` / `CUBE` aggregation shape | Adds `event_time` as a separate grouping key and watermark; data and output rows are streaming-specific | | `available-now-distinct-aggregation-errors.sql` | `group-by.sql`, `group-analytics.sql` | `COUNT(DISTINCT ...)` expression | Batch returns aggregates; streaming asserts the distinct-aggregation error even with a watermark | | `available-now-join-asof-errors.sql` | `join-asof-*.sql` | Static-left / right-side ASOF join form and match condition | Right input is streaming and watermarked; output is the unsupported-right-side error | | `available-now-join-nearest-by-errors.sql` | `join-nearest-by.sql` | `APPROX NEAREST 1 BY DISTANCE abs(...)` form | Smaller schemas/data; one side is streaming and output is `NEAREST_BY_JOIN.STREAMING_NOT_SUPPORTED` | | `available-now-join-outer-errors.sql` | `outer-join.sql` | Full outer equi-join form | Stream-static orientation and its streaming-specific error | | `available-now-set-operation-errors.sql` | `union.sql`, `intersect*.sql`, `except*.sql` | `UNION ALL`, `INTERSECT`, and `EXCEPT` forms | Mixed streaming/static inputs and streaming-specific errors | | `available-now-append-mode-errors.sql` | `union.sql`, SELECT DISTINCT coverage | `UNION` distinct and `SELECT DISTINCT` forms | Watermarked event time is intentionally projected out, so the state remains unbounded and append mode fails | | `available-now-sort-errors.sql`, `available-now-offset-errors.sql` | `order-by.sql`, offset coverage in subquery tests | `ORDER BY` and `OFFSET` clauses | Watermarked streaming input and unsupported-operation output | | `available-now-subquery-errors.sql` | EXISTS-subquery inputs | Correlated `EXISTS` shape | The streaming relation is inside the subquery, so output is `INVALID_SUBQUERY_EXPRESSION.STREAMING_QUERY` | | `available-now-window-errors.sql` | `window.sql` | `ROW_NUMBER() OVER (ORDER BY event_time)` | Watermarked streaming input and `NON_TIME_WINDOW_NOT_SUPPORTED_IN_STREAMING` output | | `available-now-tablesample-errors.sql` | `tablesample-negative.sql` only covers invalid batch fractions | Standard valid `TABLESAMPLE (50 PERCENT)` syntax | There is no matching positive batch SQL-golden statement/result; streaming asserts sampling is unsupported | | `available-now-monotonic-id-errors.sql` | No direct batch SQL-golden counterpart | The expression itself | Streaming-specific unsupported-expression output | | `available-now-stream-static-window-aggregation.sql` | No direct batch SQL-golden counterpart | N/A | Purpose-built streaming workload covering watermark advancement, checkpoint reuse, stream-static join, and window finalization | Thus the tests deliberately mirror batch **feature syntax/query shape**, but not batch **input datasets or result rows**. The positive streaming tests validate equivalent relational semantics under AvailableNow; the negative tests validate the exact streaming rejection point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
