siying commented on PR #60032:
URL: https://github.com/apache/spark/pull/60032#issuecomment-6596573825
Independent check of the result goldens at `77185e0`:
**What was checked**
- All 16 new `results/streaming/available-now-*.sql.out` files. Every
DDL/DML, `CREATE FLOW`, and `REFRESH FLOW` block has empty output with no
error, and the 16 matching analyzer goldens (236 blocks) contain no error
output.
- All 36 `SELECT` result blocks (21 sink snapshots and 15 parity checks)
were recomputed with DuckDB from the same fixture data and compared to the
goldens: rows, `ORDER BY` order (NULLS FIRST), and the schema line. Spark-only
syntax was translated (`LATERAL VIEW OUTER POSEXPLODE`, pipe operators, `SELECT
* EXCEPT`, SQL UDF/UDTF). No `ORDER BY` has ties, so the ordered outputs are
deterministic.
- Streaming behavior was modeled explicitly, after confirming it in the
source at this commit:
- Append-mode watermarked aggregations emit a group only once `event_time
<= watermark`. AvailableNow runs the trailing no-data batch that does this
eviction, so the final watermark is the max event time minus 1s (9s here).
- PIVOT is planned as two chained streaming aggregations. With watermark
propagation, late rows are judged against the previous batch's watermark and
eviction uses the current one, so rows evicted upstream are not dropped
downstream.
- On the second `REFRESH FLOW`, the 9s watermark is restored from the
checkpoint, so `(00:00:01, books, 100)` is late and dropped.
- For each of the 15 parity pairs, the batch reference is the `CREATE FLOW`
query with `STREAM(...)` removed (and `WATERMARK` for HAVING), reads the same
fixture tables, and is compared in both directions with `EXCEPT ALL`. Every
sink has 1-6 rows, so no comparison is vacuous.
**Conclusion**
All 36 result blocks are semantically correct for their queries and input
data, and all 15 `complete_output_identical` results are correctly `true`.
Notable cases:
- The watermarked aggregations keep the 00:00:10 group in state rather than
emitting it, and aggregate `FILTER` gives `NULL` (not 0) when no row qualifies.
- `LATERAL VIEW OUTER POSEXPLODE` keeps ids 3 and 4 with NULLs and 0-based
positions; `UNNEST ... WITH ORDINALITY` drops them and is 1-based.
- The second PIVOT run drops the late row, completes 00:00:10 as `(11, 7)`,
emits 00:00:11 as `(13, 17)`, and keeps 00:00:20 in state.
The local JUnit report for this commit shows `StreamingSQLQueryTestSuite --
-z available-now-` passing 42/42 (21 execution + 21 analyzer). GitHub Actions
for this commit was still running at the time of this check.
---
<!-- GITHUB_MCP_FOOTER: This attribution is automatically appended by GitHub
MCP. -->
_This comment was generated with [GitHub MCP](http://go/mcps)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
