alyssa-db opened a new pull request, #59247: URL: https://github.com/apache/spark/pull/59247
### What changes were proposed in this pull request? Add structured metadata for the configured trigger to `StreamingQueryProgress.json` and `prettyJson`, and preserve it through `QueryProgressEvent` reconstruction, event-log serialization, and the status store. An internal serializer handles the built-in trigger types. The existing `StreamingQueryProgress` constructor and public `Trigger` API are preserved. Legacy records and unknown custom trigger types omit the field. The status-store protobuf adds optional string field 13 to retain the JSON object. ### Why are the changes needed? Monitoring consumers need to distinguish trigger modes and inspect their configured intervals directly from streaming progress records. [SPARK-59986](https://issues.apache.org/jira/browse/SPARK-59986) ### Does this PR introduce _any_ user-facing change? Yes. Progress JSON gains an optional top-level `trigger` object: | Trigger | Example | |---|---| | ProcessingTime | `"trigger":{"type":"ProcessingTime","intervalMs":1000}` | | Continuous | `"trigger":{"type":"Continuous","intervalMs":2000}` | | RealTime | `"trigger":{"type":"RealTime","batchDurationMs":3000}` | | AvailableNow | `"trigger":{"type":"AvailableNow"}` | | OneTime | `"trigger":{"type":"OneTime"}` | Existing progress fields are preserved. Continuous execution currently records epochs without publishing progress records; its metadata is covered through progress construction and serialization. This PR preserves that publication behavior. ### How was this patch tested? Added 34 cases covering all built-in triggers, zero and default intervals, an interval exceeding 32-bit range, compact/pretty/Jackson JSON, event reconstruction, status-store persistence, legacy absence, custom trigger omission, live micro-batch and RealTime queries, and idle progress. Verified with Java 17 on upstream master `5926a10389e8eace9f655dc138f4bb30254f7bed`: ```sh build/sbt -Dsbt.override.build.repos=true -batch \ 'sql/testOnly *StreamingQueryStatusAndProgressSuite *KVStoreProtobufSerializerSuite *ContinuousQueryStatusAndProgressSuite' \ 'sql/testOnly *StreamingQueryListenerSuite -- -z serialization -z SPARK-59986' \ 'sql/testOnly *StreamRealTimeModeSuite -- -z SPARK-59986' ``` Results: **43/43**, **12/12**, and **1/1** passed, respectively; **56 tests passed, 0 failed**. ```sh build/sbt -Dsbt.override.build.repos=true -batch \ 'sql-api/scalaStyleOnCompile' \ 'sql/scalaStyleOnCompile' \ 'sql/scalaStyleOnTest' ``` All three checks reported **0 errors and 0 warnings**. ```sh build/mvn -B -pl sql/api scalafmt:format \ -Dscalafmt.skip=false -Dscalafmt.validateOnly=true -Dscalafmt.changedOnly=false ``` SQL API formatting validation passed. Dependency resolution used local Maven mirror settings. Protobuf schema compilation and `git diff --check` also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex CLI 0.154.0 This contribution is submitted under the project's Apache License 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
