GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/15954
[WIP][SPARK-18516][SQL] Split state and progress in streaming
This PR separates the status of a `StreamingQuery` into two separate APIs:
- `status` - describes the status of a `StreamingQuery` at this moment,
including what phase of processing is currently happening and if data is
available.
- `recentProgress` - an array of statistics about the most recent
microbatches that have executed.
A recent progress contains the following information:
```
{
"id" : "2be8670a-fce1-4859-a530-748f29553bb6",
"name" : "query-29",
"timestamp" : 1479705392724,
"inputRecordsPerSecond" : 230.76923076923077,
"processedRecordsPerSecond" : 10.869565217391303,
"durationMs" : {
"triggerExecution" : 276,
"queryPlanning" : 3,
"getBatch" : 5,
"getOffset" : 3,
"addBatch" : 234,
"walCommit" : 30
},
"currentWatermark" : 0,
"stateOperators" : [ ],
"sources" : [ {
"description" : "KafkaSource[Subscribe[topic-14]]",
"startOffset" : {
"topic-14" : {
"2" : 0,
"4" : 1,
"1" : 0,
"3" : 0,
"0" : 0
}
},
"endOffset" : {
"topic-14" : {
"2" : 1,
"4" : 2,
"1" : 0,
"3" : 0,
"0" : 1
}
},
"numRecords" : 3,
"inputRecordsPerSecond" : 230.76923076923077,
"processedRecordsPerSecond" : 10.869565217391303
} ]
}
```
Additionally, in order to make it possible to correlate progress updates
across restarts, we change the `id` field from an integer that is unique with
in the JVM to a `UUID` that is globally unique.
This is a work in progress until we finish the following:
- [ ] Port the remaining tests to the new API
- [ ] Store the query UUID in the checkpoint directory
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark queryProgress
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15954.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15954
----
commit ae8a37c2b1251eeb5ed60ebaf2b59ef7b7ea2af4
Author: Michael Armbrust <[email protected]>
Date: 2016-11-20T01:44:01Z
[SPARK-18516] Split state and progress in streaming
commit 527c8d67cf8d4e27d02377ecce90ec35e0537aa2
Author: Michael Armbrust <[email protected]>
Date: 2016-11-21T02:37:51Z
Merge remote-tracking branch 'origin/master' into queryProgress
commit 213081ac60c70a9c8bfb0ae9d0b5d255f5b12e5f
Author: Michael Armbrust <[email protected]>
Date: 2016-11-21T03:47:30Z
fix python
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
