GitHub user tdas reopened a pull request:
https://github.com/apache/spark/pull/16258
[SPARK-18834][SS] Expose event time stats through StreamingQueryProgress
## What changes were proposed in this pull request?
- Changed `StreamingQueryProgress.watermark` to
`StreamingQueryProgress.queryTimestamps` which is a `Map[String, String]`
containing the following keys: "eventTime.max", "eventTime.min",
"eventTime.avg", "processingTime", "watermark". All of them UTC formatted
strings.
- Renamed `StreamingQuery.timestamp` to
`StreamingQueryProgress.triggerTimestamp` to differentiate from
`queryTimestamps`. It has the timestamp of when the trigger was started.
## How was this patch tested?
Updated tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-18834
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16258.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16258
----
commit e9d34edb5386313c580e64fbd6c279d7f79dc381
Author: Tathagata Das <[email protected]>
Date: 2016-12-12T23:32:50Z
Added timestamps to progress
commit a14efdd71dea7da9c8a7e28a6e7f3463e2af1c1a
Author: Tathagata Das <[email protected]>
Date: 2016-12-13T01:00:26Z
Refactored
commit 9a2e941a451f212927ee7edcead8c866a619e159
Author: Tathagata Das <[email protected]>
Date: 2016-12-13T01:08:06Z
Minor changes
commit 8de17d26c947b42382ede2e6d83e7bb24fffc05a
Author: Tathagata Das <[email protected]>
Date: 2016-12-13T01:14:03Z
Minor change
commit 89389920c54b8927f397c4643245b75e0ef36740
Author: Tathagata Das <[email protected]>
Date: 2016-12-13T01:14:38Z
Add line
commit b59ab8083de3f2441133fed35658dea39cd4a759
Author: Tathagata Das <[email protected]>
Date: 2016-12-13T04:08:44Z
Addressed comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
