GitHub user vackosar opened a pull request:
https://github.com/apache/spark/pull/21919
[SPARK-24933][SS] Report numOutputRows in SinkProgress via WrittenComâ¦
## What changes were proposed in this pull request?
SinkProgress should report similar properties like SourceProgress as long
as they are available for given Sink. Count of written rows is metric availble
for all Sinks. Since relevant progress information is with respect to commited
rows, ideal object to carry this info is WriterCommitMessage. For brevity the
implementation will focus only on Sinks with API V2 and on Micro Batch mode.
Implemention for Continuous mode will be provided at later date.
### Before
```
{"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317"}
```
### After
```
{"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317","numOutputRows":5000}
```
### This PR is related to:
- https://issues.apache.org/jira/browse/SPARK-24647
- https://issues.apache.org/jira/browse/SPARK-21313
## How was this patch tested?
Existing and new unit tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AbsaOSS/spark
feature/SPARK-24933-numOutputRows
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21919.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21919
----
commit 8c513e4765691bc5e31914066b05091f90096f88
Author: Kosar, Vaclav: Functions Transformation <vaclav.kosar@...>
Date: 2018-06-26T09:41:25Z
[SPARK-24933][SS] Report numOutputRows in SinkProgress via
WrittenCommitMessage.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
