Vaclav Kosar created SPARK-24933:
------------------------------------
Summary: SinkProgress should report written rows
Key: SPARK-24933
URL: https://issues.apache.org/jira/browse/SPARK-24933
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.3.1
Reporter: Vaclav Kosar
SinkProgress should report similar properties like SourceProgress as long as
they are available for given Sink. Count of written rows is metric availble for
all Sinks. Since relevant progress information is with respect to commited
rows, ideal object to carry this info is WriterCommitMessage. For brevity the
implementation will focus only on Sinks with API V2 and on Micro Batch mode.
Implemention for Continuous mode will be provided at later date.
h4. Before
{code}
{"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317"}
{code}
h4. After
{code}
{"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317","numOutputRows":5000}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
