[ https://issues.apache.org/jira/browse/SPARK-24933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562044#comment-16562044 ]
Apache Spark commented on SPARK-24933: -------------------------------------- User 'vackosar' has created a pull request for this issue: https://github.com/apache/spark/pull/21919 > SinkProgress should report written rows > --------------------------------------- > > Key: SPARK-24933 > URL: https://issues.apache.org/jira/browse/SPARK-24933 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.3.1 > Reporter: Vaclav Kosar > Priority: Major > > SinkProgress should report similar properties like SourceProgress as long as > they are available for given Sink. Count of written rows is metric availble > for all Sinks. Since relevant progress information is with respect to > commited rows, ideal object to carry this info is WriterCommitMessage. For > brevity the implementation will focus only on Sinks with API V2 and on Micro > Batch mode. Implemention for Continuous mode will be provided at later date. > h4. Before > {code} > {"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317"} > {code} > h4. After > {code} > {"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317","numOutputRows":5000} > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org