[
https://issues.apache.org/jira/browse/SPARK-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491281#comment-16491281
]
Arun Mahadevan commented on SPARK-23887:
----------------------------------------
We could probably invoke "ProgressReporter.finishTrigger" after each epoch.
This would update the query execution stats and post StreamingQueryListener
events. However I am not clear on how the SQL metrics could be updated since it
relies on accumulator and the accumulators might not be updated unless the task
completes. We could probably post some special event to the DAGScheduler to
update the accumulators or need to figure out some other mechanisms.
[~joseph.torres] [~tdas] , what do you think?
> update query progress
> ---------------------
>
> Key: SPARK-23887
> URL: https://issues.apache.org/jira/browse/SPARK-23887
> Project: Spark
> Issue Type: Sub-task
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Jose Torres
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
