Github user arunmahadevan commented on the issue:
https://github.com/apache/spark/pull/21721
There are many unknowns to be figured out for continuous mode. Though the
way to capture the metrics would be different for continuous execution, the
interface of whats reported is not expected to change. Given that we already
report progress for micro-batch and as a user of spark the changes in the patch
are quite useful to report custom metrics for what works right now and since it
does not impact other parts of DataSourceV2 apis (only the sources that wants
to report custom metrics would add the traits) IMO, we can keep this and
continue to investigate in a time-bound manner of how to capture metrics for
continuous mode.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
