Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21721
Can someone write a design doc for the metrics support? I think this is an
important feature for data source v2 and we need to be careful here. The design
doc should explain how custom metrics fit in the abstraction of data source v2
API, how the metrics API would look like for batch, micro-batch and continuous
(I feel metrics is also important for batch sources), and how the sources
report metrics physically (via task complete event? via heartbeat? via RPC?).
@rxin just sent an email to the dev list about the data source v2 API
abstraction, it would be great if you guys can kick it and talk about the
metrics support.
It's very likely that the custom metrics API would be replaced by something
totally different after we finish the design. I don't think we should rush into
something that works but not well designed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
