Hi Spark Devs, I am reaching out to gain understanding for accessing data source CustomMetric in Java.
CustomTaskMetric documentation points out that; "The final result will be shown up in the data source scan operator in Spark UI." SQLAppStatusListenerSuite in the test cases uses physicalPlan to access the information and using scala. I've explored using SparkListener and StreamingQueryListener but with no success. They seem to give out a lot of information but no CustomMetrics so far. The Spark UI page works just fine and I see the metrics there but I would like to have access to them while the query is running to be able to represent some fine statistics between the MicroBatches. The project I work with resides here as an open source one https://github.com/teragrep/pth_06 Any replies appreciated. Kind regards, Mikko Kortelainen
