Github user boy-uber commented on the issue:
https://github.com/apache/spark/pull/22429
> @boy-uber, for structured streaming, let's do it out of this PR. I think
the actual change of this PR can be small (1.). We can change this API for
structured streaming later if needed since this is just an internal private
API. Change (2.) can be shared I guess even if we go for structured streaming
stuff.
Hi @HyukjinKwon thanks for the comment! My original suggestion was not
related to structured streaming. It was to support structured logging for Spark
plan, for example, get the Spark plan as a json blob, and send it to Spark
Listener. This will make it easy to Spark users to write a Spark Listener
plugin to get the plan and process the json in their code. Then people could
parse and analyze a large number of Spark applications based on their plans.
One use case for this is using machine learning to predict resource usage based
on the Spark plans. This may not be related to this PR. If you or other people
are interested, we could talk offline.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
