Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22353
So you need a way to reliably report some extra information like file path
in the event logs, but don't want to show it in the UI as it maybe too long.
Basically we shouldn't put such information in the event logs if it's not
used in the UI, and we should build a new mechanism to make Spark easier to be
analyzed. Also keep it mind that event logs are not reliable, Spark may drop
some events if the event bus is too busy.
I'm ok to add it back to the event logs since it was there before, but
please don't add `metadata` to `SparkPlan`, we can pattern match the
`FileSourceScanExec` in `SparkPlanInfo.fromSparkPlan`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
