Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20856 @HyukjinKwon thanks for your great analysis. I agree with you that the proposed fix is more a "workaround" than a real fix for the issue we have here. The main problem here as you pointed out is that we have a bad (invalid?) `FileSourceScanExec` on the executors. Probably this has never been an issue as on the executors we accessed only some properties which were correctly populated and we assumed that the other operation would have been performed only on driver side. I think the cleanest approach (not sure it is entirely feasible) would be to choose one of the following option: - check that all exec expression (in this case `FileSourceScanExec`) are working properly both on driver and executor side; - define which operation/attributes can be accessed on executor side too and which only on driver side, document it and enforce it (if feasible). What do you think?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org