Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22688
Another concern is, it doesn't sound to me straightforward that readsupport
is created and executed in write path.
So, at least, we should just read the schema when it's needed (when save is
append mode). The problem still exists here if we go ahead in this way. If the
target path does not exist in append mode, again, there is no way to read the
schema.
If we disallow to create the target path (or target table) and throws an
exception, then it might make sense but I think this is a breaking change in
save mode.
Another option we should think is to add an option to control it at least
or add an interface to the write side.
I hope we can at least take this out of 2.4 and bring it to 3.0 after more
discussions.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
