Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem Unfortunately, I do not have enough context about how sql writes
paths to comment on when which branch will be taken.
I was going purely from code in `HadoopMapReduceCommitProtocol`, if
(assuming) the methods are invoked, they will throw NPE due to null path. It
might be possible that in case of outputformat's without directory the method
is never invoked - or it could in future evolution of code even if as an impl
detail it currently does not or ... : I cannot unfortunately comment
authoritatively about sql data sources.
Having said that, this PR is definitely a big improvement over what
currently exists in spark - it fixes spark core and also allows a bunch of
other data source's (like shc for example, and I am sure others too - phoenix
perhaps ?) to work when they currently fail.
I want someone more knowledgable with spark sql to take a look - perhaps
minor enhancements will make it work for all cases (if it does not already).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
