Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19497
@HyukjinKwon Thanks for clarifying.
The way I look at it is:
`saveAsHadoopFile` is explicitly referring to `Output the RDD to any
Hadoop-supported file system` in its description (and name) - and so valid
`Path` is a reasonable requirement.
Additionally, in createPathFromString for `path == null` we are explicitly
throwing `IllegalArgumentException` (`new Path` will do the same now, but I
think this changed in past where it used to result in NPE ?).
The subsequent `val outputPath = new Path(path)` will do that for other
invalid input paths as well.
In contrast `saveAsHadoopDataset` is not related to file system but `Output
the RDD to any Hadoop-supported storage system` : where output being a valid
`Path` is not a requirement.
Having said that, we can always iterate in a jira if you feel there is some
confusion - it is always better to be explicitly clear about the interfaces we
expose and support !
Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
