Ngone51 commented on issue #26195: [SPARK-29537][SQL] throw exception when user defined a wrong base path URL: https://github.com/apache/spark/pull/26195#issuecomment-559097391 @cloud-fan > when did we add basePath? I have no idea what it is... After tracking the code history, I think it was introduced in #9651 from Spark 1.6. And here's documentation for `basePath`(ref: http://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery): > Starting from Spark 1.6.0, partition discovery only finds partitions under the given paths by default. For the above example, if users pass path/to/table/gender=male to either SparkSession.read.parquet or SparkSession.read.load, gender will not be considered as a partitioning column. If users need to specify the base path that partition discovery should start with, they can set basePath in the data source options. For example, when path/to/table/gender=male is the path of the data and users set basePath to path/to/table/, gender will be a partitioning column.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
