[
https://issues.apache.org/jira/browse/SPARK-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623083#comment-14623083
]
Reynold Xin commented on SPARK-8990:
------------------------------------
I merged this into master for 1.5, but leaving the ticket open for 1.4.
> DataFrameReader.parquet() ignores user specified data source options
> --------------------------------------------------------------------
>
> Key: SPARK-8990
> URL: https://issues.apache.org/jira/browse/SPARK-8990
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Cheng Lian
> Assignee: Cheng Lian
>
> A bad consequence of this is that {{sqlContext.read.parquet(path)}} always do
> schema merging. For example:
> {code}
> import sqlContext._
> import sqlContext.implicits._
> val path = "s3n://my-bucket/parquet/tiny"
> range(0, 10).coalesce(1).write.mode("overwrite").parquet(path)
> // Explicitly disables schema merging
> read.option("mergeSchema", "false").format("parquet").load(path)
> {code}
> However, we still see all files are opened for schema discovery:
> {noformat}
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening
> 's3n://databricks-lian/parquet/tiny/_metadata' for reading
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key
> 'parquet/tiny/_metadata' for reading at position '314'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening
> 's3n://databricks-lian/parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet'
> for reading
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening
> 's3n://databricks-lian/parquet/tiny/_common_metadata' for reading
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key
> 'parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet'
> for reading at position '345'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key
> 'parquet/tiny/_common_metadata' for reading at position '191'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key
> 'parquet/tiny/_metadata' for reading at position '4'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key
> 'parquet/tiny/part-r-00000-da490c43-15e2-46b5-95ff-4863e6ab1cc4.gz.parquet'
> for reading at position '97'
> 15/07/10 14:46:52 INFO s3native.NativeS3FileSystem: Opening key
> 'parquet/tiny/_common_metadata' for reading at position '4'
> {noformat}
> To workaround this issue, use the following instead:
> {noformat}
> sqlContext.read.option("mergeSchema", "false").format("parquet").load(path)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
