GitHub user caneGuy opened a pull request:
https://github.com/apache/spark/pull/20833
[SPARK-23692][SQL]Print metadata of files when infer schema failed
## What changes were proposed in this pull request?
A trivial modify.
Currently, when we had no input files to infer schema,we will throw below
exception.
For some users it may be misleading.If we can print files' metadata it will
be more clearer.
`Caused by: org.apache.spark.sql.AnalysisException: Unable to infer schema
for Parquet. It must be specified manually.;
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$8.apply(DataSource.scala:189)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$8.apply(DataSource.scala:189)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$getOrInferFileFormatSchema(DataSource.scala:188)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:387)
at
org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
at
org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:441)
at
org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:425)
at
com.xiaomi.matrix.pipeline.jobspark.importer.MatrixAdEventDailyImportJob.<init>(MatrixAdEventDailyImportJob.scala:18)`
## How was this patch tested?
Exsist tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/caneGuy/spark zhoukang/modify-log
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20833.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20833
----
commit 91e53d87b0f5503ba7e9c9bb6a7258ef30f87c9d
Author: zhoukang <zhoukang199191@...>
Date: 2018-03-15T08:53:06Z
Print metadata of files when infer schema failed
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
