[
https://issues.apache.org/jira/browse/SPARK-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-16924.
----------------------------------
Resolution: Invalid
I am resolving this JIRA. Please refer the discussion in the PR and reopen this
if I misunderstood.
> DataStreamReader can not support option("inferSchema", true/false) for csv
> and json file source
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-16924
> URL: https://issues.apache.org/jira/browse/SPARK-16924
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Xin Wu
>
> Currently DataStreamReader can not support option("inferSchema", true|false)
> for csv and json file source. It only takes SQLConf setting
> "spark.sql.streaming.schemaInference", which needs to be set at session
> level.
> For example:
> {code}
> scala> val in = spark.readStream.format("json").option("inferSchema",
> true).load("/Users/xinwu/spark-test/data/json/t1")
> java.lang.IllegalArgumentException: Schema must be specified when creating a
> streaming source DataFrame. If some files already exist in the directory,
> then depending on the file format you may be able to create a static
> DataFrame on that directory with 'spark.read.load(directory)' and infer
> schema from it.
> at
> org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:223)
> at
> org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:80)
> at
> org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:80)
> at
> org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
> at
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
> at
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
> ... 48 elided
> scala> val in = spark.readStream.format("csv").option("inferSchema",
> true).load("/Users/xinwu/spark-test/data/csv")
> java.lang.IllegalArgumentException: Schema must be specified when creating a
> streaming source DataFrame. If some files already exist in the directory,
> then depending on the file format you may be able to create a static
> DataFrame on that directory with 'spark.read.load(directory)' and infer
> schema from it.
> at
> org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:223)
> at
> org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:80)
> at
> org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:80)
> at
> org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30)
> at
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:142)
> at
> org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:153)
> ... 48 elided
> {code}
> In the example, even though users specify the option("inferSchema", true), it
> does not take it. But for batch data, DataFrameReader can take it:
> {code}
> scala> val in = spark.read.format("csv").option("header",
> true).option("inferSchema", true).load("/Users/xinwu/spark-test/data/csv1")
> in: org.apache.spark.sql.DataFrame = [signal: string, flash: int]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
