Hi Justin,
It is expected. We do not check if the provided schema matches rows since
all rows need to be scanned to give a correct answer.
Thanks,
Yin
On Fri, Feb 13, 2015 at 1:33 PM, Justin Pihony <justin.pih...@gmail.com>
wrote:
> Per the documentation:
>
> It is important to make sure that the structure of every Row of the
> provided RDD matches the provided schema. Otherwise, there will be runtime
> exception.
>
> However, it appears that this is not being enforced.
>
> import org.apache.spark.sql._
> val sqlContext = new SqlContext(sc)
> val struct = StructType(List(StructField("test", BooleanType, true)))
> val myData = sc.parallelize(List(Row(0), Row(true), Row("stuff")))
> val schemaData = sqlContext.applySchema(myData, struct) //No error
> schemaData.collect()(0).getBoolean(0) //Only now will I receive an error
>
> Is this expected or a bug?
>
> Thanks,
> Justin
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/SQLContext-applySchema-strictness-tp21650.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
