Hi all
I want to ask how exactly it differs while reading >1 tb file on standalone
cluster vs yarn or mesos cluster ?
On Wednesday 6 January 2016, Gavin Yue wrote:
> I am trying to read json files following the example:
>
> val path = "examples/src/main/resources/jsonfile"val people =
> sqlCon
If you already know the schema, then you can run the read with the schema
parameter like this:
val path = "examples/src/main/resources/jsonfile"
val jsonSchema = StructType(
StructField("id",StringType,true) ::
StructField("reference",LongType,true) ::
StructField("deta
HI ,
You can try this
sqlContext.read.format("json").option("samplingRatio","0.1").load("path")
If it still takes time , feel free to experiment with the samplingRatio.
Thanks,
Vishnu
On Wed, Jan 6, 2016 at 12:43 PM, Gavin Yue wrote:
> I am trying to read json files following the example:
>
