val test_schema = DataType.fromJson(schema).asInstanceOf[StructType]
val session = SparkHelper.getSparkSession
val df1: DataFrame = session.read
.format("json")
.schema(test_schema)
.option("inferSchema","false")
.option("mode","FAILFAST")
.load("src/test/resources/*.gz")
df1.show(80)
I've had more success exporting the schema toJson and importing that.
Something like:
val df1: DataFrame = session.read
.format("json")
.schema(test_schema)
.option("inferSchema","false")
.option("mode","FAILFAST")
.load("src/test/resources/*.gz")
df1.show(80)
On Wed, Mar 28, 2018
The to String representation look like where "someName" is unique:
StructType(StructField("someName",StringType,true),
StructField("someName",StructType(StructField("someName",StructType(StructField("someName",StringType,true),
StructField("someName",StringType,true)),true),
I've been learning spark-sql and have been trying to export and import
some of the generated schemas to edit them. I've been writing the
schemas to strings like df1.schema.toString() and
df.schema.catalogString
But I've been having trouble loading the schemas created. Does anyone
know if it's
