{"IFAM":"EQR","KTM":1430006400000,"COL":21,"DATA":[{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"}]}
{"IFAM":"EQR","KTM":1430006400000,"COL":22,"DATA":[{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"},{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"}]}________________________________ From: mélanie gallois [[email protected]] Sent: Wednesday, July 29, 2015 8:10 AM To: Young, Matthew T Cc: [email protected] Subject: Re: How to read a Json file with a specific format? Can you give an example with my extract? Mélanie Gallois 2015-07-29 16:55 GMT+02:00 Young, Matthew T <[email protected]<mailto:[email protected]>>: The built-in Spark JSON functionality cannot read normal JSON arrays. The format it expects is a bunch of individual JSON objects without any outer array syntax, with one complete JSON object per line of the input file. AFAIK your options are to read the JSON in the driver and parallelize it out to the workers or to fix your input file to match the spec. For one-off conversions I usually use a combination of jq and regex-replaces to get the source file in the right format. ________________________________________ From: SparknewUser [[email protected]<mailto:[email protected]>] Sent: Wednesday, July 29, 2015 6:37 AM To: [email protected]<mailto:[email protected]> Subject: How to read a Json file with a specific format? I'm trying to read a Json file which is like : [ {"IFAM":"EQR","KTM":1430006400000,"COL":21,"DATA":[{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ]} ,{"IFAM":"EQR","KTM":1430006400000,"COL":22,"DATA":[{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ,{"MLrate":"30","Nrout":"0","up":null,"Crate":"2"} ]} ] I've tried the command: val df = sqlContext.read.json("namefile") df.show() But this does not work : my columns are not recognized... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-a-Json-file-with-a-specific-format-tp24061.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected]<mailto:[email protected]> For additional commands, e-mail: [email protected]<mailto:[email protected]> -- Mélanie --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
