Note that if you specify the schema that you expect when reading JSON you
basically get the "relaxed" mode that you are asking for. Records that
don't match will end up with nulls.
The problem here is Spark SQL knows that the operation you are asking for
is invalid given the set of data you let i
Thanks. Once you create the jira just reply to this email with the link.
On Wednesday, March 2, 2016, Ewan Leith wrote:
> Thanks, I'll create the JIRA for it. Happy to help contribute to a patch if
> we can, not sure if my own scala skills will be up to it but perhaps one of
> my colleagues' w
Thanks, I'll create the JIRA for it. Happy to help contribute to a patch if we
can, not sure if my own scala skills will be up to it but perhaps one of my
colleagues' will :)
Ewan
I don't think that exists right now, but it's definitely a good option to have.
I myself have run into this issue
I don't think that exists right now, but it's definitely a good option to
have. I myself have run into this issue a few times.
Can you create a JIRA ticket so we can track it? Would be even better if
you are interested in working on a patch! Thanks.
On Wed, Mar 2, 2016 at 11:51 AM, Ewan Leith
w
Hi Reynold, yes that would be perfect for our use case.
I assume it doesn't exist though, otherwise I really need to go re-read the
docs!
Thanks to both of you for replying by the way, I know you must be hugely busy.
Ewan
Are you looking for "relaxed" mode that simply return nulls for fields t
Are you looking for "relaxed" mode that simply return nulls for fields that
doesn't exist or have incompatible schema?
On Wed, Mar 2, 2016 at 11:12 AM, Ewan Leith
wrote:
> Thanks Michael, it's not a great example really, as the data I'm working with
> has some source files that do fit the sche
Thanks Michael, it's not a great example really, as the data I'm working with
has some source files that do fit the schema, and some that don't (out of
millions that do work, perhaps 10 might not).
In an ideal world for us the select would probably return the valid records
only.
We're trying o
-dev +user
StructType(StructField(data,ArrayType(StructType(StructField(
> *stuff,ArrayType(*StructType(StructField(onetype,ArrayType(StructType(StructField(id,LongType,true),
> StructField(name,StringType,true)),true),true), StructField(othertype,
> ArrayType(StructType(StructField(company,String