The existing attempt used the Rapidjson library to do the parsing. Unfortunately, the Rapidjson API is not very convenient for Impala because it returns typed data, i.e., it internally converts to float/double/int/whatever which is problematic for decimal (among others). Ideally, we would use the same Impala code to convert data types from strings.
On Tue, Apr 18, 2017 at 9:27 AM, Tim Armstrong <[email protected]> wrote: > Seems like useful functionality that would be great to have in Impala. > There was an earlier attempt to do this that didn't make it in - I'm not > sure that the approach was quite right: > https://gerrit.cloudera.org/#/c/1201/1 . I'm not sure what the exact > problems were but I remember we didn't think it was quite the right > approach. > > I think we'd need to talk through a design first because there are a lot of > considerations and we want to make sure to get it right. I had some initial > questions that I'd want to think through before adding a JSON scanner. > > - What JSON does it accept? > - How do we declare a table schema and map it to the JSON > - How does it handle missing or extra fields - does it just return null > or drop the fields? What if the field type is wrong? > - How do the numeric types work? JSON only supports floating point, but > I think many people would like to store higher-precision decimal or > 64-bit > integer types (which is technically outside of the JSON standard). > - Will it support codegen? If not, is it written in a way that allows it > in future? > > Cheers, > Tim > > - Tim > > On Tue, Apr 18, 2017 at 8:52 AM, Jim Apple <[email protected]> wrote: > > > On Mon, Apr 17, 2017 at 8:02 PM, yu feng <[email protected]> wrote: > > > > > Hi impala community: > > > I am Newly join to Impala, > > > > > > Welcome! > > > > I want to know what is the attitude of impala > > > community for supporting json format. > > > > > > I am in favor of it. I am only one person, though - anybody else object > to > > JSON support? > > > > If this match the roadmap, maybe I > > > can make some contribution. > > > > > > > I do not recall much talk about Apache Impala's roadmap since we joined > the > > ASF. Perhaps I missed a thread about it? > > >
