I'm taking a look at the JIRA you filed :DRILL-3533. If I could not find a fix soon, I'll ask someone in our team to take a further look.
On Tue, Jul 21, 2015 at 2:50 PM, Stefán Baxter <[email protected]> wrote: > Hi, > > I would be super thankful if someone could fix this for us as it's impeding > our devaluation and testing (Assuming a fix is fairly doable). > I would be more than happy to take a look at it myself but I'm nowhere near > having the required understanding of your inner workings. (I have built the > project though an and can be running master/head) > > Regards, > -Stefán > > > On Tue, Jul 21, 2015 at 6:16 PM, Jacques Nadeau <[email protected]> > wrote: > > > Whoa, something strange is going on. > > > > Expected > > select * from dfs.tmp.test2; > > +-----------------+ > > | dimensions | > > +-----------------+ > > | {"adults":"A"} | > > +-----------------+ > > > > select p.dimensions as b from dfs.tmp.test2 p; > > +-----------------+ > > | b | > > +-----------------+ > > | {"adults":"A"} | > > +-----------------+ > > > > select p.dimensions as b from dfs.tmp.test2 p; > > +-----------------+ > > | b | > > +-----------------+ > > | {"adults":"A"} | > > +-----------------+ > > > > select p.dimensions.nonexist as a from dfs.tmp.test2 p; > > +-------+ > > | b | > > +-------+ > > | null | > > +-------+ > > > > Unexpected > > > > select p.dimensions.nonexist as a, p.dimensions as b from dfs.tmp.test2 > p; > > +----+-------+ > > | a | b | > > +----+-------+ > > | A | null | > > +----+-------+ > > > > <<both wrong>> > > > > select p.dimensions as b, p.dimensions.nonexist as a from dfs.tmp.test2 > p; > > +-------+-------+ > > | b | a | > > +-------+-------+ > > | null | null | > > +-------+-------+ > > > > << b is wrong, a is correct>> > > > > Stefan, not sure what is going on here. It looks like an issue with > > resolution of a nonexistent field in a multilevel structure. It exists > > whether it is Parquet or JSON. Can you file a JIRA and mark it Critical? > > > > > > On Tue, Jul 21, 2015 at 10:46 AM, Stefán Baxter < > [email protected] > > > > > wrote: > > > > > Hi, > > > > > > all kidding aside then this renders Drill+Parquet unusable for us at > the > > > moment. > > > > > > If there is a quick fix then please let me know. :) > > > > > > Regards, > > > -Stefan > > > > > > On Tue, Jul 21, 2015 at 5:37 PM, Stefán Baxter < > > [email protected]> > > > wrote: > > > > > > > Well, > > > > > > > > After some more testing it appears that this has nothing to do with > > trim. > > > > (any non existing nested-value will be pushed aside) > > > > > > > > select p.dimensions.budgetLevel as `field1`, > lower(p.dimensions.adults) > > > as > > > > `field2` from dfs.tmp.`/test/0_0_0.parquet` as p; > > > > > > > > also returns: > > > > +---------+---------+ > > > > | field1 | field2 | > > > > +---------+---------+ > > > > | a | null | > > > > +---------+---------+ > > > > > > > > I just as puzzled though :) > > > > > > > > Regards, > > > > -Stefan > > > > > > > > On Tue, Jul 21, 2015 at 5:32 PM, Stefán Baxter < > > > [email protected]> > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> Here is small trick I think you will like :). > > > >> > > > >> - With this minimal dataset as /tmp/test.json: > > > >> {"dimensions":{"adults":"A"}} > > > >> > > > >> - Running this: > > > >> select lower(p.dimensions.budgetLevel) as `field1`, > > > >> lower(p.dimensions.adults) as `field2` from dfs.tmp.`/test.json` > as > > > p; > > > >> > > > >> - To no surprise returns this: > > > >> +---------+---------+ > > > >> | field1 | field2 | > > > >> +---------+---------+ > > > >> | null | a | > > > >> +---------+---------+ > > > >> > > > >> Here comes the trick(y) part (hold your breath): > > > >> > > > >> - With the same data as a Parquet file > > > >> CREATE TABLE dfs.tmp.`/test` AS SELECT * FROM > dfs.tmp.`/test.json`; > > > >> > > > >> - The same query: > > > >> select lower(p.dimensions.budgetLevel) as `field1`, > > > >> lower(p.dimensions.adults) as `field2` from > > > dfs.tmp.`/test/0_0_0.parquet` > > > >> as p; > > > >> > > > >> - Return this: > > > >> +---------+---------+ > > > >> | field1 | field2 | > > > >> +---------+---------+ > > > >> | a | null | > > > >> +---------+---------+ > > > >> > > > >> ta ta ! > > > >> > > > >> Best regards, > > > >> -Stefan > > > >> > > > >> > > > >> > > > >> > > > > > > > > > >
