I can certainly do that, however In real world how would we go about troubleshooting and resolving issues over large data sets? Drill needs to have a better way to identify and troubleshoot such issues.
On Fri, Oct 24, 2014 at 1:41 PM, Abhishek Girish <[email protected]> wrote: > Can you try creating a new file with just one JSON record in it (copying > say the first record from the original json document) and see if you can > query the same? > > Also try creating a simple json file by copying the one on > http://json.org/example. Copy it to /tmp on HDFS and try querying the file > using Drill (specify the schema as "use dfs.tmp;"). If this works, then the > issue could be with your original json file. If not, it could be some > simple setup issue. > > Regards, > Abhishek > > On Fri, Oct 24, 2014 at 1:25 PM, Mohit Anchlia <[email protected]> > wrote: > > > Any clues? Not sure why I can't do a simple select. > > On Fri, Oct 24, 2014 at 9:19 AM, Mohit Anchlia <[email protected]> > > wrote: > > > > > Here is the exception > > > > > > 2014-10-23 20:09:08,689 [91b7d838-3128-4add-a686-7ceb05b8e765:frag:0:0] > > > ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error > > > b6f84bc1-8f18-42e9-b79f-c889fa13a40e: Screen received stop request > sent. > > > java.lang.IllegalArgumentException: null > > > at > > > > > > org.apache.drill.common.expression.PathSegment$ArraySegment.<init>(PathSegment.java:52) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$ArraySegment.cloneWithNewChild(PathSegment.java:102) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$ArraySegment.cloneWithNewChild(PathSegment.java:29) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:113) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:113) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.common.expression.SchemaPath.getUnindexedArrayChild(SchemaPath.java:163) > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.RepeatedListVector.addOrGet(RepeatedListVector.java:413) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.impl.RepeatedListWriter.float8(RepeatedListWriter.java:413) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:352) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:307) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:307) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:203) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:206) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > at > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:206) > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > > On Thu, Oct 23, 2014 at 5:35 PM, Abhishek Girish < > > > [email protected]> wrote: > > > > > >> Can you look up the drillbit.log (should be present some place in your > > >> installation log directory) and find > > >> "b6f84bc1-8f18-42e9-b79f-c889fa13a40e". > > >> Share the error that is shown. > > >> > > >> On Thu, Oct 23, 2014 at 5:10 PM, Mohit Anchlia < > [email protected]> > > >> wrote: > > >> > > >> > I moved the file to .json and now I get: > > >> > > > >> > 0: jdbc:drill:zk=local> select * from `tweets-0.json`; > > >> > Query failed: Screen received stop request sent. null > > >> > [b6f84bc1-8f18-42e9-b79f-c889fa13a40e] > > >> > Error: exception while executing query: Failure while trying to get > > next > > >> > result batch. (state=,code=0) > > >> > > > >> > On Thu, Oct 23, 2014 at 11:28 AM, Abhishek Girish < > > >> > [email protected] > > >> > > wrote: > > >> > > > >> > > Or if your data is indeed in json format, change the extension of > > your > > >> > data > > >> > > file from ".txt" to ".json" > > >> > > > > >> > > On Thu, Oct 23, 2014 at 11:25 AM, Abhishek Girish < > > >> > > [email protected] > > >> > > > wrote: > > >> > > > > >> > > > Can you try replacing "storageformat": "json" with > > "storageformat": > > >> > "csv" > > >> > > > in your plugin > > >> > > > ? > > >> > > > > > >> > > > > > >> > > > On Thu, Oct 23, 2014 at 11:11 AM, Mohit Anchlia < > > >> > [email protected]> > > >> > > > wrote: > > >> > > > > > >> > > >> I've tried that too > > >> > > >> > > >> > > >> Error: exception while executing query: Failure while trying to > > get > > >> > next > > >> > > >> result batch. (state=,code=0) > > >> > > >> 0: jdbc:drill:zk=local> select * from > hdfs.json.`/tweets-0.txt`; > > >> > > >> Oct 23, 2014 2:10:40 PM > > >> > org.eigenbase.sql.validate.SqlValidatorException > > >> > > >> <init> > > >> > > >> SEVERE: org.eigenbase.sql.validate.SqlValidatorException: Table > > >> > > >> 'hdfs.json./tweets-0.txt' not found > > >> > > >> Oct 23, 2014 2:10:40 PM org.eigenbase.util.EigenbaseException > > >> <init> > > >> > > >> SEVERE: org.eigenbase.util.EigenbaseContextException: From line > > 1, > > >> > > column > > >> > > >> 15 to line 1, column 18: Table 'hdfs.json./tweets-0.txt' not > > found > > >> > > >> Query failed: Failure while parsing sql. Table > > >> > 'hdfs.json./tweets-0.txt' > > >> > > >> not found [619f0469-0606-4e8e-9ae5-17a305f527fe] > > >> > > >> Error: exception while executing query: Failure while trying to > > get > > >> > next > > >> > > >> result batch. (state=,code=0) > > >> > > >> 0: jdbc:drill:zk=local> > > >> > > >> > > >> > > >> On Thu, Oct 23, 2014 at 11:04 AM, Neeraja Rentachintala < > > >> > > >> [email protected]> wrote: > > >> > > >> > > >> > > >> > can you just try this. > > >> > > >> > select * from hdfs.json.`/tweets-0.txt`; > > >> > > >> > > > >> > > >> > On Thu, Oct 23, 2014 at 10:59 AM, Mohit Anchlia < > > >> > > [email protected] > > >> > > >> > > > >> > > >> > wrote: > > >> > > >> > > > >> > > >> > > This is what I see, looks like that file is showing up > > >> > > >> > > > > >> > > >> > > sqlline version 1.1.6 > > >> > > >> > > 0: jdbc:drill:zk=local> use hdfs.json; > > >> > > >> > > +------------+------------+ > > >> > > >> > > | ok | summary | > > >> > > >> > > +------------+------------+ > > >> > > >> > > | true | Default schema changed to 'hdfs.json' | > > >> > > >> > > +------------+------------+ > > >> > > >> > > 1 row selected (1.112 seconds) > > >> > > >> > > 0: jdbc:drill:zk=local> show files > > >> > > >> > > . . . . . . . . . . . > ; > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ > > >> > > >> > > | name | isDirectory | isFile | length | > > owner > > >> > | > > >> > > >> > > group | permissions | accessTime | modificationTime | > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ > > >> > > >> > > | tweets-0.txt | false | true | 2097437 | > root > > >> > > | > > >> > > >> > > supergroup | rw-r--r-- | 2014-10-22 19:26:15.458 | > > 2014-10-22 > > >> > > >> > > 14:04:26.585 | > > >> > > >> > > | tweets-1.txt | false | true | 1998156 | > root > > >> > > | > > >> > > >> > > supergroup | rw-r--r-- | 2014-10-22 14:04:26.616 | > > 2014-10-22 > > >> > > >> > > 14:04:37.123 | > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ > > >> > > >> > > 2 rows selected (0.264 seconds) > > >> > > >> > > 0: jdbc:drill:zk=local> > > >> > > >> > > > > >> > > >> > > On Thu, Oct 23, 2014 at 10:56 AM, Jason Altekruse < > > >> > > >> > > [email protected]> > > >> > > >> > > wrote: > > >> > > >> > > > > >> > > >> > > > Could you try running 'show files' from the sqllline > prompt > > >> to > > >> > see > > >> > > >> if > > >> > > >> > > that > > >> > > >> > > > gives you any results for files Drill is able to find? > > >> > > >> > > > > > >> > > >> > > > On Thu, Oct 23, 2014 at 10:43 AM, Mohit Anchlia < > > >> > > >> > [email protected]> > > >> > > >> > > > wrote: > > >> > > >> > > > > > >> > > >> > > > > Could somebody look at this error and advise what might > > be > > >> > > wrong? > > >> > > >> It > > >> > > >> > > > seems > > >> > > >> > > > > I am doing everything that's documented. > > >> > > >> > > > > On Wed, Oct 22, 2014 at 2:20 PM, Mohit Anchlia < > > >> > > >> > [email protected] > > >> > > >> > > > > > >> > > >> > > > > wrote: > > >> > > >> > > > > > > >> > > >> > > > > > I am getting the following error even though that > file > > >> > exists > > >> > > in > > >> > > >> > hdfs > > >> > > >> > > > > > > > >> > > >> > > > > > 0: jdbc:drill:zk=local> select * from > > >> > > >> > > > > > hdfs.`/user/train/xd/tweets/tmp/tweets-0.txt`; > > >> > > >> > > > > > Oct 22, 2014 5:16:31 PM > > >> > > >> > > > org.eigenbase.sql.validate.SqlValidatorException > > >> > > >> > > > > > <init> > > >> > > >> > > > > > SEVERE: > > org.eigenbase.sql.validate.SqlValidatorException: > > >> > > Table > > >> > > >> > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' not > found > > >> > > >> > > > > > Oct 22, 2014 5:16:31 PM > > >> > org.eigenbase.util.EigenbaseException > > >> > > >> > <init> > > >> > > >> > > > > > SEVERE: org.eigenbase.util.EigenbaseContextException: > > >> From > > >> > > line > > >> > > >> 1, > > >> > > >> > > > column > > >> > > >> > > > > > 15 to line 1, column 18: Table > > >> > > >> > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' not > found > > >> > > >> > > > > > Query failed: Failure while parsing sql. Table > > >> > > >> > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' not > found > > >> > > >> > > > > > [7e1d5c73-0521-480e-b74b-a4fa50e3f4a7] > > >> > > >> > > > > > Error: exception while executing query: Failure while > > >> trying > > >> > > to > > >> > > >> get > > >> > > >> > > > next > > >> > > >> > > > > > result batch. (state=,code=0) > > >> > > >> > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > > I created new plugin called hdfs. > > >> > > >> > > > > > > > >> > > >> > > > > > { > > >> > > >> > > > > > "type": "file", > > >> > > >> > > > > > "enabled": true, > > >> > > >> > > > > > "connection": "hdfs://10.225.156.201:9000/", > > >> > > >> > > > > > "workspaces": { > > >> > > >> > > > > > "json": { > > >> > > >> > > > > > "location": "/user/train/xd/tweets/tmp", > > >> > > >> > > > > > "writable": false, > > >> > > >> > > > > > "storageformat": "json" > > >> > > >> > > > > > } > > >> > > >> > > > > > }, > > >> > > >> > > > > > "formats": { > > >> > > >> > > > > > "json": { > > >> > > >> > > > > > "type": "json" > > >> > > >> > > > > > } > > >> > > >> > > > > > } > > >> > > >> > > > > > } > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
