Also, In order to minimize back and fro mails, If its a json file can you post or share the json file or a few lines from the json file?
Regards Ramana On Fri, Oct 24, 2014 at 2:18 PM, Ramana Inukonda <[email protected]> wrote: > Hey, > > Sorry to hear that you are having trouble with a simple case. > I can help you debug this- Is the file a json or a txt file? > > if its a json file please have appropriate extensions. If its a txt file > can you please have an entry in your storage plugin(accessible at http:// > <drillbit>:8047) > > "formats": { > "psv": { > "type": "text", > "extensions": [ > "txt" > ], > "delimiter": "," > }, > > This is presuming the file is a comma separated file. Otherwise change to > appropriate delimiter. > > > Regards > Ramana > > > On Fri, Oct 24, 2014 at 1:47 PM, Mohit Anchlia <[email protected]> > wrote: > >> I can certainly do that, however In real world how would we go about >> troubleshooting and resolving issues over large data sets? Drill needs to >> have a better way to identify and troubleshoot such issues. >> >> >> On Fri, Oct 24, 2014 at 1:41 PM, Abhishek Girish < >> [email protected]> >> wrote: >> >> > Can you try creating a new file with just one JSON record in it (copying >> > say the first record from the original json document) and see if you >> can >> > query the same? >> > >> > Also try creating a simple json file by copying the one on >> > http://json.org/example. Copy it to /tmp on HDFS and try querying the >> file >> > using Drill (specify the schema as "use dfs.tmp;"). If this works, then >> the >> > issue could be with your original json file. If not, it could be some >> > simple setup issue. >> > >> > Regards, >> > Abhishek >> > >> > On Fri, Oct 24, 2014 at 1:25 PM, Mohit Anchlia <[email protected]> >> > wrote: >> > >> > > Any clues? Not sure why I can't do a simple select. >> > > On Fri, Oct 24, 2014 at 9:19 AM, Mohit Anchlia < >> [email protected]> >> > > wrote: >> > > >> > > > Here is the exception >> > > > >> > > > 2014-10-23 20:09:08,689 >> [91b7d838-3128-4add-a686-7ceb05b8e765:frag:0:0] >> > > > ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error >> > > > b6f84bc1-8f18-42e9-b79f-c889fa13a40e: Screen received stop request >> > sent. >> > > > java.lang.IllegalArgumentException: null >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$ArraySegment.<init>(PathSegment.java:52) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$ArraySegment.cloneWithNewChild(PathSegment.java:102) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$ArraySegment.cloneWithNewChild(PathSegment.java:29) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:113) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:113) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.common.expression.SchemaPath.getUnindexedArrayChild(SchemaPath.java:163) >> > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.RepeatedListVector.addOrGet(RepeatedListVector.java:413) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.impl.RepeatedListWriter.float8(RepeatedListWriter.java:413) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:352) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:307) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:307) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:203) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:206) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > at >> > > > >> > > >> > >> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:206) >> > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] >> > > > >> > > > On Thu, Oct 23, 2014 at 5:35 PM, Abhishek Girish < >> > > > [email protected]> wrote: >> > > > >> > > >> Can you look up the drillbit.log (should be present some place in >> your >> > > >> installation log directory) and find >> > > >> "b6f84bc1-8f18-42e9-b79f-c889fa13a40e". >> > > >> Share the error that is shown. >> > > >> >> > > >> On Thu, Oct 23, 2014 at 5:10 PM, Mohit Anchlia < >> > [email protected]> >> > > >> wrote: >> > > >> >> > > >> > I moved the file to .json and now I get: >> > > >> > >> > > >> > 0: jdbc:drill:zk=local> select * from `tweets-0.json`; >> > > >> > Query failed: Screen received stop request sent. null >> > > >> > [b6f84bc1-8f18-42e9-b79f-c889fa13a40e] >> > > >> > Error: exception while executing query: Failure while trying to >> get >> > > next >> > > >> > result batch. (state=,code=0) >> > > >> > >> > > >> > On Thu, Oct 23, 2014 at 11:28 AM, Abhishek Girish < >> > > >> > [email protected] >> > > >> > > wrote: >> > > >> > >> > > >> > > Or if your data is indeed in json format, change the extension >> of >> > > your >> > > >> > data >> > > >> > > file from ".txt" to ".json" >> > > >> > > >> > > >> > > On Thu, Oct 23, 2014 at 11:25 AM, Abhishek Girish < >> > > >> > > [email protected] >> > > >> > > > wrote: >> > > >> > > >> > > >> > > > Can you try replacing "storageformat": "json" with >> > > "storageformat": >> > > >> > "csv" >> > > >> > > > in your plugin >> > > >> > > > ? >> > > >> > > > >> > > >> > > > >> > > >> > > > On Thu, Oct 23, 2014 at 11:11 AM, Mohit Anchlia < >> > > >> > [email protected]> >> > > >> > > > wrote: >> > > >> > > > >> > > >> > > >> I've tried that too >> > > >> > > >> >> > > >> > > >> Error: exception while executing query: Failure while >> trying to >> > > get >> > > >> > next >> > > >> > > >> result batch. (state=,code=0) >> > > >> > > >> 0: jdbc:drill:zk=local> select * from >> > hdfs.json.`/tweets-0.txt`; >> > > >> > > >> Oct 23, 2014 2:10:40 PM >> > > >> > org.eigenbase.sql.validate.SqlValidatorException >> > > >> > > >> <init> >> > > >> > > >> SEVERE: org.eigenbase.sql.validate.SqlValidatorException: >> Table >> > > >> > > >> 'hdfs.json./tweets-0.txt' not found >> > > >> > > >> Oct 23, 2014 2:10:40 PM >> org.eigenbase.util.EigenbaseException >> > > >> <init> >> > > >> > > >> SEVERE: org.eigenbase.util.EigenbaseContextException: From >> line >> > > 1, >> > > >> > > column >> > > >> > > >> 15 to line 1, column 18: Table 'hdfs.json./tweets-0.txt' not >> > > found >> > > >> > > >> Query failed: Failure while parsing sql. Table >> > > >> > 'hdfs.json./tweets-0.txt' >> > > >> > > >> not found [619f0469-0606-4e8e-9ae5-17a305f527fe] >> > > >> > > >> Error: exception while executing query: Failure while >> trying to >> > > get >> > > >> > next >> > > >> > > >> result batch. (state=,code=0) >> > > >> > > >> 0: jdbc:drill:zk=local> >> > > >> > > >> >> > > >> > > >> On Thu, Oct 23, 2014 at 11:04 AM, Neeraja Rentachintala < >> > > >> > > >> [email protected]> wrote: >> > > >> > > >> >> > > >> > > >> > can you just try this. >> > > >> > > >> > select * from hdfs.json.`/tweets-0.txt`; >> > > >> > > >> > >> > > >> > > >> > On Thu, Oct 23, 2014 at 10:59 AM, Mohit Anchlia < >> > > >> > > [email protected] >> > > >> > > >> > >> > > >> > > >> > wrote: >> > > >> > > >> > >> > > >> > > >> > > This is what I see, looks like that file is showing up >> > > >> > > >> > > >> > > >> > > >> > > sqlline version 1.1.6 >> > > >> > > >> > > 0: jdbc:drill:zk=local> use hdfs.json; >> > > >> > > >> > > +------------+------------+ >> > > >> > > >> > > | ok | summary | >> > > >> > > >> > > +------------+------------+ >> > > >> > > >> > > | true | Default schema changed to 'hdfs.json' | >> > > >> > > >> > > +------------+------------+ >> > > >> > > >> > > 1 row selected (1.112 seconds) >> > > >> > > >> > > 0: jdbc:drill:zk=local> show files >> > > >> > > >> > > . . . . . . . . . . . > ; >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ >> > > >> > > >> > > | name | isDirectory | isFile | length | >> > > owner >> > > >> > | >> > > >> > > >> > > group | permissions | accessTime | modificationTime | >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ >> > > >> > > >> > > | tweets-0.txt | false | true | 2097437 | >> > root >> > > >> > > | >> > > >> > > >> > > supergroup | rw-r--r-- | 2014-10-22 19:26:15.458 | >> > > 2014-10-22 >> > > >> > > >> > > 14:04:26.585 | >> > > >> > > >> > > | tweets-1.txt | false | true | 1998156 | >> > root >> > > >> > > | >> > > >> > > >> > > supergroup | rw-r--r-- | 2014-10-22 14:04:26.616 | >> > > 2014-10-22 >> > > >> > > >> > > 14:04:37.123 | >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > >> > > >> > > >> >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ >> > > >> > > >> > > 2 rows selected (0.264 seconds) >> > > >> > > >> > > 0: jdbc:drill:zk=local> >> > > >> > > >> > > >> > > >> > > >> > > On Thu, Oct 23, 2014 at 10:56 AM, Jason Altekruse < >> > > >> > > >> > > [email protected]> >> > > >> > > >> > > wrote: >> > > >> > > >> > > >> > > >> > > >> > > > Could you try running 'show files' from the sqllline >> > prompt >> > > >> to >> > > >> > see >> > > >> > > >> if >> > > >> > > >> > > that >> > > >> > > >> > > > gives you any results for files Drill is able to find? >> > > >> > > >> > > > >> > > >> > > >> > > > On Thu, Oct 23, 2014 at 10:43 AM, Mohit Anchlia < >> > > >> > > >> > [email protected]> >> > > >> > > >> > > > wrote: >> > > >> > > >> > > > >> > > >> > > >> > > > > Could somebody look at this error and advise what >> might >> > > be >> > > >> > > wrong? >> > > >> > > >> It >> > > >> > > >> > > > seems >> > > >> > > >> > > > > I am doing everything that's documented. >> > > >> > > >> > > > > On Wed, Oct 22, 2014 at 2:20 PM, Mohit Anchlia < >> > > >> > > >> > [email protected] >> > > >> > > >> > > > >> > > >> > > >> > > > > wrote: >> > > >> > > >> > > > > >> > > >> > > >> > > > > > I am getting the following error even though that >> > file >> > > >> > exists >> > > >> > > in >> > > >> > > >> > hdfs >> > > >> > > >> > > > > > >> > > >> > > >> > > > > > 0: jdbc:drill:zk=local> select * from >> > > >> > > >> > > > > > hdfs.`/user/train/xd/tweets/tmp/tweets-0.txt`; >> > > >> > > >> > > > > > Oct 22, 2014 5:16:31 PM >> > > >> > > >> > > > org.eigenbase.sql.validate.SqlValidatorException >> > > >> > > >> > > > > > <init> >> > > >> > > >> > > > > > SEVERE: >> > > org.eigenbase.sql.validate.SqlValidatorException: >> > > >> > > Table >> > > >> > > >> > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' not >> > found >> > > >> > > >> > > > > > Oct 22, 2014 5:16:31 PM >> > > >> > org.eigenbase.util.EigenbaseException >> > > >> > > >> > <init> >> > > >> > > >> > > > > > SEVERE: >> org.eigenbase.util.EigenbaseContextException: >> > > >> From >> > > >> > > line >> > > >> > > >> 1, >> > > >> > > >> > > > column >> > > >> > > >> > > > > > 15 to line 1, column 18: Table >> > > >> > > >> > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' not >> > found >> > > >> > > >> > > > > > Query failed: Failure while parsing sql. Table >> > > >> > > >> > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' not >> > found >> > > >> > > >> > > > > > [7e1d5c73-0521-480e-b74b-a4fa50e3f4a7] >> > > >> > > >> > > > > > Error: exception while executing query: Failure >> while >> > > >> trying >> > > >> > > to >> > > >> > > >> get >> > > >> > > >> > > > next >> > > >> > > >> > > > > > result batch. (state=,code=0) >> > > >> > > >> > > > > > >> > > >> > > >> > > > > > >> > > >> > > >> > > > > > I created new plugin called hdfs. >> > > >> > > >> > > > > > >> > > >> > > >> > > > > > { >> > > >> > > >> > > > > > "type": "file", >> > > >> > > >> > > > > > "enabled": true, >> > > >> > > >> > > > > > "connection": "hdfs://10.225.156.201:9000/", >> > > >> > > >> > > > > > "workspaces": { >> > > >> > > >> > > > > > "json": { >> > > >> > > >> > > > > > "location": "/user/train/xd/tweets/tmp", >> > > >> > > >> > > > > > "writable": false, >> > > >> > > >> > > > > > "storageformat": "json" >> > > >> > > >> > > > > > } >> > > >> > > >> > > > > > }, >> > > >> > > >> > > > > > "formats": { >> > > >> > > >> > > > > > "json": { >> > > >> > > >> > > > > > "type": "json" >> > > >> > > >> > > > > > } >> > > >> > > >> > > > > > } >> > > >> > > >> > > > > > } >> > > >> > > >> > > > > > >> > > >> > > >> > > > > >> > > >> > > >> > > > >> > > >> > > >> > > >> > > >> > > >> > >> > > >> > > >> >> > > >> > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > > >> > > > >> > > >> > >> > >
