Thats some real good feedback. Thanks for that! I understand the need for identifying bad records via more verbose logging and that is something to be discussed. The extension one and a few others are known issues and are being tracked on JIRA <https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&jqlQuery=project+%3D+DRILL>. You are welcome to provide specific feedback over there.
But do continue to participate over here! On Fri, Oct 24, 2014 at 5:07 PM, Mohit Anchlia <[email protected]> wrote: > I think on error there should be more verbose logging in the logs so that > at least it's able to capture the actual reason, line in the file. This is > just one scenario but I am thinking of other scenarios as well. > > One other problem I faced was that I was getting error because I didn't > have my file extension as .json. Just by looking at the error there was no > way to say that. > > Drill should also provide options like skipping bad records, eliminating it > etc. When dealing with TBs and PBs of data we often see bad data and there > needs to be a way to deal with it more cleanly otherwise it becomes pain > really quickly. > On Fri, Oct 24, 2014 at 4:44 PM, Jason Altekruse <[email protected] > > > wrote: > > > Mohit, > > > > Thanks for all of the quick responses. Unfortunately the best we can > > suggest at this point is a binary search like approach to finding the > > issue. Try the first half of the records to see if they can be read > > successfully, then the second half. If you can keep splitting the list > with > > an issue for a while to produce a small reproduction that would be the > best > > for us. Depending on the sensitivity of the data, if you can post it > > somewhere we can take a look at it for you. > > > > Unfortunately there is no tooling in place for Drill to automate this > task. > > It might be something worthwhile to look into as we need to have users > > report issues in a way that they are reproducible for the dev team. > > > > Thanks, > > Jason Altekruse > > > > On Fri, Oct 24, 2014 at 4:38 PM, Mohit Anchlia <[email protected]> > > wrote: > > > > > Yes I did create one liner json file and it works. My previous > response > > > has the details. > > > On Fri, Oct 24, 2014 at 3:04 PM, Abhishek Girish < > > > [email protected]> > > > wrote: > > > > > > > I understand it is not easy to isolate the issue. I've personally > seen > > > > issues such as these and they are usually not due to size, but due to > > the > > > > structure of JSON and how Drill handles it. JSON being highly > flexible > > > may > > > > have some unusual records in the data which Drill fails to handle - > so > > it > > > > could be a bug. > > > > > > > > > > > > On Fri, Oct 24, 2014 at 2:41 PM, Mohit Anchlia < > [email protected] > > > > > > > wrote: > > > > > > > > > I also created a file with just one json and that seem to work, > > however > > > > it > > > > > doesn't work over the entire file. Not sure if it's the problem > with > > > the > > > > > content or the size? How can I tell? > > > > > > > > > > 0: jdbc:drill:zk=local> select * from hdfs.json.`t-0.json`; > > > > > 14/10/24 17:39:00 INFO DataNode.clienttrace: src: / > > > 10.225.156.201:50010, > > > > > dest: /10.225.156.201:55760, bytes: 2305, op: HDFS_READ, cliID: > > > > > DFSClient_NONMAPREDUCE_943548448_1, offset: 0, srvID: > > > > > 3ee44cbd-3ded-4195-8d6a-c545dc9e296f, blockid: > > > > > BP-269313242-10.225.156.201-1413995885151:blk_1073741827_1003, > > > duration: > > > > > 673844 > > > > > > > > > > > > > > > > > > > > +------------+------------+------------+------------+------------+------------+------------+---------------+----------------+------------+------------+------------+-------+ > > > > > | created_at | id | id_str | text | source | > > > > > truncated | user | retweet_count | favorite_count | > > entities | > > > > > favorited | retweeted | possi | > > > > > > > > > > > > > > > > > > > > +------------+------------+------------+------------+------------+------------+------------+---------------+----------------+------------+------------+------------+-------+ > > > > > | Wed Oct 22 18:04:43 +0000 2014 | 524984711660068864 | > > > > 524984711660068864 > > > > > | Robinho niega las acusaciones de violación sexual en Italia: El > > > > delantero > > > > > brasileño Robinho n | > > > > > > > > > > > > > > > > > > > > +------------+------------+------------+------------+------------+------------+------------+---------------+----------------+------------+------------+------------+-------+ > > > > > 1 row selected (0.662 seconds) > > > > > > > > > > 0: jdbc:drill:zk=local> select * from hdfs.json.`tweets-0.json`; > > > > > 14/10/24 17:40:51 INFO DataNode.clienttrace: src: / > > > 10.225.156.201:50010, > > > > > dest: /10.225.156.201:55778, bytes: 660480, op: HDFS_READ, cliID: > > > > > DFSClient_NONMAPREDUCE_943548448_1, offset: 0, srvID: > > > > > 3ee44cbd-3ded-4195-8d6a-c545dc9e296f, blockid: > > > > > BP-269313242-10.225.156.201-1413995885151:blk_1073741825_1001, > > > duration: > > > > > 201166843 > > > > > Query failed: Screen received stop request sent. null > > > > > [5d936aa8-e119-4e31-8d5e-69ec1714b2cb] > > > > > > > > > > > > > > > On Fri, Oct 24, 2014 at 2:36 PM, Mohit Anchlia < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Here is one of the lines from json file: > > > > > > > > > > > > [ec2-user@ip-10-225-156-201 ~]$ hadoop fs -cat > > > > > > /user/train/xd/tweets/tmp/tweets-0.json|more > > > > > > {"created_at":"Wed Oct 22 18:04:43 +0000 > > > > > > > > > > > > > > > > > > > > > 2014","id":524984711660068864,"id_str":"524984711660068864","text":"Robinho > > > > > > niega las acusaciones de violaci\u00f3n sexual en Itali > > > > > > a: El delantero brasile\u00f1o Robinho neg\u00f3 tajantemente ... > > > > > http:\/\/ > > > > > > t.co\/psgRSPbSgZ","source":"\u003ca href=\"http:\/\/ > > twitterfeed.com > > > \" > > > > > > rel=\"nofollow\"\u003etwitt > > > > > > > > > > > > > > > > > > > > > > > > > > > erfeed\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_t > > > > > > > > > > > > > > > > > > > > > o_screen_name":null,"user":{"id":2416621622,"id_str":"2416621622","name":"Daniel > > > > > > > > > > > > > > > > > > > > > Romero","screen_name":"Dany_rom5","location":"Sevilla","url":null,"description":"Verde > > > > > > paz! > > > > > > > > > > > > > > > > > > > > > ","protected":false,"verified":false,"followers_count":141,"friends_count":935,"listed_count":1,"favourites_count":4,"statuses_count":50478,"created_at":"Fri > > > > > > Mar 28 23:57:3 > > > > > > 2 +0000 > > > > > > > > > > > > > > > > > > > > > 2014","utc_offset":7200,"time_zone":"Amsterdam","geo_enabled":false,"lang":"es","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEE > > > > > > D","profile_background_image_url":"http:\/\/abs.twimg.com > > > > > > > > > > > > > > > > > > > > > \/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/ > > > > > > abs.twimg.com\/images\/themes\/th > > > > > > > > > > > > > > > > > > > > > > > > > > > eme1\/bg.png","profile_background_tile":false,"profile_link_color":"0084B4","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_colo > > > > > > > > > > > > > > > > > > > > > > > > > > > r":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/ > > > > > > pbs.twimg.com > > > > > > > > > > > > > > > > > > > > > \/profile_images\/449697097827098624\/9YmqsvgW_normal.jpeg","profile_image_url_ht > > > > > > tps":"https:\/\/pbs.twimg.com > > > > > > > > > > > > > > > > > > > > > \/profile_images\/449697097827098624\/9YmqsvgW_normal.jpeg","profile_banner_url":"https:\/\/ > > > > > > pbs.twimg.com\/profile_banners\/2416621622\/1396053 > > > > > > > > > > > > > > > > > > > > > > > > > > > 818","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"cont > > > > > > > > > > > > > > > > > > > > > > > > > > > ributors":null,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"trends":[],"urls":[{"url":"http:\/\/ > > > > > > t.co\/psgRSPbSgZ","expanded_url":"http:\/\/bit.ly\/1rgKSt > > > > > > f","display_url":"bit.ly > > > > > > > > > > > > > > > > > > > > > \/1rgKStf","indices":[114,136]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"med > > > > > > ium","lang":"es","timestamp_ms":"1414001083666"} > > > > > > > > > > > > On Fri, Oct 24, 2014 at 2:22 PM, Ramana Inukonda < > > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Also, > > > > > >> In order to minimize back and fro mails, If its a json file can > > you > > > > post > > > > > >> or > > > > > >> share the json file or a few lines from the json file? > > > > > >> > > > > > >> Regards > > > > > >> Ramana > > > > > >> > > > > > >> > > > > > >> On Fri, Oct 24, 2014 at 2:18 PM, Ramana Inukonda < > > > > > [email protected]> > > > > > >> wrote: > > > > > >> > > > > > >> > Hey, > > > > > >> > > > > > > >> > Sorry to hear that you are having trouble with a simple case. > > > > > >> > I can help you debug this- Is the file a json or a txt file? > > > > > >> > > > > > > >> > if its a json file please have appropriate extensions. If its > a > > > txt > > > > > file > > > > > >> > can you please have an entry in your storage plugin(accessible > > at > > > > > >> http:// > > > > > >> > <drillbit>:8047) > > > > > >> > > > > > > >> > "formats": { > > > > > >> > "psv": { > > > > > >> > "type": "text", > > > > > >> > "extensions": [ > > > > > >> > "txt" > > > > > >> > ], > > > > > >> > "delimiter": "," > > > > > >> > }, > > > > > >> > > > > > > >> > This is presuming the file is a comma separated file. > Otherwise > > > > change > > > > > >> to > > > > > >> > appropriate delimiter. > > > > > >> > > > > > > >> > > > > > > >> > Regards > > > > > >> > Ramana > > > > > >> > > > > > > >> > > > > > > >> > On Fri, Oct 24, 2014 at 1:47 PM, Mohit Anchlia < > > > > > [email protected]> > > > > > >> > wrote: > > > > > >> > > > > > > >> >> I can certainly do that, however In real world how would we > go > > > > about > > > > > >> >> troubleshooting and resolving issues over large data sets? > > Drill > > > > > needs > > > > > >> to > > > > > >> >> have a better way to identify and troubleshoot such issues. > > > > > >> >> > > > > > >> >> > > > > > >> >> On Fri, Oct 24, 2014 at 1:41 PM, Abhishek Girish < > > > > > >> >> [email protected]> > > > > > >> >> wrote: > > > > > >> >> > > > > > >> >> > Can you try creating a new file with just one JSON record > in > > it > > > > > >> (copying > > > > > >> >> > say the first record from the original json document) and > > see > > > if > > > > > you > > > > > >> >> can > > > > > >> >> > query the same? > > > > > >> >> > > > > > > >> >> > Also try creating a simple json file by copying the one on > > > > > >> >> > http://json.org/example. Copy it to /tmp on HDFS and try > > > > querying > > > > > >> the > > > > > >> >> file > > > > > >> >> > using Drill (specify the schema as "use dfs.tmp;"). If this > > > > works, > > > > > >> then > > > > > >> >> the > > > > > >> >> > issue could be with your original json file. If not, it > could > > > be > > > > > some > > > > > >> >> > simple setup issue. > > > > > >> >> > > > > > > >> >> > Regards, > > > > > >> >> > Abhishek > > > > > >> >> > > > > > > >> >> > On Fri, Oct 24, 2014 at 1:25 PM, Mohit Anchlia < > > > > > >> [email protected]> > > > > > >> >> > wrote: > > > > > >> >> > > > > > > >> >> > > Any clues? Not sure why I can't do a simple select. > > > > > >> >> > > On Fri, Oct 24, 2014 at 9:19 AM, Mohit Anchlia < > > > > > >> >> [email protected]> > > > > > >> >> > > wrote: > > > > > >> >> > > > > > > > >> >> > > > Here is the exception > > > > > >> >> > > > > > > > > >> >> > > > 2014-10-23 20:09:08,689 > > > > > >> >> [91b7d838-3128-4add-a686-7ceb05b8e765:frag:0:0] > > > > > >> >> > > > ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error > > > > > >> >> > > > b6f84bc1-8f18-42e9-b79f-c889fa13a40e: Screen received > > stop > > > > > >> request > > > > > >> >> > sent. > > > > > >> >> > > > java.lang.IllegalArgumentException: null > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$ArraySegment.<init>(PathSegment.java:52) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$ArraySegment.cloneWithNewChild(PathSegment.java:102) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$ArraySegment.cloneWithNewChild(PathSegment.java:29) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:113) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:113) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.PathSegment$NameSegment.cloneWithNewChild(PathSegment.java:179) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.common.expression.SchemaPath.getUnindexedArrayChild(SchemaPath.java:163) > > > > > >> >> > > > > > > > ~[drill-common-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.RepeatedListVector.addOrGet(RepeatedListVector.java:413) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.impl.RepeatedListWriter.float8(RepeatedListWriter.java:413) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:352) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:307) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:307) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:203) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:206) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > at > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:206) > > > > > >> >> > > > > > > > > ~[drill-java-exec-0.5.0-incubating-rebuffed.jar:0.5.0-incubating] > > > > > >> >> > > > > > > > > >> >> > > > On Thu, Oct 23, 2014 at 5:35 PM, Abhishek Girish < > > > > > >> >> > > > [email protected]> wrote: > > > > > >> >> > > > > > > > > >> >> > > >> Can you look up the drillbit.log (should be present > some > > > > place > > > > > >> in > > > > > >> >> your > > > > > >> >> > > >> installation log directory) and find > > > > > >> >> > > >> "b6f84bc1-8f18-42e9-b79f-c889fa13a40e". > > > > > >> >> > > >> Share the error that is shown. > > > > > >> >> > > >> > > > > > >> >> > > >> On Thu, Oct 23, 2014 at 5:10 PM, Mohit Anchlia < > > > > > >> >> > [email protected]> > > > > > >> >> > > >> wrote: > > > > > >> >> > > >> > > > > > >> >> > > >> > I moved the file to .json and now I get: > > > > > >> >> > > >> > > > > > > >> >> > > >> > 0: jdbc:drill:zk=local> select * from > `tweets-0.json`; > > > > > >> >> > > >> > Query failed: Screen received stop request sent. > null > > > > > >> >> > > >> > [b6f84bc1-8f18-42e9-b79f-c889fa13a40e] > > > > > >> >> > > >> > Error: exception while executing query: Failure > while > > > > trying > > > > > >> to > > > > > >> >> get > > > > > >> >> > > next > > > > > >> >> > > >> > result batch. (state=,code=0) > > > > > >> >> > > >> > > > > > > >> >> > > >> > On Thu, Oct 23, 2014 at 11:28 AM, Abhishek Girish < > > > > > >> >> > > >> > [email protected] > > > > > >> >> > > >> > > wrote: > > > > > >> >> > > >> > > > > > > >> >> > > >> > > Or if your data is indeed in json format, change > the > > > > > >> extension > > > > > >> >> of > > > > > >> >> > > your > > > > > >> >> > > >> > data > > > > > >> >> > > >> > > file from ".txt" to ".json" > > > > > >> >> > > >> > > > > > > > >> >> > > >> > > On Thu, Oct 23, 2014 at 11:25 AM, Abhishek Girish > < > > > > > >> >> > > >> > > [email protected] > > > > > >> >> > > >> > > > wrote: > > > > > >> >> > > >> > > > > > > > >> >> > > >> > > > Can you try replacing "storageformat": "json" > with > > > > > >> >> > > "storageformat": > > > > > >> >> > > >> > "csv" > > > > > >> >> > > >> > > > in your plugin > > > > > >> >> > > >> > > > ? > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > > > On Thu, Oct 23, 2014 at 11:11 AM, Mohit Anchlia > < > > > > > >> >> > > >> > [email protected]> > > > > > >> >> > > >> > > > wrote: > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > > >> I've tried that too > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > >> Error: exception while executing query: Failure > > > while > > > > > >> >> trying to > > > > > >> >> > > get > > > > > >> >> > > >> > next > > > > > >> >> > > >> > > >> result batch. (state=,code=0) > > > > > >> >> > > >> > > >> 0: jdbc:drill:zk=local> select * from > > > > > >> >> > hdfs.json.`/tweets-0.txt`; > > > > > >> >> > > >> > > >> Oct 23, 2014 2:10:40 PM > > > > > >> >> > > >> > org.eigenbase.sql.validate.SqlValidatorException > > > > > >> >> > > >> > > >> <init> > > > > > >> >> > > >> > > >> SEVERE: > > > > > org.eigenbase.sql.validate.SqlValidatorException: > > > > > >> >> Table > > > > > >> >> > > >> > > >> 'hdfs.json./tweets-0.txt' not found > > > > > >> >> > > >> > > >> Oct 23, 2014 2:10:40 PM > > > > > >> >> org.eigenbase.util.EigenbaseException > > > > > >> >> > > >> <init> > > > > > >> >> > > >> > > >> SEVERE: > > > org.eigenbase.util.EigenbaseContextException: > > > > > >> From > > > > > >> >> line > > > > > >> >> > > 1, > > > > > >> >> > > >> > > column > > > > > >> >> > > >> > > >> 15 to line 1, column 18: Table > > > > > 'hdfs.json./tweets-0.txt' > > > > > >> not > > > > > >> >> > > found > > > > > >> >> > > >> > > >> Query failed: Failure while parsing sql. Table > > > > > >> >> > > >> > 'hdfs.json./tweets-0.txt' > > > > > >> >> > > >> > > >> not found > [619f0469-0606-4e8e-9ae5-17a305f527fe] > > > > > >> >> > > >> > > >> Error: exception while executing query: Failure > > > while > > > > > >> >> trying to > > > > > >> >> > > get > > > > > >> >> > > >> > next > > > > > >> >> > > >> > > >> result batch. (state=,code=0) > > > > > >> >> > > >> > > >> 0: jdbc:drill:zk=local> > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > >> On Thu, Oct 23, 2014 at 11:04 AM, Neeraja > > > > > Rentachintala < > > > > > >> >> > > >> > > >> [email protected]> wrote: > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > >> > can you just try this. > > > > > >> >> > > >> > > >> > select * from hdfs.json.`/tweets-0.txt`; > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > On Thu, Oct 23, 2014 at 10:59 AM, Mohit > > Anchlia < > > > > > >> >> > > >> > > [email protected] > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > wrote: > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > > This is what I see, looks like that file is > > > > showing > > > > > >> up > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > sqlline version 1.1.6 > > > > > >> >> > > >> > > >> > > 0: jdbc:drill:zk=local> use hdfs.json; > > > > > >> >> > > >> > > >> > > +------------+------------+ > > > > > >> >> > > >> > > >> > > | ok | summary | > > > > > >> >> > > >> > > >> > > +------------+------------+ > > > > > >> >> > > >> > > >> > > | true | Default schema changed to > > > > > 'hdfs.json' > > > > > >> | > > > > > >> >> > > >> > > >> > > +------------+------------+ > > > > > >> >> > > >> > > >> > > 1 row selected (1.112 seconds) > > > > > >> >> > > >> > > >> > > 0: jdbc:drill:zk=local> show files > > > > > >> >> > > >> > > >> > > . . . . . . . . . . . > ; > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ > > > > > >> >> > > >> > > >> > > | name | isDirectory | isFile | > > > > length > > > > > >> | > > > > > >> >> > > owner > > > > > >> >> > > >> > | > > > > > >> >> > > >> > > >> > > group | permissions | accessTime | > > > > > >> modificationTime | > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ > > > > > >> >> > > >> > > >> > > | tweets-0.txt | false | true | > > > > 2097437 > > > > > >> | > > > > > >> >> > root > > > > > >> >> > > >> > > | > > > > > >> >> > > >> > > >> > > supergroup | rw-r--r-- | 2014-10-22 > > > > 19:26:15.458 > > > > > | > > > > > >> >> > > 2014-10-22 > > > > > >> >> > > >> > > >> > > 14:04:26.585 | > > > > > >> >> > > >> > > >> > > | tweets-1.txt | false | true | > > > > 1998156 > > > > > >> | > > > > > >> >> > root > > > > > >> >> > > >> > > | > > > > > >> >> > > >> > > >> > > supergroup | rw-r--r-- | 2014-10-22 > > > > 14:04:26.616 > > > > > | > > > > > >> >> > > 2014-10-22 > > > > > >> >> > > >> > > >> > > 14:04:37.123 | > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > > +------------+-------------+------------+------------+------------+------------+-------------+------------+------------------+ > > > > > >> >> > > >> > > >> > > 2 rows selected (0.264 seconds) > > > > > >> >> > > >> > > >> > > 0: jdbc:drill:zk=local> > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > On Thu, Oct 23, 2014 at 10:56 AM, Jason > > > > Altekruse < > > > > > >> >> > > >> > > >> > > [email protected]> > > > > > >> >> > > >> > > >> > > wrote: > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > Could you try running 'show files' from > the > > > > > >> sqllline > > > > > >> >> > prompt > > > > > >> >> > > >> to > > > > > >> >> > > >> > see > > > > > >> >> > > >> > > >> if > > > > > >> >> > > >> > > >> > > that > > > > > >> >> > > >> > > >> > > > gives you any results for files Drill is > > able > > > > to > > > > > >> find? > > > > > >> >> > > >> > > >> > > > > > > > > >> >> > > >> > > >> > > > On Thu, Oct 23, 2014 at 10:43 AM, Mohit > > > > Anchlia < > > > > > >> >> > > >> > > >> > [email protected]> > > > > > >> >> > > >> > > >> > > > wrote: > > > > > >> >> > > >> > > >> > > > > > > > > >> >> > > >> > > >> > > > > Could somebody look at this error and > > > advise > > > > > what > > > > > >> >> might > > > > > >> >> > > be > > > > > >> >> > > >> > > wrong? > > > > > >> >> > > >> > > >> It > > > > > >> >> > > >> > > >> > > > seems > > > > > >> >> > > >> > > >> > > > > I am doing everything that's > documented. > > > > > >> >> > > >> > > >> > > > > On Wed, Oct 22, 2014 at 2:20 PM, Mohit > > > > Anchlia > > > > > < > > > > > >> >> > > >> > > >> > [email protected] > > > > > >> >> > > >> > > >> > > > > > > > > >> >> > > >> > > >> > > > > wrote: > > > > > >> >> > > >> > > >> > > > > > > > > > >> >> > > >> > > >> > > > > > I am getting the following error even > > > > though > > > > > >> that > > > > > >> >> > file > > > > > >> >> > > >> > exists > > > > > >> >> > > >> > > in > > > > > >> >> > > >> > > >> > hdfs > > > > > >> >> > > >> > > >> > > > > > > > > > > >> >> > > >> > > >> > > > > > 0: jdbc:drill:zk=local> select * from > > > > > >> >> > > >> > > >> > > > > > > > > > > hdfs.`/user/train/xd/tweets/tmp/tweets-0.txt`; > > > > > >> >> > > >> > > >> > > > > > Oct 22, 2014 5:16:31 PM > > > > > >> >> > > >> > > >> > > > > > > > org.eigenbase.sql.validate.SqlValidatorException > > > > > >> >> > > >> > > >> > > > > > <init> > > > > > >> >> > > >> > > >> > > > > > SEVERE: > > > > > >> >> > > org.eigenbase.sql.validate.SqlValidatorException: > > > > > >> >> > > >> > > Table > > > > > >> >> > > >> > > >> > > > > > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' > > > > > >> not > > > > > >> >> > found > > > > > >> >> > > >> > > >> > > > > > Oct 22, 2014 5:16:31 PM > > > > > >> >> > > >> > org.eigenbase.util.EigenbaseException > > > > > >> >> > > >> > > >> > <init> > > > > > >> >> > > >> > > >> > > > > > SEVERE: > > > > > >> >> org.eigenbase.util.EigenbaseContextException: > > > > > >> >> > > >> From > > > > > >> >> > > >> > > line > > > > > >> >> > > >> > > >> 1, > > > > > >> >> > > >> > > >> > > > column > > > > > >> >> > > >> > > >> > > > > > 15 to line 1, column 18: Table > > > > > >> >> > > >> > > >> > > > > > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' > > > > > >> not > > > > > >> >> > found > > > > > >> >> > > >> > > >> > > > > > Query failed: Failure while parsing > > sql. > > > > > Table > > > > > >> >> > > >> > > >> > > > > > > > > > 'hdfs./user/train/xd/tweets/tmp/tweets-0.txt' > > > > > >> not > > > > > >> >> > found > > > > > >> >> > > >> > > >> > > > > > > [7e1d5c73-0521-480e-b74b-a4fa50e3f4a7] > > > > > >> >> > > >> > > >> > > > > > Error: exception while executing > query: > > > > > Failure > > > > > >> >> while > > > > > >> >> > > >> trying > > > > > >> >> > > >> > > to > > > > > >> >> > > >> > > >> get > > > > > >> >> > > >> > > >> > > > next > > > > > >> >> > > >> > > >> > > > > > result batch. (state=,code=0) > > > > > >> >> > > >> > > >> > > > > > > > > > > >> >> > > >> > > >> > > > > > > > > > > >> >> > > >> > > >> > > > > > I created new plugin called hdfs. > > > > > >> >> > > >> > > >> > > > > > > > > > > >> >> > > >> > > >> > > > > > { > > > > > >> >> > > >> > > >> > > > > > "type": "file", > > > > > >> >> > > >> > > >> > > > > > "enabled": true, > > > > > >> >> > > >> > > >> > > > > > "connection": "hdfs:// > > > > 10.225.156.201:9000/ > > > > > ", > > > > > >> >> > > >> > > >> > > > > > "workspaces": { > > > > > >> >> > > >> > > >> > > > > > "json": { > > > > > >> >> > > >> > > >> > > > > > "location": > > > > > "/user/train/xd/tweets/tmp", > > > > > >> >> > > >> > > >> > > > > > "writable": false, > > > > > >> >> > > >> > > >> > > > > > "storageformat": "json" > > > > > >> >> > > >> > > >> > > > > > } > > > > > >> >> > > >> > > >> > > > > > }, > > > > > >> >> > > >> > > >> > > > > > "formats": { > > > > > >> >> > > >> > > >> > > > > > "json": { > > > > > >> >> > > >> > > >> > > > > > "type": "json" > > > > > >> >> > > >> > > >> > > > > > } > > > > > >> >> > > >> > > >> > > > > > } > > > > > >> >> > > >> > > >> > > > > > } > > > > > >> >> > > >> > > >> > > > > > > > > > > >> >> > > >> > > >> > > > > > > > > > >> >> > > >> > > >> > > > > > > > > >> >> > > >> > > >> > > > > > > > >> >> > > >> > > >> > > > > > > >> >> > > >> > > >> > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > > > > > > > > >> >> > > >> > > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
