Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread scott
0.000s 0.000s > 0.000s 0.04% 0.04% 7,227 16KB16KB > > > -Original Message----- > From: Paul Rogers [mailto:par0...@yahoo.com.INVALID] > Sent: Tuesday, August 28, 2018 11:23 AM > To: user@drill.apache.org > Subject: Re: RE: Error: DATA_READ ERROR: Error par

RE: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread Lee, David
ter -Original Message- From: Lee, David Sent: Tuesday, August 28, 2018 12:11 PM To: user@drill.apache.org Subject: RE: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record select count(*) on a jsonl file comes back instantly /u1/my_login=> wc -l test.json

RE: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread Lee, David
Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record [EXTERNAL EMAIL] Hi Scott, Bingo. Just tried this very case with the sample file from the previous post. Got exactly the failure in the post you provided. I notice that a "select *"

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread Paul Rogers
Hi Scott, Bingo. Just tried this very case with the sample file from the previous post. Got exactly the failure in the post you provided. I notice that a "select *" query returns immediately, but a "count(*)" query hangs for the 30+ seconds before it errors out. Mine is only a two-record file,

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread scott
Paul, Thanks for prompting the right questions. I went back and took another look at my queries. It turns out that there is some condition that causes this error when running functions like "count(*)" on the data to cause this error, where a normal unqualified select does not. I also ran across

RE: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread Lee, David
A simple linux GREP command can be used to find data, but trying to GREP a JSON file with no line breaks just returns back a wall of text.. -Original Message- From: Paul Rogers [mailto:par0...@yahoo.com.INVALID] Sent: Monday, August 27, 2018 5:47 PM To: user@drill.apache.org Subject:

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-28 Thread Paul Rogers
Hi Scott, I created a file, "test.json", using the data from your e-mail: [ { "var1": "foo", "var2":"bar"},{"var1": "fo", "var2": "baz"}] The oldest build I have readily available is Drill 1.13. I ran that as a server, then connected with sqlline as a client. I ran a query: select * from

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-27 Thread scott
Paul, I'm using version 1.12. Can you tell me what version you think that was fixed in? The ticket I referenced is still open, with no comments. Scott On Mon, Aug 27, 2018 at 5:47 PM Paul Rogers wrote: > Hi David, > > JSON files are never splittable: there is no single-character way to find >

Re: RE: Error: DATA_READ ERROR: Error parsing JSON - Cannot read from the middle of a record

2018-08-27 Thread Paul Rogers
Hi David, JSON files are never splittable: there is no single-character way to find the start of a JSON record within a file. Drill is supposed to support two JSON formats: the array format from the earlier post, and the non-JSON (but very common) list of objects format in this example.