Re: Drill error

[Kunal Khatua]

That could be a possibility. See if you can skip that value in the column... 
try using a filter condition with a such as 
NOT LIKE '%' 
On 6/27/2018 11:25:10 PM, Nitin Pawar  wrote:
Could this cause an issue if one of the field in concat function has large
text ?

On Thu, Jun 28, 2018 at 11:10 AM, Nitin Pawar
wrote:

> Hi Khurram,
>
> This is a parquet table.
> all the columns in the table are string columns (even date column is
> stored as string)
>
> I am currently using drill 1.13.0. Same query used to work fine on drill
> 1.8.0
>
> Thanks,
> Nitin
>
> On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz wrote:
>
>> Hi Nitin,
>>
>> Can you please share the description of the table (i.e. column types) ?
>> Is this a parquet table or JSON ?
>> Also please share the version of Drill and the drillbit.log
>>
>> Thanks,
>> Khurram
>>
>> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar
>> wrote:
>>
>> > here are the details
>> >
>> > query:
>> >
>> > select Account Account,Name Name,CONCAT(DateString , string2)
>> > Merged_String from dfs.tmp.`/nitin/` t1
>> >
>> > There is no custom UDF in this query;
>> >
>> >
>> >
>> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar
>> > wrote:
>> >
>> > > Hi Vitalii,
>> > >
>> > > Thanks for the description.
>> > > I will try to get the log for this but as this is happening in
>> production
>> > > and drill log gets overwritten in our case very fast.
>> > >
>> > > Also I will get the query for the same while getting the log
>> > >
>> > > I will get the details and update the thread
>> > >
>> > > Thanks,
>> > > Nitin
>> > >
>> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <>
>> > > vitalii.dira...@gmail.com> wrote:
>> > >
>> > >> Hi Nitin,
>> > >>
>> > >> It happens in the process of reallocation of the size of buffers in
>> the
>> > >> memory.
>> > >> It isn't a User Exception, so it looks like a bug, if you get it in
>> some
>> > >> existed plugin.
>> > >> But to say you exactly, please describe your case. What kind of query
>> > did
>> > >> you perform, any UDF's, which data source?
>> > >> Also logs can help.
>> > >>
>> > >> Thanks.
>> > >>
>> > >> Kind regards
>> > >> Vitalii
>> > >>
>> > >>
>> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar
>> >
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > Can someone help me understand below error? and how do I not let
>> this
>> > >> > happen ??
>> > >> >
>> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
>> buffer.
>> > >> >
>> > >> > Fragment 0:0
>> > >> >
>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>> > >> > ip-10-0-103-63.ec2.internal:31010]
>> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
>> > >> unmanaged
>> > >> > buffer.
>> > >> >
>> > >> > Fragment 0:0
>> > >> >
>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>> > >> > ip-10-0-103-63.ec2.internal:31010]
>> > >> >
>> > >> > --
>> > >> > Nitin Pawar
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Nitin Pawar
>> > >
>> >
>> >
>> >
>> > --
>> > Nitin Pawar
>> >
>>
>
>
>
> --
> Nitin Pawar
>



--
Nitin Pawar

Re: Drill error

[Vitalii Diravka]

Thank you, Nitin

Kind regards
Vitalii


On Thu, Jun 28, 2018 at 6:04 PM Nitin Pawar  wrote:

> created https://issues.apache.org/jira/browse/DRILL-6551
>
> On Thu, Jun 28, 2018 at 7:43 PM, Vitalii Diravka <
> vitalii.dira...@gmail.com>
> wrote:
>
> > Hi Nitin,
> >
> > This is definitely the regression. Could you please post the Jira ticket
> > with details of your case, data and workaround [1].
> > It will help for developers to solve this issue.
> >
> > [1] https://issues.apache.org/jira/projects/DRILL
> > Thanks.
> >
> > Kind regards
> > Vitalii
> >
> >
> > On Thu, Jun 28, 2018 at 10:50 AM Nitin Pawar 
> > wrote:
> >
> > > I was able to fix this issue by doing concat(string1, ' ', string2)
> > instead
> > > of concat(string1, string2)
> > > Not sure how adding a separator helps but it solved the problem
> > >
> > > On Thu, Jun 28, 2018 at 11:55 AM, Nitin Pawar  >
> > > wrote:
> > >
> > > > Could this cause an issue if one of the field in concat function has
> > > large
> > > > text ?
> > > >
> > > > On Thu, Jun 28, 2018 at 11:10 AM, Nitin Pawar <
> nitinpawar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Khurram,
> > > >>
> > > >> This is a parquet table.
> > > >> all the columns in the table are string columns (even date column is
> > > >> stored as string)
> > > >>
> > > >> I am currently using drill 1.13.0. Same query used to work fine on
> > drill
> > > >> 1.8.0
> > > >>
> > > >> Thanks,
> > > >> Nitin
> > > >>
> > > >> On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz 
> > > >> wrote:
> > > >>
> > > >>> Hi Nitin,
> > > >>>
> > > >>> Can you please share the description of the table (i.e. column
> > types) ?
> > > >>> Is this a parquet table or JSON ?
> > > >>> Also please share the version of Drill and the drillbit.log
> > > >>>
> > > >>> Thanks,
> > > >>> Khurram
> > > >>>
> > > >>> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar <
> > nitinpawar...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > here are the details
> > > >>> >
> > > >>> > query:
> > > >>> >
> > > >>> > select Account Account,Name Name,CONCAT(DateString , string2)
> > > >>> > Merged_String from dfs.tmp.`/nitin/` t1
> > > >>> >
> > > >>> > There is no custom UDF in this query;
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar <
> > > nitinpawar...@gmail.com>
> > > >>> > wrote:
> > > >>> >
> > > >>> > > Hi Vitalii,
> > > >>> > >
> > > >>> > > Thanks for the description.
> > > >>> > > I will try to get the log for this but as this is happening in
> > > >>> production
> > > >>> > > and drill log gets overwritten in our case very fast.
> > > >>> > >
> > > >>> > > Also I will get the query for the same while getting the log
> > > >>> > >
> > > >>> > > I will get the details and update the thread
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > > Nitin
> > > >>> > >
> > > >>> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
> > > >>> > > vitalii.dira...@gmail.com> wrote:
> > > >>> > >
> > > >>> > >> Hi Nitin,
> > > >>> > >>
> > > >>> > >> It happens in the process of reallocation of the size of
> buffers
> > > in
> > > >>> the
> > > >>> > >> memory.
> > > >>> > >> It isn't a User Exception, so it looks like a bug, if you get
> it
> > > in
> > > >>> some
> > > >>> > >> existed plugin.
> > > >>> > >> But to say you exactly, please describe your case. What kind
> of
> > > >>> query
> > > >>> > did
> > > >>> > >> you perform, any UDF's, which data source?
> > > >>> > >> Also logs can help.
> > > >>> > >>
> > > >>> > >> Thanks.
> > > >>> > >>
> > > >>> > >> Kind regards
> > > >>> > >> Vitalii
> > > >>> > >>
> > > >>> > >>
> > > >>> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar <
> > > >>> nitinpawar...@gmail.com>
> > > >>> > >> wrote:
> > > >>> > >>
> > > >>> > >> > Hi,
> > > >>> > >> >
> > > >>> > >> > Can someone help me understand below error? and how do I not
> > let
> > > >>> this
> > > >>> > >> > happen ??
> > > >>> > >> >
> > > >>> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove
> unmanaged
> > > >>> buffer.
> > > >>> > >> >
> > > >>> > >> > Fragment 0:0
> > > >>> > >> >
> > > >>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > > >>> > >> > ip-10-0-103-63.ec2.internal:31010]
> > > >>> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to
> > remove
> > > >>> > >> unmanaged
> > > >>> > >> > buffer.
> > > >>> > >> >
> > > >>> > >> > Fragment 0:0
> > > >>> > >> >
> > > >>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > > >>> > >> > ip-10-0-103-63.ec2.internal:31010]
> > > >>> > >> >
> > > >>> > >> > --
> > > >>> > >> > Nitin Pawar
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > --
> > > >>> > > Nitin Pawar
> > > >>> > >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > --
> > > >>> > Nitin Pawar
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Nitin Pawar
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Nitin Pawar
> > > >
> > >
> > >
> > >
> > > --
> > > Nitin Pawar
> 

Re: Drill error

[Nitin Pawar]

created https://issues.apache.org/jira/browse/DRILL-6551

On Thu, Jun 28, 2018 at 7:43 PM, Vitalii Diravka 
wrote:

> Hi Nitin,
>
> This is definitely the regression. Could you please post the Jira ticket
> with details of your case, data and workaround [1].
> It will help for developers to solve this issue.
>
> [1] https://issues.apache.org/jira/projects/DRILL
> Thanks.
>
> Kind regards
> Vitalii
>
>
> On Thu, Jun 28, 2018 at 10:50 AM Nitin Pawar 
> wrote:
>
> > I was able to fix this issue by doing concat(string1, ' ', string2)
> instead
> > of concat(string1, string2)
> > Not sure how adding a separator helps but it solved the problem
> >
> > On Thu, Jun 28, 2018 at 11:55 AM, Nitin Pawar 
> > wrote:
> >
> > > Could this cause an issue if one of the field in concat function has
> > large
> > > text ?
> > >
> > > On Thu, Jun 28, 2018 at 11:10 AM, Nitin Pawar  >
> > > wrote:
> > >
> > >> Hi Khurram,
> > >>
> > >> This is a parquet table.
> > >> all the columns in the table are string columns (even date column is
> > >> stored as string)
> > >>
> > >> I am currently using drill 1.13.0. Same query used to work fine on
> drill
> > >> 1.8.0
> > >>
> > >> Thanks,
> > >> Nitin
> > >>
> > >> On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz 
> > >> wrote:
> > >>
> > >>> Hi Nitin,
> > >>>
> > >>> Can you please share the description of the table (i.e. column
> types) ?
> > >>> Is this a parquet table or JSON ?
> > >>> Also please share the version of Drill and the drillbit.log
> > >>>
> > >>> Thanks,
> > >>> Khurram
> > >>>
> > >>> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar <
> nitinpawar...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > here are the details
> > >>> >
> > >>> > query:
> > >>> >
> > >>> > select Account Account,Name Name,CONCAT(DateString , string2)
> > >>> > Merged_String from dfs.tmp.`/nitin/` t1
> > >>> >
> > >>> > There is no custom UDF in this query;
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar <
> > nitinpawar...@gmail.com>
> > >>> > wrote:
> > >>> >
> > >>> > > Hi Vitalii,
> > >>> > >
> > >>> > > Thanks for the description.
> > >>> > > I will try to get the log for this but as this is happening in
> > >>> production
> > >>> > > and drill log gets overwritten in our case very fast.
> > >>> > >
> > >>> > > Also I will get the query for the same while getting the log
> > >>> > >
> > >>> > > I will get the details and update the thread
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Nitin
> > >>> > >
> > >>> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
> > >>> > > vitalii.dira...@gmail.com> wrote:
> > >>> > >
> > >>> > >> Hi Nitin,
> > >>> > >>
> > >>> > >> It happens in the process of reallocation of the size of buffers
> > in
> > >>> the
> > >>> > >> memory.
> > >>> > >> It isn't a User Exception, so it looks like a bug, if you get it
> > in
> > >>> some
> > >>> > >> existed plugin.
> > >>> > >> But to say you exactly, please describe your case. What kind of
> > >>> query
> > >>> > did
> > >>> > >> you perform, any UDF's, which data source?
> > >>> > >> Also logs can help.
> > >>> > >>
> > >>> > >> Thanks.
> > >>> > >>
> > >>> > >> Kind regards
> > >>> > >> Vitalii
> > >>> > >>
> > >>> > >>
> > >>> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar <
> > >>> nitinpawar...@gmail.com>
> > >>> > >> wrote:
> > >>> > >>
> > >>> > >> > Hi,
> > >>> > >> >
> > >>> > >> > Can someone help me understand below error? and how do I not
> let
> > >>> this
> > >>> > >> > happen ??
> > >>> > >> >
> > >>> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
> > >>> buffer.
> > >>> > >> >
> > >>> > >> > Fragment 0:0
> > >>> > >> >
> > >>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > >>> > >> > ip-10-0-103-63.ec2.internal:31010]
> > >>> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to
> remove
> > >>> > >> unmanaged
> > >>> > >> > buffer.
> > >>> > >> >
> > >>> > >> > Fragment 0:0
> > >>> > >> >
> > >>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > >>> > >> > ip-10-0-103-63.ec2.internal:31010]
> > >>> > >> >
> > >>> > >> > --
> > >>> > >> > Nitin Pawar
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > --
> > >>> > > Nitin Pawar
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Nitin Pawar
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Nitin Pawar
> > >>
> > >
> > >
> > >
> > > --
> > > Nitin Pawar
> > >
> >
> >
> >
> > --
> > Nitin Pawar
> >
>



-- 
Nitin Pawar

Re: Drill error

[Vitalii Diravka]

Hi Nitin,

This is definitely the regression. Could you please post the Jira ticket
with details of your case, data and workaround [1].
It will help for developers to solve this issue.

[1] https://issues.apache.org/jira/projects/DRILL
Thanks.

Kind regards
Vitalii


On Thu, Jun 28, 2018 at 10:50 AM Nitin Pawar 
wrote:

> I was able to fix this issue by doing concat(string1, ' ', string2) instead
> of concat(string1, string2)
> Not sure how adding a separator helps but it solved the problem
>
> On Thu, Jun 28, 2018 at 11:55 AM, Nitin Pawar 
> wrote:
>
> > Could this cause an issue if one of the field in concat function has
> large
> > text ?
> >
> > On Thu, Jun 28, 2018 at 11:10 AM, Nitin Pawar 
> > wrote:
> >
> >> Hi Khurram,
> >>
> >> This is a parquet table.
> >> all the columns in the table are string columns (even date column is
> >> stored as string)
> >>
> >> I am currently using drill 1.13.0. Same query used to work fine on drill
> >> 1.8.0
> >>
> >> Thanks,
> >> Nitin
> >>
> >> On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz 
> >> wrote:
> >>
> >>> Hi Nitin,
> >>>
> >>> Can you please share the description of the table (i.e. column types) ?
> >>> Is this a parquet table or JSON ?
> >>> Also please share the version of Drill and the drillbit.log
> >>>
> >>> Thanks,
> >>> Khurram
> >>>
> >>> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar 
> >>> wrote:
> >>>
> >>> > here are the details
> >>> >
> >>> > query:
> >>> >
> >>> > select Account Account,Name Name,CONCAT(DateString , string2)
> >>> > Merged_String from dfs.tmp.`/nitin/` t1
> >>> >
> >>> > There is no custom UDF in this query;
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar <
> nitinpawar...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Vitalii,
> >>> > >
> >>> > > Thanks for the description.
> >>> > > I will try to get the log for this but as this is happening in
> >>> production
> >>> > > and drill log gets overwritten in our case very fast.
> >>> > >
> >>> > > Also I will get the query for the same while getting the log
> >>> > >
> >>> > > I will get the details and update the thread
> >>> > >
> >>> > > Thanks,
> >>> > > Nitin
> >>> > >
> >>> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
> >>> > > vitalii.dira...@gmail.com> wrote:
> >>> > >
> >>> > >> Hi Nitin,
> >>> > >>
> >>> > >> It happens in the process of reallocation of the size of buffers
> in
> >>> the
> >>> > >> memory.
> >>> > >> It isn't a User Exception, so it looks like a bug, if you get it
> in
> >>> some
> >>> > >> existed plugin.
> >>> > >> But to say you exactly, please describe your case. What kind of
> >>> query
> >>> > did
> >>> > >> you perform, any UDF's, which data source?
> >>> > >> Also logs can help.
> >>> > >>
> >>> > >> Thanks.
> >>> > >>
> >>> > >> Kind regards
> >>> > >> Vitalii
> >>> > >>
> >>> > >>
> >>> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar <
> >>> nitinpawar...@gmail.com>
> >>> > >> wrote:
> >>> > >>
> >>> > >> > Hi,
> >>> > >> >
> >>> > >> > Can someone help me understand below error? and how do I not let
> >>> this
> >>> > >> > happen ??
> >>> > >> >
> >>> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
> >>> buffer.
> >>> > >> >
> >>> > >> > Fragment 0:0
> >>> > >> >
> >>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> >>> > >> > ip-10-0-103-63.ec2.internal:31010]
> >>> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
> >>> > >> unmanaged
> >>> > >> > buffer.
> >>> > >> >
> >>> > >> > Fragment 0:0
> >>> > >> >
> >>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> >>> > >> > ip-10-0-103-63.ec2.internal:31010]
> >>> > >> >
> >>> > >> > --
> >>> > >> > Nitin Pawar
> >>> > >> >
> >>> > >>
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Nitin Pawar
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Nitin Pawar
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Nitin Pawar
> >>
> >
> >
> >
> > --
> > Nitin Pawar
> >
>
>
>
> --
> Nitin Pawar
>

Re: Drill error

[Nitin Pawar]

I was able to fix this issue by doing concat(string1, ' ', string2) instead
of concat(string1, string2)
Not sure how adding a separator helps but it solved the problem

On Thu, Jun 28, 2018 at 11:55 AM, Nitin Pawar 
wrote:

> Could this cause an issue if one of the field in concat function has large
> text ?
>
> On Thu, Jun 28, 2018 at 11:10 AM, Nitin Pawar 
> wrote:
>
>> Hi Khurram,
>>
>> This is a parquet table.
>> all the columns in the table are string columns (even date column is
>> stored as string)
>>
>> I am currently using drill 1.13.0. Same query used to work fine on drill
>> 1.8.0
>>
>> Thanks,
>> Nitin
>>
>> On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz 
>> wrote:
>>
>>> Hi Nitin,
>>>
>>> Can you please share the description of the table (i.e. column types) ?
>>> Is this a parquet table or JSON ?
>>> Also please share the version of Drill and the drillbit.log
>>>
>>> Thanks,
>>> Khurram
>>>
>>> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar 
>>> wrote:
>>>
>>> > here are the details
>>> >
>>> > query:
>>> >
>>> > select Account Account,Name Name,CONCAT(DateString , string2)
>>> > Merged_String from dfs.tmp.`/nitin/` t1
>>> >
>>> > There is no custom UDF in this query;
>>> >
>>> >
>>> >
>>> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar 
>>> > wrote:
>>> >
>>> > > Hi Vitalii,
>>> > >
>>> > > Thanks for the description.
>>> > > I will try to get the log for this but as this is happening in
>>> production
>>> > > and drill log gets overwritten in our case very fast.
>>> > >
>>> > > Also I will get the query for the same while getting the log
>>> > >
>>> > > I will get the details and update the thread
>>> > >
>>> > > Thanks,
>>> > > Nitin
>>> > >
>>> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
>>> > > vitalii.dira...@gmail.com> wrote:
>>> > >
>>> > >> Hi Nitin,
>>> > >>
>>> > >> It happens in the process of reallocation of the size of buffers in
>>> the
>>> > >> memory.
>>> > >> It isn't a User Exception, so it looks like a bug, if you get it in
>>> some
>>> > >> existed plugin.
>>> > >> But to say you exactly, please describe your case. What kind of
>>> query
>>> > did
>>> > >> you perform, any UDF's, which data source?
>>> > >> Also logs can help.
>>> > >>
>>> > >> Thanks.
>>> > >>
>>> > >> Kind regards
>>> > >> Vitalii
>>> > >>
>>> > >>
>>> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar <
>>> nitinpawar...@gmail.com>
>>> > >> wrote:
>>> > >>
>>> > >> > Hi,
>>> > >> >
>>> > >> > Can someone help me understand below error? and how do I not let
>>> this
>>> > >> > happen ??
>>> > >> >
>>> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
>>> buffer.
>>> > >> >
>>> > >> > Fragment 0:0
>>> > >> >
>>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>>> > >> > ip-10-0-103-63.ec2.internal:31010]
>>> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
>>> > >> unmanaged
>>> > >> > buffer.
>>> > >> >
>>> > >> > Fragment 0:0
>>> > >> >
>>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>>> > >> > ip-10-0-103-63.ec2.internal:31010]
>>> > >> >
>>> > >> > --
>>> > >> > Nitin Pawar
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Nitin Pawar
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Nitin Pawar
>>> >
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Nitin Pawar
>



-- 
Nitin Pawar

Re: Drill error

[Nitin Pawar]

Could this cause an issue if one of the field in concat function has large
text ?

On Thu, Jun 28, 2018 at 11:10 AM, Nitin Pawar 
wrote:

> Hi Khurram,
>
> This is a parquet table.
> all the columns in the table are string columns (even date column is
> stored as string)
>
> I am currently using drill 1.13.0. Same query used to work fine on drill
> 1.8.0
>
> Thanks,
> Nitin
>
> On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz  wrote:
>
>> Hi Nitin,
>>
>> Can you please share the description of the table (i.e. column types) ?
>> Is this a parquet table or JSON ?
>> Also please share the version of Drill and the drillbit.log
>>
>> Thanks,
>> Khurram
>>
>> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar 
>> wrote:
>>
>> > here are the details
>> >
>> > query:
>> >
>> > select Account Account,Name Name,CONCAT(DateString , string2)
>> > Merged_String from dfs.tmp.`/nitin/` t1
>> >
>> > There is no custom UDF in this query;
>> >
>> >
>> >
>> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar 
>> > wrote:
>> >
>> > > Hi Vitalii,
>> > >
>> > > Thanks for the description.
>> > > I will try to get the log for this but as this is happening in
>> production
>> > > and drill log gets overwritten in our case very fast.
>> > >
>> > > Also I will get the query for the same while getting the log
>> > >
>> > > I will get the details and update the thread
>> > >
>> > > Thanks,
>> > > Nitin
>> > >
>> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
>> > > vitalii.dira...@gmail.com> wrote:
>> > >
>> > >> Hi Nitin,
>> > >>
>> > >> It happens in the process of reallocation of the size of buffers in
>> the
>> > >> memory.
>> > >> It isn't a User Exception, so it looks like a bug, if you get it in
>> some
>> > >> existed plugin.
>> > >> But to say you exactly, please describe your case. What kind of query
>> > did
>> > >> you perform, any UDF's, which data source?
>> > >> Also logs can help.
>> > >>
>> > >> Thanks.
>> > >>
>> > >> Kind regards
>> > >> Vitalii
>> > >>
>> > >>
>> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar > >
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > Can someone help me understand below error? and how do I not let
>> this
>> > >> > happen ??
>> > >> >
>> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
>> buffer.
>> > >> >
>> > >> > Fragment 0:0
>> > >> >
>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>> > >> > ip-10-0-103-63.ec2.internal:31010]
>> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
>> > >> unmanaged
>> > >> > buffer.
>> > >> >
>> > >> > Fragment 0:0
>> > >> >
>> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>> > >> > ip-10-0-103-63.ec2.internal:31010]
>> > >> >
>> > >> > --
>> > >> > Nitin Pawar
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Nitin Pawar
>> > >
>> >
>> >
>> >
>> > --
>> > Nitin Pawar
>> >
>>
>
>
>
> --
> Nitin Pawar
>



-- 
Nitin Pawar

Re: Drill error

[Nitin Pawar]

Hi Khurram,

This is a parquet table.
all the columns in the table are string columns (even date column is stored
as string)

I am currently using drill 1.13.0. Same query used to work fine on drill
1.8.0

Thanks,
Nitin

On Thu, Jun 28, 2018 at 12:17 AM, Khurram Faraaz  wrote:

> Hi Nitin,
>
> Can you please share the description of the table (i.e. column types) ?
> Is this a parquet table or JSON ?
> Also please share the version of Drill and the drillbit.log
>
> Thanks,
> Khurram
>
> On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar 
> wrote:
>
> > here are the details
> >
> > query:
> >
> > select Account Account,Name Name,CONCAT(DateString , string2)
> > Merged_String from dfs.tmp.`/nitin/` t1
> >
> > There is no custom UDF in this query;
> >
> >
> >
> > On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar 
> > wrote:
> >
> > > Hi Vitalii,
> > >
> > > Thanks for the description.
> > > I will try to get the log for this but as this is happening in
> production
> > > and drill log gets overwritten in our case very fast.
> > >
> > > Also I will get the query for the same while getting the log
> > >
> > > I will get the details and update the thread
> > >
> > > Thanks,
> > > Nitin
> > >
> > > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
> > > vitalii.dira...@gmail.com> wrote:
> > >
> > >> Hi Nitin,
> > >>
> > >> It happens in the process of reallocation of the size of buffers in
> the
> > >> memory.
> > >> It isn't a User Exception, so it looks like a bug, if you get it in
> some
> > >> existed plugin.
> > >> But to say you exactly, please describe your case. What kind of query
> > did
> > >> you perform, any UDF's, which data source?
> > >> Also logs can help.
> > >>
> > >> Thanks.
> > >>
> > >> Kind regards
> > >> Vitalii
> > >>
> > >>
> > >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar 
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Can someone help me understand below error? and how do I not let
> this
> > >> > happen ??
> > >> >
> > >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
> buffer.
> > >> >
> > >> > Fragment 0:0
> > >> >
> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > >> > ip-10-0-103-63.ec2.internal:31010]
> > >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
> > >> unmanaged
> > >> > buffer.
> > >> >
> > >> > Fragment 0:0
> > >> >
> > >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > >> > ip-10-0-103-63.ec2.internal:31010]
> > >> >
> > >> > --
> > >> > Nitin Pawar
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Nitin Pawar
> > >
> >
> >
> >
> > --
> > Nitin Pawar
> >
>



-- 
Nitin Pawar

Re: Drill error

[Khurram Faraaz]

Hi Nitin,

Can you please share the description of the table (i.e. column types) ?
Is this a parquet table or JSON ?
Also please share the version of Drill and the drillbit.log

Thanks,
Khurram

On Wed, Jun 27, 2018 at 9:45 AM, Nitin Pawar 
wrote:

> here are the details
>
> query:
>
> select Account Account,Name Name,CONCAT(DateString , string2)
> Merged_String from dfs.tmp.`/nitin/` t1
>
> There is no custom UDF in this query;
>
>
>
> On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar 
> wrote:
>
> > Hi Vitalii,
> >
> > Thanks for the description.
> > I will try to get the log for this but as this is happening in production
> > and drill log gets overwritten in our case very fast.
> >
> > Also I will get the query for the same while getting the log
> >
> > I will get the details and update the thread
> >
> > Thanks,
> > Nitin
> >
> > On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
> > vitalii.dira...@gmail.com> wrote:
> >
> >> Hi Nitin,
> >>
> >> It happens in the process of reallocation of the size of buffers in the
> >> memory.
> >> It isn't a User Exception, so it looks like a bug, if you get it in some
> >> existed plugin.
> >> But to say you exactly, please describe your case. What kind of query
> did
> >> you perform, any UDF's, which data source?
> >> Also logs can help.
> >>
> >> Thanks.
> >>
> >> Kind regards
> >> Vitalii
> >>
> >>
> >> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar 
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Can someone help me understand below error? and how do I not let this
> >> > happen ??
> >> >
> >> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer.
> >> >
> >> > Fragment 0:0
> >> >
> >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> >> > ip-10-0-103-63.ec2.internal:31010]
> >> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
> >> unmanaged
> >> > buffer.
> >> >
> >> > Fragment 0:0
> >> >
> >> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> >> > ip-10-0-103-63.ec2.internal:31010]
> >> >
> >> > --
> >> > Nitin Pawar
> >> >
> >>
> >
> >
> >
> > --
> > Nitin Pawar
> >
>
>
>
> --
> Nitin Pawar
>

Re: Drill error

[Nitin Pawar]

here are the details

query:

select Account Account,Name Name,CONCAT(DateString , string2)
Merged_String from dfs.tmp.`/nitin/` t1

There is no custom UDF in this query;



On Wed, Jun 27, 2018 at 2:22 PM, Nitin Pawar 
wrote:

> Hi Vitalii,
>
> Thanks for the description.
> I will try to get the log for this but as this is happening in production
> and drill log gets overwritten in our case very fast.
>
> Also I will get the query for the same while getting the log
>
> I will get the details and update the thread
>
> Thanks,
> Nitin
>
> On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka <
> vitalii.dira...@gmail.com> wrote:
>
>> Hi Nitin,
>>
>> It happens in the process of reallocation of the size of buffers in the
>> memory.
>> It isn't a User Exception, so it looks like a bug, if you get it in some
>> existed plugin.
>> But to say you exactly, please describe your case. What kind of query did
>> you perform, any UDF's, which data source?
>> Also logs can help.
>>
>> Thanks.
>>
>> Kind regards
>> Vitalii
>>
>>
>> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar 
>> wrote:
>>
>> > Hi,
>> >
>> > Can someone help me understand below error? and how do I not let this
>> > happen ??
>> >
>> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer.
>> >
>> > Fragment 0:0
>> >
>> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>> > ip-10-0-103-63.ec2.internal:31010]
>> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove
>> unmanaged
>> > buffer.
>> >
>> > Fragment 0:0
>> >
>> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
>> > ip-10-0-103-63.ec2.internal:31010]
>> >
>> > --
>> > Nitin Pawar
>> >
>>
>
>
>
> --
> Nitin Pawar
>



-- 
Nitin Pawar

Re: Drill error

[Nitin Pawar]

Hi Vitalii,

Thanks for the description.
I will try to get the log for this but as this is happening in production
and drill log gets overwritten in our case very fast.

Also I will get the query for the same while getting the log

I will get the details and update the thread

Thanks,
Nitin

On Tue, Jun 26, 2018 at 7:32 PM, Vitalii Diravka 
wrote:

> Hi Nitin,
>
> It happens in the process of reallocation of the size of buffers in the
> memory.
> It isn't a User Exception, so it looks like a bug, if you get it in some
> existed plugin.
> But to say you exactly, please describe your case. What kind of query did
> you perform, any UDF's, which data source?
> Also logs can help.
>
> Thanks.
>
> Kind regards
> Vitalii
>
>
> On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar 
> wrote:
>
> > Hi,
> >
> > Can someone help me understand below error? and how do I not let this
> > happen ??
> >
> > SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer.
> >
> > Fragment 0:0
> >
> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > ip-10-0-103-63.ec2.internal:31010]
> > Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
> > buffer.
> >
> > Fragment 0:0
> >
> > [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> > ip-10-0-103-63.ec2.internal:31010]
> >
> > --
> > Nitin Pawar
> >
>



-- 
Nitin Pawar

Re: Drill error

[Vitalii Diravka]

Hi Nitin,

It happens in the process of reallocation of the size of buffers in the
memory.
It isn't a User Exception, so it looks like a bug, if you get it in some
existed plugin.
But to say you exactly, please describe your case. What kind of query did
you perform, any UDF's, which data source?
Also logs can help.

Thanks.

Kind regards
Vitalii


On Tue, Jun 26, 2018 at 1:27 PM Nitin Pawar  wrote:

> Hi,
>
> Can someone help me understand below error? and how do I not let this
> happen ??
>
> SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged buffer.
>
> Fragment 0:0
>
> [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> ip-10-0-103-63.ec2.internal:31010]
> Caused By: SYSTEM ERROR: IllegalStateException: Tried to remove unmanaged
> buffer.
>
> Fragment 0:0
>
> [Error Id: bcd510f6-75ee-49a7-b723-7b35d8575623 on
> ip-10-0-103-63.ec2.internal:31010]
>
> --
> Nitin Pawar
>

RE: drill error connecting to Hbase

[Shai Shapira]

We re-install drill, with newer version ( 1.11) and played a bit in with the 
configuration using the Web access and made it work.

Thanks a lot for your help!!

Thanks,
Shai

-Original Message-
From: Dor Ben Dov 
Sent: Sunday, August 06, 2017 2:09 PM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

Hi Kunal,

I am assisting Shai with the drill, I followed you instructions but once I am 
running maven with the profile of cloudera aka 'cdh' I am receiving this
[dor@dor-fedora64 drill]$ mvn -U -DskipTests clean install -Pcdh [INFO] 
Scanning for projects...
Downloading: 
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/14/apache-14.pom
Downloading: http://conjars.org/repo/org/apache/apache/14/apache-14.pom
Downloading: http://repository.mapr.com/maven/org/apache/apache/14/apache-14.pom
Downloading: http://repo.dremio.com/release/org/apache/apache/14/apache-14.pom
Downloading: 
http://repository.mapr.com/nexus/content/repositories/drill/org/apache/apache/14/apache-14.pom
Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: Could 
not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11  @ [ERROR] The build could not 
read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.apache.drill:drill-root:1.11.0 
(/home/dor/Downloads/drill/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: 
Could not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11: Unknown host 
repository.cloudera.com: Name or service not known -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[dor@dor-fedora64 drill]$


** I am using fedora 26 **

Regards,
Dor

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com]
Sent: יום ה 03 אוגוסט 2017 20:53
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

The failure appears to be coming from this:

Caused by: java.lang.IllegalAccessError: Class 
org/apache/hadoop/hbase/zookeeper/MetaTableLocator illegally accessing "package 
private" member of class com/google/common/base/Stopwatch


Scrolling up a bit, I noticed that during the startup, there is an error here:
2017-08-03 11:04:58,957 [main] WARN  o.a.drill.exec.util.GuavaPatcher - Unable 
to patch Guava classes.
javassist.CannotCompileException: by java.lang.LinkageError: 
com.google.common.base.Stopwatch

When you build your Drill package, you can specify a profile. 

https://github.com/apache/drill/blob/master/pom.xml#L953

You can choose the platform you need Drill for and build with that. This tells 
Maven to apply specific versions of some dependencies that will work.

e.g.
mvn -U -DskipTests clean install -P

If there is an issue, let us know the specs of the platform you are building 
against. It is possible that there might have been upgrades to the dependencies 
within these platforms that are not being reflected in the pon.xml.



-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com]
Sent: Thursday, August 03, 2017 1:13 AM
To: user@drill.apache.org; Kunal Khatua <kkha...@mapr.com>
Subject: RE: drill error connecting to Hbase

Attached the relevant part from the sqlline.log, Hope it helps


Thanks,
Shai


-Original Message-
From: Shai Shapira
Sent: Thursday, August 03, 2017 11:04 AM
To: kkha...@mapr.com
Cc: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

Hi,

My versions are:
Hbase - 1.2.0   
Hive - 1.1.0 

I'll send the complete stack trace.

Is Drill is so version sensitive?
Can I build a solution for production based on Drill? Or should I stick to what 
is coming with the Cloudera/Hortonworks distribution?

Thanks,
Shai


hbase shell
17/08/03 10:54:05 INFO Configuration.deprecation: hadoop.native.lib is 
deprecated. Instead, use io.native.lib.available HBase Shell; enter 
'help' for list of supported commands.
Type "exit" to leave the HBase Shell Version 1.2.0-cdh5.8.2, rUnknown, 
Sun Sep 11 11:52:54 PDT 2016


hive 

RE: drill error connecting to Hbase

[Dor Ben Dov]

Hi Kunal,

I am assisting Shai with the drill, I followed you instructions but once I am 
running maven with the profile of cloudera aka 'cdh' I am receiving this
[dor@dor-fedora64 drill]$ mvn -U -DskipTests clean install -Pcdh
[INFO] Scanning for projects...
Downloading: 
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/apache/14/apache-14.pom
Downloading: http://conjars.org/repo/org/apache/apache/14/apache-14.pom
Downloading: http://repository.mapr.com/maven/org/apache/apache/14/apache-14.pom
Downloading: http://repo.dremio.com/release/org/apache/apache/14/apache-14.pom
Downloading: 
http://repository.mapr.com/nexus/content/repositories/drill/org/apache/apache/14/apache-14.pom
Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: Could 
not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.apache.drill:drill-root:1.11.0 
(/home/dor/Downloads/drill/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.apache.drill:drill-root:1.11.0: 
Could not transfer artifact org.apache:apache:pom:14 from/to cloudera 
(https://repository.cloudera.com/artifactory/cloudera-repos/): 
repository.cloudera.com: Name or service not known and 'parent.relativePath' 
points at wrong local POM @ line 15, column 11: Unknown host 
repository.cloudera.com: Name or service not known -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
[dor@dor-fedora64 drill]$


** I am using fedora 26 **

Regards,
Dor

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com] 
Sent: יום ה 03 אוגוסט 2017 20:53
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

The failure appears to be coming from this:

Caused by: java.lang.IllegalAccessError: Class 
org/apache/hadoop/hbase/zookeeper/MetaTableLocator illegally accessing "package 
private" member of class com/google/common/base/Stopwatch


Scrolling up a bit, I noticed that during the startup, there is an error here:
2017-08-03 11:04:58,957 [main] WARN  o.a.drill.exec.util.GuavaPatcher - Unable 
to patch Guava classes.
javassist.CannotCompileException: by java.lang.LinkageError: 
com.google.common.base.Stopwatch

When you build your Drill package, you can specify a profile. 

https://github.com/apache/drill/blob/master/pom.xml#L953

You can choose the platform you need Drill for and build with that. This tells 
Maven to apply specific versions of some dependencies that will work.

e.g.
mvn -U -DskipTests clean install -P

If there is an issue, let us know the specs of the platform you are building 
against. It is possible that there might have been upgrades to the dependencies 
within these platforms that are not being reflected in the pon.xml.



-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Thursday, August 03, 2017 1:13 AM
To: user@drill.apache.org; Kunal Khatua <kkha...@mapr.com>
Subject: RE: drill error connecting to Hbase

Attached the relevant part from the sqlline.log, Hope it helps


Thanks,
Shai


-Original Message-
From: Shai Shapira 
Sent: Thursday, August 03, 2017 11:04 AM
To: kkha...@mapr.com
Cc: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

Hi,

My versions are:
Hbase - 1.2.0   
Hive - 1.1.0 

I'll send the complete stack trace.

Is Drill is so version sensitive?
Can I build a solution for production based on Drill? Or should I stick to what 
is coming with the Cloudera/Hortonworks distribution?

Thanks,
Shai


hbase shell
17/08/03 10:54:05 INFO Configuration.deprecation: hadoop.native.lib is 
deprecated. Instead, use io.native.lib.available HBase Shell; enter 
'help' for list of supported commands.
Type "exit" to leave the HBase Shell Version 1.2.0-cdh5.8.2, rUnknown, 
Sun Sep 11 11:52:54 PDT 2016


hive shell

Logging initialized using configuration in 
jar:file:/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p1980.2070/jars/hive-common-1.1.0-cdh5.8.2.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com]

RE: drill error connecting to Hbase

[Shai Shapira]

Attached the relevant part from the sqlline.log, 
Hope it helps


Thanks,
Shai


-Original Message-
From: Shai Shapira 
Sent: Thursday, August 03, 2017 11:04 AM
To: kkha...@mapr.com
Cc: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

Hi,

My versions are:
Hbase - 1.2.0   
Hive - 1.1.0 

I'll send the complete stack trace.

Is Drill is so version sensitive?
Can I build a solution for production based on Drill? Or should I stick to what 
is coming with the Cloudera/Hortonworks distribution?

Thanks,
Shai


hbase shell
17/08/03 10:54:05 INFO Configuration.deprecation: hadoop.native.lib is 
deprecated. Instead, use io.native.lib.available HBase Shell; enter 
'help' for list of supported commands.
Type "exit" to leave the HBase Shell Version 1.2.0-cdh5.8.2, rUnknown, 
Sun Sep 11 11:52:54 PDT 2016


hive shell

Logging initialized using configuration in 
jar:file:/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p1980.2070/jars/hive-common-1.1.0-cdh5.8.2.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com]
Sent: Wednesday, July 26, 2017 8:58 PM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

The bundled projects (HBase, Hive) in CDH have their own versions. I'm 
wondering if that is what is the difference.

Drill has been tested with HBase 1.1.1 and Hive 1.2.1 . For higher versions, as 
long as APIs have not changed, things should be backward compatible. 

Also, the error message you see in the SQLLine session... there is a complete 
stack trace in the Drill logs. Can you share that stack trace as well?


-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com]
Sent: Wednesday, July 26, 2017 5:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

It is CDH 5.8.2

I believe it is reliable versions, isn't it?

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com]
Sent: Monday, July 24, 2017 8:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

This means that the connectivity with ZK appears to be working. 

What are the HBase, ZK and Hadoop versions that you are working with? I presume 
that the student table is otherwise accessible.

-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com]
Sent: Sunday, July 23, 2017 2:58 AM
To: user@drill.apache.org
Cc: Shai Shapira <shai.shap...@amdocs.com>
Subject: RE: drill error connecting to Hbase

Hi,

I installed Drill and started to work with it, my goal is to use it to connect 
to Hbase.
I checked it a bit locally, csv files, Json files, works great.
When I am trying to connect to Hbase, I am getting error.

It seems that it is connecting to the Hbase/ZK, but fails somehow there.
The errors when trying to select from non-exist table ( stud ) and when 
accessing an existing table ( students ) are different.
For existing table, the error is in the zookeeper.MetaTableLocator.

Any ideas?

Thanks,
Shai




illin4620 STABDB05 54 > drill
Jul 20, 2017 6:17:02 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"just drill it"
0: jdbc:drill:zk=local> use hbase;
+---++
|  ok   |  summary   |
+---++
| true  | Default schema changed to [hbase]  |
+---++
1 row selected (0.895 seconds)
0: jdbc:drill:zk=local> select * from students ;
Error: SYSTEM ERROR: IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator


[Error Id: 71a4a041-4f5d-4e68-9400-78c2faeac6f9 on illin4620:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> select * from stud ;
Error: DATA_READ ERROR: Failure while loading table stud in database hbase.

Message:  stud
SQL Query null

[Error Id: f0a6591d-9068-4490-95c0-b0aea41365b4 on illin4620:31010] 
(state=,code=0)


Thanks,
Shai

From: Shai Shapira
Sent: Sunday, July 23, 2017 12:49 PM
To: Shai Shapira <shai.shap...@amdocs.com>
Subject: drill error connecting to Hbase



Shai Shapira
*  shai.shap...@amdocs.com<mailto:shai.shap...@amdocs.com>
* +972 9 776 4171

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

This message and the information

RE: drill error connecting to Hbase

[Shai Shapira]

Hi,

My versions are:
Hbase - 1.2.0   
Hive - 1.1.0 

I'll send the complete stack trace.

Is Drill is so version sensitive?
Can I build a solution for production based on Drill? Or should I stick to what 
is coming with the Cloudera/Hortonworks distribution?

Thanks,
Shai


hbase shell
17/08/03 10:54:05 INFO Configuration.deprecation: hadoop.native.lib is 
deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help' for list of supported commands.
Type "exit" to leave the HBase Shell
Version 1.2.0-cdh5.8.2, rUnknown, Sun Sep 11 11:52:54 PDT 2016


hive shell

Logging initialized using configuration in 
jar:file:/opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p1980.2070/jars/hive-common-1.1.0-cdh5.8.2.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com] 
Sent: Wednesday, July 26, 2017 8:58 PM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

The bundled projects (HBase, Hive) in CDH have their own versions. I'm 
wondering if that is what is the difference.

Drill has been tested with HBase 1.1.1 and Hive 1.2.1 . For higher versions, as 
long as APIs have not changed, things should be backward compatible. 

Also, the error message you see in the SQLLine session... there is a complete 
stack trace in the Drill logs. Can you share that stack trace as well?


-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Wednesday, July 26, 2017 5:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

It is CDH 5.8.2

I believe it is reliable versions, isn't it?

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com] 
Sent: Monday, July 24, 2017 8:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

This means that the connectivity with ZK appears to be working. 

What are the HBase, ZK and Hadoop versions that you are working with? I presume 
that the student table is otherwise accessible.

-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Sunday, July 23, 2017 2:58 AM
To: user@drill.apache.org
Cc: Shai Shapira <shai.shap...@amdocs.com>
Subject: RE: drill error connecting to Hbase

Hi,

I installed Drill and started to work with it, my goal is to use it to connect 
to Hbase.
I checked it a bit locally, csv files, Json files, works great.
When I am trying to connect to Hbase, I am getting error.

It seems that it is connecting to the Hbase/ZK, but fails somehow there.
The errors when trying to select from non-exist table ( stud ) and when 
accessing an existing table ( students ) are different.
For existing table, the error is in the zookeeper.MetaTableLocator.

Any ideas?

Thanks,
Shai




illin4620 STABDB05 54 > drill
Jul 20, 2017 6:17:02 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"just drill it"
0: jdbc:drill:zk=local> use hbase;
+---++
|  ok   |  summary   |
+---++
| true  | Default schema changed to [hbase]  |
+---++
1 row selected (0.895 seconds)
0: jdbc:drill:zk=local> select * from students ;
Error: SYSTEM ERROR: IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator


[Error Id: 71a4a041-4f5d-4e68-9400-78c2faeac6f9 on illin4620:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> select * from stud ;
Error: DATA_READ ERROR: Failure while loading table stud in database hbase.

Message:  stud
SQL Query null

[Error Id: f0a6591d-9068-4490-95c0-b0aea41365b4 on illin4620:31010] 
(state=,code=0)


Thanks,
Shai

From: Shai Shapira
Sent: Sunday, July 23, 2017 12:49 PM
To: Shai Shapira <shai.shap...@amdocs.com>
Subject: drill error connecting to Hbase



Shai Shapira
*  shai.shap...@amdocs.com<mailto:shai.shap...@amdocs.com>
* +972 9 776 4171

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

RE: drill error connecting to Hbase

[Kunal Khatua]

The bundled projects (HBase, Hive) in CDH have their own versions. I'm 
wondering if that is what is the difference.

Drill has been tested with HBase 1.1.1 and Hive 1.2.1 . For higher versions, as 
long as APIs have not changed, things should be backward compatible. 

Also, the error message you see in the SQLLine session... there is a complete 
stack trace in the Drill logs. Can you share that stack trace as well?


-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Wednesday, July 26, 2017 5:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

It is CDH 5.8.2

I believe it is reliable versions, isn't it?

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com] 
Sent: Monday, July 24, 2017 8:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

This means that the connectivity with ZK appears to be working. 

What are the HBase, ZK and Hadoop versions that you are working with? I presume 
that the student table is otherwise accessible.

-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Sunday, July 23, 2017 2:58 AM
To: user@drill.apache.org
Cc: Shai Shapira <shai.shap...@amdocs.com>
Subject: RE: drill error connecting to Hbase

Hi,

I installed Drill and started to work with it, my goal is to use it to connect 
to Hbase.
I checked it a bit locally, csv files, Json files, works great.
When I am trying to connect to Hbase, I am getting error.

It seems that it is connecting to the Hbase/ZK, but fails somehow there.
The errors when trying to select from non-exist table ( stud ) and when 
accessing an existing table ( students ) are different.
For existing table, the error is in the zookeeper.MetaTableLocator.

Any ideas?

Thanks,
Shai




illin4620 STABDB05 54 > drill
Jul 20, 2017 6:17:02 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"just drill it"
0: jdbc:drill:zk=local> use hbase;
+---++
|  ok   |  summary   |
+---++
| true  | Default schema changed to [hbase]  |
+---++
1 row selected (0.895 seconds)
0: jdbc:drill:zk=local> select * from students ;
Error: SYSTEM ERROR: IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator


[Error Id: 71a4a041-4f5d-4e68-9400-78c2faeac6f9 on illin4620:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> select * from stud ;
Error: DATA_READ ERROR: Failure while loading table stud in database hbase.

Message:  stud
SQL Query null

[Error Id: f0a6591d-9068-4490-95c0-b0aea41365b4 on illin4620:31010] 
(state=,code=0)


Thanks,
Shai

From: Shai Shapira
Sent: Sunday, July 23, 2017 12:49 PM
To: Shai Shapira <shai.shap...@amdocs.com>
Subject: drill error connecting to Hbase



Shai Shapira
*  shai.shap...@amdocs.com<mailto:shai.shap...@amdocs.com>
* +972 9 776 4171

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

RE: drill error connecting to Hbase

[Shai Shapira]

It is CDH 5.8.2

I believe it is reliable versions, isn't it?

Thanks,
Shai

-Original Message-
From: Kunal Khatua [mailto:kkha...@mapr.com] 
Sent: Monday, July 24, 2017 8:50 AM
To: user@drill.apache.org
Subject: RE: drill error connecting to Hbase

This means that the connectivity with ZK appears to be working. 

What are the HBase, ZK and Hadoop versions that you are working with? I presume 
that the student table is otherwise accessible.

-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Sunday, July 23, 2017 2:58 AM
To: user@drill.apache.org
Cc: Shai Shapira <shai.shap...@amdocs.com>
Subject: RE: drill error connecting to Hbase

Hi,

I installed Drill and started to work with it, my goal is to use it to connect 
to Hbase.
I checked it a bit locally, csv files, Json files, works great.
When I am trying to connect to Hbase, I am getting error.

It seems that it is connecting to the Hbase/ZK, but fails somehow there.
The errors when trying to select from non-exist table ( stud ) and when 
accessing an existing table ( students ) are different.
For existing table, the error is in the zookeeper.MetaTableLocator.

Any ideas?

Thanks,
Shai




illin4620 STABDB05 54 > drill
Jul 20, 2017 6:17:02 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"just drill it"
0: jdbc:drill:zk=local> use hbase;
+---++
|  ok   |  summary   |
+---++
| true  | Default schema changed to [hbase]  |
+---++
1 row selected (0.895 seconds)
0: jdbc:drill:zk=local> select * from students ;
Error: SYSTEM ERROR: IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator


[Error Id: 71a4a041-4f5d-4e68-9400-78c2faeac6f9 on illin4620:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> select * from stud ;
Error: DATA_READ ERROR: Failure while loading table stud in database hbase.

Message:  stud
SQL Query null

[Error Id: f0a6591d-9068-4490-95c0-b0aea41365b4 on illin4620:31010] 
(state=,code=0)


Thanks,
Shai

From: Shai Shapira
Sent: Sunday, July 23, 2017 12:49 PM
To: Shai Shapira <shai.shap...@amdocs.com>
Subject: drill error connecting to Hbase



Shai Shapira
*  shai.shap...@amdocs.com<mailto:shai.shap...@amdocs.com>
* +972 9 776 4171

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>
This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

RE: drill error connecting to Hbase

[Kunal Khatua]

This means that the connectivity with ZK appears to be working. 

What are the HBase, ZK and Hadoop versions that you are working with? I presume 
that the student table is otherwise accessible.

-Original Message-
From: Shai Shapira [mailto:shai.shap...@amdocs.com] 
Sent: Sunday, July 23, 2017 2:58 AM
To: user@drill.apache.org
Cc: Shai Shapira <shai.shap...@amdocs.com>
Subject: RE: drill error connecting to Hbase

Hi,

I installed Drill and started to work with it, my goal is to use it to connect 
to Hbase.
I checked it a bit locally, csv files, Json files, works great.
When I am trying to connect to Hbase, I am getting error.

It seems that it is connecting to the Hbase/ZK, but fails somehow there.
The errors when trying to select from non-exist table ( stud ) and when 
accessing an existing table ( students ) are different.
For existing table, the error is in the zookeeper.MetaTableLocator.

Any ideas?

Thanks,
Shai




illin4620 STABDB05 54 > drill
Jul 20, 2017 6:17:02 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"just drill it"
0: jdbc:drill:zk=local> use hbase;
+---++
|  ok   |  summary   |
+---++
| true  | Default schema changed to [hbase]  |
+---++
1 row selected (0.895 seconds)
0: jdbc:drill:zk=local> select * from students ;
Error: SYSTEM ERROR: IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator


[Error Id: 71a4a041-4f5d-4e68-9400-78c2faeac6f9 on illin4620:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> select * from stud ;
Error: DATA_READ ERROR: Failure while loading table stud in database hbase.

Message:  stud
SQL Query null

[Error Id: f0a6591d-9068-4490-95c0-b0aea41365b4 on illin4620:31010] 
(state=,code=0)


Thanks,
Shai

From: Shai Shapira
Sent: Sunday, July 23, 2017 12:49 PM
To: Shai Shapira <shai.shap...@amdocs.com>
Subject: drill error connecting to Hbase



Shai Shapira
*  shai.shap...@amdocs.com<mailto:shai.shap...@amdocs.com>
* +972 9 776 4171

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 
<https://www.amdocs.com/about/email-disclaimer>

RE: drill error connecting to Hbase

[Shai Shapira]

Hi,

I installed Drill and started to work with it, my goal is to use it to connect 
to Hbase.
I checked it a bit locally, csv files, Json files, works great.
When I am trying to connect to Hbase, I am getting error.

It seems that it is connecting to the Hbase/ZK, but fails somehow there.
The errors when trying to select from non-exist table ( stud ) and when 
accessing an existing table ( students ) are different.
For existing table, the error is in the zookeeper.MetaTableLocator.

Any ideas?

Thanks,
Shai




illin4620 STABDB05 54 > drill
Jul 20, 2017 6:17:02 PM org.glassfish.jersey.server.ApplicationHandler 
initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.10.0
"just drill it"
0: jdbc:drill:zk=local> use hbase;
+---++
|  ok   |  summary   |
+---++
| true  | Default schema changed to [hbase]  |
+---++
1 row selected (0.895 seconds)
0: jdbc:drill:zk=local> select * from students ;
Error: SYSTEM ERROR: IllegalAccessError: tried to access method 
com.google.common.base.Stopwatch.()V from class 
org.apache.hadoop.hbase.zookeeper.MetaTableLocator


[Error Id: 71a4a041-4f5d-4e68-9400-78c2faeac6f9 on illin4620:31010] 
(state=,code=0)
0: jdbc:drill:zk=local> select * from stud ;
Error: DATA_READ ERROR: Failure while loading table stud in database hbase.

Message:  stud
SQL Query null

[Error Id: f0a6591d-9068-4490-95c0-b0aea41365b4 on illin4620:31010] 
(state=,code=0)


Thanks,
Shai

From: Shai Shapira
Sent: Sunday, July 23, 2017 12:49 PM
To: Shai Shapira 
Subject: drill error connecting to Hbase



Shai Shapira
*  shai.shap...@amdocs.com
* +972 9 776 4171

This message and the information contained herein is proprietary and 
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer 

RE: Drill error with large sort

[Paul Friedman]

Thanks for this, these parameters fixed it!

---Paul


-Original Message-
From: Abdel Hakim Deneche [mailto:adene...@maprtech.com]
Sent: Thursday, February 25, 2016 5:32 PM
To: user <user@drill.apache.org>
Subject: Re: Drill error with large sort

Not so short answer:

In Drill 1.5 (I assume you are using 1.5) we have an improved allocator that
better tracks how much memory each operator is using. In your case it seems
that the date has very wide columns that are causing Sort to choke on the
very first batch of data (1024 records taking up 224MB!!!) because it's way
more than it's memory limit (around 178MB in your particular case).
Drill uses a fancy equation to compute this limit and increasing the
aforementioned option will increase the sort limit. More details here:

http://drill.apache.org/docs/configuring-drill-memory/

On Thu, Feb 25, 2016 at 5:26 PM, Abdel Hakim Deneche <adene...@maprtech.com>
wrote:

> Short answer:
>
> increase the value of planner.memory.max_query_memory_per_node, by
> default it's set to 2GB, try setting to 4 or even 8GB. This should get
> the query to pass.
>
> On Thu, Feb 25, 2016 at 5:24 PM, Jeff Maass <jma...@cccis.com> wrote:
>
>>
>> If you are open to changing the query:
>>   # try removing the functions on the 5th column
>>   # is there any way you could further limit the query?
>>   # does the query finish if u add a limit / top clause?
>>   # what do the logs say?
>>
>> 
>> From: Paul Friedman <paul.fried...@streetlightdata.com>
>> Sent: Thursday, February 25, 2016 7:07:12 PM
>> To: user@drill.apache.org
>> Subject: Drill error with large sort
>>
>> I’ve got a query reading from a large directory of parquet files (41
>> GB) and I’m consistently getting this error:
>>
>>
>>
>> Error: RESOURCE ERROR: One or more nodes ran out of memory while
>> executing the query.
>>
>>
>>
>> Unable to allocate sv2 for 1023 records, and not enough batchGroups
>> to spill.
>>
>> batchGroups.size 0
>>
>> spilledBatchGroups.size 0
>>
>> allocated memory 224287987
>>
>> allocator limit 178956970
>>
>> Fragment 0:0
>>
>>
>>
>> [Error Id: 878d604c-4656-4a5a-8b46-ff38a6ae020d on
>> chai.dev.streetlightdata.com:31010] (state=,code=0)
>>
>>
>>
>> Direct memory is set to 48GB and heap is 8GB.
>>
>>
>>
>> The query is:
>>
>>
>>
>> select probe_id, provider_id, is_moving, mode,
>> cast(convert_to(points,
>> 'JSON') as varchar(1))
>>
>> from dfs.`/home/paul/data`
>>
>> where
>>
>> start_lat between 24.4873780449008 and 60.0108911181433 and
>>
>> start_lon between -139.065890469841 and -52.8305074899881 and
>>
>> provider_id = '343' and
>>
>> mod(abs(hash(probe_id)),  100) = 0
>>
>> order by probe_id, start_time;
>>
>>
>>
>> I’m also using the “example” drill-override configuration.
>>
>>
>>
>> Any help would be appreciated.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> ---Paul
>>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <http://www.mapr.com/training?utm_source=Email_medium=Signature
> m_campaign=Free%20available>
>



-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email_medium=Signature_campaign=Free%20available>

Re: Drill error with large sort

[Abdel Hakim Deneche]

Not so short answer:

In Drill 1.5 (I assume you are using 1.5) we have an improved allocator
that better tracks how much memory each operator is using. In your case it
seems that the date has very wide columns that are causing Sort to choke on
the very first batch of data (1024 records taking up 224MB!!!) because it's
way more than it's memory limit (around 178MB in your particular case).
Drill uses a fancy equation to compute this limit and increasing the
aforementioned option will increase the sort limit. More details here:

http://drill.apache.org/docs/configuring-drill-memory/

On Thu, Feb 25, 2016 at 5:26 PM, Abdel Hakim Deneche 
wrote:

> Short answer:
>
> increase the value of planner.memory.max_query_memory_per_node, by default
> it's set to 2GB, try setting to 4 or even 8GB. This should get the query to
> pass.
>
> On Thu, Feb 25, 2016 at 5:24 PM, Jeff Maass  wrote:
>
>>
>> If you are open to changing the query:
>>   # try removing the functions on the 5th column
>>   # is there any way you could further limit the query?
>>   # does the query finish if u add a limit / top clause?
>>   # what do the logs say?
>>
>> 
>> From: Paul Friedman 
>> Sent: Thursday, February 25, 2016 7:07:12 PM
>> To: user@drill.apache.org
>> Subject: Drill error with large sort
>>
>> I’ve got a query reading from a large directory of parquet files (41 GB)
>> and I’m consistently getting this error:
>>
>>
>>
>> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing
>> the query.
>>
>>
>>
>> Unable to allocate sv2 for 1023 records, and not enough batchGroups to
>> spill.
>>
>> batchGroups.size 0
>>
>> spilledBatchGroups.size 0
>>
>> allocated memory 224287987
>>
>> allocator limit 178956970
>>
>> Fragment 0:0
>>
>>
>>
>> [Error Id: 878d604c-4656-4a5a-8b46-ff38a6ae020d on
>> chai.dev.streetlightdata.com:31010] (state=,code=0)
>>
>>
>>
>> Direct memory is set to 48GB and heap is 8GB.
>>
>>
>>
>> The query is:
>>
>>
>>
>> select probe_id, provider_id, is_moving, mode,  cast(convert_to(points,
>> 'JSON') as varchar(1))
>>
>> from dfs.`/home/paul/data`
>>
>> where
>>
>> start_lat between 24.4873780449008 and 60.0108911181433 and
>>
>> start_lon between -139.065890469841 and -52.8305074899881 and
>>
>> provider_id = '343' and
>>
>> mod(abs(hash(probe_id)),  100) = 0
>>
>> order by probe_id, start_time;
>>
>>
>>
>> I’m also using the “example” drill-override configuration.
>>
>>
>>
>> Any help would be appreciated.
>>
>>
>>
>> Thanks.
>>
>>
>>
>> ---Paul
>>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   
>
>
> Now Available - Free Hadoop On-Demand Training
> 
>



-- 

Abdelhakim Deneche

Software Engineer

  


Now Available - Free Hadoop On-Demand Training

Re: Drill error with large sort

[Jeff Maass]

If you are open to changing the query:
  # try removing the functions on the 5th column
  # is there any way you could further limit the query?
  # does the query finish if u add a limit / top clause?
  # what do the logs say?


From: Paul Friedman 
Sent: Thursday, February 25, 2016 7:07:12 PM
To: user@drill.apache.org
Subject: Drill error with large sort

I’ve got a query reading from a large directory of parquet files (41 GB)
and I’m consistently getting this error:



Error: RESOURCE ERROR: One or more nodes ran out of memory while executing
the query.



Unable to allocate sv2 for 1023 records, and not enough batchGroups to
spill.

batchGroups.size 0

spilledBatchGroups.size 0

allocated memory 224287987

allocator limit 178956970

Fragment 0:0



[Error Id: 878d604c-4656-4a5a-8b46-ff38a6ae020d on
chai.dev.streetlightdata.com:31010] (state=,code=0)



Direct memory is set to 48GB and heap is 8GB.



The query is:



select probe_id, provider_id, is_moving, mode,  cast(convert_to(points,
'JSON') as varchar(1))

from dfs.`/home/paul/data`

where

start_lat between 24.4873780449008 and 60.0108911181433 and

start_lon between -139.065890469841 and -52.8305074899881 and

provider_id = '343' and

mod(abs(hash(probe_id)),  100) = 0

order by probe_id, start_time;



I’m also using the “example” drill-override configuration.



Any help would be appreciated.



Thanks.



---Paul