Re: hive concurrency not working

[Mich Talebzadeh]

-- I am pretty sure that they will support it because the Spark option is
supported

Hortonworks support Spark but not Hive on Spark.  Their official distro is
Hive on Tez + LLAP

Not sure where you get your information from though

I got it from Hortonworks and I know that




Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 7 August 2016 at 06:51, Jörn Franke  wrote:

> I am pretty sure that they will support it because the Spark option is
> supported.
> Not sure where you get your information from though...
>
> On 05 Aug 2016, at 11:01, Mich Talebzadeh 
> wrote:
>
> great in that case they can try it and I am pretty sure if they are stuck
> they can come and ask you for expert advice since Hortonworks do not
> support Hive on Spark and I know that
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 5 August 2016 at 09:01, Jörn Franke  wrote:
>
>> That is not correct the option is there to install it.
>>
>> On 05 Aug 2016, at 08:41, Mich Talebzadeh 
>> wrote:
>>
>> You won't have this problem if you use Spark as the execution engine!
>> This set up handles concurrency but Hive with Spark is not part of the HW
>> distro.
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 5 August 2016 at 07:39, Mich Talebzadeh 
>> wrote:
>>
>>> you won't have this problem if you use Spark as the execution engine?
>>> That handles concurrency OK
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 5 August 2016 at 06:23, Raj hadoop  wrote:
>>>
 Thanks everyone..

 we are raising case with Hortonworks

 On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop 
 wrote:

> Dear All,
>
> In need or your help,
>
> we have horton works 4 node cluster,and the problem is hive is
> allowing only one user at a time,
>
> if any second resource need to login hive is not working,
>
> could someone please help me in this
>
> Thanks,
> Rajesh
>


>>>
>>
>

Re: hive concurrency not working

[Jörn Franke]

I am pretty sure that they will support it because the Spark option is 
supported.
Not sure where you get your information from though...

> On 05 Aug 2016, at 11:01, Mich Talebzadeh  wrote:
> 
> great in that case they can try it and I am pretty sure if they are stuck 
> they can come and ask you for expert advice since Hortonworks do not support 
> Hive on Spark and I know that
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
>> On 5 August 2016 at 09:01, Jörn Franke  wrote:
>> That is not correct the option is there to install it.
>> 
>>> On 05 Aug 2016, at 08:41, Mich Talebzadeh  wrote:
>>> 
>>> You won't have this problem if you use Spark as the execution engine! This 
>>> set up handles concurrency but Hive with Spark is not part of the HW distro.
>>> 
>>> HTH
>>> 
>>> Dr Mich Talebzadeh
>>>  
>>> LinkedIn  
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>  
>>> http://talebzadehmich.wordpress.com
>>> 
>>> Disclaimer: Use it at your own risk. Any and all responsibility for any 
>>> loss, damage or destruction of data or any other property which may arise 
>>> from relying on this email's technical content is explicitly disclaimed. 
>>> The author will in no case be liable for any monetary damages arising from 
>>> such loss, damage or destruction.
>>>  
>>> 
 On 5 August 2016 at 07:39, Mich Talebzadeh  
 wrote:
 you won't have this problem if you use Spark as the execution engine? That 
 handles concurrency OK
 
 Dr Mich Talebzadeh
  
 LinkedIn  
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
  
 http://talebzadehmich.wordpress.com
 
 Disclaimer: Use it at your own risk. Any and all responsibility for any 
 loss, damage or destruction of data or any other property which may arise 
 from relying on this email's technical content is explicitly disclaimed. 
 The author will in no case be liable for any monetary damages arising from 
 such loss, damage or destruction.
  
 
> On 5 August 2016 at 06:23, Raj hadoop  wrote:
> Thanks everyone..
> 
> we are raising case with Hortonworks
> 
>> On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:
>> Dear All,
>> 
>> In need or your help,
>> 
>> we have horton works 4 node cluster,and the problem is hive is allowing 
>> only one user at a time,
>> 
>> if any second resource need to login hive is not working,
>> 
>> could someone please help me in this
>> 
>> Thanks,
>> Rajesh
> 

Re: hive concurrency not working

[Mich Talebzadeh]

Actually that is correct with regard to MR. The standard MapReduce as I
know reads the data from HDFS, apply map-reduce algorithm and writes back
to HDFS. If there are many iterations of map-reduce then, there will be
many intermediate writes to HDFS. This is all serial writes to disk. Each
map-reduce step is completely independent of other steps, and the executing
engine does not have any global knowledge of what map-reduce steps are
going to come after each map-reduce step.

However, having said that I thought Tez is essentially MR with DAG and
sounds like it still has some issues.

I wonder whether hive.support.concurrency is set to true with zookeeper
running and hive.lock.manager set to
org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

HTH

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 5 August 2016 at 23:46, Gopal Vijayaraghavan  wrote:

>
> > Depends on how you configured scheduling in yarn ...
> ...
>
> >> you won't have this problem if you use Spark as the execution engine?
> >>That handles concurrency OK
>
> If I read this right, it is unlikely to be related to YARN configs.
>
> The Hue issue is directly related to how many Tez/Spark sessions are
> supported per-connection-handle.
>
> hive.server2.parallel.ops.in.session
>
> I would guess that this is queuing up in  the
> getSessionManager().submitBackgroundOperation() call talking to
> SparkSessionManagerImpl/TezSessionPoolManager.
>
>
> MR has no equivalent of a "session" in relation to the cluster, because as
> a non-DAG engine  it has to have the "parallel job queue" built over to
> support parallel stages.
>
>
> Cheers,
> Gopal
>
>
>

Re: hive concurrency not working

[Gopal Vijayaraghavan]

> Depends on how you configured scheduling in yarn ...
...

>> you won't have this problem if you use Spark as the execution engine?
>>That handles concurrency OK

If I read this right, it is unlikely to be related to YARN configs.

The Hue issue is directly related to how many Tez/Spark sessions are
supported per-connection-handle.

hive.server2.parallel.ops.in.session

I would guess that this is queuing up in  the
getSessionManager().submitBackgroundOperation() call talking to
SparkSessionManagerImpl/TezSessionPoolManager.


MR has no equivalent of a "session" in relation to the cluster, because as
a non-DAG engine  it has to have the "parallel job queue" built over to
support parallel stages.


Cheers,
Gopal

Re: hive concurrency not working

[Mich Talebzadeh]

great in that case they can try it and I am pretty sure if they are stuck
they can come and ask you for expert advice since Hortonworks do not
support Hive on Spark and I know that

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 5 August 2016 at 09:01, Jörn Franke  wrote:

> That is not correct the option is there to install it.
>
> On 05 Aug 2016, at 08:41, Mich Talebzadeh 
> wrote:
>
> You won't have this problem if you use Spark as the execution engine! This
> set up handles concurrency but Hive with Spark is not part of the HW distro.
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 5 August 2016 at 07:39, Mich Talebzadeh 
> wrote:
>
>> you won't have this problem if you use Spark as the execution engine?
>> That handles concurrency OK
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 5 August 2016 at 06:23, Raj hadoop  wrote:
>>
>>> Thanks everyone..
>>>
>>> we are raising case with Hortonworks
>>>
>>> On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:
>>>
 Dear All,

 In need or your help,

 we have horton works 4 node cluster,and the problem is hive is allowing
 only one user at a time,

 if any second resource need to login hive is not working,

 could someone please help me in this

 Thanks,
 Rajesh

>>>
>>>
>>
>

Re: hive concurrency not working

[Jörn Franke]

Depends on how you configured scheduling in yarn ...

> On 05 Aug 2016, at 08:39, Mich Talebzadeh  wrote:
> 
> you won't have this problem if you use Spark as the execution engine? That 
> handles concurrency OK
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
>> On 5 August 2016 at 06:23, Raj hadoop  wrote:
>> Thanks everyone..
>> 
>> we are raising case with Hortonworks
>> 
>>> On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:
>>> Dear All,
>>> 
>>> In need or your help,
>>> 
>>> we have horton works 4 node cluster,and the problem is hive is allowing 
>>> only one user at a time,
>>> 
>>> if any second resource need to login hive is not working,
>>> 
>>> could someone please help me in this
>>> 
>>> Thanks,
>>> Rajesh
> 

Re: hive concurrency not working

[Jörn Franke]

That is not correct the option is there to install it.

> On 05 Aug 2016, at 08:41, Mich Talebzadeh  wrote:
> 
> You won't have this problem if you use Spark as the execution engine! This 
> set up handles concurrency but Hive with Spark is not part of the HW distro.
> 
> HTH
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
>> On 5 August 2016 at 07:39, Mich Talebzadeh  wrote:
>> you won't have this problem if you use Spark as the execution engine? That 
>> handles concurrency OK
>> 
>> Dr Mich Talebzadeh
>>  
>> LinkedIn  
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>  
>> http://talebzadehmich.wordpress.com
>> 
>> Disclaimer: Use it at your own risk. Any and all responsibility for any 
>> loss, damage or destruction of data or any other property which may arise 
>> from relying on this email's technical content is explicitly disclaimed. The 
>> author will in no case be liable for any monetary damages arising from such 
>> loss, damage or destruction.
>>  
>> 
>>> On 5 August 2016 at 06:23, Raj hadoop  wrote:
>>> Thanks everyone..
>>> 
>>> we are raising case with Hortonworks
>>> 
 On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:
 Dear All,
 
 In need or your help,
 
 we have horton works 4 node cluster,and the problem is hive is allowing 
 only one user at a time,
 
 if any second resource need to login hive is not working,
 
 could someone please help me in this
 
 Thanks,
 Rajesh
> 

Re: hive concurrency not working

[Mich Talebzadeh]

You won't have this problem if you use Spark as the execution engine! This
set up handles concurrency but Hive with Spark is not part of the HW distro.

HTH

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 5 August 2016 at 07:39, Mich Talebzadeh 
wrote:

> you won't have this problem if you use Spark as the execution engine? That
> handles concurrency OK
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 5 August 2016 at 06:23, Raj hadoop  wrote:
>
>> Thanks everyone..
>>
>> we are raising case with Hortonworks
>>
>> On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:
>>
>>> Dear All,
>>>
>>> In need or your help,
>>>
>>> we have horton works 4 node cluster,and the problem is hive is allowing
>>> only one user at a time,
>>>
>>> if any second resource need to login hive is not working,
>>>
>>> could someone please help me in this
>>>
>>> Thanks,
>>> Rajesh
>>>
>>
>>
>

Re: hive concurrency not working

[Mich Talebzadeh]

you won't have this problem if you use Spark as the execution engine? That
handles concurrency OK

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 5 August 2016 at 06:23, Raj hadoop  wrote:

> Thanks everyone..
>
> we are raising case with Hortonworks
>
> On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:
>
>> Dear All,
>>
>> In need or your help,
>>
>> we have horton works 4 node cluster,and the problem is hive is allowing
>> only one user at a time,
>>
>> if any second resource need to login hive is not working,
>>
>> could someone please help me in this
>>
>> Thanks,
>> Rajesh
>>
>
>

Re: hive concurrency not working

[Raj hadoop]

Thanks everyone..

we are raising case with Hortonworks

On Wed, Aug 3, 2016 at 6:44 PM, Raj hadoop  wrote:

> Dear All,
>
> In need or your help,
>
> we have horton works 4 node cluster,and the problem is hive is allowing
> only one user at a time,
>
> if any second resource need to login hive is not working,
>
> could someone please help me in this
>
> Thanks,
> Rajesh
>

Re: hive concurrency not working

[Divakar Reddy]

Reg " hive concurrency not working" in HDP.

yes, it's known issue in HDP and with Hue 2.6

I have below information on this issue and hope it will help you.

When you are running Hive queries through Hue (Beeswax), users are unable
to run multiple queries concurrently. In practice, this doesn't matter if
it is separate browser sessions, separate clients, etc. it seems to be tied
to the user.
In looking at the way Tez works and looking through the code for the patch
in Hive 0.14 that supports concurrent queries in general with Tez, it does
not support parallel queries in a particular TezSession, only serial
queries. This is also documented in Tez documentation. It seems the way
that Hive creates a session is based upon the user.  Upon further digging,
we found a ticket HIVE-9223 that is in open state which describes this
issue.
---

Regards,
Divakar


On Wed, Aug 3, 2016 at 11:50 AM, Sergey Shelukhin 
wrote:

> Can you elaborate on not working? Is it giving an error, or hanging (and
> if so, does it queue and eventually execute); are you using HS2; what
> commands/actions do the users perform?
> Also, what version of Hive is this?
>
> From: Raj hadoop 
> Reply-To: "user@hive.apache.org" 
> Date: Wednesday, August 3, 2016 at 06:14
> To: "user@hive.apache.org" 
> Subject: hive concurrency not working
>
> Dear All,
>
> In need or your help,
>
> we have horton works 4 node cluster,and the problem is hive is allowing
> only one user at a time,
>
> if any second resource need to login hive is not working,
>
> could someone please help me in this
>
> Thanks,
> Rajesh
>

Re: hive concurrency not working

[Jörn Franke]

You need to configure the yarn scheduler (fair or capacity depending on your 
needs)

> On 03 Aug 2016, at 15:14, Raj hadoop  wrote:
> 
> Dear All,
> 
> In need or your help,
> 
> we have horton works 4 node cluster,and the problem is hive is allowing only 
> one user at a time,
> 
> if any second resource need to login hive is not working,
> 
> could someone please help me in this
> 
> Thanks,
> Rajesh

RE: hive concurrency not working

[Amit Bajpai]

You need to increase the value for the below hive property value in Ambari

hive.server2.tez.sessions.per.default.queue

If this does not fix the issue then you need to update the capacity scheduler 
property values.

From: Raj hadoop [mailto:raj.had...@gmail.com]
Sent: Wednesday, August 03, 2016 8:15 AM
To: user@hive.apache.org
Subject: hive concurrency not working

Dear All,

In need or your help,

we have horton works 4 node cluster,and the problem is hive is allowing only 
one user at a time,

if any second resource need to login hive is not working,

could someone please help me in this

Thanks,
Rajesh

Legal Disclaimer:
The information contained in this message may be privileged and confidential. 
It is intended to be read only by the individual or entity to whom it is 
addressed or by their designee. If the reader of this message is not the 
intended recipient, you are on notice that any distribution of this message, in 
any form, is strictly prohibited. If you have received this message in error, 
please immediately notify the sender and delete or destroy any copy of this 
message!

Re: hive concurrency not working

[Sergey Shelukhin]

Can you elaborate on not working? Is it giving an error, or hanging (and if so, 
does it queue and eventually execute); are you using HS2; what commands/actions 
do the users perform?
Also, what version of Hive is this?

From: Raj hadoop >
Reply-To: "user@hive.apache.org" 
>
Date: Wednesday, August 3, 2016 at 06:14
To: "user@hive.apache.org" 
>
Subject: hive concurrency not working

Dear All,

In need or your help,

we have horton works 4 node cluster,and the problem is hive is allowing only 
one user at a time,

if any second resource need to login hive is not working,

could someone please help me in this

Thanks,
Rajesh