RE: Mappers spawning Hive queries

2016-04-18 Thread Ryan Harris
ay uncover unexpected edge-cases. From: Shirish Tatikonda [mailto:shirish.tatiko...@gmail.com] Sent: Monday, April 18, 2016 3:44 PM To: user@hive.apache.org Subject: Re: Mappers spawning Hive queries I am using Hive 1.2.1 with MR backend. Ryan, I hear you. I totally agree. This is not the best approach

Re: Mappers spawning Hive queries

2016-04-18 Thread Shirish Tatikonda
atikonda [mailto:shirish.tatiko...@gmail.com] > *Sent:* Monday, April 18, 2016 2:00 PM > *To:* user@hive.apache.org > *Subject:* Re: Mappers spawning Hive queries > > > > Hi John, > > > > 2) The shell script is invoked in the mappers of a Hadoop streaming job. > >

RE: Mappers spawning Hive queries

2016-04-18 Thread Ryan Harris
, April 18, 2016 2:00 PM To: user@hive.apache.org Subject: Re: Mappers spawning Hive queries Hi John, 2) The shell script is invoked in the mappers of a Hadoop streaming job. 1) The use case is that I have to process multiple entities in parallel. Each entity is associated with its own data set

Re: Mappers spawning Hive queries

2016-04-18 Thread Mich Talebzadeh
What is the version of Hive and the execution engine (MR, Tez, Spark)? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Mappers spawning Hive queries

2016-04-18 Thread Shirish Tatikonda
Hi John, 2) The shell script is invoked in the mappers of a Hadoop streaming job. 1) The use case is that I have to process multiple entities in parallel. Each entity is associated with its own data set. The processing involves a few hive queries to do joins and aggregations, which is followed

Re: Mappers spawning Hive queries

2016-04-16 Thread Jörn Franke
Just out of curiosity, what is the use case behind this? How do you call the shell script? > On 16 Apr 2016, at 00:24, Shirish Tatikonda > wrote: > > Hello, > > I am trying to run multiple hive queries in parallel by submitting them > through a map-reduce job.

Mappers spawning Hive queries

2016-04-15 Thread Shirish Tatikonda
Hello, I am trying to run multiple hive queries in parallel by submitting them through a map-reduce job. More specifically, I have a map-only hadoop streaming job where each mapper runs a shell script that does two things -- 1) parses input lines obtained via streaming; and 2) submits a very