Thanks for quick reply. I will take a look at stream job and transform
functions.
One more question:
I have multiple csv files ( same structure, dir added as partition) mapped to
hive table. Then I run different group by jobs on same data like below. All
these are spanned as different jobs. So multiple mappers read/fetch data from
disk and then computes different group/aggregation jobs.
Each below job fetch same data from disk. Can this be avoided by reading split
only once and mapper computing different group by jobs in same mapper itself.
That may no of mappers will come down drastically and also mainly multiple disk
seeks for same data avoided. Do I need to write custom map reduce job to do
this?
1) Insert into temptable1 select TAG,col2,SUM(col5) as SUM_col5,SUM(col6)
as SUM_col6,SUM(col7) as SUM_col7,ts from raw_data_by_epoch where
ts=${hivevar:collectiontimestamp} group by TAG,col2,TS
2) Insert into temptable2 select TAG,col2,col3,SUM(col5) as
SUM_col5,SUM(col6) as SUM_col6,SUM(col7) as SUM_col7,ts from raw_data_by_epoch
where ts=${hivevar:collectiontimestamp} group by TAG,col2,col3,TS
3) Insert into temptable3 select TAG,col2,col3,col4,SUM(col5) as
SUM_col5,SUM(col6) as SUM_col6,SUM(col7) as SUM_col7,ts from raw_data_by_epoch
where ts=${hivevar:collectiontimestamp} group by TAG,col2,col3,col4,TS
Thanks,
Chandra
From: Stephen Sprague [mailto:[email protected]]
Sent: Friday, January 17, 2014 11:39 AM
To: [email protected]
Subject: Re: complex datatypes filling
remember you can always setup a stream job to do any wild and crazy custom
thing you want. see the tranform() function documentation. Its really quite
easy. honest.
On Thu, Jan 16, 2014 at 9:39 PM, Bogala, Chandra Reddy
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I found lot of examples to map json data into hive complex data types (map,
array , struct etc). But I don't see anywhere filling complex data types with
nested sql query ( I.e group by few columns(key) and array of struct(multiple
columns) containing result values ).
So that it will be easy for me to map back into embedded/nested json document.
Thanks,
Chandra
