This could be a result of the data being skewed. When you look at the job tracker page are you noticing just a few lurking around?
Thanks, Ranjith On May 18, 2012, at 1:57 PM, Balaji Rao <sbalaji...@gmail.com> wrote: > I have a simple query running on HIVE using Amazon EMR where the > reduce operation seems to be stuck. The table is small with just > 300,000+ rows (data_analysis). > I intend to work on a 30,000,000 row table once I'm able to fix this. > The query (modified to hide some information) is as follows: > > INSERT OVERWRITE TABLE data_analysis > SELECT > a.clientid, a.isi, a.os, a.bro, a.engaged, a.iframed, > hover, hower1, hower5, hower60, > Visible, a.x1, a.x2, a.x3, > NoViewMeasure, a.x4, a.x5, a.x6, > a.alerttype, a.x7, a.x9, a.x11, a.x13, a.x17, a.x21, a.x27, a.x28, > COUNT(a.auction_id_64), SUM(CASE WHEN b.auction_id_64 IS NULL THEN > 1 ELSE 0 END) > FROM > data_subset_2 a > LEFT OUTER JOIN > OrderFields b on a.auction_id_64=b.auction_id_64 > GROUP BY > a.clientid, a.isi, a.os, a.bro, a.engaged, a.iframed, > hover, hower1, hower5, hower60, > Visible, a.x1, a.x2, a.x3, > NoViewMeasure, a.x4, a.x5, a.x6, > a.alerttype, a.x7, a.x9, a.x11, a.x13, a.x17, a.x21, a.x27, a.x28; > > > > > 2012-05-18 17:49:32,440 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:50:32,813 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:51:33,176 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:52:33,538 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:53:33,906 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:54:34,270 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:55:34,644 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:56:35,017 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:57:35,387 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:58:35,754 Stage-1 map = 100%, reduce = 97% > 2012-05-18 17:59:36,125 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:00:36,503 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:01:36,873 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:02:36,990 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:03:37,355 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:04:37,719 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:05:38,081 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:06:38,432 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:07:38,777 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:08:39,135 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:09:39,489 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:10:39,852 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:11:40,221 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:12:40,591 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:13:40,962 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:14:41,336 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:15:41,700 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:16:42,070 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:17:42,440 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:18:42,801 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:19:43,156 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:20:43,531 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:21:43,899 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:22:44,255 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:23:44,619 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:24:44,978 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:25:45,347 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:26:45,720 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:27:46,088 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:28:46,446 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:29:46,814 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:30:47,183 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:31:47,546 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:32:47,918 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:33:48,282 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:34:48,408 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:35:48,766 Stage-1 map = 100%, reduce = 97% > 2012-05-18 18:36:49,118 Stage-1 map = 100%, reduce = 97% > > Thanks, > Balaji
