Thanks Bill and Norbert that seems like what I was looking for. I'm a bit worried about how much data/io this could create. But I'll see ;)
Cheers -Marco On Thu, Oct 27, 2011 at 6:03 PM, Norbert Burger <[email protected]>wrote: > In case what you're looking for is an analysis over the full learning > duration, and not just the start interval, then one further insight is > that each original record can be transformed into a sequence of > records, where the size of the sequence corresponds to the session > duration. In other words, you can use a UDF to "explode" the original > record: > > 1,marco,1319708213,500,math > > into: > > 1,marco,1319708190,500,math > 1,marco,1319708220,500,math > 1,marco,1319708250,500,math > 1,marco,1319708280,500,math > 1,marco,1319708310,500,math > 1,marco,1319708340,500,math > 1,marco,1319708370,500,math > 1,marco,1319708400,500,math > 1,marco,1319708430,500,math > 1,marco,1319708460,500,math > 1,marco,1319708490,500,math > 1,marco,1319708520,500,math > 1,marco,1319708550,500,math > 1,marco,1319708580,500,math > 1,marco,1319708610,500,math > 1,marco,1319708640,500,math > 1,marco,1319708670,500,math > 1,marco,1319708700,500,math > > and then use Bill's suggestion to group by course, interval. > > Norbert > > On Thu, Oct 27, 2011 at 11:05 AM, Bill Graham <[email protected]> > wrote: > > You can pass your time to a udf that rounds it down to the nearest 30 > second > > interval and then group by course, interval to get counts for each > course, > > interval. > > > > On Thursday, October 27, 2011, Marco Cadetg <[email protected]> wrote: > >> I have a problem where I don't know how or if pig is even suitable to > > solve > >> it. > >> > >> I have a schema like this: > >> > >> student-id,student-name,start-time,duration,course > >> 1,marco,1319708213,500,math > >> 2,ralf,1319708111,112,english > >> 3,greg,1319708321,333,french > >> 4,diva,1319708444,80,english > >> 5,susanne,1319708123,2000,math > >> 1,marco,1319708564,500,french > >> 2,ralf,1319708789,123,french > >> 7,fred,1319708213,5675,french > >> 8,laura,1319708233,123,math > >> 10,sab,1319708999,777,math > >> 11,fibo,1319708789,565,math > >> 6,dan,1319708456,50,english > >> 9,marco,1319708123,60,english > >> 12,bo,1319708456,345,math > >> 1,marco,1319708789,673,math > >> ... > >> ... > >> > >> I would like to retrieve a graph (interpolation) over time grouped by > >> course. Meaning how many students are learning for a course based on a > 30 > >> sec interval. > >> The grouping by course is easy but from there I've no clue how I would > >> achieve the rest. I guess the rest needs to be achieved via some UDF > >> or is there any way how to this in pig? I often think that I need a "for > >> loop" or something similar in pig. > >> > >> Thanks for your help! > >> -Marco > >> > > >
