RE: how to get the size of a data bag

[Olga Natkovich]

Sorry, group should be after generate:

C = foreach B generate group, COUNT(A); 

> -----Original Message-----
> From: charles du [EMAIL PROTECTED] 
> Sent: Thursday, August 07, 2008 4:20 PM
> To: pig-user@incubator.apache.org
> Subject: Re: how to get the size of a data bag
> 
> Hi:
> 
> I tried the command, and pig threw an exception complaining 
> about the grammar.
> 
> Also, just want to make sure. Should the command be?
> 
>   A = LOAD <my file> as (type, ...);
>   B = GROUP  A  BY  type;
>   C = foreach *B *group, generate COUNT(A);
> 
> How does pig decide when to use a combiner, and when not to?
> 
> Thanks.
> 
> Chuang
> 
> 
> On Thu, Aug 7, 2008 at 2:59 PM, Olga Natkovich 
> <[EMAIL PROTECTED]> wrote:
> 
> > The way your query is formulated combiner is not called and 
> that would 
> > account for the slowness.
> >
> > Try this:
> >
> > A = LOAD <my file> as (type, ...);
> > B = GROUP  A  BY  type;
> > C = foreach A group, generate COUNT(A);
> >
> > You can check if combiner will be called by running
> >
> > Explain C;
> >
> > Olga
> >
> > > -----Original Message-----
> > > From: charles du [EMAIL PROTECTED]
> > > Sent: Thursday, August 07, 2008 2:55 PM
> > > To: pig-user@incubator.apache.org
> > > Subject: Re: how to get the size of a data bag
> > >
> > > Thanks. It works.
> > >
> > > My concern right now is the performance. For example, I have
> > > 2 million records that belongs to two types. If I want to 
> count the 
> > > number of records for each type, I need group records 
> based on the 
> > > type as follows:
> > >
> > > A = LOAD <my file> as (type, ...);
> > > B = GROUP  A  BY  type;
> > > C = foreach A generate COUNT(A);
> > >
> > > I notices it usually takes hadoop a long time to get the results 
> > > back. My experience with hadoop is that if there are a 
> large number 
> > > of values for a key, hadoop is very slow on the reduce 
> function. I 
> > > understand it is a more hadoop problem, instead of pig's. Do you 
> > > guys know any way to speedup the calculation?
> > >
> > >
> > > Thanks.
> > >
> > > Chuang
> > >
> > > On Fri, Jul 18, 2008 at 2:10 PM, Olga Natkovich 
> > > <[EMAIL PROTECTED]> wrote:
> > >
> > > > How was you bag created?
> > > >
> > > > Normally, you would load the data then group it into a bag
> > > using group
> > > > by or group all and then apply the count:
> > > >
> > > > A = load 'input';
> > > > B = group A all;
> > > > C = foreach A generate COUNT(A);
> > > >
> > > > Olga
> > > >
> > > > > -----Original Message-----
> > > > > From: charles du [EMAIL PROTECTED]
> > > > > Sent: Friday, July 18, 2008 12:23 PM
> > > > > To: pig-user@incubator.apache.org
> > > > > Subject: how to get the size of a data bag
> > > > >
> > > > > Hi:
> > > > >
> > > > > Just start learning hadoop and pig latin. How can I get
> > > the number
> > > > > of elements in a data bag?
> > > > >
> > > > > For example, a data bag like follow has four elements.
> > > > >   B= {1, 2, 3, 5}
> > > > >
> > > > > I tried C = COUNT(B), it did not work. Thanks.
> > > > >
> > > > > --
> > > > > tp
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > tp
> > >
> >
> 
> 
> 
> --
> tp
>