This is the output I am expecting NC,28613,55
On Fri, Apr 13, 2012 at 4:28 PM, Dmitriy Ryaboy <[email protected]> wrote: > Could you type out the actual results you would like to see? I am not > sure what you expect the results of " foreach rel GENERATE FIELD == > ST, FIELD == ZIP, FIELD == AGE" to look like. > > Also, your Pig scripts do not have to be all caps. Using something > other than all caps will make then a lot more readable... > > D > > On Fri, Apr 13, 2012 at 3:41 PM, Mohit Anchlia <[email protected]> > wrote: > > This is my pig script so far that gives me output. What I want to do is > > arrange them in this order NC,28613,55 from below output. > > > > My question is from this relation how can I extract specific fields from > > bags and tuples? Essentially I want to do something like: > > > > foreach rel GENERATE FIELD == ST, FIELD == ZIP, FIELD == AGE --I want > > fields in this order from a given relation. But the problem is it's > > arranged in a bag and multiple tuples > > > > > > > (1333477861077/home/hadoop/pigtest/./formml_dat/999000093_tax_return.xml,04/03/12 > > 11:36:25) > > > {(1333477861077/home/hadoop/pigtest/./formml_dat/999000093_tax_return.xml,FINFOWKS,PER,FINFOWKS,04/ > > 03/12 > > > 11:36:25,ST,NC),(1333477861077/home/hadoop/pigtest/./formml_dat/999000093_tax_return.xml,FINFOWKS,PER,FINFOWKS,04/03/12 > > 11:36:25,ZIP,28613),(1333477861077/home/hadoop/pigtest/./formml_dat/9 > > 99000093_tax_return.xml,FINFOWKS,PER,FINFOWKS,04/03/12 > > > 11:36:25,CITY,Xxxxxxx),(1333477861077/home/hadoop/pigtest/./formml_dat/999000093_tax_return.xml,FINFOWKS,PER,FINFOWKS,04/03/12 > > 11 > > :36:25,NAM2,Xxxxx X &xxx; Xxxxx X Xxxxxx)} > > > {(1333477861077/home/hadoop/pigtest/./formml_dat/999000093_tax_return.xml,WKS,PER,WKS,04/03/12 > > 11:36:25,AGE,55),(1333477861077/home/hadoo > > p/pigtest/./formml_dat/999000093_tax_return.xml,WKS,PER,WKS,04/03/12 > > 11:36:25,OCCUP,xxxxxxx > > > xxxxx),(1333477861077/home/hadoop/pigtest/./formml_dat/999000093_tax_return.xml,WKS,S201 > > 1US1040PER,WKS,04/03/12 11:36:25,MARITAL,Married)} > > > > snippet of the script > > > > D = FILTER A by F_ID == 'FINFOWKS' AND FIELD_ID == 'TSN'; > > NM_CT_ST_FILTER = FILTER A by (FIELD_ID == 'NAM2' OR FIELD_ID == 'CITY' > OR > > FIELD_ID == 'ST' OR FIELD_ID == 'ZIP'); > > AG_OC_MT_FILTER = FILTER A by (FIELD_ID == 'AGE' OR FIELD_ID == 'OCCUP' > OR > > FIELD_ID == 'MARITAL') AND F_ID == 'WKS' AND F_COPY_NUM == '1'; > > NM_CT_ST_FIELDS = FOREACH NM_CT_ST_FILTER GENERATE FILE_NAME as > > A_FILE_NAME, F_ID as A_F_ID, FSET_ID as A_FSET_ID, F_ID_ROOT as > > A_F_ID_ROOT, CREATED_DATE as A_CREATED_DATE,FIELD_ID as > > A_FIELD_ID,FIELD_VALUE as A_FIELD_VALUE; > > AG_OC_MT_FIELDS = FOREACH AG_OC_MT_FILTER GENERATE FILE_NAME as > > B_FILE_NAME,F_ID as B_F_ID, FSET_ID as B_FSET_ID, F_ID_ROOT as > B_F_ID_ROOT, > > CREATED_DATE as B_CREATED_DATE,FIELD_ID as B_FIELD_ID,FIELD_VALUE as > > B_FIELD_VALUE; > > A_JOIN = JOIN NM_CT_ST_FIELDS BY > > (A_FILE_NAME,A_CREATED_DATE,A_F_ID,A_F_ID_ROOT), D BY > > (FILE_NAME,CREATED_DATE,F_ID,F_ID_ROOT); > > B_JOIN = JOIN AG_OC_MT_FIELDS BY (B_FILE_NAME,B_CREATED_DATE), D BY > > (FILE_NAME,CREATED_DATE); > > A_JOIN_F = FOREACH A_JOIN GENERATE A_FILE_NAME, A_F_ID, A_FSET_ID, > > A_F_ID_ROOT, A_CREATED_DATE,A_FIELD_ID,A_FIELD_VALUE,FIELD_VALUE; > > B_JOIN_F = FOREACH B_JOIN GENERATE > > > B_FILE_NAME,B_F_ID,B_FSET_ID,B_F_ID_ROOT,B_CREATED_DATE,B_FIELD_ID,B_FIELD_VALUE; > > FINAL = COGROUP A_JOIN_F BY (A_FILE_NAME,A_CREATED_DATE), B_JOIN_F BY > > (B_FILE_NAME,B_CREATED_DATE); > > FINAL_DISTINCT = DISTINCT FINAL; > > > > > > On Thu, Apr 12, 2012 at 7:37 PM, Dmitriy Ryaboy <[email protected]> > wrote: > > > >> It's not clear to me what exactly you are trying to accomplish. Could > you > >> provide some sample inputs and expected outputs? > >> > >> You can use filter inside a foreach: > >> > >> Foreach foo { a = filter bag_in_foo by condition; generate a; } > >> > >> On Apr 11, 2012, at 5:27 PM, Mohit Anchlia <[email protected]> > wrote: > >> > >> > I am new to pig and I have gone through the reference. I am getting > used > >> to > >> > how this works but I keep getting questions as I write my scripts. I > have > >> > couple of questions: > >> > > >> > i) I use FILTER with FOREACH? Below I am trying to FILTER, JOIN and > MERGE > >> > into one row. But in the end I get all the fields in form of row which > >> > seems to have Bags inside tuples. In the end all I want is to output > >> values > >> > of some of the fields from each row in "a,b,c" format. How can I do > that? > >> > > >> > > >> > NM_CT_ST_FILTER = FILTER A by (FIELD_ID == 'NAM2' OR FIELD_ID == > 'CITY' > >> OR > >> > FIELD_ID == 'ST' OR FIELD_ID == 'ZIP'); > >> > > >> > AG_OC_MT_FILTER = FILTER A by (FIELD_ID == 'AGE' OR FIELD_ID == > 'OCCUP' > >> OR > >> > FIELD_ID == 'MARITAL') AND FORM_ID == 'FPERSWKS' AND FORM_COPY_NUM == > >> '1'; > >> > > >> > NM_CT_ST = JOIN NM_CT_ST_FILTER BY (FILE_NAME,CREATED_DATE), D BY > >> > (FILE_NAME,CREATED_DATE); > >> > > >> > AG_OC_MT = JOIN AG_OC_MT_FILTER BY > >> > (FILE_NAME,CREATED_DATE,FORM_ID,FORM_ID_ROOT), D BY > >> > (FILE_NAME,CREATED_DATE,FORM_ID,FORM_ID_ROOT); > >> > > >> > FINAL = COGROUP NM_CT_ST BY (D::FILE_NAME,D::CREATED_DATE), AG_OC_MT > BY > >> > (D::FILE_NAME,D::CREATED_DATE); > >> > > >> > 2) Is it possible to use FILTER with foreach? something like foreach A > >> > GENERATE B FILTER FIELD BY .. OR FIELD BY .. > >> >
