CB: does the dynamic partitioning fill your need? I don't totally understand it but if it does, awesome.
Otherwise there isn't a for/each construct in HiveQL. You'd have to write an external program. I'm curious though, do you have to reprocess each partition each day or is there a partition created each day and then never touched again? I think it's the later as Hive doesn't have an update statement either but you might be doing a full drop partition and rebuilding from scratch with the new data. Pat -----Original Message----- From: Namit Jain [mailto:nj...@fb.com] Sent: Wednesday, February 09, 2011 8:24 AM To: user@hive.apache.org Subject: Re: for each partition You can use dynamic partitioning: insert overwrite table item_view_aggregate partition (date_hour) select iv.sid, count(*), date_hour from item_view iv where (iv.date_hour='2011310116' or date_hour=''' or date_hour='.....) group by iv.sid, date_hour; On 2/9/11 5:49 AM, "Cam Bazz" <camb...@gmail.com> wrote: >Well, I designed my dataflow to work incrementally based on >partitions. But I have a number of datafiles now, >and for the first run, I have to for example: > >insert overwrite table item_view_aggregate partition >(date_hour=2011310116) select iv.sid, count(*) from item_view iv where >iv.date_hour='2011310116' group by iv.sid; > >.. > >I have to repeat this for each partition. so I need something like a >for loop, or do it with an external program. > >best regards, >-c.b. > >On Wed, Feb 9, 2011 at 7:33 AM, Christopher, Pat ><patrick.christop...@hp.com> wrote: >> If you want to operate over all partitions in a table you don't need to >>specify the partitions at all. Run your query and enjoy! >> >> If you want to specify the partition mapping of the output dataset from >>a query, I think you can derive that value on a per row basis like so: >> >> Partition=substr(dateval,1,14) >> >> Assuming dateval is a field in your table and partition is your >>partition field. >> >> Lemme know if you need more or if it doesn't work. I'll check it out >>tomorrow at work. >> >> Pat >> >> >> >> -- Sent from my Palm Pre >> >> ________________________________ >> On Feb 8, 2011 8:57 PM, Cam Bazz <camb...@gmail.com> wrote: >> >> Hello, >> >> How can I do some process for each partition in some other table. >> >> for example lets say table A has partitions 1,2,3 >> >> I want to be able to say >> >> for each partition in A do { >> select * from A where partition is ? into some othertable where >>partition is ? >> } >> >> Best Regards, >> C.B. >>
