Thanks for the quick reply. My alternate approach is to just let the mappers do their writes, and to retrieve values based on the timestamp of the last *successful* run, which should effectively filter out the results of failed runs. It sounds like it may be my only option.
I suppose it might also be possible to generate the output to a separate table, and then merge it back into the main table only if the m/r finishes successfully, but that sounds a lot hairier... Thanks Mark Clint Morgan-3 wrote: > > Unfortunately, there is no more documentation on transactions. > > I've never used transactions in M/Rs. You are correct that if you only > start/commit a transaction in mapper then you will only get atomicity > across > the individual map. > > One thing to keep in mind is that, in the current impl, all the writes of > a > pending transaction are keep in memory. A big M/R job could quickly blow > up > with an OOME. > > -clint > > On Wed, Jun 24, 2009 at 8:45 PM, schnitzi > <[email protected]>wrote: > >> >> I have some scenarios involving map/reduce that update HBase tables. >> (They >> all involve creating or modifying a number of individual records, so I'm >> planning to do it all in the mapper.) >> >> What I'm hoping to do is to only commit at the end of the map/reduce, if >> it >> succeeds. If I start a TransactionManager transaction in the mapper and >> commit it there, in my understanding, it will only guarantee atomicity >> for >> the set of rows that that mapper updates. >> >> Has anyone attempted anything like this before? Is there a way to do it? >> >> Also, is there any general documentation about transactions in HBase >> besides >> the javadoc and the test cases? There doesn't seem to be any on the >> wiki... >> I'm muddling through without it but it would be nice to see some. >> >> >> Cheers >> Mark >> -- >> View this message in context: >> http://www.nabble.com/Transactions-and-Map-Reduce-tp24196501p24196501.html >> Sent from the HBase User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/Transactions-and-Map-Reduce-tp24196501p24213337.html Sent from the HBase User mailing list archive at Nabble.com.
