Digging into this bug a bit, I think I have a feel for what's happening, but I want to check.
It seems like since the MR job is writing to two HBase tables, it's using two instances of TableOutputFormat in the same thread. This means two instances of HTable in the same thread. From previous discussions(1), HTable shares connection state unless HTablePool is used. Since TableOutputFormat doesn't use HTablePool though, it seems like the first call to TableOutputFormat.close (which calls HTable.flushCommits) will close the connection and the second call to it will then fail. Hence the second table never gets written to If this is the case, then the question becomes just how do you write to two tables in one MR job using TableOutputFormat? 1- http://search-hadoop.com/m/IsdwtMF2pV/HTable+reuse/v=plain On Fri, May 20, 2011 at 2:29 PM, Bill Graham <[email protected]> wrote: > Yes, that's what it seems. I've opened a Pig JIRA for it: > > https://issues.apache.org/jira/browse/PIG-2085 > > > On Thu, May 19, 2011 at 1:31 PM, Jean-Daniel Cryans > <[email protected]>wrote: > >> Your attachement didn't make it, it rarely does on the mailing lists. >> I suggest you use a gist.github or a pastebin. >> >> Regarding the error, looks like something closed the HCM and someone >> else is trying to use it. Since this is client side, it would point to >> a Pig problem. >> >> J-D >> >> On Thu, May 19, 2011 at 12:28 PM, Keric Donnelly >> <[email protected]> wrote: >> > To All, >> > I'm running into IO issues when trying to write to an Hbase table using >> > multiple STORE commands in PIG script. I can comment out any 2 of the >> STORE >> > statements and run the script and then the data inserts fine. If I try >> and >> > run with all 3 get the following: >> > java.io.IOException: >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@26c1186f >> > closed >> > >> > I've attached a simplified version of the script to illustrate the >> problem. >> > >> > Any thoughts or comments are appreciated. >> > >> > Thanks >> > >> > Keric >> > >> > >> > -- >> > >> > Keric Donnelly >> > Senior Data Architect >> > T 954-689-3291 C 954.683.5445 >> > >> > 1401 West Cypress Creek Road, Fort Lauderdale, FL 33309 >> > >> > >
