Bill:
TableOutputFormat instances are mutually exclusive due to this line in
close():
HConnectionManager.deleteAllConnections(true);
In trunk, after HBASE-3777, we have:
public void close(TaskAttemptContext context)
throws IOException {
table.close();
}
Keric can remove the HConnectionManager.deleteAllConnections() call so that
more than one STORE commands can be used at the same time.
But I am not sure how HConnectionManager.deleteAllConnections() itself can
be triggered by a PIG command.
Cheers
On Fri, May 20, 2011 at 11:07 PM, Bill Graham <[email protected]> wrote:
> Digging into this bug a bit, I think I have a feel for what's happening,
> but
> I want to check.
>
> It seems like since the MR job is writing to two HBase tables, it's using
> two instances of TableOutputFormat in the same thread. This means two
> instances of HTable in the same thread. From previous discussions(1),
> HTable
> shares connection state unless HTablePool is used. Since TableOutputFormat
> doesn't use HTablePool though, it seems like the first call to
> TableOutputFormat.close (which calls HTable.flushCommits) will close the
> connection and the second call to it will then fail. Hence the second table
> never gets written to
>
> If this is the case, then the question becomes just how do you write to two
> tables in one MR job using TableOutputFormat?
>
> 1- http://search-hadoop.com/m/IsdwtMF2pV/HTable+reuse/v=plain
>
>
> On Fri, May 20, 2011 at 2:29 PM, Bill Graham <[email protected]> wrote:
>
> > Yes, that's what it seems. I've opened a Pig JIRA for it:
> >
> > https://issues.apache.org/jira/browse/PIG-2085
> >
> >
> > On Thu, May 19, 2011 at 1:31 PM, Jean-Daniel Cryans <[email protected]
> >wrote:
> >
> >> Your attachement didn't make it, it rarely does on the mailing lists.
> >> I suggest you use a gist.github or a pastebin.
> >>
> >> Regarding the error, looks like something closed the HCM and someone
> >> else is trying to use it. Since this is client side, it would point to
> >> a Pig problem.
> >>
> >> J-D
> >>
> >> On Thu, May 19, 2011 at 12:28 PM, Keric Donnelly
> >> <[email protected]> wrote:
> >> > To All,
> >> > I'm running into IO issues when trying to write to an Hbase table
> using
> >> > multiple STORE commands in PIG script. I can comment out any 2 of the
> >> STORE
> >> > statements and run the script and then the data inserts fine. If I try
> >> and
> >> > run with all 3 get the following:
> >> > java.io.IOException:
> >> >
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@26c1186f
> >> > closed
> >> >
> >> > I've attached a simplified version of the script to illustrate the
> >> problem.
> >> >
> >> > Any thoughts or comments are appreciated.
> >> >
> >> > Thanks
> >> >
> >> > Keric
> >> >
> >> >
> >> > --
> >> >
> >> > Keric Donnelly
> >> > Senior Data Architect
> >> > T 954-689-3291 C 954.683.5445
> >> >
> >> > 1401 West Cypress Creek Road, Fort Lauderdale, FL 33309
> >> >
> >>
> >
> >
>
