You can also try upgrading to 0.94.8 - there are other fixes for replication that you will benefit.
Cheers On May 28, 2013, at 5:23 AM, Álvaro Recuero <[email protected]> wrote: > Hi Ted, > > I see, let me check that out then and try to integrate that patch-code into > my release. For sure it will make a difference just by seeing the lines in > the patch mentioned. Thank you. > > Best, > Álvaro > > > On 28 May 2013 04:26, Ted Yu <[email protected]> wrote: > >> There was a recently integrated JIRA (0.94.8): >> HBASE-7122 Proper warning message when opening a log file with no entries >> (idle cluster) >> >> The stack trace in above JIRA looks very similar to yours. >> >> On Mon, May 27, 2013 at 8:40 AM, Álvaro Recuero <[email protected]> >> wrote: >> >>> Hi All, >>> >>> From the shell: >>> >>> ~/hbase-0.94.4$ ./bin/hbase shell >>> HBase Shell; enter 'help<RETURN>' for list of supported commands. >>> Type "exit<RETURN>" to leave the HBase Shell >>> Version 0.94.4, r, Mon May 27 15:29:11 WEST 2013 >>> >>> I have seen today it comes up as soon as there is a new item to be >>> replicated, and even happens if an old Hlog is rolled. I am not really >> sure >>> what is going there behind the scenes of Hbase. >>> >>> Regards, >>> Álvaro >>> >>> On 27 May 2013 14:43, Ted Yu <[email protected]> wrote: >>> >>>> The warning came from this loop in ReplicationSource: >>>> >>>> // Loop until we close down >>>> while (isActive()) { >>>> ... >>>> // Open a reader on it >>>> if (!openReader(sleepMultiplier)) { >>>> // Reset the sleep multiplier, else it'd be reused for the next >>>> file >>>> sleepMultiplier = 1; >>>> continue; >>>> } >>>> >>>> // If we got a null reader but didn't continue, then sleep and >>>> continue >>>> if (this.reader == null) { >>>> if (sleepForRetries("Unable to open a reader", >> sleepMultiplier)) >>> { >>>> sleepMultiplier++; >>>> } >>>> continue; >>>> } >>>> >>>> How often did the following warning come up in the server log ? >>>> Which version of HBase are you using ? >>>> >>>> Cheers >>>> >>>> On Mon, May 27, 2013 at 5:15 AM, Álvaro Recuero <[email protected]> >>>> wrote: >>>> >>>>> Does anyone get this when replication is enabled in HBase? If it is a >>>>> warning but does not stop coming up, I wonder what that loop is that >>>>> iterating through and if this is really a critical problem. >>>>> >>>>> cheers, >>>>> Álvaro >>>>> >>>>> 2013-05-27 13:10:40,406 WARN >>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: >>>> Waited >>>>> too long for this file, considering dumping >>>>> 2013-05-27 13:10:40,406 DEBUG >>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: >>>> Unable >>>>> to open a reader, sleeping 1000 times 10 >>>>> 2013-05-27 13:10:50,411 WARN >>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 >>>> Got: >>>>> java.io.EOFException >>>>> at java.io.DataInputStream.readFully(DataInputStream.java:197) >>>>> at java.io.DataInputStream.readFully(DataInputStream.java:169) >>>>> at >>> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508) >>>>> at >>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486) >>>>> at >>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475) >>>>> at >>>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470) >>>>> at >> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55) >>>>> at >> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177) >>>>> at >>> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:721) >>>>> at >> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:66) >>>>> at >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:562) >>>>> at >> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:339) >>
