JD hi The system is under heavy load only via hbase (no map reduce running). Another thing is that we have hard constraints in latencies, therefore if this point on performance problem i would like to "fix" it. The GC seems fine too. I see around 300 message like this in 10 hours of run, approx each 2 minutes intervals. So now the question - what should i do to reduce the time taken (knowing that the data to be written will remain the same in volume)?
Regards, Mikael.S On Wed, Dec 21, 2011 at 7:44 PM, Jean-Daniel Cryans <[email protected]>wrote: > That line means that appending to the log took x ms, we print it out > when it takes more than 1 second. I'd be worried to see that a lot in > my logs if I had a realtime-ish system because it would mean that my > writes are taking seconds. > > The usual reasons: > > - Heavy IO, I would expect those lines if I was writing a lot of data > into HBase or running MR jobs that don't use HBase but do push a lot > of data to their datanodes. > - GC, for example you have a pretty busy system with not a lot of > memory so it's running long-ish GCs every now and then well if it > happens during the append operation it may appear that it took x ms, > but in fact could have been super short but the JVM was blocked for > some time. > > Hope this helps, > > J-D > > On Wed, Dec 21, 2011 at 1:11 AM, Mikael Sitruk <[email protected]> > wrote: > > Hi > > > > I have my log with lot of the following message in my hbase logs. > > > > 2011-12-21 00:32:05,035 WARN > org.apache.hadoop.hbase.regionserver.wal.HLog: > > IPC Server handler 38 on 60020 took 1247 ms appending an edit to hlog; > > editcount=625, len~=61.4k > > 2011-12-21 00:32:06,378 WARN > org.apache.hadoop.hbase.regionserver.wal.HLog: > > IPC Server handler 33 on 60020 took 1342 ms appending an edit to hlog; > > editcount=627, len~=61.4k > > > > Can you explain the what does it mean? What should i do to remove this > > warning? Does it have performance impact? > > > > Thanks > > Mikael.S > -- Mikael.S
