FSHLog: Slow sync cost:143 ms, current ... I have seen this messages at recent performance test. This showed up when hdfs had hard time catching up. (I did a lot of put request. ) check read/write byte of hdfs jmx to confirm.
Maybe when you added new RS server, loadbalance kicked in and many region got moved. when new assigned region compacted, data got moved a lot for locality and caused this message. regard, 2015. 10. 18. 오전 3:24에 "Artem Ervits" <[email protected]>님이 작성: > The slow sync messages appeared across all servers as soon new RS were > added. Would it make a difference if I said it was on Azure using standard > storage? Should I look at GC settings across DN, RS and NN? > On Oct 17, 2015 12:23 PM, "Ted Yu" <[email protected]> wrote: > > > From your first post, I got the impression that 'Slow sync' appeared in > new > > region server logs. > > > > Can you monitor disk activities across the 6 datanodes ? > > Take into consideration of YARN usage. > > > > Cheers > > > > On Sat, Oct 17, 2015 at 9:07 AM, Artem Ervits <[email protected]> > > wrote: > > > > > The initial three RS were on Kafka brokers with two disks shared > between > > > Kafka logs and HDFS. HBase 1.1.1. The new RS servers are 3 disks each, > > with > > > RS, DN and NodeManagers on them. > > > On Oct 17, 2015 11:36 AM, "Ted Yu" <[email protected]> wrote: > > > > > > bq. once I added more regionservers > > > > > > Were any of the new regionservers on the Kafka broker nodes ? > > > > > > Which release of hbase are you using ? Looks like 1.x since the log was > > > added by HBASE-11240 > > > > > > Thanks > > > > > > On Sat, Oct 17, 2015 at 8:22 AM, Artem Ervits <[email protected]> > > > wrote: > > > > > > > Hello all, trying to address a sudden change in performance, > processing > > > > Kafka, Storm, HBase pipeline. I'm seeing error wal.FSHLog: Slow sync > > > cost: > > > > 143 ms, current pipeline:, this started appearing once I added more > > > > regionservers. Is there a problem with small Xmx value for datanode? > > > That's > > > > what I've found so far in searches. Table is presplit with no > > hotspotting > > > > to be seen. I See the message on all RS. I do have 3 RS servers > > colocated > > > > with Kafka brokers and Kafka disks share storage with datanode but > > > > performance was never an issue before. > > > > > > > > > >
