The slow sync messages appeared across all servers as soon new RS were added. Would it make a difference if I said it was on Azure using standard storage? Should I look at GC settings across DN, RS and NN? On Oct 17, 2015 12:23 PM, "Ted Yu" <[email protected]> wrote:
> From your first post, I got the impression that 'Slow sync' appeared in new > region server logs. > > Can you monitor disk activities across the 6 datanodes ? > Take into consideration of YARN usage. > > Cheers > > On Sat, Oct 17, 2015 at 9:07 AM, Artem Ervits <[email protected]> > wrote: > > > The initial three RS were on Kafka brokers with two disks shared between > > Kafka logs and HDFS. HBase 1.1.1. The new RS servers are 3 disks each, > with > > RS, DN and NodeManagers on them. > > On Oct 17, 2015 11:36 AM, "Ted Yu" <[email protected]> wrote: > > > > bq. once I added more regionservers > > > > Were any of the new regionservers on the Kafka broker nodes ? > > > > Which release of hbase are you using ? Looks like 1.x since the log was > > added by HBASE-11240 > > > > Thanks > > > > On Sat, Oct 17, 2015 at 8:22 AM, Artem Ervits <[email protected]> > > wrote: > > > > > Hello all, trying to address a sudden change in performance, processing > > > Kafka, Storm, HBase pipeline. I'm seeing error wal.FSHLog: Slow sync > > cost: > > > 143 ms, current pipeline:, this started appearing once I added more > > > regionservers. Is there a problem with small Xmx value for datanode? > > That's > > > what I've found so far in searches. Table is presplit with no > hotspotting > > > to be seen. I See the message on all RS. I do have 3 RS servers > colocated > > > with Kafka brokers and Kafka disks share storage with datanode but > > > performance was never an issue before. > > > > > >
