Hello- I am evaluating HBase for storage of time-series data and see periodic degradation of the cluster happening at the start of the hour, every hour, for about 10mins. I see "WALProcedureStoreSyncThread" (wal rolling?) in the logs during this time like so:
2016-03-01 06:09:35,151 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove log: hdfs://hdmaster01:54310/hbase/MasterProcWALs/state-00000000000000000009.log In my tests, I'm pushing 1million data points (each about 200bytes in size) into the cluster each minute at the same time using 5000 threads. I am also querying 1000 of these data points each minute to test latency, errors, and performance. The cluster runs absolutely fine except for those ~10mins at the start of every hour when both reads and writes are affected. Could someone suggest which part of HBase needs tuning? Thanks! -Ganesh
