Ganesh Viswanathan wrote:
3) When all the nodes in the cluster were restarted with Ambari, locality dropped to ~13% and Hbase was almost non-responsive. Only triggering a manual major compaction seems to help improve the locality after this. But the data-locality increase is very gradual (about 4% every hour). Is there any way to speed up major compaction (increase the number of threads etc) in HDP distribution?
See the properties hbase.regionserver.thread.compaction.small and hbase.regionserver.thread.compaction.large in CompactSplitThread. By default, there is one thread for small compactions and one thread for large compactions. You can consider increasing these values in hbase-site.xml to run more compactions concurrently.
Be aware that increasing these values may also require you to increase other properties (such as JVM max heap).
