Presuming you're using the Lily indexer[1], yes it relies on hbase's built in cross-cluster replication.
The replication system stores WALs until it can successfully send them for replication. If you look in ZK you should be able to see which regionserver(s) are waiting to send those WALs over. The easiest way to do this is probably to look at the "zk dump" web page on the Master's web ui[2]. Once you have the particular region server(s), take a look at their logs for messages about difficulty sending edits to the replication peer you have set up for the destination solr collection. If you remove the WALs then the solr collection will have a hole in it. Depending on how far behind you are, it might be quicker to 1) remove the replication peer, 2) wait for old wals to clear, 3) reenable replication, 4) use a batch indexing tool to index data already in the table. [1]: http://ngdata.github.io/hbase-indexer/ [2]: The specifics will vary depending on your installation, but the page is essentially at a URL like https://active-master-host.example.com:22002/zk.jsp the link is on the master UI landing page, near the bottom, in the description of the "ZooKeeper Quorum" row. it's the end of "Addresses of all registered ZK servers. For more, see zk dump." On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh <manjeet.chand...@gmail.com> wrote: > Hi All > > I have a query regarding Hbase replication and OldWals > > Hbase version 1.2.1 > > To enable Hbase indexing we use below command on table > > alter '<NameOfTable>', {NAME => 'CF1', REPLICATION_SCOPE => 1} > > By Doing this actually replication get enabled as hbase-indexer required > it, as per my understanding indexer use hbase WAL (Please correct me if I > am wrong). > > so question is How Hbase syncronize with Solr Indexer? What is the role of > replication? what optimization we can apply in order to reduce data size? > > > I can see that our OldWals are getting filled , if Hmaster it self taking > care why it's reached to 7.2 TB? what if I delete it, does it impact solr > indexing? > > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > > > > Thanks > Manjeet Singh
