Thanks Sean for your reply I still have some question un answered like Q1: How Hbase syncronized with Hbase indexer. Q2 What optimization I can apply. Q3 As it's clear from my stats, data in OldWals is quite huge so it's not getting clear my HMaster., how can I improve my HDFS space issue due to this?
Thanks Manjeet Singh On Wed, Jul 11, 2018 at 9:33 PM, Sean Busbey <bus...@apache.org> wrote: > Presuming you're using the Lily indexer[1], yes it relies on hbase's > built in cross-cluster replication. > > The replication system stores WALs until it can successfully send them > for replication. If you look in ZK you should be able to see which > regionserver(s) are waiting to send those WALs over. The easiest way > to do this is probably to look at the "zk dump" web page on the > Master's web ui[2]. > > Once you have the particular region server(s), take a look at their > logs for messages about difficulty sending edits to the replication > peer you have set up for the destination solr collection. > > If you remove the WALs then the solr collection will have a hole in > it. Depending on how far behind you are, it might be quicker to 1) > remove the replication peer, 2) wait for old wals to clear, 3) > reenable replication, 4) use a batch indexing tool to index data > already in the table. > > [1]: > > http://ngdata.github.io/hbase-indexer/ > > [2]: > > The specifics will vary depending on your installation, but the page > is essentially at a URL like > https://active-master-host.example.com:22002/zk.jsp > > the link is on the master UI landing page, near the bottom, in the > description of the "ZooKeeper Quorum" row. it's the end of "Addresses > of all registered ZK servers. For more, see zk dump." > > On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh > <manjeet.chand...@gmail.com> wrote: > > Hi All > > > > I have a query regarding Hbase replication and OldWals > > > > Hbase version 1.2.1 > > > > To enable Hbase indexing we use below command on table > > > > alter '<NameOfTable>', {NAME => 'CF1', REPLICATION_SCOPE => 1} > > > > By Doing this actually replication get enabled as hbase-indexer required > > it, as per my understanding indexer use hbase WAL (Please correct me if I > > am wrong). > > > > so question is How Hbase syncronize with Solr Indexer? What is the role > of > > replication? what optimization we can apply in order to reduce data size? > > > > > > I can see that our OldWals are getting filled , if Hmaster it self taking > > care why it's reached to 7.2 TB? what if I delete it, does it impact solr > > indexing? > > > > 7.2 K 21.5 K /hbase/.hbase-snapshot > > 0 0 /hbase/.tmp > > 0 0 /hbase/MasterProcWALs > > 18.3 G 60.2 G /hbase/WALs > > 28.7 G 86.1 G /hbase/archive > > 0 0 /hbase/corrupt > > 1.7 T 5.2 T /hbase/data > > 42 126 /hbase/hbase.id > > 7 21 /hbase/hbase.version > > 7.2 T 21.6 T /hbase/oldWALs > > > > > > > > > > Thanks > > Manjeet Singh > -- luv all
