I have one more question If solr is having its own data mean its maintaining data in their shards and hbase is maintaining in data folder... Why still oldWals need?
Thanks Manjeet singh On Wed, 11 Jul 2018, 23:19 Manjeet Singh, <manjeet.chand...@gmail.com> wrote: > Thanks Sean for your reply > > I still have some question un answered like > Q1: How Hbase syncronized with Hbase indexer. > Q2 What optimization I can apply. > Q3 As it's clear from my stats, data in OldWals is quite huge so it's not > getting clear my HMaster., how can I improve my HDFS space issue due to > this? > > Thanks > Manjeet Singh > > On Wed, Jul 11, 2018 at 9:33 PM, Sean Busbey <bus...@apache.org> wrote: > >> Presuming you're using the Lily indexer[1], yes it relies on hbase's >> built in cross-cluster replication. >> >> The replication system stores WALs until it can successfully send them >> for replication. If you look in ZK you should be able to see which >> regionserver(s) are waiting to send those WALs over. The easiest way >> to do this is probably to look at the "zk dump" web page on the >> Master's web ui[2]. >> >> Once you have the particular region server(s), take a look at their >> logs for messages about difficulty sending edits to the replication >> peer you have set up for the destination solr collection. >> >> If you remove the WALs then the solr collection will have a hole in >> it. Depending on how far behind you are, it might be quicker to 1) >> remove the replication peer, 2) wait for old wals to clear, 3) >> reenable replication, 4) use a batch indexing tool to index data >> already in the table. >> >> [1]: >> >> http://ngdata.github.io/hbase-indexer/ >> >> [2]: >> >> The specifics will vary depending on your installation, but the page >> is essentially at a URL like >> https://active-master-host.example.com:22002/zk.jsp >> >> the link is on the master UI landing page, near the bottom, in the >> description of the "ZooKeeper Quorum" row. it's the end of "Addresses >> of all registered ZK servers. For more, see zk dump." >> >> On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh >> <manjeet.chand...@gmail.com> wrote: >> > Hi All >> > >> > I have a query regarding Hbase replication and OldWals >> > >> > Hbase version 1.2.1 >> > >> > To enable Hbase indexing we use below command on table >> > >> > alter '<NameOfTable>', {NAME => 'CF1', REPLICATION_SCOPE => 1} >> > >> > By Doing this actually replication get enabled as hbase-indexer required >> > it, as per my understanding indexer use hbase WAL (Please correct me if >> I >> > am wrong). >> > >> > so question is How Hbase syncronize with Solr Indexer? What is the role >> of >> > replication? what optimization we can apply in order to reduce data >> size? >> > >> > >> > I can see that our OldWals are getting filled , if Hmaster it self >> taking >> > care why it's reached to 7.2 TB? what if I delete it, does it impact >> solr >> > indexing? >> > >> > 7.2 K 21.5 K /hbase/.hbase-snapshot >> > 0 0 /hbase/.tmp >> > 0 0 /hbase/MasterProcWALs >> > 18.3 G 60.2 G /hbase/WALs >> > 28.7 G 86.1 G /hbase/archive >> > 0 0 /hbase/corrupt >> > 1.7 T 5.2 T /hbase/data >> > 42 126 /hbase/hbase.id >> > 7 21 /hbase/hbase.version >> > 7.2 T 21.6 T /hbase/oldWALs >> > >> > >> > >> > >> > Thanks >> > Manjeet Singh >> > > > > -- > luv all >
