Thanks, Appreciate your help. It can be confusing as I am waiting here but it is not replicating I will reduce those values temporarily and see what happens. Interesting part is Files Needing replication is stuck at 3 so it is possible that problem is some where else.
Thanks again -S From: Adam J. Shook <adamjsh...@gmail.com> Sent: Thursday, September 23, 2021 1:50 PM To: user@accumulo.apache.org Subject: Re: [External] RE: accumulo 1.10 replication issue Yes, if it is not heavily used then you may see a significant delay. You can change the defaults using tserver.walog.max.age [1] and tserver.walog.max.size [2]. If I recall you can change these via the shell and a restart is not required. If you aren't seeing much ingestion, then the max age would be what you want to set to ensure data is replicated within the window you want it to be replicated. Please keep in mind that setting either of these values to very low thresholds will cause the WALs to roll over frequently and could negatively impact the system, particularly for large Accumulo clusters. In my experience, using Accumulo replication is a good fit when you have longer SLAs on replication. If you are looking for anything in the near-real-time realm (milliseconds to seconds to maybe even a few minutes), you'd be better off double writing to multiple Accumulo instances. --Adam [1] https://accumulo.apache.org/1.10/accumulo_user_manual.html#_tserver_walog_max_age<https://urldefense.com/v3/__https:/accumulo.apache.org/1.10/accumulo_user_manual.html*_tserver_walog_max_age__;Iw!!May37g!abBCLknLxFVCzoPfcMJQ_DMnbdLbmOa-oYeMQf0CbQTNSS8yF4jOrBV_yNn1zEjBog$> [2] https://accumulo.apache.org/1.10/accumulo_user_manual.html#_tserver_walog_max_size<https://urldefense.com/v3/__https:/accumulo.apache.org/1.10/accumulo_user_manual.html*_tserver_walog_max_size__;Iw!!May37g!abBCLknLxFVCzoPfcMJQ_DMnbdLbmOa-oYeMQf0CbQTNSS8yF4jOrBV_yNnD3o4pgA$> On Thu, Sep 23, 2021 at 1:31 PM Ligade, Shailesh [USA] <ligade_shail...@bah.com<mailto:ligade_shail...@bah.com>> wrote: Thanks Adam, System is not heavily used, Does that mean it will wait for 1G data in wal file, (or 24 hours) before it will replicate? I don’t see any error In any log source master, tserver or target master,tserver Monitor replication page has correct status, and once in while I see In-Progress Replication section flashing by. But don’t see any new data in the target table. ☹ -S From: Adam J. Shook <adamjsh...@gmail.com<mailto:adamjsh...@gmail.com>> Sent: Thursday, September 23, 2021 12:10 PM To: user@accumulo.apache.org<mailto:user@accumulo.apache.org> Subject: Re: [External] RE: accumulo 1.10 replication issue Yes, inserting via the shell will be enough to test it. Note that the replication system uses the write-ahead logs (WAL) to replicate the data. These logs must be closed before any replication can occur, so there will be a delay before it shows up in the peer table. How long of a delay depends on how much data is actively being written to the TabletServers (and therefore the WAL) and/or how much time has passed since the WAL was opened. The default max WAL data size is 1 GB and the max age is 24 hours. --Adam On Thu, Sep 23, 2021 at 11:13 AM Ligade, Shailesh [USA] <ligade_shail...@bah.com<mailto:ligade_shail...@bah.com>> wrote: Thanks Adam, I am setting accumulo.name<https://urldefense.com/v3/__http:/accumulo.name__;!!May37g!au2NJ_bzRengNQZdhTHO9O38cNfzpFNzN8DFC49SzxW58cbMe9Vl20i58oJkg1wZWQ$> property in accumulo-site.xml. I think this property must be set to “Instance Name” value, I tried to set to “primary” and I saw error stating that instance id was not found in zookeeper I have few tables to replicate so I am thinking I will set all others properties using shell config command To test this, I just insert value using shell right? Or do I need to flush or compact on the table to see those values on the other side? -S From: Adam J. Shook <adamjsh...@gmail.com<mailto:adamjsh...@gmail.com>> Sent: Thursday, September 23, 2021 11:08 AM To: user@accumulo.apache.org<mailto:user@accumulo.apache.org> Subject: Re: [External] RE: accumulo 1.10 replication issue Your configurations look correct to me, and it sounds like it is partially working as you are seeing files that need replicated in the Accumulo Monitor. I do have the replication.name<https://urldefense.com/v3/__http:/replication.name__;!!May37g!YaShfRxRNA1m14PM-_NQTaWWuL-fcis6RlUI9RKQU68Q2oWUTZuh-Q1EkHA0XIO8vA$> and all replication.peer.* properties defined in accumulo-site.xml. Do you have all these properties defined there? If not, try setting them in accumulo-site.xml and restarting your Accumulo services, particularly the Master and TabletServers. The Master may not be queuing work and/or the TabletServers may not be looking for work. You should see DEBUG-level logs in the TabletServers that say "Looking for work in /accumulo/<instanceId>/replication/workqueue", so enable debug logging if you haven't done so already in the generic_logger.xml file. --Adam On Thu, Sep 23, 2021 at 6:53 AM Ligade, Shailesh [USA] <ligade_shail...@bah.com<mailto:ligade_shail...@bah.com>> wrote: Thanks for reply I am using insert command from shell to insert data. Also, a quick question, replication.name<https://urldefense.com/v3/__http:/replication.name__;!!May37g!YaShfRxRNA1m14PM-_NQTaWWuL-fcis6RlUI9RKQU68Q2oWUTZuh-Q1EkHA0XIO8vA$> property can it be set using cli? Will that work or it must be defined in accumilo-site.xml? Thanks -S From: d...@etcoleman.com<mailto:d...@etcoleman.com> <d...@etcoleman.com<mailto:d...@etcoleman.com>> Sent: Thursday, September 23, 2021 6:50 AM To: user@accumulo.apache.org<mailto:user@accumulo.apache.org> Subject: [External] RE: accumulo 1.10 replication issue How are you inserting the data? From: Ligade, Shailesh [USA] <ligade_shail...@bah.com<mailto:ligade_shail...@bah.com>> Sent: Wednesday, September 22, 2021 10:22 PM To: user@accumulo.apache.org<mailto:user@accumulo.apache.org> Subject: accumulo 1.10 replication issue Hello, I am following Apache Accumulo® User Manual Version 1.10<https://urldefense.com/v3/__https:/accumulo.apache.org/1.10/accumulo_user_manual.html*_replication__;Iw!!May37g!epzOA4Zxtj4kjXvE1dPTtAae7AAFXbiZCVMxVk6_yQ3O-AlaG8GkML6q5OX1nt3O0A$> I want to setup replication from accumulo instance inst1, table source, TO inst2, table target I created a replication user,( same password) on both instances and grant Table.READ/WRITE for source and target respectively I set replication.name<https://urldefense.com/v3/__http:/replication.name__;!!May37g!YaShfRxRNA1m14PM-_NQTaWWuL-fcis6RlUI9RKQU68Q2oWUTZuh-Q1EkHA0XIO8vA$> property to be same as inst on both instances On inst1 Set following properties replication.peer.inst1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,inst2,inst2zoo1:2181,inst2zoo2:2181,inst2zoo3:2181 replication.peer.user.inst2=replication replication.peer.password.inst2=replication set the source table for replication config -t source -s table.replication=true config -t source -s table.replication.target.inst2=(number I got for target table from inst2 tables -l command) and finally I did online accumulo.replication Now when I insert data in source, I get feiles needing replication 1 on the monitor replication section. All other values are correct, TABLE – source, PEER – inst2 REMOTE ID as number I set However my In-Progress Replication always stay empty and I don’t see any data in inst2 target table No errors that I can see in master log or tserver log where tablet exist. Any idea what may be wrong? Is there any way to debug this? -S
