Re: Use of TLog
There’s also the Collections API MOVEREPLICA command that does this all as a single command. > On Nov 19, 2019, at 5:21 AM, Sripra deep wrote: > > Got it, Thanks for helping me Emir. > > On Tue, Nov 19, 2019 at 12:36 PM Emir Arnautović < > emir.arnauto...@sematext.com> wrote: > >> Hi, >> Copying indices will work and it is a fine approach. An alternative would >> be to join new node to a cluster, use add replica to copy cores to this new >> node and then remove replicas from old nodes, if you want to move cores. >> >> Regards, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 18 Nov 2019, at 13:22, Sripra deep >> wrote: >>> >>> Hi Emir, >>> >>> Thank you so much. Now I got it cleared with the TLOG purpose. >>> I am trying to copy an index of one solr cluster to use it to build >>> other solr cluster. I am able to make that work but Is this design okay? >> or >>> any other approach I can try to get a new cluster spin up with the same >>> data as in the old one. >>> >>> Thanks, >>> Sripradeep P >>> >>> >>> On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović < >>> emir.arnauto...@sematext.com> wrote: >>> >>>> Hi Sripradeep, >>>> Simplified: TLog files are used to replay index updates from the last >>>> successful hard commit in case of some Solr crashes. It is used on the >> next >>>> Solr startup. It does not contain all updates, otherwise, it would >>>> duplicate the index size. >>>> If you start from these premises, you will understand why it is not >> copied >>>> when adding replicas and why it is not needed and why you cannot use >> TLog >>>> to spin up a new cluster. >>>> >>>> HTH, >>>> Emir >>>> -- >>>> Monitoring - Log Management - Alerting - Anomaly Detection >>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >>>> >>>> >>>> >>>>> On 18 Nov 2019, at 06:35, Sripra deep >>>> wrote: >>>>> >>>>> Hi Guys, >>>>> >>>>> I observed a scenario with the tlog creation and usage and couldn't >> find >>>>> any usage for the tlog. >>>>> >>>>> Solr version: 7.1.0 >>>>> Number of shards = 3 >>>>> Number of replica = 1 >>>>> I indexed the about 10k docs into the collection. >>>>> >>>>> Scenario 1: >>>>> Using add replica collection API, I created one more replica (tried >> with >>>>> both nrt and tlog) neither of the replicas doesn't pull the tlog files. >>>>> Only the index files are pulled from master. >>>>> * If the tlog is not present in a replica then during ungraceful >>>> shutdown >>>>> of the solr server how the replicas will regain the index without tlog >>>>> files. >>>>> * To verify the above scenario, I killed the newly added replica server >>>>> with kill -9 command and started back >>>>> also stopped the leader node. >>>>> >>>>> Questions: >>>>> 1) TLog files are not used even in the case of ungraceful shutdown, >>>> where >>>>> else it will be used? >>>>> 2) Tlog files doesn't get copied to the newly added replica so adding a >>>>> new replica to the already created collection with data/index is not >>>>> advisable? >>>>> 3) Is there a way to make the newly added slave node to replicate the >>>>> tlog file as it does for the data/index files from leader? >>>>> 4) Is it possible to use the Tlog files /index files from an existing >>>>> solr server to spin up a new solr cluster? >>>>> >>>>> >>>>> It would be much helpful for me to understand the core working of Solr >>>>> server. >>>>> >>>>> Thanks, >>>>> Sripradeep P >>>> >>>> >> >>
Re: Use of TLog
Got it, Thanks for helping me Emir. On Tue, Nov 19, 2019 at 12:36 PM Emir Arnautović < emir.arnauto...@sematext.com> wrote: > Hi, > Copying indices will work and it is a fine approach. An alternative would > be to join new node to a cluster, use add replica to copy cores to this new > node and then remove replicas from old nodes, if you want to move cores. > > Regards, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > On 18 Nov 2019, at 13:22, Sripra deep > wrote: > > > > Hi Emir, > > > > Thank you so much. Now I got it cleared with the TLOG purpose. > > I am trying to copy an index of one solr cluster to use it to build > > other solr cluster. I am able to make that work but Is this design okay? > or > > any other approach I can try to get a new cluster spin up with the same > > data as in the old one. > > > > Thanks, > > Sripradeep P > > > > > > On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović < > > emir.arnauto...@sematext.com> wrote: > > > >> Hi Sripradeep, > >> Simplified: TLog files are used to replay index updates from the last > >> successful hard commit in case of some Solr crashes. It is used on the > next > >> Solr startup. It does not contain all updates, otherwise, it would > >> duplicate the index size. > >> If you start from these premises, you will understand why it is not > copied > >> when adding replicas and why it is not needed and why you cannot use > TLog > >> to spin up a new cluster. > >> > >> HTH, > >> Emir > >> -- > >> Monitoring - Log Management - Alerting - Anomaly Detection > >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > >> > >> > >> > >>> On 18 Nov 2019, at 06:35, Sripra deep > >> wrote: > >>> > >>> Hi Guys, > >>> > >>> I observed a scenario with the tlog creation and usage and couldn't > find > >>> any usage for the tlog. > >>> > >>> Solr version: 7.1.0 > >>> Number of shards = 3 > >>> Number of replica = 1 > >>> I indexed the about 10k docs into the collection. > >>> > >>> Scenario 1: > >>> Using add replica collection API, I created one more replica (tried > with > >>> both nrt and tlog) neither of the replicas doesn't pull the tlog files. > >>> Only the index files are pulled from master. > >>> * If the tlog is not present in a replica then during ungraceful > >> shutdown > >>> of the solr server how the replicas will regain the index without tlog > >>> files. > >>> * To verify the above scenario, I killed the newly added replica server > >>> with kill -9 command and started back > >>> also stopped the leader node. > >>> > >>> Questions: > >>> 1) TLog files are not used even in the case of ungraceful shutdown, > >> where > >>> else it will be used? > >>> 2) Tlog files doesn't get copied to the newly added replica so adding a > >>> new replica to the already created collection with data/index is not > >>> advisable? > >>> 3) Is there a way to make the newly added slave node to replicate the > >>> tlog file as it does for the data/index files from leader? > >>> 4) Is it possible to use the Tlog files /index files from an existing > >>> solr server to spin up a new solr cluster? > >>> > >>> > >>> It would be much helpful for me to understand the core working of Solr > >>> server. > >>> > >>> Thanks, > >>> Sripradeep P > >> > >> > >
Re: Use of TLog
Hi, Copying indices will work and it is a fine approach. An alternative would be to join new node to a cluster, use add replica to copy cores to this new node and then remove replicas from old nodes, if you want to move cores. Regards, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 18 Nov 2019, at 13:22, Sripra deep wrote: > > Hi Emir, > > Thank you so much. Now I got it cleared with the TLOG purpose. > I am trying to copy an index of one solr cluster to use it to build > other solr cluster. I am able to make that work but Is this design okay? or > any other approach I can try to get a new cluster spin up with the same > data as in the old one. > > Thanks, > Sripradeep P > > > On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović < > emir.arnauto...@sematext.com> wrote: > >> Hi Sripradeep, >> Simplified: TLog files are used to replay index updates from the last >> successful hard commit in case of some Solr crashes. It is used on the next >> Solr startup. It does not contain all updates, otherwise, it would >> duplicate the index size. >> If you start from these premises, you will understand why it is not copied >> when adding replicas and why it is not needed and why you cannot use TLog >> to spin up a new cluster. >> >> HTH, >> Emir >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> >> >> >>> On 18 Nov 2019, at 06:35, Sripra deep >> wrote: >>> >>> Hi Guys, >>> >>> I observed a scenario with the tlog creation and usage and couldn't find >>> any usage for the tlog. >>> >>> Solr version: 7.1.0 >>> Number of shards = 3 >>> Number of replica = 1 >>> I indexed the about 10k docs into the collection. >>> >>> Scenario 1: >>> Using add replica collection API, I created one more replica (tried with >>> both nrt and tlog) neither of the replicas doesn't pull the tlog files. >>> Only the index files are pulled from master. >>> * If the tlog is not present in a replica then during ungraceful >> shutdown >>> of the solr server how the replicas will regain the index without tlog >>> files. >>> * To verify the above scenario, I killed the newly added replica server >>> with kill -9 command and started back >>> also stopped the leader node. >>> >>> Questions: >>> 1) TLog files are not used even in the case of ungraceful shutdown, >> where >>> else it will be used? >>> 2) Tlog files doesn't get copied to the newly added replica so adding a >>> new replica to the already created collection with data/index is not >>> advisable? >>> 3) Is there a way to make the newly added slave node to replicate the >>> tlog file as it does for the data/index files from leader? >>> 4) Is it possible to use the Tlog files /index files from an existing >>> solr server to spin up a new solr cluster? >>> >>> >>> It would be much helpful for me to understand the core working of Solr >>> server. >>> >>> Thanks, >>> Sripradeep P >> >>
Re: Use of TLog
Hi Emir, Thank you so much. Now I got it cleared with the TLOG purpose. I am trying to copy an index of one solr cluster to use it to build other solr cluster. I am able to make that work but Is this design okay? or any other approach I can try to get a new cluster spin up with the same data as in the old one. Thanks, Sripradeep P On Mon, Nov 18, 2019 at 2:12 PM Emir Arnautović < emir.arnauto...@sematext.com> wrote: > Hi Sripradeep, > Simplified: TLog files are used to replay index updates from the last > successful hard commit in case of some Solr crashes. It is used on the next > Solr startup. It does not contain all updates, otherwise, it would > duplicate the index size. > If you start from these premises, you will understand why it is not copied > when adding replicas and why it is not needed and why you cannot use TLog > to spin up a new cluster. > > HTH, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > On 18 Nov 2019, at 06:35, Sripra deep > wrote: > > > > Hi Guys, > > > > I observed a scenario with the tlog creation and usage and couldn't find > > any usage for the tlog. > > > > Solr version: 7.1.0 > > Number of shards = 3 > > Number of replica = 1 > > I indexed the about 10k docs into the collection. > > > > Scenario 1: > > Using add replica collection API, I created one more replica (tried with > > both nrt and tlog) neither of the replicas doesn't pull the tlog files. > > Only the index files are pulled from master. > > * If the tlog is not present in a replica then during ungraceful > shutdown > > of the solr server how the replicas will regain the index without tlog > > files. > > * To verify the above scenario, I killed the newly added replica server > > with kill -9 command and started back > > also stopped the leader node. > > > > Questions: > > 1) TLog files are not used even in the case of ungraceful shutdown, > where > > else it will be used? > > 2) Tlog files doesn't get copied to the newly added replica so adding a > > new replica to the already created collection with data/index is not > > advisable? > > 3) Is there a way to make the newly added slave node to replicate the > > tlog file as it does for the data/index files from leader? > > 4) Is it possible to use the Tlog files /index files from an existing > > solr server to spin up a new solr cluster? > > > > > > It would be much helpful for me to understand the core working of Solr > > server. > > > > Thanks, > > Sripradeep P > >
Re: Use of TLog
Hi Sripradeep, Simplified: TLog files are used to replay index updates from the last successful hard commit in case of some Solr crashes. It is used on the next Solr startup. It does not contain all updates, otherwise, it would duplicate the index size. If you start from these premises, you will understand why it is not copied when adding replicas and why it is not needed and why you cannot use TLog to spin up a new cluster. HTH, Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 18 Nov 2019, at 06:35, Sripra deep wrote: > > Hi Guys, > > I observed a scenario with the tlog creation and usage and couldn't find > any usage for the tlog. > > Solr version: 7.1.0 > Number of shards = 3 > Number of replica = 1 > I indexed the about 10k docs into the collection. > > Scenario 1: > Using add replica collection API, I created one more replica (tried with > both nrt and tlog) neither of the replicas doesn't pull the tlog files. > Only the index files are pulled from master. > * If the tlog is not present in a replica then during ungraceful shutdown > of the solr server how the replicas will regain the index without tlog > files. > * To verify the above scenario, I killed the newly added replica server > with kill -9 command and started back > also stopped the leader node. > > Questions: > 1) TLog files are not used even in the case of ungraceful shutdown, where > else it will be used? > 2) Tlog files doesn't get copied to the newly added replica so adding a > new replica to the already created collection with data/index is not > advisable? > 3) Is there a way to make the newly added slave node to replicate the > tlog file as it does for the data/index files from leader? > 4) Is it possible to use the Tlog files /index files from an existing > solr server to spin up a new solr cluster? > > > It would be much helpful for me to understand the core working of Solr > server. > > Thanks, > Sripradeep P
Use of TLog
Hi Guys, I observed a scenario with the tlog creation and usage and couldn't find any usage for the tlog. Solr version: 7.1.0 Number of shards = 3 Number of replica = 1 I indexed the about 10k docs into the collection. Scenario 1: Using add replica collection API, I created one more replica (tried with both nrt and tlog) neither of the replicas doesn't pull the tlog files. Only the index files are pulled from master. * If the tlog is not present in a replica then during ungraceful shutdown of the solr server how the replicas will regain the index without tlog files. * To verify the above scenario, I killed the newly added replica server with kill -9 command and started back also stopped the leader node. Questions: 1) TLog files are not used even in the case of ungraceful shutdown, where else it will be used? 2) Tlog files doesn't get copied to the newly added replica so adding a new replica to the already created collection with data/index is not advisable? 3) Is there a way to make the newly added slave node to replicate the tlog file as it does for the data/index files from leader? 4) Is it possible to use the Tlog files /index files from an existing solr server to spin up a new solr cluster? It would be much helpful for me to understand the core working of Solr server. Thanks, Sripradeep P
