[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2018-02-16 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367882#comment-16367882
 ] 

Amrit Sarkar commented on SOLR-8389:


Hi [~prusko],

I apologise I should have put down the details on how we are approaching on 
this JIRA with your patch. We are leveraging the patch provided by you to 
improve the CDCR configuration but obviously it has other use-cases and I see 
Tomas is reviewing on SOLR-11960. Well if SOLR-11960 goes in, then yes, I 
believe too we don't need a seperate cdcr.json, hence I will wait for others in 
the community to weigh in on agreeing to the common properties having cdcr ones.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, Screen 
> Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2018-02-09 Thread Peter Rusko (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359068#comment-16359068
 ] 

Peter Rusko commented on SOLR-8389:
---

Hi Amrit,

You've completely removed all the collection properties code, which could be 
useful not just for this patch, but some other use cases too. I've created 
SOLR-11960 to track that feature. I still think storing the properties that way 
has a lot of value and you could use it for CDCR properties too. It is more 
generic, and should be easily extendable, should you need to add validation of 
the property, or a way to set them at collection creation time.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, Screen 
> Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2018-01-24 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338025#comment-16338025
 ] 

Amrit Sarkar commented on SOLR-8389:


Took me a while to refractor APIs and write decent tests which are running 100% 
successfully. Patch uploaded, still lot of unfinished tests and documentation 
are due, but core code is in place. I have cleaned and moduled as much I could.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, Screen 
> Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2018-01-21 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333516#comment-16333516
 ] 

Amrit Sarkar commented on SOLR-8389:


Meanwhile while writing tests, CDCR API have to move outside 
{{CdcrRequestHandler}} to the collection api, where Collection Reload is 
imminent. Need some advice on this.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, Screen Shot 2017-12-21 at 
> 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2018-01-18 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331057#comment-16331057
 ] 

Amrit Sarkar commented on SOLR-8389:


Fully working clean code. Entire CDCR is working perfectly with back-combat. I 
am finishing up documentation and tests for this.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> Screen Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2018-01-15 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326208#comment-16326208
 ] 

Amrit Sarkar commented on SOLR-8389:


Fresh patch uploaded with no tests and documentation to have clean code for 
review and feedback.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-8389.patch, SOLR-8389.patch, SOLR-8389.patch, 
> Screen Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-12-27 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304519#comment-16304519
 ] 

Amrit Sarkar commented on SOLR-8389:


I would need some advice on designing the new APIs for CDCR specially adding / 
modifying target configs.

this is sample API I have designed for now which is very ineffective:

 
{{/cdcr/action=MODIFY=zkhost:zkpost/chroot=targetColName}}

and add them sequentially to add all the target collection information ONE BY 
ONE.

*Should I configure JSON payload as request or use V2 API to pass multiple 
target configs at once?* I know this configuration will be passed just one, and 
wouldn't hurt if target configs be passed one by one.

Looking forward to suggestions, I am still cleaning code to support this and 
then will start modifying tests around all components.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Attachments: SOLR-8389.patch, Screen Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-12-21 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299924#comment-16299924
 ] 

Amrit Sarkar commented on SOLR-8389:


I started working on extending the patch and collaborated with CDCR module, but 
was pulled away to professional duties and was not able to work further. I 
wrote a very rough patch, cleaning it now and should be able to get it ready 
soon. *PFA screenshot* of how {{cdcr.json}} looks like and sample CREATE 
collection command:
{code}
http://localhost:8983/solr/admin/collections?action=CREATE=source_col=1=1=true=source_col=target_col=localhost:8574
{code}

[~prusko], I have extended your code to support nested json formatted 
properties like below:
{code}
{
  "replica":[{
  "source":"source_col",
  "zkHost":"localhost:8574",
  "target":"target_col"}],
  "replicator":{
"schedule":1000,
"threadPoolSize":2,
"batchSize":128},
  "buffer":{"defaultState":"disabled"},
  "updateLogSynchronizer":{"schedule":6}}
{code}

In this example, not a single cdcr property mention is required in 
{{solrconfig.xml}} and using default configuration, a very significant and long 
time improvement coming.

[~erickerickson], I think I will be able to incorporate the target collection 
being optional and use the same source collection name as target if not 
specified quite easily.

[~prusko] [~erickerickson] [~varunthacker] [~shalinmangar], I will be posting 
the updated patch real soon, hopefully before the year end and will be looking 
forward to your feedback and comments.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Attachments: SOLR-8389.patch, Screen Shot 2017-12-21 at 5.44.36 PM.png
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-12-18 Thread Peter Rusko (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295829#comment-16295829
 ] 

Peter Rusko commented on SOLR-8389:
---

This patch above is just providing a way for properties to be assigned to 
collections. It is pretty generic at the moment, the value can be anything, so 
you can assign any JSON value you want, it won't require a target collection to 
be set. The interpretation of the value comes down to the code reading the 
value.

It looks like it's missing two things. None of those seem to be important for 
the first iteration to me. One is the ability to assign a value at the 
collection creation time and the other is a support of a property value 
validator, which could check that the assigned value has certain fields for 
instance.

Is it okay if we merge this patch as it is so far, provided someone can review 
it? Then I, or someone else with better ideas can add the two missing pieces 
above later, if they are needed.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Attachments: SOLR-8389.patch
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-12-18 Thread Erick Erickson (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295497#comment-16295497
 ] 

Erick Erickson commented on SOLR-8389:
--

Haven't looked at all, but a comment on the user's list 

WDYT about making the target collection optional? If omitted it would be the 
same as the source collection



> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Attachments: SOLR-8389.patch
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-11-17 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257705#comment-16257705
 ] 

Amrit Sarkar commented on SOLR-8389:


[~prusko],

Thank you for coming up with the patch. Allow me sometime to go through the 
improvement and would definitely seek your help and collaboration.

Thanks
Amrit Sarkar

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Attachments: SOLR-8389.patch
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-11-16 Thread Peter Rusko (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256127#comment-16256127
 ] 

Peter Rusko commented on SOLR-8389:
---

Hi Amrit,

I didn't have as much time as I wanted, but finally, here's the patch. This 
doesn't let you set up the properties at the collection creation, but I can add 
that too if it's needed. Let me know what you think.

Thanks,
Peter

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Attachments: SOLR-8389.patch
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-10-13 Thread Peter Rusko (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203842#comment-16203842
 ] 

Peter Rusko commented on SOLR-8389:
---

The main concern was exactly the update frequency, that's why I separated this 
from state.json, to clusterprops.json. It will have its own watcher and won't 
be affected by clusterstate changes. The intention was to put user-defined 
properties here, which can be changed without restarting solr.

Varun, I'm not sure what you mean by that. Currently unrelated collection 
property changes still trigger all the listeners, but given that they are all 
supposed to be rare (though nothing prevents anyone from writing properties 
from the code I guess), I don't see this as a problem.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-10-13 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203745#comment-16203745
 ] 

Amrit Sarkar commented on SOLR-8389:


Peter, how often the collection properties will change? Are these properties 
comprise of {{user-defined}} ones or subjected to machine / node level 
properties? If unrelated updates are too frequent., I think Varun is right; it 
won't be helpful in this case atleast. Let us know.



> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-10-12 Thread Varun Thacker (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203007#comment-16203007
 ] 

Varun Thacker commented on SOLR-8389:
-

bq. ZkStateReader keeps a set of watchers for each of them (one per 
collection), in a similar way to state.json. But on top of caching the values, 
it provides a method where you can add listeners, that will be called each time 
the znode is changed.

Will it suffer from the same problem as putting the configs in state.json? 
Unrelated updates in the collection properties api might trigger a bootstrap ? 

 

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-10-11 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1613#comment-1613
 ] 

Amrit Sarkar commented on SOLR-8389:


Hi Peter,

Absolutely, this serves and solves the purpose. I was about to go to down the 
lines of autoscaling, glad you already worked that out. Couple of points I want 
to lay out specific for CDCR:

1. The CDCR configurations will be uploaded at the time of collection creation. 
Calling the api, collection-props, you are working on, implicitly can be called 
if cdcr is involved.
2. The idea is to keep cdcr away from {{state.json}}, so that it doesn't get 
rewritten by any node status change. I think the motivation of creating 
collection properties is the same, if not wrong.

Looking forward to your patch and ultimately in the code, so that I can extend 
the same to regulate cdcr configurations and actions. Thanks Peter.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-10-09 Thread Peter Rusko (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197742#comment-16197742
 ] 

Peter Rusko commented on SOLR-8389:
---

Hi Amrit,

I’m working on something that would solve this problem. I already have a 
working prototype and can upload a patch soon.

The idea is to add the concept of collection properties. It’s a key value map 
that would be stored in the ```collections//collectionprops.json``` znode 
for each collection in json format. The properties can be changed via 
/solr/admin/collections?action=COLLECTIONPROP?name===.

ZkStateReader keeps a set of watchers for each of them (one per collection), in 
a similar way to state.json. But on top of caching the values, it provides a 
method where you can add listeners, that will be called each time the znode is 
changed.

I think this pretty much aligns with your idea, but will be a more scalable 
approach for the watchers. It can accommodate the CDCR properties as well as 
any other properties that could be needed for the collection. Let me know if 
this could work for you or if you have any questions.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-09-28 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184000#comment-16184000
 ] 

Amrit Sarkar commented on SOLR-8389:


Proposed design:

As state.json for collections gets modified / overwritten every time there is a 
state change of the nodes, it is somehow risky to put the cdcr configurations 
in the same place, as it can unnecessary bootstrap and other inconsistencies.

On an offline discussion with [~shalinmangar], we plan to:

1. have seperate config json for cdcr at root level dir in zookeeper: 
{{/cdcr.json}}
2. they should be collection specific in following format:
{code}
{
"collection1": {
"replica": {
"zkHost":"target-zk-host:target-zk-port/chroot-to-solr",
"source":"collection1",
"target":"collection1"
},
"replicator": {
"threadPoolSize":"8",
"schedule":"1000",
"batchSize":"128"
},
"updateLogSynchronizer": {
"schedule":"1000"
}
}
}
{code}
3. the cdcr config should be passed with *CREATE* collection command as the 
collection must use {{CdcrUpdateLog}} and not the conventional one, 
{{UpdateLog}}.
4. expose collections api like "SETCDCR" to modify CDCR configuration. It will 
require collection reload, hence will be implicit. Removing CDCR api should not 
be exposed, as it will be of no use since the collection is already created 
with {{CdcrUpdateLog}}.

The back-combat will be established, with cdcr json config overriding 
solrconfig ones.

If no objection with the design, will move forward with development.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-09-20 Thread Shalin Shekhar Mangar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173546#comment-16173546
 ] 

Shalin Shekhar Mangar commented on SOLR-8389:
-

Sounds great! Looking forward to it.

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8389) Convert CDCR peer cluster and other configurations into collection properties modifiable via APIs

2017-09-19 Thread Amrit Sarkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/SOLR-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172562#comment-16172562
 ] 

Amrit Sarkar commented on SOLR-8389:


Shalin,

I will give it a go to get this done. Will post patch and more information once 
I have something of substance.

- Amrit

> Convert CDCR peer cluster and other configurations into collection properties 
> modifiable via APIs
> -
>
> Key: SOLR-8389
> URL: https://issues.apache.org/jira/browse/SOLR-8389
> Project: Solr
>  Issue Type: Improvement
>  Components: CDCR, SolrCloud
>Reporter: Shalin Shekhar Mangar
> Fix For: 6.0
>
>
> CDCR configuration is kept inside solrconfig.xml which makes it difficult to 
> add or change peer cluster configuration.
> I propose to move all CDCR config to collection level properties in cluster 
> state so that they can be modified using the existing modify collection API.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org