[jira] [Comment Edited] (SOLR-11487) Collection Alias metadata for time partitioned collections

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434467#comment-16434467
 ] 

Varun Thacker edited comment on SOLR-11487 at 4/11/18 7:43 PM:
---

Hi David,

I've added the fix version as 7.2 for this Jira for reference

 

We recently had a user who ran into a race condition when updating aliases on a 
Solr 5.x . Looking at master today it looks like we're dealing with race 
conditions and this comment confirms that we fixed it as part of this Jira 

 
{quote}Decomposing aliases.json has pros/cons, but it won't remove the 
possibility of races between modifying some portion of the aliases state; it 
just makes it more rare. So we still need to deal with races in code using 
a zkVersion with a retry and eventual timeout, etc.
{quote}


was (Author: varunthacker):
Hi David,

I've added the fix version as 7.2 for this Jira for reference

 

We recently had a user who ran into a race condition when updating aliases on a 
Solr 5.x . Looking at master today it looks like we're dealing with race 
conditions and this comment confirms that we fixed it as part of this Jira 

 
{quote}Decomposing aliases.json has pros/cons, but it won't remove the 
possibility of races between modifying some portion of the aliases state; it 
just makes it more rare. So we still need to deal with races in code using 
a zkVersion with a retry and eventual timeout, etc.
{quote}
 

 

 

> Collection Alias metadata for time partitioned collections
> --
>
> Key: SOLR-11487
> URL: https://issues.apache.org/jira/browse/SOLR-11487
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Fix For: 7.2
>
> Attachments: SOLR_11487.patch, SOLR_11487.patch, SOLR_11487.patch, 
> SOLR_11487.patch, SOLR_11487.patch, SOLR_11487.patch, SOLR_11487.patch, 
> SOLR_11487.patch, SOLR_11487.patch
>
>
> SOLR-11299 outlines an approach to using a collection Alias to refer to a 
> series of collections of a time series. We'll need to store some metadata 
> about these time series collections, such as which field of the document 
> contains the timestamp to route on.
> The current {{/aliases.json}} is a Map with a key {{collection}} which is in 
> turn a Map of alias name strings to a comma delimited list of the collections.
> _If we change the comma delimited list to be another Map to hold the existing 
> list and more stuff, older CloudSolrClient (configured to talk to ZooKeeper) 
> will break_.  Although if it's configured with an HTTP Solr URL then it would 
> not break.  There's also some read/write hassle to worry about -- we may need 
> to continue to read an aliases.json in the older format.
> Alternatively, we could add a new map entry to aliases.json, say, 
> {{collection_metadata}} keyed by alias name?
> Perhaps another very different approach is to attach metadata to the 
> configset in use?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-11487) Collection Alias metadata for time partitioned collections

[Gus Heck (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224353#comment-16224353
 ] 

Gus Heck edited comment on SOLR-11487 at 10/30/17 3:54 AM:
---

Attaching revised patch. 

Highlights:
* Immutability was restored
* After getting irritated with a number of issues caused by miss-matches 
between the Map version of the aliaes and 
Map notions of aliases that were being kept, including a 
need to clone, tweak and re-clone to get the list version back in sync, I nixed 
the duplicated state, and made the method providing the Map view 
construct that view on the fly. There is now only one canonical map, nothing to 
keep in sync, and it contains the List values. I could be talked into 
flip flopping to keep comma strings and construct lists on the fly, but at the 
moment it all passes (and still writes comma strings to zookeeper). I don't 
think it matters much which way we do it.
* I fixed immutability leakage with the Map version where 
the Map was immutable, but the list elements were still mutable, and thus could 
be added/removed producing state (reflected in the original Alias instance) out 
of sync with the Map version of aliases held in the 
"collections" key of the main aliasMap field. New code now provides Immutable 
map with immutable list values.
* TimeOut class removed, replaced with a OneTimeListener class that allows 
easy, temporary piggybacking on the AliasWatcher I extracted in the prior 
patch. This is coded in a way that if other watchers simply extend Observer 
they can also use this too. The net effect is rather than polling on a loop we 
wait on a countdown latch which is released after a timeout or if the watcher 
hears an update that passes a test (written by the implementor of the 
OneTimeListener). The listener is automatically removed on successful test or 
timeout. This means we don't evaluate the test condition any more times than 
aliases are updated. This could be more executions of the test condition if 
aliases were under heavy updates, but that seems like an oddball condition so 
this will probably result in many fewer tests. In any case it will be way less 
expensive than the hot loop.


was (Author: gus_heck):
Attaching revised patch. 

Highlights:
* Immutability was restored
* After getting irritated with a number of issues caused by miss-matches 
between the Map version of the aliaes and 
Map notions of aliases that were being kept, including a 
need to clone, tweak and re-clone to get the list version back in sync, I nixed 
the duplicated state, and made the method providing the Map view 
construct that view on the fly. There is now only one canonical map, nothing to 
keep in sync, and it contains the List values. I could be talked into 
flip flopping to keep comma strings and construct lists on the fly, but at the 
moment it all passes (and still writes comma strings to zookeeper). I don't 
think it matters much which way we do it.
* I fixed immutability leakage with the Map version where 
the Map was immutable, but the list elements were still mutable, and thus could 
be added/removed producing state (reflected in the original Alias instance) out 
of sync with the Map version of aliases held in the 
"collections" key of the main aliasMap field. New code now provides Immutable 
map with immutable list values.
* TimeOut class removed, replaced with a OneTimeListener class that allows 
easy, temporary piggybacking on the AliasWatcher I extracted in the prior 
patch. This is coded in a way that if other watchers simply extend Observer 
they can also use this too. The net effect is rather than polling on a loop we 
wait on a countdown latch which is released after a timeout or if the watcher 
hears an update that passes a test (written by the implementor of the 
OneTimeListener). The listener is automatically removed on successful test or 
timeout. This means we don't evaluate the test condition any more times than 
aliases are updated. This could executions of the test condition if aliases 
were under heavy updates, but that seems like an oddball condition so this will 
probably result in many fewer tests. In any case it will be way less expensive 
than the hot loop.

> Collection Alias metadata for time partitioned collections
> --
>
> Key: SOLR-11487
> URL: https://issues.apache.org/jira/browse/SOLR-11487
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR_11487.patch, SOLR_11487.patch, SOLR_11487.patch
>
>
> SOLR-11299 outlines an approach to 

[jira] [Comment Edited] (SOLR-11487) Collection Alias metadata for time partitioned collections

[Gus Heck (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205218#comment-16205218
 ] 

Gus Heck edited comment on SOLR-11487 at 10/15/17 5:30 PM:
---

In zk nodes can have both values and children right? So the value of the node 
called aliases.json can remain the same json text, but it could also have a 
list of children corresponding to each alias and then each member of the alias 
containing that metadata... yes some duplication there, but this would mean 
that any older clients reading the value from the node will still get what they 
expect... newer code could simply ignore the old json...


was (Author: gus_heck):
In zk nodes can have both values and children right? So the value of the node 
called aliases.json can remain the same json text, but it could also have a 
list of children corresponding to each member of the alias containing that 
metadata... yes some duplication there, but this would mean that any older 
clients reading the value from the node will still get what they expect... 
newer code could simply ignore the old json...

> Collection Alias metadata for time partitioned collections
> --
>
> Key: SOLR-11487
> URL: https://issues.apache.org/jira/browse/SOLR-11487
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
>
> SOLR-11299 outlines an approach to using a collection Alias to refer to a 
> series of collections of a time series. We'll need to store some metadata 
> about these time series collections, such as which field of the document 
> contains the timestamp to route on.
> The current {{/aliases.json}} is a Map with a key {{collection}} which is in 
> turn a Map of alias name strings to a comma delimited list of the collections.
> _If we change the comma delimited list to be another Map to hold the existing 
> list and more stuff, older CloudSolrClient (configured to talk to ZooKeeper) 
> will break_.  Although if it's configured with an HTTP Solr URL then it would 
> not break.  There's also some read/write hassle to worry about -- we may need 
> to continue to read an aliases.json in the older format.
> Alternatively, we could add a new map entry to aliases.json, say, 
> {{collection_metadata}} keyed by alias name?
> Perhaps another very different approach is to attach metadata to the 
> configset in use?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org