Re: zookeeper data and collection properties were lost
On 7/20/2020 10:30 AM, yaswanth kumar wrote: 1# I did make sure that zoo.cfg got the proper data dir and its not pointing to temp folder; do I need to set the variables in ZK_ENV.sh. as well on top of the zoo.cfg ?? Those are questions about the ZK server, which we are not completely qualified to answer. ZK and Solr are separate Apache projects, with separate mailing lists. We have some familiarity with ZK because it is required to run Solr in cloud mode, but are not experts. We can only provide minimal help with standalone ZK servers ... you would need to talk to the ZK project for the best information. Here are my confusions, as I said we are in two node architecture in DEV but maintaining only one instance of zookeeper, is that true that I need to maintain the same folder structure that we specify on the dataDir of zoo.cfg on both the nodes ?? Each ZK server is independent of the others and should have its own data directory. ZK will handle creating the contents of that directory, it is likely not something you would do. Each server could have a different setting for the data directory, or the same setting. Note that if the setting is the same on multiple servers, that each of those directories should point to separate storage. If you try to use a shared directory (perhaps with NFS) then I would imagine that ZK will not function correctly. A fault tolerant install of ZK cannot be created with only two servers. It requires a minimum of three. For the Solr part, only two servers are required for minimal fault tolerance. Each Solr server must be configured with the addresses and ports of all 3 (or more) zookeeper servers. See the Note in the following sections of the ZK documentation: https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html#sc_zkMulitServerSetup https://zookeeper.apache.org/doc/r3.5.8/zookeeperStarted.html#sc_RunningReplicatedZooKeeper Thanks, Shawn
Re: zookeeper data and collection properties were lost
Thanks Erick for a quick response. Here are my responses for your questions 1# I did make sure that zoo.cfg got the proper data dir and its not pointing to temp folder; do I need to set the variables in ZK_ENV.sh. as well on top of the zoo.cfg ?? 2# I can confirm that we are not using the embedded one but we are using a standalone zookeeper 3.4.14 and also the admin UI is showing what we configured (port 2181) Here are my confusions, as I said we are in two node architecture in DEV but maintaining only one instance of zookeeper, is that true that I need to maintain the same folder structure that we specify on the dataDir of zoo.cfg on both the nodes ?? Thanks, On Mon, Jul 20, 2020 at 12:22 PM Erick Erickson wrote: > Some possibilities: > > 1> you haven’t changed your data dir for Zookeeper from the default > "/tmp/zookeeper” > > 2> you aren’t pointing to the Zookeepers you think you are. In particular > are you running embedded zookeeper? This should be apparent if you look on > the admin page ant the zookeeper URLs you’re pointing at are on port 9983 > > this is almost certainly some kind of misconfiguration, zookeeper data > doesn’t just disappear on its own that I know of. The admin UI will also > show you the exact parameters that Solr starts up with, check that they’re > all pointing to the ZK ensemble you expect and that the data directory is > preserved across restarts/reboots etc. > > Best, > Erick > > > On Jul 20, 2020, at 12:02 PM, yaswanth kumar > wrote: > > > > HI Team, > > > > Can someone help me understand on what could be the reason to lose both > > zookeeper data and also the collection information that will be stored > for > > each collection in the path ../solr/server/solr/ > > > > Here are the details of what versions that we use > > > > Solr - 8.2 > > Zookeeper 3.4.14 > > > > Two node solr cloud with zookeeper on single node, and when ever we see > an > > issue with networking between these two nodes, and once the connectivity > is > > restored, but when we restart the zookeeper service , everything was lost > > under /zookeeper_data/version-2/ and also the collection folders that > used > > to exists under ../solr/server/solr/ > > > > *Note*: We are testing this in DEV environment, but with this behavior we > > are afraid of moving this to production without knowing if that's an > issue > > with some configuration or zookeeper behavior and we need to adjust > > something else to not to wipe out the configs. > > > > -- > > Thanks & Regards, > > Yaswanth Kumar Konathala. > > yaswanth...@gmail.com > > -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
Re: zookeeper data and collection properties were lost
Some possibilities: 1> you haven’t changed your data dir for Zookeeper from the default "/tmp/zookeeper” 2> you aren’t pointing to the Zookeepers you think you are. In particular are you running embedded zookeeper? This should be apparent if you look on the admin page ant the zookeeper URLs you’re pointing at are on port 9983 this is almost certainly some kind of misconfiguration, zookeeper data doesn’t just disappear on its own that I know of. The admin UI will also show you the exact parameters that Solr starts up with, check that they’re all pointing to the ZK ensemble you expect and that the data directory is preserved across restarts/reboots etc. Best, Erick > On Jul 20, 2020, at 12:02 PM, yaswanth kumar wrote: > > HI Team, > > Can someone help me understand on what could be the reason to lose both > zookeeper data and also the collection information that will be stored for > each collection in the path ../solr/server/solr/ > > Here are the details of what versions that we use > > Solr - 8.2 > Zookeeper 3.4.14 > > Two node solr cloud with zookeeper on single node, and when ever we see an > issue with networking between these two nodes, and once the connectivity is > restored, but when we restart the zookeeper service , everything was lost > under /zookeeper_data/version-2/ and also the collection folders that used > to exists under ../solr/server/solr/ > > *Note*: We are testing this in DEV environment, but with this behavior we > are afraid of moving this to production without knowing if that's an issue > with some configuration or zookeeper behavior and we need to adjust > something else to not to wipe out the configs. > > -- > Thanks & Regards, > Yaswanth Kumar Konathala. > yaswanth...@gmail.com
zookeeper data and collection properties were lost
HI Team, Can someone help me understand on what could be the reason to lose both zookeeper data and also the collection information that will be stored for each collection in the path ../solr/server/solr/ Here are the details of what versions that we use Solr - 8.2 Zookeeper 3.4.14 Two node solr cloud with zookeeper on single node, and when ever we see an issue with networking between these two nodes, and once the connectivity is restored, but when we restart the zookeeper service , everything was lost under /zookeeper_data/version-2/ and also the collection folders that used to exists under ../solr/server/solr/ *Note*: We are testing this in DEV environment, but with this behavior we are afraid of moving this to production without knowing if that's an issue with some configuration or zookeeper behavior and we need to adjust something else to not to wipe out the configs. -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
Re: collection properties
I opened SOLR-12224 for this: https://issues.apache.org/jira/browse/SOLR-12224 On 14.04.2018 01:49, Shawn Heisey wrote: On 4/13/2018 5:07 PM, Tomás Fernández Löbbe wrote: Yes... Unfortunately there is no GET API :S Can you open a Jira? Patch should be trivial My suggestion would be to return the list of properties for a collection when a URL like this is used: /solr/admin/collections?action=COLLECTIONPROP=gettingstarted At the moment, this complains that he propertyName parameter is required and missing. If the "name" parameter is omitted, it should return the properties for ALL collections. The format of the single-collection response should be the same as the all-collection response -- in the JSON, have a key for the collection name and then under that, keys for each property. It would be nice to allow multiple "name" parameters for the list (when propertyName is not present). Do we also need a specific parameter to explicitly tell Solr to list the properties? Or maybe an explicit action value for listing them, like LISTCOLLECTIONPROP? Thanks, Shawn
Re: collection properties
On 4/13/2018 5:07 PM, Tomás Fernández Löbbe wrote: > Yes... Unfortunately there is no GET API :S Can you open a Jira? Patch > should be trivial My suggestion would be to return the list of properties for a collection when a URL like this is used: /solr/admin/collections?action=COLLECTIONPROP=gettingstarted At the moment, this complains that he propertyName parameter is required and missing. If the "name" parameter is omitted, it should return the properties for ALL collections. The format of the single-collection response should be the same as the all-collection response -- in the JSON, have a key for the collection name and then under that, keys for each property. It would be nice to allow multiple "name" parameters for the list (when propertyName is not present). Do we also need a specific parameter to explicitly tell Solr to list the properties? Or maybe an explicit action value for listing them, like LISTCOLLECTIONPROP? Thanks, Shawn
Re: collection properties
Yes... Unfortunately there is no GET API :S Can you open a Jira? Patch should be trivial On Fri, Apr 13, 2018 at 3:05 PM, Hendrik Haddorp <hendrik.hadd...@gmx.net> wrote: > Hi, > > with Solr 7.3 it is possible to set arbitrary collection properties using > https://lucene.apache.org/solr/guide/7_3/collections-api. > html#collectionprop > But how do I read out the properties again? So far I could not find a REST > call that would return the properties. I do see my property in the ZK file > collectionprops.json below my collection though. > > thanks, > Hendrik >
collection properties
Hi, with Solr 7.3 it is possible to set arbitrary collection properties using https://lucene.apache.org/solr/guide/7_3/collections-api.html#collectionprop But how do I read out the properties again? So far I could not find a REST call that would return the properties. I do see my property in the ZK file collectionprops.json below my collection though. thanks, Hendrik
Re: SolrCloud collection properties
We are currently having many custom properties defined in the
core.properties which are used in our solrconfig.xml, e.g.
str name=enabled${solr.enable.cachewarming:true}/str
Now we want to migrate to SolrCloud and want to define these properties for
a collection. But defining properties when creating a collection just
writes them into the core.properties of the created cores. This is a pain,
because we have a lot of properties and you have to specify each as an URL
parameter. Furthermore it seems that these properties are not propagated to
the cores for new shards, if you e.g. split a shard - error-prone.
As you already mentioned, we could resolve this properties ourselves by
using many configsets instead of just one. My question was, if it is
possible to use just one configset in this case and specify collection
specific properties at the collection level? This seems for me the better
way to handle the configuration complexity.
Markus
2015-05-06 3:48 GMT+02:00 Erick Erickson erickerick...@gmail.com:
_What_ properties? Details matter
And how do you do this now? Assuming you do this with separate conf
directories, these are then just configsets in Zookeeper and you can
have as many of them as you want. Problem here is that each one of
them is a complete set of schema and config files, AFAIK the config
set is the finest granularity that you have OOB.
Best,
Erick
On Tue, May 5, 2015 at 6:55 AM, Markus Heiden markus.hei...@s24.com
wrote:
Hi,
we are trying to migrate from Solr 4.10 to SolrCloud 4.10. I understood
that SolrCloud uses collections as abstraction from the cores. What I am
missing is a possibility to store collection-specific properties in
Zookeeper. Using property.foo=bar in CREATE-URLs just sets core-specific
properties which are not distributed, e.g. if I migrate a shard from one
node to another.
How do I define collection-specific properties (to be used in
solrconfig.xml and schema.xml) which get distributed with the collection
to
all nodes?
Why do I try that? Currently we have different cores which structure is
identical, but have each having some specific properties. I would like to
have a single configuration for them in Zookeeper from which I want to
create different collections, which just differ in the value of some
properties.
Markus
Re: SolrCloud collection properties
_What_ properties? Details matter And how do you do this now? Assuming you do this with separate conf directories, these are then just configsets in Zookeeper and you can have as many of them as you want. Problem here is that each one of them is a complete set of schema and config files, AFAIK the config set is the finest granularity that you have OOB. Best, Erick On Tue, May 5, 2015 at 6:55 AM, Markus Heiden markus.hei...@s24.com wrote: Hi, we are trying to migrate from Solr 4.10 to SolrCloud 4.10. I understood that SolrCloud uses collections as abstraction from the cores. What I am missing is a possibility to store collection-specific properties in Zookeeper. Using property.foo=bar in CREATE-URLs just sets core-specific properties which are not distributed, e.g. if I migrate a shard from one node to another. How do I define collection-specific properties (to be used in solrconfig.xml and schema.xml) which get distributed with the collection to all nodes? Why do I try that? Currently we have different cores which structure is identical, but have each having some specific properties. I would like to have a single configuration for them in Zookeeper from which I want to create different collections, which just differ in the value of some properties. Markus
SolrCloud collection properties
Hi, we are trying to migrate from Solr 4.10 to SolrCloud 4.10. I understood that SolrCloud uses collections as abstraction from the cores. What I am missing is a possibility to store collection-specific properties in Zookeeper. Using property.foo=bar in CREATE-URLs just sets core-specific properties which are not distributed, e.g. if I migrate a shard from one node to another. How do I define collection-specific properties (to be used in solrconfig.xml and schema.xml) which get distributed with the collection to all nodes? Why do I try that? Currently we have different cores which structure is identical, but have each having some specific properties. I would like to have a single configuration for them in Zookeeper from which I want to create different collections, which just differ in the value of some properties. Markus
