Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!
Congratulations, Jan and thank you! Thanks, Anshum for all your work as chair. On Fri, Feb 19, 2021 at 12:26 AM Anshum Gupta wrote: > Hi everyone, > > I’d like to inform everyone that the newly formed Apache Solr PMC nominated > and elected Jan Høydahl for the position of the Solr PMC Chair and Vice > President. This decision was approved by the board in its February 2021 > meeting. > > Congratulations Jan! > > -- > Anshum Gupta > -- Regards, Shalin Shekhar Mangar.
Re: Solrcloud load balancing / failover
No, the load balancing is based on random selection of replicas and CPU is not consulted. There are limited ways to influence the replica selection, see https://lucene.apache.org/solr/guide/8_4/distributed-requests.html#shards-preference-parameter If a replica fails then the query fails and an error is returned. I think (but I am not sure) that SolrJ retries the request on some specific errors in which case a different replica may be selected and the request may succeed. IMO, these are two weak areas of Solr right now. Suggestions/patches are welcome :-) On 12/11/20, Dominique Bejean wrote: > Hi, > > Is there in Solrcloud any load balancing based on CPU load on Solr nodes ? > > If for shard a replica fails to handle a query, the query is sent to > another replica in order to be completed ? > > Regards > > Dominique > -- Regards, Shalin Shekhar Mangar.
Re: solrcloud with EKS kubernetes
FWIW, I have seen Solr exhaust the IOPS burst quota on AWS causing slow replication and high latency for search and indexing operations. You may want to dig into cloud watch metrics and see if you are running into a similar issue. The default IOPS quota on gp2 is very low (100?). Another thing to check is whether you have DNS TTLs for both positive and negative lookups configured. When nodes go down and come back up in Kubernetes the address of the pod remains the same but the IP can change and the JVM caches DNS lookups. This can cause timeouts. On 12/14/20, Abhishek Mishra wrote: > Hi Houston, > Sorry for the late reply. Each shard has a 9GB size around. > Yeah, we are providing enough resources to pods. We are currently > using c5.4xlarge. > XMS and XMX is 16GB. The machine is having 32 GB and 16 core. > No, I haven't run it outside Kubernetes. But I do have colleagues who did > the same on 7.2 and didn't face any issue regarding it. > Storage volume is gp2 50GB. > It's not the search query where we are facing inconsistencies or timeouts. > Seems some internal admin APIs sometimes have issues. So while adding new > replica in clusters sometimes result in inconsistencies. Like recovery > takes some time more than one hour. > > Regards, > Abhishek > > On Thu, Dec 10, 2020 at 10:23 AM Houston Putman > wrote: > >> Hello Abhishek, >> >> It's really hard to provide any advice without knowing any information >> about your setup/usage. >> >> Are you giving your Solr pods enough resources on EKS? >> Have you run Solr in the same configuration outside of kubernetes in the >> past without timeouts? >> What type of storage volumes are you using to store your data? >> Are you using headless services to connect your Solr Nodes, or ingresses? >> >> If this is the first time that you are using this data + Solr >> configuration, maybe it's just that your data within Solr isn't optimized >> for the type of queries that you are doing. >> If you have run it successfully in the past outside of Kubernetes, then I >> would look at the resources that you are giving your pods and the storage >> volumes that you are using. >> If you are using Ingresses, that might be causing slow connections >> between >> nodes, or between your client and Solr. >> >> - Houston >> >> On Wed, Dec 9, 2020 at 3:24 PM Abhishek Mishra >> wrote: >> >> > Hello guys, >> > We are kind of facing some of the issues(Like timeout etc.) which are >> very >> > inconsistent. By any chance can it be related to EKS? We are using solr >> 7.7 >> > and zookeeper 3.4.13. Should we move to ECS? >> > >> > Regards, >> > Abhishek >> > >> > -- Regards, Shalin Shekhar Mangar.
Re: CDCR
If you manually issue a commit operation on the remote clusters, do you see any updates? If yes, then you should set autoCommit on the remote clusters. If no, then check the logs on the cluster which is receiving the indexing operations and see if there are any errors. On Wed, Nov 25, 2020 at 6:11 PM Gell-Holleron, Daniel < daniel.gell-holle...@gb.unisys.com> wrote: > Hello, > > Does anybody have advice on why CDCR would say its Forwarding updates > (with no errors) even though the solr servers its replicating to aren't > updating? > > We have just under 50 million documents, that are spread across 4 servers. > Each server has a node each. > > One side is updating happily so would think that sharding wouldn't be > needed at this point? > > We are using Solr version 7.7.1. > > Thanks, > > Daniel > > -- Regards, Shalin Shekhar Mangar.
Re: _version_ taking too much memory
You should change the _version_ field to be of type docValues and re-index data. This way you don't have to pay the memory cost of un-inversion. On Wed, Nov 18, 2020 at 9:51 AM sanjay dutt wrote: > Solr Collection setup > Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too > much)Heap Space :- 12GB > So basically, above collection is having OutOfMemory issues. And upon > inspection, I got to know > that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field > "_version_" is taking around 50 - 60% memory(5GB). > Now I am trying to understand How "_version_" internally using the field > cache? > Around the same time, In Splunk logs I can see "Found MAX value from Terms > for _version_ in index". and also few "firstSearcher" queries. Now I think > upon core initialization "firstSearcher" triggered which then try to find > the Max value for version using Field Cache but because of too many records > it occupy lot of memory to carry out the task. > Can someone please validate Is what I am talking in above para is the > problem or problem lies in somewhere else? > > Best Regards,Sanjay > -- Regards, Shalin Shekhar Mangar.
Re: Massively unbalanced CPU by different SOLR Nodes
Good to hear that. Thanks for closing the loop! On Tue, Oct 27, 2020 at 11:14 AM Jonathan Tan wrote: > Hi Shalin, > > Moving to 8.6.3 fixed it! > > Thank you very much for that. :) > We'd considered an upgrade - just because - but we won't have done so so > quickly without your information. > > Cheers > > On Sat, Oct 24, 2020 at 11:37 PM Shalin Shekhar Mangar < > shalinman...@gmail.com> wrote: > > > Hi Jonathan, > > > > Are you using the "shards.preference" parameter by any chance? There is a > > bug that causes uneven request distribution during fan-out. Can you check > > the number of requests using the /admin/metrics API? Look for the /select > > handler's distrib and local request times for each core in the node. > > Compare those across different nodes. > > > > The bug I refer to is https://issues.apache.org/jira/browse/SOLR-14471 > and > > it is fixed in Solr 8.5.2 > > > > On Fri, Oct 23, 2020 at 9:05 AM Jonathan Tan wrote: > > > > > Hi, > > > > > > We've got a 3 node SolrCloud cluster running on GKE, each on their own > > > kube node (which is in itself, relatively empty of other things). > > > > > > Our collection has ~18m documents of 36gb in size, split into 6 shards > > > with 2 replicas each, and they are evenly distributed across the 3 > nodes. > > > Our JVMs are currently sized to ~14gb min & max , and they are running > on > > > SSDs. > > > > > > > > > [image: Screen Shot 2020-10-23 at 2.15.48 pm.png] > > > > > > Graph also available here: https://pasteboard.co/JwUQ98M.png > > > > > > Under perf testing of ~30 requests per second, we start seeing really > bad > > > response times (around 3s in the 90th percentile, and *one* of the > nodes > > > would be fully maxed out on CPU. At about 15 requests per second, our > > > response times are reasonable enough for our purposes (~0.8-1.1s), but > as > > > is visible in the graph, it's definitely *not* an even distribution of > > the > > > CPU load. One of the nodes is running at around 13cores, whilst the > > other 2 > > > are running at ~8cores and 6 cores respectively. > > > > > > We've tracked in our monitoring tools that the 3 nodes *are* getting an > > > even distribution of requests, and we're using a Kube service which is > in > > > itself a fairly well known tool for load balancing pods. We've also > used > > > kube services heaps for load balancing of other apps and haven't seen > > such > > > a problem, so we doubt it's the load balancer that is the problem. > > > > > > All 3 nodes are built from the same kubernetes statefulset deployment > so > > > they'd all have the same configuration & setup. Additionally, over the > > > course of the day, it may suddenly change so that an entirely different > > > node is the one that is majorly overloaded on CPU. > > > > > > All this is happening only under queries, and we are doing no indexing > at > > > that time. > > > > > > We'd initially thought it might be the overseer that is being majorly > > > overloaded when under queries (although we were surprised) until we did > > > more testing and found that even the nodes that weren't overseer would > > > sometimes have that disparity. We'd also tried using the `ADDROLE` API > to > > > force an overseer change in the middle of a test, and whilst the tree > > > updated to show that the overseer had changed, it made no difference to > > the > > > highest CPU load. > > > > > > Directing queries directly to the non-busy nodes do actually give us > back > > > decent response times. > > > > > > We're quite puzzled by this and would really like some help figuring > out > > > *why* the CPU on one is so much higher. I did try to get the jaeger > > tracing > > > working (we already have jaeger in our cluster), but we just kept > getting > > > errors on startup with solr not being able to load the main function... > > > > > > > > > Thank you in advance! > > > Cheers > > > Jonathan > > > > > > > > > > > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > -- Regards, Shalin Shekhar Mangar.
Re: Massively unbalanced CPU by different SOLR Nodes
Hi Jonathan, Are you using the "shards.preference" parameter by any chance? There is a bug that causes uneven request distribution during fan-out. Can you check the number of requests using the /admin/metrics API? Look for the /select handler's distrib and local request times for each core in the node. Compare those across different nodes. The bug I refer to is https://issues.apache.org/jira/browse/SOLR-14471 and it is fixed in Solr 8.5.2 On Fri, Oct 23, 2020 at 9:05 AM Jonathan Tan wrote: > Hi, > > We've got a 3 node SolrCloud cluster running on GKE, each on their own > kube node (which is in itself, relatively empty of other things). > > Our collection has ~18m documents of 36gb in size, split into 6 shards > with 2 replicas each, and they are evenly distributed across the 3 nodes. > Our JVMs are currently sized to ~14gb min & max , and they are running on > SSDs. > > > [image: Screen Shot 2020-10-23 at 2.15.48 pm.png] > > Graph also available here: https://pasteboard.co/JwUQ98M.png > > Under perf testing of ~30 requests per second, we start seeing really bad > response times (around 3s in the 90th percentile, and *one* of the nodes > would be fully maxed out on CPU. At about 15 requests per second, our > response times are reasonable enough for our purposes (~0.8-1.1s), but as > is visible in the graph, it's definitely *not* an even distribution of the > CPU load. One of the nodes is running at around 13cores, whilst the other 2 > are running at ~8cores and 6 cores respectively. > > We've tracked in our monitoring tools that the 3 nodes *are* getting an > even distribution of requests, and we're using a Kube service which is in > itself a fairly well known tool for load balancing pods. We've also used > kube services heaps for load balancing of other apps and haven't seen such > a problem, so we doubt it's the load balancer that is the problem. > > All 3 nodes are built from the same kubernetes statefulset deployment so > they'd all have the same configuration & setup. Additionally, over the > course of the day, it may suddenly change so that an entirely different > node is the one that is majorly overloaded on CPU. > > All this is happening only under queries, and we are doing no indexing at > that time. > > We'd initially thought it might be the overseer that is being majorly > overloaded when under queries (although we were surprised) until we did > more testing and found that even the nodes that weren't overseer would > sometimes have that disparity. We'd also tried using the `ADDROLE` API to > force an overseer change in the middle of a test, and whilst the tree > updated to show that the overseer had changed, it made no difference to the > highest CPU load. > > Directing queries directly to the non-busy nodes do actually give us back > decent response times. > > We're quite puzzled by this and would really like some help figuring out > *why* the CPU on one is so much higher. I did try to get the jaeger tracing > working (we already have jaeger in our cluster), but we just kept getting > errors on startup with solr not being able to load the main function... > > > Thank you in advance! > Cheers > Jonathan > > > > -- Regards, Shalin Shekhar Mangar.
Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)
Perhaps this bug in the kernel? https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64 Can you check if you are running an affected kernel? On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani < daniel.oliveira.mantov...@gmail.com> wrote: > I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs. > > $ java -version > java version "1.8.0_231" > Java(TM) SE Runtime Environment (build 1.8.0_231-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode > > $ sudo strace -f -c -p 5972 > % time seconds usecs/call callserrors syscall > -- --- --- - - > 71.02 33.329124 662 50335 13317 futex > 21.43 10.055086 3351703028 restart_syscall > 7.493.512846 14760 238 epoll_wait > 0.070.030526254412 poll > 0.000.001571 5 322 write > 0.000.000439 4 117 times > 0.000.000355 2 183 4 read > 0.000.000236 372 mprotect > 0.000.000161 1 118 epoll_ctl > 0.000.000126 0 352 stat > 0.000.89 11 8 getdents > 0.000.77 10 8 sendto > 0.000.69 69 1 statfs > 0.000.43 22 2 writev > 0.000.29 4 7 openat > 0.000.21 4 6 fstat > 0.000.18 122 lstat > 0.000.18 9 2 dup > 0.000.10 1 9 close > 0.000.10 3 4 fcntl > 0.000.05 3 2 access > 0.000.02 2 1 mmap > 0.000.02 026 sched_yield > 0.000.01 0 8 recvfrom > 0.000.01 1 2 readlink > -- --- --- - - > 100.00 46.930865 51887 13349 total > > $ sudo strace -f -p 5972 > strace: Process 5972 attached with 86 threads > [pid 7524] restart_syscall(<... resuming interrupted futex ...> > > [pid 7523] restart_syscall(<... resuming interrupted futex ...> > > [pid 7511] epoll_wait(142, > [pid 7471] epoll_wait(160, > [pid 7266] restart_syscall(<... resuming interrupted futex ...> > > [pid 7267] epoll_wait(151, > [pid 7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 7265] epoll_wait(157, > [pid 6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6281] epoll_wait(154, > [pid 6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6277] accept(136, > [pid 6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6271] epoll_wait(208, > [pid 6270] epoll_wait(205, > [pid 6262] epoll_wait(199, > [pid 6261] epoll_wait(196, > [pid 6258] epoll_wait(193, > [pid 6269] epoll_wait(202, > [pid 6257] epoll_wait(190, > [pid 6256] epoll_wait(187, > [pid 6255] epoll_wait(184, > [pid 6253] epoll_wait(178, > [pid 6250] restart_syscall(<... resuming interrupted futex ...> > > [pid 6252] epoll_wait(175, > [pid 6251] epoll_wait(172, > [pid 6254] epoll_wait(181, > [pid 6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6238] restart_syscall(<... resuming interrupted futex ...> > > [pid 6237] restart_syscall(<... resuming interrupted futex ...> > > [pid 6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6234] restart_syscall(<... resuming interrupted futex ...> > > [pid 6233] restart_syscall(<... resuming interrupted futex ...> > > [pid 6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6230] epoll_wait(134, > [pid 6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL ...> > [pid 6047] epoll_wait(130, > [pid 6046] restart_syscall(<... resuming interrupted futex ...> > > [pid 6045] restart_syscall(<... resuming interrupted futex ...> > > > JVM args: > > >- >
Re: Can replicas retry for a longer time?
Assuming this is SolrCloud, the replicas retry upto 500 times and wait an exponentially increasing time between retries starting from 2 seconds and upto a minute. Did that not happen in your case? Did the JVM had DNS ttls set? On Sun, Apr 28, 2019 at 7:51 PM Walter Underwood wrote: > Hello! Any response on this? > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Apr 11, 2019, at 7:46 PM, Walter Underwood > wrote: > > > > We had a bad situation with our prod cluster. There was a DNS failure in > AWS and all the replicas went “brown”. Only the leaders were taking traffic. > > > > If the replicas had continued to attempt recovery every five minutes or > so, they would have come back online automatically. > > > > Is there a way to configure that? > > > > wunder > > Walter Underwood > > wun...@wunderwood.org > > http://observer.wunderwood.org/ (my blog) > > > > -- Regards, Shalin Shekhar Mangar.
Re: Solr 7 not removing a node completely due to too small thread pool
Thanks Roger. This was reported earlier but missed our attention.
The issue is https://issues.apache.org/jira/browse/SOLR-11208
On Tue, Apr 2, 2019 at 5:56 PM Roger Lehmann
wrote:
> To be more specific: I currently have 19 collections, where each node has
> exactly one replica per collection. A new node will automatically create
> new replicas on itself, one for each existing collection (see
> cluster-policy above).
> So when removing a node, all 19 collection replicas of it need to be
> removed. This can't be done in one go due to thread count (parallel
> synchronous execution) being only 10 and is not scaling up when necessary.
>
> On Fri, 29 Mar 2019 at 14:20, Roger Lehmann
> wrote:
>
> > Situation
> >
> > I'm currently trying to set up SolrCloud in an AWS Autoscaling Group, so
> > that it can scale dynamically.
> >
> > I've also added the following triggers to Solr, so that each node will
> > have 1 (and only one) replication of each collection:
> >
> > {
> > "set-cluster-policy": [
> > {"replica": "<2", "shard": "#EACH", "node": "#EACH"}
> > ],
> > "set-trigger": [{
> > "name": "node_added_trigger",
> > "event": "nodeAdded",
> > "waitFor": "5s",
> > "preferredOperation": "ADDREPLICA"
> > },{
> > "name": "node_lost_trigger",
> > "event": "nodeLost",
> > "waitFor": "120s",
> > "preferredOperation": "DELETENODE"
> > }]
> > }
> >
> > This works pretty well. But my problem is that when the a node gets
> > removed, it doesn't remove all 19 replicas from this node and I have
> > problems when accessing the "nodes" page:
> >
> > [image: enter image description here]
> > <https://i.stack.imgur.com/QyJrY.png>
> >
> > In the logs, this exception occurs:
> >
> > Operation deletenode
> failed:java.util.concurrent.RejectedExecutionException: Task
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1104948431@467049e2
> rejected from
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@773563df[Running,
> pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 1]
> > at
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> > at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> > at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> > at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194)
> > at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
> > at
> org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteCore(DeleteReplicaCmd.java:276)
> > at
> org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:95)
> > at
> org.apache.solr.cloud.api.collections.DeleteNodeCmd.cleanupReplicas(DeleteNodeCmd.java:109)
> > at
> org.apache.solr.cloud.api.collections.DeleteNodeCmd.call(DeleteNodeCmd.java:62)
> > at
> org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:292)
> > at
> org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:496)
> > at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
> > Problem description
> >
> > So, the problem is that it only has a pool size of 10, of which 10 are
> > busy and nothing gets queued (synchronous execution). In fact, it really
> > only removed 10 replicas and the other 9 replicas stayed there. When
> > manually sending the API command to delete this node it works fine, since
> > Solr only needs to remove the remaining 9 replicas and everything is good
> > again.
> > Question
> >
> > How can I either increase this (small) thread pool size and/or activate
> > queueing the remaining deletion tasks? Another solution might be to retry
> > the failed task until it succeeds.
> >
> > Using Solr 7.7.1 on Ubuntu Server installed with the installation script
> > from Solr (so I guess it's using Jetty?).
> >
> > Thanks for your help!
> >
>
--
Regards,
Shalin Shekhar Mangar.
Re: Solr 7.5 DeleteShard not working when all cores are down
What Shawn said.
DeleteShard API is supposed to be used either when using implicit routing
or when you have compositeId router but the shard has already been split
and therefore in an inactive state.
Delete Replica API is what you need if you want to delete an individual
replica.
On Fri, Mar 15, 2019 at 12:39 AM Shawn Heisey wrote:
> On 3/14/2019 12:47 PM, Aroop Ganguly wrote:
> > I am trying to delete a shard from a collection using the collections
> > api for the same.
> > On the solr ui, all the replicas are in “downed” state.
> >
> > However, when I run the delete shard
> >
> command: /solr/admin/collections?action=DELETESHARD=x=shard84
> > I get this exception:
> > {
> >"responseHeader":{
> > "status":400,
> > "QTime":14},
> >"Operation deleteshard caused
> >
> exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
>
> > The slice: shard35 is currently active. Only non-active
> > (or custom-hashed) slices can be deleted.",
>
>
>
> > Why is this api thinking this slice is active ? When the Solr UI shows
> > all replicas down ?
>
> Active means the shard is considered part of the whole collection --
> included when you run a query, etc.
>
> Even though all replicas are down, the shard is still an active part of
> the index. So you can't delete it.
>
> If your collection is typical and has compositeId routing, deleting a
> shard is really only possible after you have run SPLITSHARD and then you
> will only be able to delete the original shard that gets split.
>
> Aside from SPLITSHARD, I really have no idea how to mark a shard as
> inactive, but that will be required before you can delete it.
>
> Thanks,
> Shawn
>
--
Regards,
Shalin Shekhar Mangar.
Re: Solr cloud - poweroff procedure
In case you are using a recent Solr 7.x version with collections that have autoAddReplicas=true, you should disable the auto add replicas feature before powering off so that Solr does not decide to move replicas around because nodes have been lost. See https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-auto-add-replicas.html#using-cluster-property-to-enable-autoaddreplicas On Wed, Oct 31, 2018 at 3:27 AM lstusr 5u93n4 wrote: > Hi All, > > We have a solr cloud running 3 shards, 3 hosts, 6 total NRT replicas, and > the data director on hdfs. It has 950 million documents in the index, > occupying 700GB of disk space. > > We need to completely power off the system to move it. > > Are there any actions we should take on shutdown to help the process? > Anyhing we should expect on power on? > > Thanks, > > Kyle > -- Regards, Shalin Shekhar Mangar.
Re: SolrCloud scaling/optimization for high request rate
What does your cache statistics look like? What's the hit ratio, size, evictions etc? More comments inline: On Sat, Oct 27, 2018 at 8:23 AM Erick Erickson wrote: > Sofiya: > > I haven't said so before, but it's a great pleasure to work with > someone who's done a lot of homework before pinging the list. The only > unfortunate bit is that it usually means the simple "Oh, I can fix > that without thinking about it much" doesn't work ;) > > 2. I'll clarify a bit here. Any TLOG replica can become the leader. > Here's the process for an update: > > doc comes in to the leader (may be TLOG) > > doc is forwarded to all TLOG replicas, _but it is not indexed there_. > > If the leader fails, the other TLOG replicas have enough documents in > _their_ tlogs to "catch up" and one is elected > > You're totally right that PULL replicas cannot become leaders > > having all TLOG replicas means that the CPU cycles otherwise consumed by > indexing are available for query processing. > > The point here is that TLOG replicas don't need to expend CPU cycles > to index documents, freeing up all those cycles for serving queries. > > Now, that said you report that QPS rate doesn't particularly seem to > be affected by whether you're indexing or not, so that makes using > TLOG and PULL replicas less likely to solve your problem. I was > thinking about your statement that you index as fast as possible > > > 6. This is a little surprising. Here's my guess: You're indexing in > large batches and the batch is only really occupying a thread or two > so it's effectively serialized thus not consuming a huge amount of > resources. > The CloudSolrClient parallelizes updates to each shard leader. But in this case, there is only 1 shard so all updates are serialized. All indexing activity is therefore being performed by a single CPU at a time. > > So unless G1 really solves a lot of problems, more replicas are > indicated. On machines with large amounts of RAM and lots of CPUs, one > other option is to run multiple JVMs per physical node that's > sometimes helpful. > > One other possibility. In Solr 7.5, you have a ton of metrics > available. If you hit the admin/metrics end point you'll see 150-200 > available metrics. Apart from running a profiler to see what's > consuming the most cycles, the metrics can give you a view into what > Solr is doing and may help you pinpoint what's using the most cycles. > > Best, > Erick > On Fri, Oct 26, 2018 at 12:23 PM Toke Eskildsen wrote: > > > > David Hastings wrote: > > > Would adding the docValues in the schema, but not reindexing, cause > > > errors? IE, only apply the doc values after the next reindex, but in > the > > > meantime keep functioning as there were none until then? > > > > As soon as you specify in the schema that a field has docValues=true, > Solr treats all existing documents as having docValues enabled for that > field. As there is no docValue content, DocValues-aware functionality such > as sorting and faceting will not work for that field, until the documents > has been re-indexed. > > > > - Toke Eskildsen > -- Regards, Shalin Shekhar Mangar.
Re: AW: AW: 6.6 -> 7.5 SolrJ, seeing many "Connection evictor"-Threads
You can expect as many connection evictor threads as the number of http client instances. This is true for both Solr 6.6 and 7.x. I was intrigued as to why you were not seeing the same threads in both versions. It turns out that I made a mistake in the patch I committed in SOLR-9290 where instead of using Solr's DefaultSolrThreadFactory which names threads with a proper prefix, I used Java's DefaultThreadFactory which names threads like pool-123-thread-1282. So if you take a thread dump from Solr 6.6, you should be able to find threads named like these which are sleeping at a similar place in the stack. On Tue, Oct 23, 2018 at 9:14 AM Clemens Wyss DEV wrote: > On 10/22/2018 6:15 AM, Shawn Heisey wrote: > > autoSoftCommit is pretty aggressive . If your commits are taking 1-2 > seconds or les > well, some take minutes (re-index)! > > > autoCommit is quite long. I'd probably go with 60 seconds > Which means every 1min the "pending"/"soft" commits are effectively saved? > > One additional question: having auto(soft)commits in place, do I at all > need to explicitly commit UpdateRequest from SolrJ? > > > added in 5.5.3 and 6.2.0 by this issue > hmmm, I have never seen these threads before, not even in 6.6 > > > Shalin worked on that issue, maybe they can shed some light on it and > >indicate whether there should be many threads running that code > I'd appreciate > > Yet again, many thanks. > - Clemens > > -- Regards, Shalin Shekhar Mangar.
Re: 7.3 to 7.5
Upgrade between minor releases does not require a re-index but please read the upgrade notes for 7.4 and 7.5 releases in the CHANGES.txt in case there are configuration changes you need to make. On Fri, Oct 19, 2018 at 12:33 AM Dave wrote: > Would a minor solr upgrade such as this require a reindexing in order to > take advantage of the skg functionality, or would it work regardless? A > full reindex is quite a large operation in my use case > -- Regards, Shalin Shekhar Mangar.
Re: Something odd with async request status for BACKUP operation on Collections API
The responses are collected by node so subsequent responses from the same node overwrite previous responses. Definitely a bug. Please open an issue. On Mon, Oct 15, 2018 at 6:24 AM Shawn Heisey wrote: > On 10/14/2018 6:25 PM, dami...@gmail.com wrote: > > I had an issue with async backup on solr 6.5.1 reporting that the backup > > was complete when clearly it was not. I was using 12 shards across 6 > nodes. > > I only noticed this issue when one shard was much larger than the others. > > There were no answers here > > http://lucene.472066.n3.nabble.com/async-backup-td4342776.html > > One detail I thought I had written but isn't there: The backup did > fully complete -- all 30 shards were in the backup location. Not a lot > in each shard backup -- the collection was empty. It would be easy > enough to add a few thousand documents to the collection before doing > the backup. > > If the backup process reports that it's done before it's ACTUALLY done, > that's a bad thing. It's hard to say whether that problem is related to > the problem I described. Since I haven't dived into the code, I cannot > say for sure, but it honestly would not surprise me to find they are > connected. Every time I try to understand Collections API code, I find > it extremely difficult to follow. > > I'm sorry that you never got resolution on your problem. Do you know > whether that is still a problem in 7.x? Setting up a reproduction where > one shard is significantly larger than the others will take a little bit > of work. > > > I was focusing on the STATUS returned from the REQUESTSTATUS command, but > > looking again now I can see a response from only 6 shards, and each shard > > is from a different node. So this fits with what you're seeing. I assume > > your shards 1, 7, 9 are all on different nodes. > > I did not actually check, and the cloud example I was using isn't around > any more, but each of the shards in the status response were PROBABLY on > separate nodes. The cloud example was 3 nodes. It's an easy enough > scenario to replicate, and I provided enough details for anyone to do it. > > The person on IRC that reported this problem had a cluster of 15 nodes, > and the status response had ten shards (out of 30) mentioned. It was > shards 1-9 and shard 20. The suspicion is that there's something > hard-coded that limits it to 10 responses ... because without that, I > would expect the number of shards in the response to match the number of > nodes. > > Thanks, > Shawn > > -- Regards, Shalin Shekhar Mangar.
Re: Autoscaling with core properties
Hi Jim,
Very interesting scenario that we didn't anticipate. I think this is a
limitation of the MoveReplica API which does not move core properties.
But it also raises questions such as whether to always move all core
properties? I personally think core properties are an artifact that was
suitable for non-cloud Solr but does not lend well to cloud environments.
We also have replica properties in Solr and corresponding APIs to
add/update/delete them. See
https://lucene.apache.org/solr/guide/7_4/collections-api.html#addreplicaprop
(however, I think even these are not carried forward by move replica
today). Do you mind opening a Jira to discuss how we can fix the current
behavior?
Would using the Config API to set user properties work for your use-case?
See
https://lucene.apache.org/solr/guide/7_4/configuring-solrconfig-xml.html#substituting-properties-in-solr-config-files
and
https://lucene.apache.org/solr/guide/7_4/config-api.html#commands-for-user-defined-properties
We can improve autoscaling actions such as ComputePlanAction to add custom
core properties to any add replica or move replica command. That is
probably worth another Jira as well.
On Wed, Sep 5, 2018 at 11:54 PM James Strassburg
wrote:
> Hello,
>
> We're creating a SolrCloud in AWS and attempting to use autoscaling to add
> replicas during nodeAdded/nodeLost. This was working fine for us until we
> started creating collections specifying core properties (e.g.
>
> /solr/admin/collections?action=CREATE_datasource=REDACTED).
> It seems that when the nodeLost/Added trigger fires the properties don't
> manifest in the core create invocation and we get errors like the
> following:
>
> products_20180904200015_shard1_replica_n39:
> org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> Could not load conf for core products_20180904200015_shard1_replica_n39:
> Can't load schema schema.xml: No system property or default value specified
> for synonyms_datasource value:jdbc/${synonyms_datasource}
>
> The autoscaling API also doesn't appear to have a way to set the properties
> when we create the triggers.
>
> Are we missing something or is this not supported at this time? I couldn't
> find a relevant JIRA or other documentation or solr-user discussion on
> this.
>
> thanks,
>
> JiM
>
--
Regards,
Shalin Shekhar Mangar.
Re: node sort order and policy
The "sortedNodes" shows the nodes in the system from the most loaded to
least loaded order. All other things being equal, a node having the least
free disk space will come first and the last node will be the one with the
most available disk space.
On Wed, Sep 5, 2018 at 3:01 AM Kudrettin Güleryüz
wrote:
> Given exhibit A which shows the current sort order on a 7.3.1 cloud
> installation, why would the sort order be from less freedisk to more
> freedisk? Shouldn't it be the reverse since the number of cores is the same
> for all nodes? preferences and policies are listed in exhibit B.
>
> Thank you,
>
>
> Exhibit A: (autoscaling/diagnostics)
> {
>
> "responseHeader":{
> "status":0,
> "QTime":281},
> "diagnostics":{
> "sortedNodes":[{
> "node":"test-46:8983_solr",
> "cores":138,
> "freedisk":289.32603454589844,
> "sysLoadAvg":88.0},
> {
> "node":"test-44:8983_solr",
> "cores":138,
> "freedisk":303.9487609863281,
> "sysLoadAvg":122.0},
> {
> "node":"test-45:8983_solr",
> "cores":138,
> "freedisk":323.1516571044922,
> "sysLoadAvg":166.0},
> {
> "node":"test-43:8983_solr",
> "cores":138,
> "freedisk":325.2588005065918,
> "sysLoadAvg":107.0},
> {
> "node":"test-41:8983_solr",
> "cores":138,
> "freedisk":334.3892288208008,
> "sysLoadAvg":56.99},
> {
> "node":"test-42:8983_solr",
> "cores":138,
> "freedisk":360.2049674987793,
> "sysLoadAvg":98.0}],
> "violations":[]},
> "WARNING":"This response format is experimental. It is likely to
> change in the future."}
>
>
> Exhibit B: (admin/autoscaling)
>
> "cluster-preferences":[{
> "minimize":"cores",
> "precision":1}
> ,{
> "maximize":"freedisk",
> "precision":10}
> ,{
> "minimize":"sysLoadAvg",
> "precision":3}],
> "cluster-policy":[{
> "replica":"<2",
> "node":"#ANY"}
> ,{
> "replica":"0",
> "freedisk":"<10",
> "strict":"true"}],
>
--
Regards,
Shalin Shekhar Mangar.
Re: Need Help on Solr Client connection Pooling
You should create a single HttpSolrClient and re-use for all requests. It is thread safe and creates an Http connection pool internally (well Apache HttpClient does). On Thu, Aug 30, 2018 at 2:28 PM Gembali Satish kumar < gembalisatishku...@gmail.com> wrote: > Hi Team, > > Need some help on Client connection object pooling > I am using SolrJ API to connect the Solr. > > This below snippet I used to create the client object. > > *SolrClient client = new HttpSolrClient.Builder(* > * SolrUtil.getSolrURL(tsConfigUtil.getClusterAdvertisedAddress(), > aInCollectionName)).build();* > > after my job search done, I am closing my client. > *client.close();* > > but from UI getting more requests to search the data > I think to create the *client object *on every request is costly is there > any way to pool the *SolrClient objects?*? > If there kindly share me the reference > > Thanks and Regards, > Satish > -- Regards, Shalin Shekhar Mangar.
Re: cloud disk space utilization
There is a bad oversight on our part which causes preferences to not be
used for placing replicas unless a cluster policy also exists. We hope to
fix it in the next release (Solr 7.5). See
https://issues.apache.org/jira/browse/SOLR-12648
You may also be interested in
https://issues.apache.org/jira/browse/SOLR-12592
On Tue, Aug 28, 2018 at 2:47 AM Kudrettin Güleryüz
wrote:
> Hi,
>
> We have six Solr nodes with ~1TiB disk space on each mounted as ext4. The
> indexers sometimes update the collections and create new ones if update
> wouldn't be faster than scratch indexing. (up to around 5 million documents
> are indexed for each collection) On average there are around 130
> collections on this SolrCloud. Collection sizes vary from 1GiB to 150GiB.
>
> Preferences set:
>
> "cluster-preferences":[{
> "maximize":"freedisk",
> "precision":10}
> ,{
> "minimize":"cores",
> "precision":1}
> ,{
> "minimize":"sysLoadAvg",
> "precision":3}],
>
> * Is it be possible to run out of disk space on one of the nodes while
> others would have plenty? I observe some are getting close to ~80%
> utilization while others stay at ~60%
> * Would this difference be due to collection index size differences or due
> to error on my side to come up with a useful policy/preferences?
>
> Thank you
>
--
Regards,
Shalin Shekhar Mangar.
Re: Handshake for NRT?
There are a single persistent HTTP connection open from the leader to each replica in the shard. All updates coming to the leader are expanded (for atomic updates) and streamed over that single connection. When using in-place docvalues updates, there is a possibility of the replica making a request to the leader if updates has been re-ordered and the replica does not have enough context to process the update. Can you quantify the "tons of internal traffic"? Are you seeing higher number of open connections as well? On Fri, Aug 17, 2018 at 11:17 PM Walter Underwood wrote: > How many messages are sent back and forth between a leader and replica > with NRT? > > We have a collection that gets frequent updates and we are seeing a ton of > internal > cluster traffic. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > -- Regards, Shalin Shekhar Mangar.
Re: Solr fails even ZK quorum has majority
gt;> > org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1441) > > >> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> > > >> at > > >> sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java: > > >> 62) > > >> > > >> at > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke( > > >> DelegatingMethodAccessorImpl.java:43) > > >> > > >> at java.lang.reflect.Method.invoke(Method.java:498) > > >> > > >> at org.eclipse.jetty.start.Main.invokeMain(Main.java:214) > > >> > > >> at org.eclipse.jetty.start.Main.start(Main.java:457) > > >> > > >> at org.eclipse.jetty.start.Main.main(Main.java:75) > > >> > > >> Caused by: org.apache.solr.common.SolrException: > > >> java.net.UnknownHostException: ditsearch001.es.com: Name or service > not > > >> known > > >> > > >> at > > >> > org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:171) > > >> > > >> at > > >> > org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:117) > > >> > > >> at > > >> > org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:112) > > >> > > >> at > > >> org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:99) > > >> > > >> at > > >> org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig( > > >> SolrDispatchFilter.java:263) > > >> > > >> ... 44 more > > >> > > >> Caused by: java.net.UnknownHostException: ditsearch001.es.com: Name > or > > >> service not known > > >> > > >> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) > > >> > > >> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress. > > java:928) > > >> > > >> at > > >> > java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) > > >> > > >> at java.net.InetAddress.getAllByName0(InetAddress.java:1276) > > >> > > >> at java.net.InetAddress.getAllByName(InetAddress.java:1192) > > >> > > >> at java.net.InetAddress.getAllByName(InetAddress.java:1126) > > >> > > >> at > > >> org.apache.zookeeper.client.StaticHostProvider.( > > >> StaticHostProvider.java:61) > > >> > > >> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) > > >> > > >> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) > > >> > > >> at > > >> org.apache.solr.common.cloud.SolrZooKeeper.( > > SolrZooKeeper.java:43) > > >> > > >> at > > >> org.apache.solr.common.cloud.ZkClientConnectionStrategy. > > >> createSolrZooKeeper(ZkClientConnectionStrategy.java:105) > > >> > > >> at > > >> org.apache.solr.common.cloud.DefaultConnectionStrategy.connect( > > >> DefaultConnectionStrategy.java:37) > > >> > > >> at > > >> > org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:146) > > >> > > >> ... 48 more > > >> > > > -- Regards, Shalin Shekhar Mangar.
Re: AddReplica to shard with lowest node count
The rule based replica placement was deprecated. The autoscaling APIs are the way to go. Please see http://lucene.apache.org/solr/guide/7_3/solrcloud-autoscaling.html Your use-case is interesting. By default, the trigger for nodeAdded event will move replicas from the most loaded nodes to the new node. That does not take care of your use-case. Can you please open a Jira to add this feature? On Thu, Jul 5, 2018 at 6:45 AM Gus Heck wrote: > Perhaps the rule based replica placement stuff would do the trick? > > https://lucene.apache.org/solr/guide/7_3/rule-based-replica-placement.html > > I haven't used it myself but I've seen lots of work going into it lately... > > On Wed, Jul 4, 2018 at 12:35 PM, Duncan, Adam > wrote: > > > Hi all, > > > > Our team use Solrcloud for Solr 5.1 and are investigating an upgrade to > 7.3 > > Currently we have a working scale-up approach for adding a new server to > > the cluster beyond the initial collection creation. > > We’ve automated the install of Solr on new servers and, following that, > we > > register the new instance with zookeeper so that the server will be > > included in the list of live nodes. > > Finally we use the CoreAdmin API ‘Create’ command to associate the new > > node with our collection. Solr 5.1's CoreAdmin Create command would > > conveniently auto-assign the new node to the shard with the least nodes. > > > > In Solr 7.3, the CoreAdmin API documentation warns us not to use the > > Create command with SolrCloud. > > We tried 7.3’s CoreAdmin API Create command regardless and, > > unsurprisingly, it did not work. > > The 7.3 documentation suggests we use the Collections API AddReplica > > command.The problem with AddReplica is that it expects us to specify the > > shard name. > > This is unfortunate as it makes it hard for us to keep shards balanced. > It > > puts the onus on us to work out the least populated shard via a call to > the > > cluster status endpoint. > > With that we now face the problem managing this correctly when scaling up > > multiple servers at once. > > > > Are we missing something here? Is there really no way for a node to be > > auto-assigned to a shard in 7.3? > > And if so, are there any recommendations for an approach to reliably > doing > > this ourselves? > > > > Thanks! > > Adam > > > > > > -- > http://www.the111shift.com > -- Regards, Shalin Shekhar Mangar.
Re: Querying in Solrcloud
You send your queries and updates directly to Solr's collection e.g. http://host:port/solr/. You can use any Solr node for this request. If the node does not have the collection being queried then the request will be forwarded internally to a Solr instance which has that collection. ZooKeeper is used by Solr's Java client to look up the list of Solr nodes having the collection being queried. But if you are using PHP then you can probably keep a list of Solr nodes in configuration and randomly choose one. A better implementation would be to setup a load balancer and put all Solr nodes behind it and query the load balancer URL in your application. On Sat, Jun 30, 2018 at 7:31 AM Sushant Vengurlekar < svengurle...@curvolabs.com> wrote: > I have a question regarding querying in solrcloud. > > I am working on php code to query solrcloud for search results. Do I send > the query to zookeeper or send it to a particular solr node? How does the > querying process work in general. > > Thank you > -- Regards, Shalin Shekhar Mangar.
Re: Autoscaling and inactive shards
Yes, I believe Noble is working on this. See
https://issues.apache.org/jira/browse/SOLR-11985
On Wed, Jun 13, 2018 at 1:35 PM Jan Høydahl wrote:
> Ok, get the meaning of preferences.
>
> Would there be a way to write a generic rule that would suggest moving
> shards to obtain balance, without specifying absolute core counts? I.e. if
> you have three nodes
> A: 3 cores
> B: 5 cores
> C: 3 cores
>
> Then that rule would suggest two moves to end up with 4 cores on all three
> (unless that would violate disk space or load limits)?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 12. jun. 2018 kl. 08:10 skrev Shalin Shekhar Mangar <
> shalinman...@gmail.com>:
> >
> > Hi Jan,
> >
> > Comments inline:
> >
> > On Tue, Jun 12, 2018 at 2:19 AM Jan Høydahl <mailto:jan@cominvent.com>> wrote:
> >
> >> Hi
> >>
> >> I'm trying to have Autoscaling move a shard to another node after
> manually
> >> splitting.
> >> We have two nodes, one has a shard1 and the other node is empty.
> >>
> >> After SPLITSHARD you have
> >>
> >> * shard1 (inactive)
> >> * shard1_0
> >> * shard1_1
> >>
> >> For autoscaling we have the {"minimize" : "cores"} cluster preference
> >> active. Because of that I'd expect that Autoscaling would suggest to
> move
> >> e.g. shard1_1 to the other (empty) node, but it doesn't. Then I create a
> >> rule just to test {"cores": "<2", "node": "#ANY"}, but still no
> >> suggestions. Not until I delete the inactive shard1, then it suggests to
> >> move one of the two remaining shards to the other node.
> >>
> >> So my two questions are
> >> 1. Is it by design that inactive shards "count" wrt #cores?
> >> I understand that it consumes disk but it is not active otherwise,
> >> so one could argue that it should not be counted in core/replica
> rules?
> >>
> >
> > Today, inactive slices also count towards the number of cores -- though
> > technically correct, it is probably an oversight.
> >
> >
> >> 2. Why is there no suggestion to move a shard due to the "minimize
> cores"
> >> reference itself?
> >>
> >
> > The /autoscaling/suggestions end point only suggests if there are policy
> > violations. Preferences such as minimize:cores are more of a sorting
> order
> > so they aren't really being violated. After you add the rule, the
> framework
> > still cannot give a suggestion that satisfies your rule. This is because
> > even if shard1_1 is moved to node2, node1 still has shard1 and shard1_0.
> So
> > the system ends up not suggesting anything. You should get a suggestion
> if
> > you add a third node to the cluster though.
> >
> > Also see SOLR-11997 <https://issues.apache.org/jira/browse/SOLR-11997 <
> https://issues.apache.org/jira/browse/SOLR-11997>> which
> > will tell users that a suggestion could not be returned because we cannot
> > satisfy the policy. There are a slew of other improvements to suggestions
> > planned that will return suggestions even when there are no policy
> > violations.
> >
> >
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com <http://www.cominvent.com/>
> >>
> >>
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Autoscaling and inactive shards
Hi Jan,
Comments inline:
On Tue, Jun 12, 2018 at 2:19 AM Jan Høydahl wrote:
> Hi
>
> I'm trying to have Autoscaling move a shard to another node after manually
> splitting.
> We have two nodes, one has a shard1 and the other node is empty.
>
> After SPLITSHARD you have
>
> * shard1 (inactive)
> * shard1_0
> * shard1_1
>
> For autoscaling we have the {"minimize" : "cores"} cluster preference
> active. Because of that I'd expect that Autoscaling would suggest to move
> e.g. shard1_1 to the other (empty) node, but it doesn't. Then I create a
> rule just to test {"cores": "<2", "node": "#ANY"}, but still no
> suggestions. Not until I delete the inactive shard1, then it suggests to
> move one of the two remaining shards to the other node.
>
> So my two questions are
> 1. Is it by design that inactive shards "count" wrt #cores?
>I understand that it consumes disk but it is not active otherwise,
>so one could argue that it should not be counted in core/replica rules?
>
Today, inactive slices also count towards the number of cores -- though
technically correct, it is probably an oversight.
> 2. Why is there no suggestion to move a shard due to the "minimize cores"
> reference itself?
>
The /autoscaling/suggestions end point only suggests if there are policy
violations. Preferences such as minimize:cores are more of a sorting order
so they aren't really being violated. After you add the rule, the framework
still cannot give a suggestion that satisfies your rule. This is because
even if shard1_1 is moved to node2, node1 still has shard1 and shard1_0. So
the system ends up not suggesting anything. You should get a suggestion if
you add a third node to the cluster though.
Also see SOLR-11997 <https://issues.apache.org/jira/browse/SOLR-11997> which
will tell users that a suggestion could not be returned because we cannot
satisfy the policy. There are a slew of other improvements to suggestions
planned that will return suggestions even when there are no policy
violations.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: SolrJ and autoscaling
Yes, we don't have Solrj support for changing autoscaling configuration today. It'd be nice to have for sure. Can you please file a Jira? Patches are welcome too! On Wed, Jun 6, 2018 at 8:33 PM, Hendrik Haddorp wrote: > Hi, > > I'm trying to read and modify the autoscaling config. The API on > https://lucene.apache.org/solr/guide/7_3/solrcloud-autoscaling-api.html > does only mention the REST API. The read part does however also work via > SolrJ: > > cloudSolrClient.getZkStateReader().getAutoScalingConfig() > > Just for the write part I could not find anything in the API. Is this > still a gap? > > regards, > Hendrik > -- Regards, Shalin Shekhar Mangar.
Re: Configuring aliases in ZooKeeper first
Thanks for the report. This sounds like a bug. At least on startup, we should refresh the configuration from ZK without looking at local config versions. Can you please open a Jira issue? On Wed, May 23, 2018 at 5:35 PM, Gael Jourdan-Weil < gael.jourdan-w...@kelkoogroup.com> wrote: > Hello everyone, > > We are running a SolrCloud cluster with ZooKeeper. > This SolrCloud cluster is down most of the time (backup environment) but > the ZooKeeper instances are always up so that we can easily update > configuration. > > This has been working fine for a long time with Solr 6.4.0 then 6.6.0, but > since upgrading to 7.2.1, we ran into an issue where Solr ignores > aliases.json stored in ZooKeeper. > > Steps to reproduce the problem: > 1/ SolrCloud cluster is down > 2/ Direct update of aliases.json file in ZooKeeper with Solr ZkCLI > *without using Collections API* : > java ... org.apache.solr.cloud.ZkCLI -zkhost ... -cmd clear /aliases.json > java ... org.apache.solr.cloud.ZkCLI -zkhost ... -cmd put /aliases.json > "new content" > 3/ SolrCloud cluster is started => aliases.json not taken into account > > Digging a bit in the code, what is actually causing the issue is that, > when starting, Solr now checks for the metadata of the aliases.json file > and if the version metadata from ZooKeeper is lower or equal to local > version, it keeps the local version. > When it starts, Solr has a local version of 0 for the aliases but > ZooKeeper also has a version of 0 of the file because we just recreated it. > So Solr ignores ZooKeeper configuration and never has a chance to load > aliases. > > Relevant parts of Solr code are: > - https://github.com/apache/lucene-solr/blob/branch_7_2/ > solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java : > line 4562 : method setIfNewer > - https://github.com/apache/lucene-solr/blob/branch_7_2/ > solr/solrj/src/java/org/apache/solr/common/cloud/Aliases.java : line 45 : > the "empty" Aliases object with default version 0 > > Obviously, a workaround is to force ZooKeeper to have a version greater > than 0 for aliases.json file (for instance by not clearing the file and > just overwriting it again and again). > > > But we were wondering, is this the intended behavior for Solr ? > > Thanks for reading, > > Gaël -- Regards, Shalin Shekhar Mangar.
Re: CDCR setup with Custom Document Routing
Setups using implicit routers are not considered in the design so I don't think they will work today. That being said, it should be a simple enhancement to the CdcrReplicator to add the shard name to the UpdateRequest object. But ensure that both target and source have the exact same number and name of shards. On Fri, May 18, 2018 at 11:48 PM, Atita Arora <atitaar...@gmail.com> wrote: > Hi, > > I am to setup the CDCR for a Solr Cluster which uses Custom Document > Routing. > Has anyone tried that before ? > Do we have any caveats to know well before ? > I will be setting up Uni Directional in Solr 7.3. > > Per documentation - > > > The current design works most robustly if both the Source and target > > clusters have the same number of shards. There is no requirement that the > > shards in the Source and target collection have the same number of > replicas. > > Having different numbers of shards on the Source and target cluster is > > possible, but is also an "expert" configuration as that option imposes > > certain constraints and is not recommended. Most of the scenarios where > > having differing numbers of shards are contemplated are better > accomplished > > by hosting multiple shards on each target Solr instance. > > > I am precisely little curious to know how would this fare if this isn't > followed. > Would highly appreciate any pointers around this. > > Sincerely, > Atita > -- Regards, Shalin Shekhar Mangar.
Re: SolrCloud replicaition
The min_rf parameter does not fail indexing. It only tells you how many replicas received the live update. So if the value is less than what you wanted then it is up to you to retry the update later. On Wed, May 2, 2018 at 3:33 PM, Greenhorn Techie <greenhorntec...@gmail.com> wrote: > Hi, > > Good Morning!! > > In the case of a SolrCloud setup with sharing and replication in place, > when a document is sent for indexing, what happens when only the shard > leader has indexed the document, but the replicas failed, for whatever > reason. Will the document be resent by the leader to the replica shards to > index the document after sometime or how is scenario addressed? > > Also, given the above context, when I set the value of min_rf parameter to > say 2, does that mean the calling application will be informed that the > indexing failed? > -- Regards, Shalin Shekhar Mangar.
Re: 7.3 appears to leak
Hi Markus, Can you give an idea of what your filter queries look like? Any custom plugins or things we should be aware of? Simple indexing artificial docs, querying and committing doesn't seem to reproduce the issue for me. On Thu, Apr 26, 2018 at 10:13 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hello, > > We just finished upgrading our three separate clusters from 7.2.1 to 7.3, > which went fine, except for our main text search collection, it appears to > leak memory on commit! > > After initial upgrade we saw the cluster slowly starting to run out of > memory within about an hour and a half. We increased heap in case 7.3 just > requires more of it, but the heap consumption graph is still growing on > each commit. Heap space cannot be reclaimed by forcing the garbage > collector to run, everything just piles up in the OldGen. Running with this > slightly larger heap, the first nodes will run out of memory in about two > and a half hours after cluster restart. > > The heap eating cluster is a 2shard/3replica system on separate nodes. > Each replica is about 50 GB in size and about 8.5 million documents. On > 7.2.1 it ran fine with just a 2 GB heap. With 7.3 and 2.5 GB heap, it will > take just a little longer for it to run out of memory. > > I inspected reports shown by the sampler of VisualVM and spotted one > peculiarity, the number of instances of SortedIntDocSet kept growing on > each commit by about the same amount as the number of cached filter > queries. But this doesn't happen on the logs cluster, SortedIntDocSet > instances are neatly collected there. The number of instances also accounts > for the number of commits since start up times the cache sizes > > Our other two clusters don't have this problem, one of them receives very > few commits per day, but the other receives data all the time, it logs user > interactions so a large amount of data is coming in all the time. I cannot > reproduce it locally by indexing data and committing all the time, the peak > usage in OldGen stays about the same. But, i can reproduce it locally when > i introduce queries, and filter queries while indexing pieces of data and > committing it. > > So, what is the problem? I dug in the CHANGES.txt of both Lucene and Solr, > but nothing really caught my attention. Does anyone here have an idea where > to look? > > Many thanks, > Markus > -- Regards, Shalin Shekhar Mangar.
Re: solr 5.4.1 - updates/inserts suddenly very slow. Search still fast
You can get a thread dump by calling http://localhost:8983/solr/admin/threads or by using the Admin UI. On Wed, Apr 18, 2018 at 9:11 PM, Felix XY <felix.xy.xy...@gmail.com> wrote: > Thank you Emir, but I'm not able to make a thread dump while doing updates > because the updates are very fast again: > > > While I wrote this email my colleague was googling around. > > He found this > http://lucene.472066.n3.nabble.com/HttpSolrServer- > commit-is-taking-more-time-td4330954.html > > and my colleague changed some values > > from: > > > > to: > size="512" > initialSize="512" > autowarmCount="0"/> > from: > > > > to: >size="512" > initialSize="512" > autowarmCount="0"/> > > > and it seems, that our problems are gone completely. Updates fast. > Search seems not to be much slower. > > > But I'm still curious why our problems started so suddenly and what > negative side effects these changes could have. > > Cheers > > Felix > > > > 2018-04-18 17:11 GMT+02:00 Emir Arnautović <emir.arnauto...@sematext.com>: > > > Hi Felix, > > Did you try to do thread dump while doing update. Did it show anything? > > > > Emir > > -- > > Monitoring - Log Management - Alerting - Anomaly Detection > > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > > > > > On 18 Apr 2018, at 17:06, Felix XY <felix.xy.xy...@gmail.com> wrote: > > > > > > Hello group, > > > > > > since two days we have huge problems with our solr 5.4.1 installation. > > > > > > ( yes, we have to update it. But this will not be a solution right now > ) > > > > > > All path=/select requests are still very fast. But all /update Requests > > > take >30sec up to 3 minutes. > > > > > > The index is not very big (1.000.000 documents) and its size on disk is > > > about 1GB > > > > > > The virtual server (ESX) has 8GB RAM and 8 cores. IO is good. > > > > > > solr was started with -Xms4096M -Xmx4096M > > > ( but we changed it to higher and lower values during our tests ) > > > > > > We have a lot of /select requests at the moment (10.000/Minute) but > this > > is > > > not unusual for this installation and we didn't have this update > problems > > > before. > > > > > > On another identical sleeping core on the same server, we are able to > > make > > > fast updates. We experience slow updates only on the core with high > > > selecting traffic. So it seems not to be a general problem with java, > GC, > > > > > > > > > We disabled all other insert/updates and we are able to reproduce this > > slow > > > update behaviour in the Solr Admin console with a single update of one > > > document. > > > > > > We are lost. > > > > > > We didn't change the Solr configuration. > > > The load seems to be not higher then during previous peaks > > > The developers didn't change anything (so they say) > > > Search is still fast. > > > > > > But single simple updates takes >30sec > > > > > > Any ideas about this? We tried quite a lot the last two days > > > > > > Cheers > > > Felix > > > > > -- Regards, Shalin Shekhar Mangar.
Re: Howto change log level with Solr Admin UI ?
The changes made using the admin logging UI are local to the node. It will not change logging settings on other nodes and these changes do not persist between restarts. On Wed, Apr 18, 2018 at 7:33 PM, Bernd Fehling < bernd.fehl...@uni-bielefeld.de> wrote: > I just tried to change the log level with Solr Admin UI but it > does not change any logging on my running SolrCloud. > It just shows the changes in the Admin UI and the commands in the > request log, but no changes in the level of logging. > > Do I have to RELOAD the collection after changing log level? > > I tried all setting from ALL, TRACE, DEBUG, ... > > Also the Reference Guide 6.6 shows the Admin UI as I see it, but > the table below the image has levels FINEST, FINE, CONFIG, ... > https://lucene.apache.org/solr/guide/6_6/configuring-logging.html > This is confusing. > > > Regards, > Bernd > -- Regards, Shalin Shekhar Mangar.
Re: CLUSTERSTATUS API and Error loading specified collection / config in Solr 5.3.2.
I think you are running into race conditions in the API which have been fixed. See SOLR-8804 and SOLR-10720. The first is available in 5.5.1 but the latter fix will be released in the upcoming 7.3 release. The best workaround for your version is to just retry a few times until the API succeeds. On Sun, Mar 11, 2018 at 11:57 PM, Atita Arora <atitaar...@gmail.com> wrote: > Hi , > > I am working on an application which involves working on a highly > distributed Solr cloud environment. The application supports multi-tenancy > and we have around 250-300 collections on Solr where each client has their > own collection with a new shard being created as clientid- > where the timestamp is whenever the new data comes in for the client > (typically every 4-8 hrs) , the reason for this convention is to make sure > when the Indexes are being built (on demand) the timestamp matches closely > to the time when the last indexing was run (the earlier shard is > de-provisioned as soon as the new one is created). Whenever the indexing is > triggered it first makes a DB entry and then creates a catalog with > timestamp in solr. > The Solr cloud has 10 Nodes distributed geographically among 10 > datacenters. > The replication factor is 2. The Solr version is 5.3.2. > Coming to my problem - I had to write a utility to ensure that the DB > insert timestamp matches closely to the Solr index timestamp wherein I can > ensure that if the difference between DB timestamp and Solr Index tinestamp > is <= 2 hrs , we have fresh index. The new index contains revised prices of > products or offers etc which are critical to be updated as in when they > come. Hence this utility is to track that the required updates have been > successfully made. > I used *CLUSTERSTATUS* api for this task. It is serving the purpose well so > far , but pretty recently our solr cloud started complaining of strange > things because of which the *CLUSTERSTATUS* api keeps returning as error. > > The error claims to be of missing config & sometime missing collections > like. > > org.apache.solr.common.SolrException: Could not find collection : > > 1785-1520548816454 > > org.apache.solr.common.SolrException: Could not find collection : > 1785-1520548816454 > at > org.apache.solr.common.cloud.ClusterState.getCollection( > ClusterState.java:165) > at > org.apache.solr.handler.admin.ClusterStatus.getClusterStatus( > ClusterStatus.java:110) > at > org.apache.solr.handler.admin.CollectionsHandler$ > CollectionOperation$19.call(CollectionsHandler.java:614) > at > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody( > CollectionsHandler.java:166) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest( > RequestHandlerBase.java:143) > at > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest( > HttpSolrCall.java:678) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:215) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:179) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1652) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) > > The other times it would complain of missing the config for same or > different client id- timestamp like : > > 1532-1518669619526_shard1_replica3: > org.apache.solr.common.cloud.ZooKeeperException:org.apache. > solr.common.cloud.ZooKeeperException: > Specified config does not exist in ZooKeeper:1532-1518669619526I > > I would really appreciate if : > > >1. Someone can possibly guide me as to whats going on Solr Cloud >2. If CLUSTERSTATUS is the right pick to build such utility. Do we > have any other option? > > > Thanks for any pointers and suggestions. > > Appreciate your attention looking this through. > > Atita > -- Regards, Shalin Shekhar Mangar.
Re: /var/solr/data has lots of index* directories
You can look inside the index.properties. The directory name mentioned in
that properties file is the one being used actively. The rest are old
directories that should be cleaned up on Solr restart but you can delete
them yourself without any issues.
On Mon, Mar 5, 2018 at 11:43 PM, Tom Peters <tpet...@synacor.com> wrote:
> While trying to debug an issue with CDCR, I noticed that the
> /var/solr/data directories on my source cluster have wildly different sizes.
>
> % for i in solr2-{a..e}; do echo -n "$i: "; ssh -A $i du -sh
> /var/solr/data; done
> solr2-a: 9.5G /var/solr/data
> solr2-b: 29G/var/solr/data
> solr2-c: 6.6G /var/solr/data
> solr2-d: 9.7G /var/solr/data
> solr2-e: 19G/var/solr/data
>
> The leader is currently "solr2-a"
>
> Here's the actual index size:
>
> Master (Searching)
> 1520273178244 # version
> 73034 # gen
> 3.66 GB # size
>
> When I look inside /var/solr/data/ on solr2-b, I see a bunch of index.*
> directories:
>
> % ls | grep index
> index.20180223021742634
> index.20180223024901983
> index.20180223033852960
> index.20180223034811193
> index.20180223035648403
> index.20180223041040318
> index.properties
>
> On solr2-a, I only see one index directory (index.20180222192820572).
>
> Does anyone know why this will happen and how I can clean it up without
> potentially causing any issues? We're currently on version Solr 7.1.
>
>
> This message and any attachment may contain information that is
> confidential and/or proprietary. Any use, disclosure, copying, storing, or
> distribution of this e-mail or any attached file by anyone other than the
> intended recipient is strictly prohibited. If you have received this
> message in error, please notify the sender by reply email and delete the
> message and any attachments. Thank you.
>
--
Regards,
Shalin Shekhar Mangar.
Re: Changing Leadership in SolrCloud
When you say it is active, I presume you mean the "state" as returned by the Cluster Status API or as shown on the UI. But is it still the leader? Are you sure the firewall rules are correct? Do you see disconnected or session expiry exceptions in the leader logs? On Wed, Feb 28, 2018 at 12:21 PM, Zahra Aminolroaya <z.aminolro...@gmail.com > wrote: > Thanks Shalin. our "zkClientTimeout" is 3, so the leader should be > changed by now; However, the previous leader is still active. > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html > -- Regards, Shalin Shekhar Mangar.
Re: Changing Leadership in SolrCloud
When you block communication between Zookeeper and the leader, the ZK client inside Solr will disconnect and its session will expire after the session timeout. At this point a new leader should be elected automatically. The default timeout is 30 seconds. You should be able to see the value in solr.xml property named "zkClientTimeout". On Tue, Feb 27, 2018 at 2:06 PM, zahra121 <z.aminolro...@gmail.com> wrote: > Suppose I have a node which is a leader in SolrCloud. > > When I block this leader's SolrCloud and Zookeeper ports by the command > "firewall-cmd --remove-port=/tcp --permanent", the leader does > not > change automatically and this leader status remains active in solr admin > UI. > > Thus, I decided to change the leader manually. I tried REBALANCELEADERS and > ADDROLE commands in solrCloud, however the leader did not change! > > How can I manually change the leader if the firewall blocks the SolrCloud > ports from being listened? > > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html > -- Regards, Shalin Shekhar Mangar.
Re: Could not find collection when upgrading to Solr 7
Bootstrapping is not broken but the default in Solr 7 is to set the cluster property legacyCloud=false which means that bootstrapped cores cannot register new collections automatically anymore. If you want to rely on the old behavior then you need to set the cluster property legacyCloud=true. This was also mentioned in the upgrade section for Solr 7 in CHANGES.txt On Mon, Feb 26, 2018 at 1:28 AM, Marvin Bredal Lillehaug < marvin.lilleh...@gmail.com> wrote: > Thank your for your reply. > I will take a look at both the code for bootstrapping, and look at what you > are suggesting to determine what to do. > > On Sun, Feb 25, 2018 at 5:17 PM, Shawn Heisey <elyog...@elyograg.org> > wrote: > > > On 2/24/2018 8:05 AM, Marvin Bredal Lillehaug wrote: > > > >> This is for doing local development and running tests. > >> So starting with embedded zookeeper in SolrCloud mode is intentional. > The > >> idea is to just run the script setting up the folder structure for the > >> cores and everything is up and running, ready for indexing and querying > >> locally. > >> The applications using Solr also use Zookeeper for configuration and > >> coordination of their own stuff. > >> > >> There are no persmission issues. > >> > >> I have put the scripts here https://github.com/computerlove/start-solr > >> > >> The file config specifies what version to use. When running > >> run-dev-solr.sh > >> that version is downloaded, solrhome is copied to the right place and > the > >> instance is started. > >> > > > > I grabbed the git repo and proceeded to fiddle with it. > > > > As you said, it works when the solr version that is downloaded is 6.6.2, > > but fails with 7.2.1. Based on what gets logged and what I know about > how > > Solr does initialization, I'm betting the bootstrap feature is broken. I > > can't tell what's wrong, but I think that's probably where the problem > is. > > > > But as I already said, I strongly recommend that you don't use that > > feature. At all. > > > > The cores that are in the git repo are empty -- there's no index. If the > > source cores are empty when you run the script "for real", then I have an > > alternate idea for you to try: > > > > Instead of copying the cores and bootstrapping a non-cloud install into a > > cloud install: Remove the bootstrap option. Create the collections using > > "bin/solr create", with -replicationFactor 1 -shards 1, and the -d option > > pointing at the config you want uploaded for the collection. You could > > even go with a higher shard count if you want. Increasing > > replicationFactor is probably possible, but as all this would be running > on > > a single server, there's no point. > > > > Thanks, > > Shawn > > > > > > > -- > med vennlig hilsen, > Marvin B. Lillehaug > -- Regards, Shalin Shekhar Mangar.
Re: regarding exposing merge metrics
The merge metrics were enabled by default in 6.4 but they were turned off in 6.4.2 because of large performance degradations. For more details, see https://issues.apache.org/jira/browse/SOLR-10130 On Tue, Jan 9, 2018 at 9:11 AM, S G <sg.online.em...@gmail.com> wrote: > Yes, this is actually confusing and the documentation ( > https://lucene.apache.org/solr/guide/7_2/metrics-reporting.html) does not > help either: > > *Index Merge Metrics* : These metrics are collected in respective > registries for each core (e.g., solr.core.collection1….), under the INDEX > category. > Basic metrics are always collected - collection of additional metrics can > be turned on using boolean parameters in the /config/indexConfig/metrics. > > However, we do not see the merge-metrics being collected if the above > config is absent. So what basic metrics are always collected for merge? > And why do the merge metrics require an additional config while most of the > others are reported directly? > > Thanks > SG > > > > On Mon, Jan 8, 2018 at 2:02 PM, suresh pendap <sureshfors...@gmail.com> > wrote: > >> Hi, >> I am following the instructions from >> https://lucene.apache.org/solr/guide/7_1/metrics-reporting.html >> in order to expose the Index merge related metrics. >> >> The document says that we have to add the below snippet in order to expose >> the merge metrics >> >> >> ... >> >> >> 524288 >> true >> >> ... >> >> ... >> >> >> >> I would like to know why is this metrics not exposed by default just like >> all the other metrics? >> >> Is there any performance overhead that we should be concerned about it? >> >> If there was no particular reason, then can we expose it by default? >> >> >> >> Regards >> Suresh >> -- Regards, Shalin Shekhar Mangar.
Re: docValues with stored and useDocValuesAsStored
Hi Bernd, If Solr can fetch a field from both stored and docValues then it chooses docValues only if such field is single-valued and that allows Solr to avoid accessing the stored document altogether for *all* fields to be returned. Otherwise stored values are preferred. This is the behavior since 7.1+ On Mon, Jan 8, 2018 at 2:25 PM, Bernd Fehling <bernd.fehl...@uni-bielefeld.de> wrote: > What is the precedence when docValues with stored=true is used? > e.g. > docValues="true" /> > > My guess, because of useDocValuesAsStored=true is default, that stored=true is > ignored and the values are pulled from docValues. > > And only if useDocValuesAsStored=false is explicitly used then stored=true > comes > into play. > > Or short, useDocValuesAsStored=true (the default) has precedence over > stored=true. > I this right? > > Regards > Bernd -- Regards, Shalin Shekhar Mangar.
Re: Replication Factor Bug in Collections Restore API?
Sounds like a bug. Can you please open a Jira issue?
On Thu, Jan 4, 2018 at 8:37 PM, Ansgar Wiechers
<solr-u...@planetcobalt.net> wrote:
> Hi all.
>
> I'm running Solr 7.1 in SolrCloud mode ona a 3-node cluster and tried
> using the backup/restore API for the first time. Backup worked fine, but
> when trying to restore the backed-up collection I ran into an unexpected
> problem with the replication factor setting.
>
> Below command attempts to restore a backup of the collection "demo" with
> 3 shards, creating 2 replicas per shard:
>
> # curl -s -k
> 'https://localhost:8983/solr/admin/collections?action=restore=demo=/srv/backup/solr/solr-dev=demo=2=2'
> {
> "error": {
> "code": 400,
> "msg": "Solr cloud with available number of nodes:3 is insufficient for
> restoring a collection with 3 shards, total replicas per shard 6 and
> maxShardsPerNode 2. Consider increasing maxShardsPerNode value OR number of
> available nodes.",
> "metadata": [
> "error-class",
> "org.apache.solr.common.SolrException",
> "root-error-class",
> "org.apache.solr.common.SolrException"
> ]
> },
> "exception": {
> "rspCode": 400,
> "msg": "Solr cloud with available number of nodes:3 is insufficient for
> restoring a collection with 3 shards, total replicas per shard 6 and
> maxShardsPerNode 2. Consider increasing maxShardsPerNode value OR number of
> available nodes."
> },
> "Operation restore caused exception:":
> "org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> Solr cloud with available number of nodes:3 is insufficient for restoring a
> collection with 3 shards, total replicas per shard 6 and maxShardsPerNode 2.
> Consider increasing maxShardsPerNode value OR number of available nodes.",
> "responseHeader": {
> "QTime": 28,
> "status": 400
> }
> }
>
> It looks to me like the restore API multiplies the replication factor
> with the number of nodes, which is not how the replication factor
> behaves in other contexts. The documentation[1] also didn't lead me to
> expect this behavior:
>
>> replicationFactor
>>
>>The number of replicas to be created for each shard.
>
> Is this expected behavior (by anyone but me)?
> Should I report it as a bug?
>
> [1]: https://lucene.apache.org/solr/guide/7_1/collections-api.html
>
> Regards
> Ansgar Wiechers
> --
> "Abstractions save us time working, but they don't save us time learning."
> --Joel Spolsky
--
Regards,
Shalin Shekhar Mangar.
Re: New replica types
Comments inline: On Wed, Jan 3, 2018 at 11:39 AM, S G <sg.online.em...@gmail.com> wrote: > AFAIK, tlog file is truncated with a hard-commit. > So if the TLOG replica is only pulling the tlog-file, it would become out > of date if it does not pull the full index too. > That means that the TLOG replica would do a full copy every time there is a > commit on the leader. TLOG replica does not pull the tlog file. Instead, each update is pushed from the leader to all TLOG replicas as a synchronous operation. The same thing happens for NRT replicas as well. Such updates are appended to the transaction logs on the PULL replicas. Similar to the old master/slave model, the leader (hard) commits its index frequently and the TLOG replica polls the leader to check if a newer index version is available for download. If there is a new version on the leader then the replica downloads only the new segments from the leader that are not present locally. > > PULL replica, by definition copies index files only and so it would do full > recoveries often too. PULL replicas also download only the newest segments present on the leader which haven't been copied over previously. At no time does a full recovery happen unless the replica is new i.e. it has an empty index or it has been out of sync for so long that leader has completely new segments due to merges. One thing to note is that you should never mix NRT replicas with other types. Either have a collection with only NRT replicas or have a mix of TLOG and PULL replicas. This way you ensure that the leader is never different enough for a full recovery to be required. > > > How intelligent are the two replica types in determining that they need to > do a full recovery vs partial recovery? > Does full recovery happen every hard-commit on the leader? > Or does it happen with segment merges on the leader? (because index files > will look much different after a segment-merge) > > > > NRT replicas will typically have very different files in their on-disk >> indexes even though they contain the same documents. > > > This is something which has caused full recoveries many times in my > clusters and I wish there was a solution to this one. > Do you think it would make sense for all replicas of a shard to agree upon > the segment where a document should go to? > Coupling this with an agreed cadence on segment merges, Solr would never do > full recovery. (It's a very high level view of course and will need lot of > refinements if implemented). > > Getting a cadence on segment merges could possibly be implemented by a > time-based merging strategy where documents arriving within a particular > time-range only will form a particular segment. > So documents arriving between 1pm-2pm go to segment 1, those between > 2pm-3pm go to segment 2 and so on. > That ways replicas will only copy the last N segments (with N being 1 > generally) instead of doing a full recovery. > Even if merging happens on the leader, the last N segments should not be > cleared to avoid full recoveries on the replicas. > (I know something like this happens today, but not very sure about the > internal details and it's nowhere documented clearly). > > Currently, I see my replicas go into full-recovery even when I dynamically > add a field to a collection or a replica missed updates for a few seconds. > (I do have high values for catchup rather than the default 100) > > > Thanks > SG > > > > > > > On Tue, Jan 2, 2018 at 8:58 PM, Shawn Heisey <apa...@elyograg.org> wrote: > >> On 1/2/2018 8:02 PM, S G wrote: >> >>> If the above is incorrect, can someone please point that out? >>> >> >> Assuming I have a correct understanding of how the different replica types >> work, I have some small clarifications. If my understanding is incorrect, >> I hope somebody will point out my errors. >> >> TLOG is leader eligible because it keeps transaction logs from ongoing >> indexing, although it does not perform that indexing on its own index >> unless it becomes the leader. Transaction logs are necessary for operation >> as leader. >> >> PULL does not keep transaction logs, which is why it is not leader >> eligible. It only copies the index data. >> >> Either TLOG or PULL would do a full index copy if the local index is >> suddenly very different from the leader. This could happen in situations >> where you have NRT replicas and the leader changes -- NRT replicas will >> typically have very different files in their on-disk indexes even though >> they contain the same documents. When the leader changes to a different >> NRT replica, TLOG/PULL replicas will suddenly find that they have a very >> different list of index files, so they will fetch the entire index. >> >> Thanks, >> Shawn >> -- Regards, Shalin Shekhar Mangar.
CVE-2016-6809: Java code execution for serialized objects embedded in MATLAB files parsed by Apache Solr using Apache Tika
CVE-2016-6809: Java code execution for serialized objects embedded in MATLAB files parsed by Apache Solr using Tika Severity: Important Vendor: The Apache Software Foundation Versions Affected: Solr 5.0.0 to 5.5.4 Solr 6.0.0 to 6.6.1 Solr 7.0.0 to 7.0.1 Description: Apache Solr uses Apache Tika for parsing binary file types such as doc, xls, pdf etc. Apache Tika wraps the jmatio parser (https://github.com/gradusnikov/jmatio) to handle MATLAB files. The parser uses native deserialization on serialized Java objects embedded in MATLAB files. A malicious user could inject arbitrary code into a MATLAB file that would be executed when the object is deserialized. This vulnerability was originally described at http://mail-archives.apache.org/mod_mbox/tika-user/201611.mbox/%3C2125912914.1308916.1478787314903%40mail.yahoo.com%3E Mitigation: Users are advised to upgrade to either Solr 5.5.5 or Solr 6.6.2 or Solr 7.1.0 releases which have fixed this vulnerability. Solr 5.5.5 upgrades the jmatio parser to v1.2 and disables the Java deserialisation support to protect against this vulnerability. Solr 6.6.2 and Solr 7.1.0 have upgraded the bundled Tika to v1.16. Once upgrade is complete, no other steps are required. References: https://issues.apache.org/jira/browse/SOLR-11486 https://issues.apache.org/jira/browse/SOLR-10335 https://wiki.apache.org/solr/SolrSecurity -- Regards, Shalin Shekhar Mangar.
[ANNOUNCE] [SECURITY] CVE-2017-12629: Several critical vulnerabilities discovered in Apache Solr (XXE & RCE)
CVE-2017-12629: Several critical vulnerabilities discovered in Apache Solr (XXE & RCE) Severity: Critical Vendor: The Apache Software Foundation Versions Affected: Solr 5.5.0 to 5.5.4 Solr 6.0.0 to 6.6.1 Solr 7.0.0 to 7.0.1 Description: The details of this vulnerability were reported on public mailing lists. See https://s.apache.org/FJDl The first vulnerability relates to XML external entity expansion in the XML Query Parser which is available, by default, for any query request with parameters deftype=xmlparser. This can be exploited to upload malicious data to the /upload request handler. It can also be used as Blind XXE using ftp wrapper in order to read arbitrary local files from the solr server. The second vulnerability relates to remote code execution using the RunExecutableListener available on all affected versions of Solr. At the time of the above report, this was a 0-day vulnerability with a working exploit affecting the versions of Solr mentioned in the previous section. However, mitigation steps were announced to protect Solr users the same day. See https://lucene.apache.org/solr/news.html#12-october-2017-please-secure-your-apache-solr-servers-since-a-zero-day-exploit-has-been-reported-on-a-public-mailing-list Mitigation: Users are advised to upgrade to either Solr 6.6.2 or Solr 7.1.0 releases both of which address the two vulnerabilities. Once upgrade is complete, no other steps are required. If users are unable to upgrade to Solr 6.6.2 or Solr 7.1.0 then they are advised to restart their Solr instances with the system parameter `-Ddisable.configEdit=true`. This will disallow any changes to be made to your configurations via the Config API. This is a key factor in this vulnerability, since it allows GET requests to add the RunExecutableListener to your config. Users are also advised to re-map the XML Query Parser to another parser to mitigate the XXE vulnerability. For example, adding the following to the solrconfig.xml file re-maps the xmlparser to the edismax parser: Credit: Michael Stepankin (JPMorgan Chase) Olga Barinova (Gotham Digital Science) References: https://issues.apache.org/jira/browse/SOLR-11482 https://issues.apache.org/jira/browse/SOLR-11477 https://wiki.apache.org/solr/SolrSecurity -- Regards, Shalin Shekhar Mangar.
[ANNOUNCE] Apache Solr 7.1.0 released
17 October 2017, Apache Solr™ 7.1.0 available The Lucene PMC is pleased to announce the release of Apache Solr 7.1.0 Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly scalable, providing fault tolerant distributed search and indexing, and powers the search and navigation features of many of the world's largest internet sites. Solr 7.1.0 is available for immediate download at: http://lucene.apache.org/solr/mirrors-solr-latest-redir.html See http://lucene.apache.org/solr/7_1_0/changes/Changes.html for a full list of details. Solr 7.1.0 Release Highlights: * Critical Security Update: Fix for CVE-2017-12629 which is a working 0-day exploit reported on the public mailing list. See https://s.apache.org/FJDl * Auto-scaling: Solr can now move replicas automatically when a new node is added or an existing node is removed using the auto scaling policy framework introduced in 7.0 * Auto-scaling: The 'autoAddReplicas' feature which was limited to shared file systems is now available for all file systems. It has been ported to use the new autoscaling framework internally. * Auto-scaling: New set-trigger, remove-trigger, set-listener, remove-listener, suspend-trigger, resume-trigger APIs * Auto-scaling: New /autoscaling/history API to show past autoscaling actions and cluster events * New JSON based Query DSL for Solr that extends JSON Request API to also support all query parsers and their nested parameters * JSON Facet API: min/max aggregations are now supported on single-valued date fields * Lucene's Geo3D (surface of sphere & ellipsoid) is now supported on spatial RPT fields by setting spatialContextFactory="Geo3D". Furthermore, this is the first time Solr has out of the box support for polygons * Expanded support for statistical stream evaluators such as various distributions, rank correlations, distances and more. * Multiple other optimizations and bug fixes You are encouraged to thoroughly read the "Upgrade Notes" at http://lucene.apache.org/solr/7_1_0/changes/Changes.html or in the CHANGES.txt file accompanying the release. Solr 7.1 also includes many other new features as well as numerous optimizations and bugfixes of the corresponding Apache Lucene release. Please report any feedback to the mailing lists (http://lucene.apache.org/solr/discussion.html) Note: The Apache Software Foundation uses an extensive mirroring network for distributing releases. It is possible that the mirror you are using may not have replicated the release yet. If that is the case, please try another mirror. This also goes for Maven access. -- Regards, Shalin Shekhar Mangar.
Re: zero-day exploit security issue
Yes, there is but it is private i.e. only the Apache Lucene PMC members can see it. This is standard for all security issues in Apache land. The fixes for this issue has been applied to the release branches and the Solr 7.1.0 release candidate is already up for vote. Barring any unforeseen circumstances, a 7.1.0 release with the fixes should be expected this week. On Fri, Oct 13, 2017 at 8:14 PM, Xie, Sean <sean@finra.org> wrote: > Is there a tracking to address this issue for SOLR 6.6.x and 7.x? > > https://lucene.apache.org/solr/news.html#12-october-2017-please-secure-your-apache-solr-servers-since-a-zero-day-exploit-has-been-reported-on-a-public-mailing-list > > Sean > > Confidentiality Notice:: This email, including attachments, may include > non-public, proprietary, confidential or legally privileged information. If > you are not an intended recipient or an authorized agent of an intended > recipient, you are hereby notified that any dissemination, distribution or > copying of the information contained in or transmitted with this e-mail is > unauthorized and strictly prohibited. If you have received this email in > error, please notify the sender by replying to this message and permanently > delete this e-mail, its attachments, and any copies of it immediately. You > should not retain, copy or use this e-mail or any attachment for any purpose, > nor disclose all or any part of the contents to any other person. Thank you. -- Regards, Shalin Shekhar Mangar.
Re: question about an entry in the log file
That log shows that the searcher being opened is the "realtime"
searcher as opposed to the "main" searcher. The realtime searcher is
quite lightweight. It causes a flush of built index segments from the
memory to the disk and opens a new searcher over them. No autowarming
or fsync happens for realtime searcher. It is used for realtime gets,
atomic updates and some other internal housekeeping tasks. This
searcher is essential and unavoidable.
The "main" searcher, on the other hand, is used for searching and is
heavy-weight. This is controlled by the "openSearcher" request
parameter and configuration.
On Wed, Sep 20, 2017 at 12:07 PM, kaveh minooie <ka...@plutoz.com> wrote:
> Hi Erick
>
> Thanks for your response. I understand your point, but what I was asking was
> does solr reopen searchers after a commit call even if the commit was called
> with openSearcher=false since this is what seems to be happening based on
> these log entries?
>
> Also, it seems that if autocommit is configured with
> false and softAutoCommit is set to -1, absence
> of any other commit call, the new updates, although committed, would remain
> invisible forever. my problem here is that I am going for maximum indexing
> performance, millions of document per each batch job, couple of batch jobs
> per day. their immediate visibility or even delayed visibility is not a
> priority, but they have to become visible at some point. preferably at the
> end of each batch job. what do you think would be the best way to go about
> this?
>
> thanks,
>
>
> On 09/20/2017 10:17 AM, Erick Erickson wrote:
>>
>> First, I would not recommend you call commit from the client. It's
>> usually far better to let your autocommit settings in solrconfig.xml
>> deal with it. When you need to search, you either need to configure
>> with true
>>
>> or set to something other than -1.
>>
>>
>> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>
>> Best,
>> Erick
>>
>> On Tue, Sep 19, 2017 at 4:53 PM, kaveh minooie <ka...@plutoz.com> wrote:
>>>
>>> Hi eveyone
>>>
>>> I am trying to figure out why calling commit from my client takes a very
>>> long time in an environment with concurrent updates, and I see the
>>> following
>>> snippet in the solr log files when client calls for commit. my question
>>> is
>>> regarding the third info. what is it opening? and how can make solr to
>>> stop
>>> doing that?
>>>
>>>
>>> INFO - 2017-09-19 16:42:20.557; [c:dosweb2016 s:shard2 r:core_node5
>>> x:dosweb2016] org.apache.solr.update.DirectUpdateHandler2; start
>>>
>>> commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>>> INFO - 2017-09-19 16:42:20.557; [c:dosweb2016 s:shard2 r:core_node5
>>> x:dosweb2016] org.apache.solr.update.SolrIndexWriter; Calling
>>> setCommitData
>>> with IW:org.apache.solr.update.SolrIndexWriter@3ee73284
>>> INFO - 2017-09-19 16:42:20.660; [c:dosweb2016 s:shard2 r:core_node5
>>> x:dosweb2016] org.apache.solr.search.SolrIndexSearcher; Opening
>>> [Searcher@644a8d33[dosweb2016] realtime]
>>> INFO - 2017-09-19 16:42:20.668; [c:dosweb2016 s:shard2 r:core_node5
>>> x:dosweb2016] org.apache.solr.update.DirectUpdateHandler2;
>>> end_commit_flush
>>>
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>
>
> --
> Kaveh Minooie
--
Regards,
Shalin Shekhar Mangar.
CVE-2017-9803: Security vulnerability in kerberos delegation token functionality
CVE-2017-9803: Security vulnerability in kerberos delegation token functionality Severity: Important Vendor: The Apache Software Foundation Versions Affected: Apache Solr 6.2.0 to 6.6.0 Description: Solr's Kerberos plugin can be configured to use delegation tokens, which allows an application to reuse the authentication of an end-user or another application. There are two issues with this functionality (when using SecurityAwareZkACLProvider type of ACL provider e.g. SaslZkACLProvider), Firstly, access to the security configuration can be leaked to users other than the solr super user. Secondly, malicious users can exploit this leaked configuration for privilege escalation to further expose/modify private data and/or disrupt operations in the Solr cluster. The vulnerability is fixed from Solr 6.6.1 onwards. Mitigation: 6.x users should upgrade to 6.6.1 Credit: This issue was discovered by Hrishikesh Gadre of Cloudera Inc. References: https://issues.apache.org/jira/browse/SOLR-11184 https://wiki.apache.org/solr/SolrSecurity -- The Lucene PMC
Re: Solr uses lots of shared memory!
This impossible situation >> means that something is being misreported somewhere. If I deduct that >> 11GB of SHR from the RES value, then all the numbers work. >> >> The screenshot was almost 3 years ago, so I do not know what machine it >> came from, and therefore I can't be sure what the actual heap size was. >> I think it was about 6GB -- the difference between RES and SHR. I have >> used a 6GB heap on some of my production servers in the past. The >> server where I got this screenshot was not having any noticeable >> performance or memory problems, so I think that I can trust that the >> main numbers above the process list (which only come from the OS) are >> correct. >> >> Thanks, >> Shawn >> >> -- Regards, Shalin Shekhar Mangar.
Re: 6.6.0 getNumShards() NPE?!
Can you please post the steps to reproduce this? On Thu, Aug 10, 2017 at 4:12 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > I can now reproduce it on the two shard, two replica cluster. > > It does NOT happen on the collection_shard1_replica1 and > collection_shard2_replica1 nodes. > > It happens consistently on the collection_shard1_replica2 and > collection_shard2_replica2 nodes. > > Any ideas? > > -Original message- >> From:Markus Jelsma <markus.jel...@openindex.io> >> Sent: Thursday 10th August 2017 12:34 >> To: Solr-user <solr-user@lucene.apache.org> >> Subject: 6.6.0 getNumShards() NPE?! >> >> Hello, >> >> Having trouble, again, with CloudDescriptor and friend, getting the number >> of shards of the collection. It sometimes returns 1 for a collection of two >> shards. Having this code: >> >> cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor(); >> return cloudDescriptor.getNumShards(); >> >> In some cases cloudDescriptor.getNumShards() throws an NPE. Am i doing >> something wrong? >> >> Thanks, >> Markus >> -- Regards, Shalin Shekhar Mangar.
Re: identifying source of queries
There is no need to change anything related to query parsers. A search component should be enough. On Wed, Aug 9, 2017 at 9:56 PM, suresh pendap <sureshfors...@gmail.com> wrote: > Thanks Shalin for the reply. > Do I need to also update the query parsers in order to handle the new query > param? > I can build a custom component but dabbling with query parsers would be way > too much for me to handle. > > Thanks > Suresh > > > > > On Tue, Aug 8, 2017 at 9:49 PM, Shalin Shekhar Mangar < > shalinman...@gmail.com> wrote: > >> There is no in-built way but if you are willing to create a custom >> query component then it should be easy to mandate that every query >> must have a tag parameter by throwing an exception otherwise. Any >> query param you pass to a distributed query request should be >> propagated to all query nodes as well. >> >> On Wed, Aug 9, 2017 at 8:58 AM, suresh pendap <sureshfors...@gmail.com> >> wrote: >> > Hi, >> > We have found that application teams often fire ad-hoc queries, some of >> > these are very expensive queries and can bring the solr cluster down. >> Some >> > times they just build custom scripts which does some offline analytics by >> > firing expensive queries, the solr cluster was originally not sized for >> > executing such queries. >> > >> > When an issue happens we identify from the solr logs the query which is >> > taking long time. But some times we do not even know who is firing these >> > queries and hence it takes a while to stop them. >> > >> > We would like be able to identify the source of the solr queries. >> > >> > Is there a way to tag the solr queries? >> > Can we associate some tags or query identifier with the query? >> > These tags should be made mandatory without which the solr query should >> > fail? >> > >> > We would like to build a custom component which logs the query, the query >> > identifier (the tag which user provides) and the IP address of the client >> > machine which fired this query. >> > >> > >> > Thanks >> > Suresh >> >> >> >> -- >> Regards, >> Shalin Shekhar Mangar. >> -- Regards, Shalin Shekhar Mangar.
Re: identifying source of queries
There is no in-built way but if you are willing to create a custom query component then it should be easy to mandate that every query must have a tag parameter by throwing an exception otherwise. Any query param you pass to a distributed query request should be propagated to all query nodes as well. On Wed, Aug 9, 2017 at 8:58 AM, suresh pendap <sureshfors...@gmail.com> wrote: > Hi, > We have found that application teams often fire ad-hoc queries, some of > these are very expensive queries and can bring the solr cluster down. Some > times they just build custom scripts which does some offline analytics by > firing expensive queries, the solr cluster was originally not sized for > executing such queries. > > When an issue happens we identify from the solr logs the query which is > taking long time. But some times we do not even know who is firing these > queries and hence it takes a while to stop them. > > We would like be able to identify the source of the solr queries. > > Is there a way to tag the solr queries? > Can we associate some tags or query identifier with the query? > These tags should be made mandatory without which the solr query should > fail? > > We would like to build a custom component which logs the query, the query > identifier (the tag which user provides) and the IP address of the client > machine which fired this query. > > > Thanks > Suresh -- Regards, Shalin Shekhar Mangar.
Re: Metrics in 6.5.1 names and stuff
Hi Walter,
You need the metrics with the prefix "QUERY./srp" -- those will give
you request counts and percentiles for the /srp handler for a node.
Comments inline:
On Fri, Aug 4, 2017 at 3:23 AM, Walter Underwood <wun...@wunderwood.org> wrote:
> I’m trying to get what I want out of the metrics reporting in Solr. I want
> the counts and percentiles for each request handler in each collection. If I
> have “/srp”, “/suggest”, and “/seo”, I want three sets of metrics.
>
> I’m getting a lot of weird stuff. For counts for /srp in an eight node
> cluster, one node (new-solr-c01) is reporting all of these graphite metric
> names:
>
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c01.test3.cloud.cheggnet.com:8983/solr/questions_shard1_replica2/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c02.test3.cloud.cheggnet.com:8983/solr/questions_shard2_replica4/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c03.test3.cloud.cheggnet.com:8983/solr/questions_shard4_replica3/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c04.test3.cloud.cheggnet.com:8983/solr/questions_shard2_replica2/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c05.test3.cloud.cheggnet.com:8983/solr/questions_shard4_replica4/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c06.test3.cloud.cheggnet.com:8983/solr/questions_shard3_replica1/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c07.test3.cloud.cheggnet.com:8983/solr/questions_shard1_replica3/srp.post.requests.count
> new-solr-c01.solr.node.QUERY.httpShardHandler.http://new-solr-c08.test3.cloud.cheggnet.com:8983/solr/questions_shard3_replica3/srp.post.requests.count
> new-solr-c01.solr.node.UPDATE.updateShardHandler./solr/questions/srp.get.requests.count
>
> 1. What are the metrics with the URLs? Are those distributed queries to other
> shards? If so, why are they going to the local shard, too?
Yes, these are distributed queries to other shards. Any thing under
QUERY.httpShardHandler is either a distributed search request or an
admin request made by the overseer (in solrcloud).
The local shard isn't short circuited. A request to a local core (as
part of a distributed query) also goes through the HttpShardHandler.
This is why the local shard shows up here.
>
> 2. Why is the last one under UPDATE, when it is a query request handler?
It is not under just UPDATE. It is under UPDATE.updateShardHandler.
These are requests made using the UpdateShardHandler which is
typically used for distributed updates. I am not sure why a call was
made to the /srp handler using the update shard handler.
>
> 3. The jetty metrics seem to lump everything together under the HTTP method.
Is there a question here?
>
> 4. I enabled the http group, but don’t get any metrics from it.
The http group is not used currently. It was supposed to be used for
collecting metrics from http clients used inside Solr but since then
those have been folded into QUERY.httpShardHandler and
UPDATE.updateShardHandler metrics.
>
> Here is the config. The prefix is set to the nodename. For this test, I sent
> the metrics to localhost and logged them with “nc -l 2003 > graphite.log”.
>
>
> class="org.apache.solr.metrics.reporters.SolrGraphiteReporter">
>${graphite_host:NO_HOST_SET}
> 2003
>${graphite_prefix:NO_PREFIX_SET}
>http://observer.wunderwood.org/ (my blog)
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: logging support in Lucene code
Lucene does not use a logger framework. But if you are using Solr then you can route the infoStream logging to Solr's log files by setting an option in the solrconfig.xml. See http://lucene.apache.org/solr/guide/6_6/indexconfig-in-solrconfig.html#IndexConfiginSolrConfig-OtherIndexingSettings On Fri, Jul 28, 2017 at 11:13 AM, Nawab Zada Asad Iqbal <khi...@gmail.com> wrote: > Any doughnut for me ? > > > Regards > Nawab > > On Thu, Jul 27, 2017 at 9:57 AM Nawab Zada Asad Iqbal <khi...@gmail.com> > wrote: > > > Hi, > > > > I see a lot of discussion on this topic from almost 10 years ago: e.g., > > https://issues.apache.org/jira/browse/LUCENE-1482 > > > > For 4.5, I relied on 'System.out.println' for writing information for > > debugging in production. > > > > In 6.6, I notice that some classes in Lucene are instantiating a Logger, > > should I use Logger instead? I tried to log with it, but I don't see any > > output in logs. > > > > > > Regards > > Nawab > > > -- Regards, Shalin Shekhar Mangar.
[ANNOUNCE] [SECURITY] CVE-2017-7660: Security Vulnerability in secure inter-node communication in Apache Solr
CVE-2017-7660: Security Vulnerability in secure inter-node communication in Apache Solr Severity: Important Vendor: The Apache Software Foundation Versions Affected: Solr 5.3 to 5.5.4 Solr 6.0 to 6.5.1 Description: Solr uses a PKI based mechanism to secure inter-node communication when security is enabled. It is possible to create a specially crafted node name that does not exist as part of the cluster and point it to a malicious node. This can trick the nodes in cluster to believe that the malicious node is a member of the cluster. So, if Solr users have enabled BasicAuth authentication mechanism using the BasicAuthPlugin or if the user has implemented a custom Authentication plugin, which does not implement either "HttpClientInterceptorPlugin" or "HttpClientBuilderPlugin", his/her servers are vulnerable to this attack. Users who only use SSL without basic authentication or those who use Kerberos are not affected. Mitigation: 6.x users should upgrade to 6.6 5.x users should obtain the latest source from git and apply this patch: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2f5ecbcf Credit: This issue was discovered by Noble Paul of Lucidworks Inc. References: https://issues.apache.org/jira/browse/SOLR-10624 https://wiki.apache.org/solr/SolrSecurity -- The Lucene PMC
Re: Solr 5.2+ using SSL and non-SSL ports
No, Solr cannot use both SSL and non-SSL at the same time. You must choose one. On Mon, Jul 3, 2017 at 10:29 PM, sputul <sputu...@gmail.com> wrote: > I have SSL enabled in Solr 5 but Zookeeper needs to be started with proper > url scheme. Does this imply Solr Cloud cannot use SSL and non-SSL at the > same time as Zookeeper itself need separate ports? > > Thanks. > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/Solr-5-2-using-SSL-and-non-SSL-ports-tp4312859p4343997.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Regards, Shalin Shekhar Mangar.
Re: Performance warning: Overlapping onDeskSearchers=2 solr
This has been changed already in 6.4.0. From the CHANGES.txt entry: SOLR-9712: maxWarmingSearchers now defaults to 1, and more importantly commits will now block if this limit is exceeded instead of throwing an exception (a good thing). Consequently there is no longer a risk in overlapping commits. Nonetheless users should continue to avoid excessive committing. Users are advised to remove any pre-existing maxWarmingSearchers entries from their solrconfig.xml files. On Wed, May 17, 2017 at 8:45 PM, Jason Gerlowski <gerlowsk...@gmail.com> wrote: > Hey Shawn, others. > > This is a pitfall that Solr users seem to run into with some > frequency. (Anecdotally, I've bookmarked the Lucidworks article you > referenced because I end up referring people to it often enough.) > > The immediate first advice when someone encounters these > onDeckSearcher error messages is to examine their commit settings. Is > there any other possible cause for those messages? If not, can we > consider changing the log/exception error message to be more explicit > about the cause? > > A strawman new message could be: "Performance warning: Overlapping > onDeskSearchers=2; consider reducing commit frequency if performance > problems encountered" > > Happy to create a JIRA/patch for this; just wanted to get some > feedback first in case there's an obvious reason the messages don't > get explicit about the cause. > > Jason > > On Wed, May 17, 2017 at 8:49 AM, Shawn Heisey <apa...@elyograg.org> wrote: >> On 5/17/2017 5:57 AM, Srinivas Kashyap wrote: >>> We are using Solr 5.2.1 version and are currently experiencing below >>> Warning in Solr Logging Console: >>> >>> Performance warning: Overlapping onDeskSearchers=2 >>> >>> Also we encounter, >>> >>> org.apache.solr.common.SolrException: Error opening new searcher. exceeded >>> limit of maxWarmingSearchers=2, try again later. >>> >>> >>> The reason being, we are doing mass update on our application and solr >>> experiencing the higher loads at times. Data is being indexed using DIH(sql >>> queries). >>> >>> In solrconfig.xml below is the code. >>> >>> >>> >>> Should we be uncommenting the above lines and try to avoid this error? >>> Please help me. >> >> This warning means that you are committing so frequently that there are >> already two searchers warming when you start another commit. >> >> DIH does a commit exactly once -- at the end of the import. One import will >> not cause the warning message you're seeing, so if there is one import >> happening at a time, either you are sending explicit commit requests during >> the import, or you have autoSoftCommit enabled with values that are far too >> small. >> >> You should definitely have autoCommit configured, but I would remove >> maxDocs and set maxTime to something like 6 -- one minute. The >> autoCommit should also set openSearcher to false. This kind of commit >> will not make new changes visible, but it will start a new transaction >> log frequently. >> >> >> 6 >> false >> >> >> An automatic commit (soft or hard) with a one second interval is going to >> cause that warning you're seeing. >> >> https://lucidworks.com/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> Thanks, >> Shawn >> -- Regards, Shalin Shekhar Mangar.
Re: Seeing odd behavior with implicit routing
On Sun, May 14, 2017 at 7:40 PM, Chris Troullis <cptroul...@gmail.com> wrote: > Hi, > > I've been experimenting with various sharding strategies with Solr cloud > (6.5.1), and am seeing some odd behavior when using the implicit router. I > am probably either doing something wrong or misinterpreting what I am > seeing in the logs, but if someone could help clarify that would be awesome. > > I created a collection using the implicit router, created 10 shards, named > shard1, shard2, etc. I indexed 3000 documents to each shard, routed by > setting the _route_ field on the documents in my schema. All works fine, I > verified there are 3000 documents in each shard. > > The odd behavior I am seeing is when I try to route a query to a specific > shard. I submitted a simple query to shard1 using the request parameter > _route_=shard1. The query comes back fine, but when I looked in the logs, > it looked like it was issuing 3 separate requests: > > 1. The original query to shard1 > 2. A 2nd query to shard1 with the parameter ids=a bunch of document ids > 3. The original query to a random shard (changes every time I run the query) > > It looks like the first query is getting back a list of ids, and the 2nd > query is retrieving the documents for those ids? I assume this is some solr > cloud implementation detail. > > What I don't understand is the 3rd query. Why is it issuing the original > query to a random shard every time, when I am specifying the _route_? The > _route_ parameter is definitely doing something, because if I remove it, it > is querying all shards (which I would expect). > > Any ideas? I can provide the actual queries from the logs if required. How many nodes is this collection distributed across? I suspect that you are using a single node for experimentation? What happens with _route_=shard1 parameter and implicit routing is that the _route_ parameter is resolved to a list of replicas of shard1. But, SolrJ uses only the node name of the replica along with the collection name to make the request (this is important, we'll come back to this later). So, ordinarily, that node hosts a single shard (shard1) and when it receives the request, it will optimize the search to go the non-distributed code path (since the replica has all the data needed to satisfy the search). But interesting things happen when the node hosts more than one shard (say shard1 and shard3 both). When we query such a node using just the collection name, the collection name can be resolved to either shard1 or shard3 -- this is picked randomly without looking at _route_ parameter at all. If shard3 is picked, it looks at the request, sees that it doesn't have all the necessary data and decides to follow the two-phase distributed search path where phase 1 is to get the ids and score of the documents matching the query from all participating shards (the list of such shards is limited by _route_ parameter, which in our case will be only shard1) and a second phase where we get the actual stored fields to be returned to the user. So you get three queries in the log, 1) phase 1 of distributed search hitting shard1, 2) phase two of distributed search hitting shard1 and 3) the distributed scatter-gather search run by shard3. So to recap, this is happening because you have more than one shard1 hosted on a node. Easy workaround is to have each shard hosted on a unique node. But we can improve things on the solr side as well by 1) having SolrJ resolve requests down to node name and core name, 2) having the collection name to core name resolution take _route_ param into account. Both 1 and 2 can solve the problem. Can you please open a Jira issue? > > Thanks, > > Chris -- Regards, Shalin Shekhar Mangar.
Re: SPLITSHARD Working
No, split always happens on the original node. But you can move the sub-shard leader to a new node once the split is complete by using AddReplica/DeleteReplica collection API. On Mon, May 8, 2017 at 1:02 PM, vrindavda <vrinda...@gmail.com> wrote: > Hi, > > I need to SPLITSHARD such that one split remains on the same machine as > original and another uses new machines for leader and replicas. Is this > possible ? Please let me know what properties do I need to specify in > Collection API to achieve this. > > Thank you, > Vrinda Davda > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/SPLITSHARD-Working-tp4333876.html > Sent from the Solr - User mailing list archive at Nabble.com. -- Regards, Shalin Shekhar Mangar.
Re: logging in SolrCloud
I'm not a fan of auto-archiving myself and we definitely shouldn't be doing it before checking if an instance is running. Can you please open an issue? On Thu, May 4, 2017 at 12:52 PM, Bernd Fehling <bernd.fehl...@uni-bielefeld.de> wrote: > Hi Shalin, > > sounds like all or nothing method :-) > > How about a short check if an instance is still running > and using that log file before moving it to archived? > > Regards > Bernd > > Am 04.05.2017 um 09:07 schrieb Shalin Shekhar Mangar: >> Yes this is expected. On startup old console logs and gc logs are >> moved into the archived folder by default. This can be disabled by >> setting SOLR_LOG_PRESTART_ROTATION=false as a environment variable >> (search for its usage in bin/solr) but it will also disable all log >> rotation. >> >> On Wed, May 3, 2017 at 5:59 PM, Bernd Fehling >> <bernd.fehl...@uni-bielefeld.de> wrote: >>> While looking into SolrCloud I noticed that my logging >>> gets moved to archived dir by starting a new node. >>> >>> E.g.: >>> bin/solr start -cloud -p 8983 >>> -> server/logs/ has solr-8983-console.log >>> >>> bin/solr start -cloud -p 7574 >>> -> solr-8983-console.log is moved to server/logs/archived/ >>> -> server/logs/ has solr-7574-console.log >>> >>> Is this how it should be or do I have a misconfig? >>> >>> Regards >>> Bernd >> >> >> -- Regards, Shalin Shekhar Mangar.
Re: logging in SolrCloud
Yes this is expected. On startup old console logs and gc logs are moved into the archived folder by default. This can be disabled by setting SOLR_LOG_PRESTART_ROTATION=false as a environment variable (search for its usage in bin/solr) but it will also disable all log rotation. On Wed, May 3, 2017 at 5:59 PM, Bernd Fehling <bernd.fehl...@uni-bielefeld.de> wrote: > While looking into SolrCloud I noticed that my logging > gets moved to archived dir by starting a new node. > > E.g.: > bin/solr start -cloud -p 8983 > -> server/logs/ has solr-8983-console.log > > bin/solr start -cloud -p 7574 > -> solr-8983-console.log is moved to server/logs/archived/ > -> server/logs/ has solr-7574-console.log > > Is this how it should be or do I have a misconfig? > > Regards > Bernd -- Regards, Shalin Shekhar Mangar.
Re: Suggester uses lots of 'Page cache' memory
On Tue, May 2, 2017 at 10:22 AM, Damien Kamerman <dami...@gmail.com> wrote: > Hi all, > > I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each replica > uses about 14GB disk usage. I'm using Solaris 11 and I see the 'Page cache' > grow by about 7GB for each suggester replica I build. The suggester index > itself is very small. The 'Page cache' memory is freed when the node is > stopped. > > I guess the Suggester component is mmap'ing the entire Lucene index into > memory and holding it? Is this expected behavior? Is there a workaround? > Yes, this is expected. The suggester opens the index using Lucene's MMapDirectory which ends up memory mapping the index. But the memory mapped pages can be shared across everyone using the same index which basically means that the replica's usual index searcher can also use these pages. If you were not building the suggester index, even then the replica's search index would have been mmapped but perhaps only on demand instead of all at once. > I use this command to build the suggester for just the replica > 'target1_shard1_replica1': > curl " > http://localhost:8983/solr/collection1/suggest?suggest.dictionary=mySuggester=true=localhost:8983/solr/target1_shard1_replica1 > " > > BTW: Without the 'shards' param the distributed request will randomly hit > half the replicas. Yes, this is a problem. I recently opened https://issues.apache.org/jira/browse/SOLR-10532 but unfortunately I don't have the time to fix it soon. Patches are always welcome. > > From my solrconfig.xml: > > > mySuggester > AnalyzingInfixLookupFactory > mySuggester > DocumentDictionaryFactory > mySuggest > x > suggestTypeLc > false > > > > Cheers, > Damien. -- Regards, Shalin Shekhar Mangar.
Re: Index and query time suggester behavior in a SolrCloud environment
I also opened https://issues.apache.org/jira/browse/SOLR-10532 to fix this annoying and confusing behavior of SuggestComponent. On Thu, Apr 20, 2017 at 8:40 PM, Andrea Gazzarini <gxs...@gmail.com> wrote: > Ah great, many thanks again! > > > > On 20/04/17 17:09, Shalin Shekhar Mangar wrote: >> >> Hi Andrea, >> >> Looks like I have you some bad information. I looked at the code and >> ran a test locally. The suggest.build and suggest.reload params are in >> fact distributed across to all shards but only to one replica of each >> shard. This is still bad enough and you should use buildOnOptimize as >> suggested but I just wanted to correct the wrong information I gave >> earlier. >> >> On Thu, Apr 20, 2017 at 6:23 PM, Andrea Gazzarini <gxs...@gmail.com> >> wrote: >>> >>> Perfect, I don't need NRT at this moment so that fits perfectly >>> >>> Thanks, >>> Andrea >>> >>> >>> On 20/04/17 14:37, Shalin Shekhar Mangar wrote: >>>> >>>> Yeah, if it is just once a day then you can afford to do an optimize. >>>> For a more NRT indexing approach, I wouldn't recommend optimize at >>>> all. >>>> >>>> On Thu, Apr 20, 2017 at 5:29 PM, Andrea Gazzarini <gxs...@gmail.com> >>>> wrote: >>>>> >>>>> Ok, many thanks >>>>> >>>>> I see / read that it should be better to rely on the background merging >>>>> instead of issuing explicit optimizes, but I think in this case one >>>>> optimize >>>>> in a day it shouldn't be a problem. >>>>> >>>>> Did I get you correctly? >>>>> >>>>> Thanks again, >>>>> Andrea >>>>> >>>>> >>>>> On 20/04/17 13:17, Shalin Shekhar Mangar wrote: >>>>>> >>>>>> Can the client not send an optimize command explicitly after all >>>>>> indexing/deleting is complete? >>>>> >>>>> >>>> >> >> > -- Regards, Shalin Shekhar Mangar.
Re: Index and query time suggester behavior in a SolrCloud environment
Hi Andrea, Looks like I have you some bad information. I looked at the code and ran a test locally. The suggest.build and suggest.reload params are in fact distributed across to all shards but only to one replica of each shard. This is still bad enough and you should use buildOnOptimize as suggested but I just wanted to correct the wrong information I gave earlier. On Thu, Apr 20, 2017 at 6:23 PM, Andrea Gazzarini <gxs...@gmail.com> wrote: > Perfect, I don't need NRT at this moment so that fits perfectly > > Thanks, > Andrea > > > On 20/04/17 14:37, Shalin Shekhar Mangar wrote: >> >> Yeah, if it is just once a day then you can afford to do an optimize. >> For a more NRT indexing approach, I wouldn't recommend optimize at >> all. >> >> On Thu, Apr 20, 2017 at 5:29 PM, Andrea Gazzarini <gxs...@gmail.com> >> wrote: >>> >>> Ok, many thanks >>> >>> I see / read that it should be better to rely on the background merging >>> instead of issuing explicit optimizes, but I think in this case one >>> optimize >>> in a day it shouldn't be a problem. >>> >>> Did I get you correctly? >>> >>> Thanks again, >>> Andrea >>> >>> >>> On 20/04/17 13:17, Shalin Shekhar Mangar wrote: >>>> >>>> Can the client not send an optimize command explicitly after all >>>> indexing/deleting is complete? >>> >>> >> >> > -- Regards, Shalin Shekhar Mangar.
Re: Index and query time suggester behavior in a SolrCloud environment
Yeah, if it is just once a day then you can afford to do an optimize. For a more NRT indexing approach, I wouldn't recommend optimize at all. On Thu, Apr 20, 2017 at 5:29 PM, Andrea Gazzarini <gxs...@gmail.com> wrote: > Ok, many thanks > > I see / read that it should be better to rely on the background merging > instead of issuing explicit optimizes, but I think in this case one optimize > in a day it shouldn't be a problem. > > Did I get you correctly? > > Thanks again, > Andrea > > > On 20/04/17 13:17, Shalin Shekhar Mangar wrote: >> >> Can the client not send an optimize command explicitly after all >> indexing/deleting is complete? > > -- Regards, Shalin Shekhar Mangar.
Re: Index and query time suggester behavior in a SolrCloud environment
On Thu, Apr 20, 2017 at 4:27 PM, Andrea Gazzarini <gxs...@gmail.com> wrote: > Hi Shalin, > many thanks for your response. This is my scenario: > > * I build my index once in a day, it could be a delta or a full >re-index.In any case, that takes some time; > * I have an auto-commit (hard, no soft-commits) set to a given period >and during the indexing cycle, several hard commits are executed. So >the buildOnCommit (I guess) it's not an option because it will >rebuild that suggest index several times. Yes, you're right, multiple commits will cause the suggest index to be rebuilt needlessly. > > But I have a doubt on the second point: the reference guide says: > > /"Use buildOnCommit to rebuild the dictionary with every soft-commit"/ > > As I said, I have no soft-commits only hard-commits: does the rebuild happen > after hard commits (with buildOnCommit=true)? I peeked at the code and yes, actually the rebuild happens whenever a new searcher is created which means that it happens on soft-commits or on a hard commit with openSearcher=true. > > The other option, buildOnOptimize, makes me curious: in the scenario above, > let's say documents are indexed / deleted every morning at 4am, in a window > that takes 1 max 3 hours, how can I build the suggest index (more or less) > just after that window? I'm ok if the build happens after a reasonable delay > (e.g. 1, max 2 hours) Can the client not send an optimize command explicitly after all indexing/deleting is complete? > > Many thanks, > Andrea > > > > On 20/04/17 11:11, Shalin Shekhar Mangar wrote: >> >> Comments inline: >> >> >> On Wed, Apr 19, 2017 at 2:46 PM, Andrea Gazzarini <gxs...@gmail.com> >> wrote: >>> >>> Hi, >>> any help out there? >>> >>> BTW I forgot the Solr version: 6.5.0 >>> >>> Thanks, >>> Andrea >>> >>> >>> On 18/04/17 11:45, Andrea Gazzarini wrote: >>>> >>>> Hi, >>>> I have a project, with SolrCloud, where I'm going to use the Suggester >>>> component (BlendedInfixLookupFactory with DocumentDictionaryFactory). >>>> Some info: >>>> >>>>* I will have a suggest-only collection, with no NRT requirements >>>> (indexes will be updated with a daily frequency) >>>>* I'm not yet sure about the replication factor (I have to do some >>>> checks) >>>>* I'm using Solrj on the client side >>>> >>>> After reading some documentation I have a couple of doubts: >>>> >>>>* how the *suggest.build* command is working? Can I issue this >>>> command towards just one node, and have that node forward the >>>> request to the other nodes (so each of them can build its own >>>> suggester index portion)? >> >> The suggest.build only builds locally in the node to which you sent >> the request. This makes it a bit tricky because if you send that >> command with just the collection name, it will be resolved to a local >> core and executed there. The safest/easiest way is to set >> buildOnCommit or buildOnOptimize in the suggester configuration. >> >>>>* how things are working at query time? Can I use send a request >>>> with only suggest.q=... to my /suggest request handler and get >>>> back distributed suggestions? >> >> The SuggestComponent works in distributed mode and it will request and >> merge results from all shards. >> >>>> Thanks in advance >>>> Andrea >>> >>> >> >> > -- Regards, Shalin Shekhar Mangar.
Re: Index and query time suggester behavior in a SolrCloud environment
Comments inline: On Wed, Apr 19, 2017 at 2:46 PM, Andrea Gazzarini <gxs...@gmail.com> wrote: > Hi, > any help out there? > > BTW I forgot the Solr version: 6.5.0 > > Thanks, > Andrea > > > On 18/04/17 11:45, Andrea Gazzarini wrote: >> >> Hi, >> I have a project, with SolrCloud, where I'm going to use the Suggester >> component (BlendedInfixLookupFactory with DocumentDictionaryFactory). >> Some info: >> >> * I will have a suggest-only collection, with no NRT requirements >> (indexes will be updated with a daily frequency) >> * I'm not yet sure about the replication factor (I have to do some >> checks) >> * I'm using Solrj on the client side >> >> After reading some documentation I have a couple of doubts: >> >> * how the *suggest.build* command is working? Can I issue this >> command towards just one node, and have that node forward the >> request to the other nodes (so each of them can build its own >> suggester index portion)? The suggest.build only builds locally in the node to which you sent the request. This makes it a bit tricky because if you send that command with just the collection name, it will be resolved to a local core and executed there. The safest/easiest way is to set buildOnCommit or buildOnOptimize in the suggester configuration. >> * how things are working at query time? Can I use send a request >> with only suggest.q=... to my /suggest request handler and get >> back distributed suggestions? The SuggestComponent works in distributed mode and it will request and merge results from all shards. >> >> Thanks in advance >> Andrea > > -- Regards, Shalin Shekhar Mangar.
Re: Solr 6.x leaking one SolrZkClient instance per second?
Please open a Jira issue. Thanks! On Tue, Apr 4, 2017 at 7:16 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hi, > > One of our nodes became berzerk after a restart, Solr went completely nuts! > So i opened VisualVM to keep an eye on it and spotted a different problem > that occurs in all our Solr 6.4.2 and 6.5.0 nodes. > > It appears Solr is leaking one SolrZkClient instance per second via > DistributedQueue$ChildWatcher. That one per second is quite accurate for all > nodes, there are about the same amount of instances as there are seconds > since Solr started. I know VisualVM's instance count includes > objects-to-be-collected, the instance count does not drop after a forced > garbed collection round. > > It doesn't matter how many cores or collections the nodes carry or how heavy > traffic is. > > Is this a known issue? Ticket? > > Thanks, > Markus -- Regards, Shalin Shekhar Mangar.
Re: Indexing CPU performance
According to the profiler output, a significant amount of cpu is being spent in JSON parsing but your previous email said that you use SolrJ. SolrJ uses the javabin binary format to send documents to Solr and it never ever uses JSON so there is definitely some other indexing process that you have not accounted for. On Tue, Mar 14, 2017 at 12:31 AM, Mahmoud Almokadem <prog.mahm...@gmail.com> wrote: > Thanks Erick, > > I've commented out the line SolrClient.add(doclist) and get 5500+ docs per > second from single producer. > > Regarding more shards, you mean use 2 nodes with 8 shards per node so we > got 16 shards on the same 2 nodes or spread shards over more nodes? > > I'm using solr 6.4.1 with zookeeper on the same nodes. > > Here's what I got from sematext profiler > > 51% > Thread.java:745java.lang.Thread#run > > 42% > QueuedThreadPool.java:589 > org.eclipse.jetty.util.thread.QueuedThreadPool$2#run > Collapsed 29 calls (Expand) > > 43% > UpdateRequestHandler.java:97 > org.apache.solr.handler.UpdateRequestHandler$1#load > > 30% > JsonLoader.java:78org.apache.solr.handler.loader.JsonLoader#load > > 30% > JsonLoader.java:115 > org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader#load > > 13% > JavabinLoader.java:54org.apache.solr.handler.loader.JavabinLoader#load > > 9% > ThreadPoolExecutor.java:617 > java.util.concurrent.ThreadPoolExecutor$Worker#run > > 9% > ThreadPoolExecutor.java:1142 > java.util.concurrent.ThreadPoolExecutor#runWorker > > 33% > ConcurrentMergeScheduler.java:626 > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread#run > > 33% > ConcurrentMergeScheduler.java:588 > org.apache.lucene.index.ConcurrentMergeScheduler#doMerge > > 33% > SolrIndexWriter.java:233org.apache.solr.update.SolrIndexWriter#merge > > 33% > IndexWriter.java:3920org.apache.lucene.index.IndexWriter#merge > > 33% > IndexWriter.java:4343org.apache.lucene.index.IndexWriter#mergeMiddle > > 20% > SegmentMerger.java:101org.apache.lucene.index.SegmentMerger#merge > > 11% > SegmentMerger.java:89org.apache.lucene.index.SegmentMerger#merge > > 2% > SegmentMerger.java:144org.apache.lucene.index.SegmentMerger#merge > > > On Mon, Mar 13, 2017 at 5:12 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > >> Note that 70,000 docs/second pretty much guarantees that there are >> multiple shards. Lots of shards. >> >> But since you're using SolrJ, the very first thing I'd try would be >> to comment out the SolrClient.add(doclist) call so you're doing >> everything _except_ send the docs to Solr. That'll tell you whether >> there's any bottleneck on getting the docs from the system of record. >> The fact that you're pegging the CPUs argues that you are feeding Solr >> as fast as Solr can go so this is just a sanity check. But it's >> simple/fast. >> >> As far as what on Solr could be the bottleneck, no real way to know >> without profiling. But 300+ fields per doc probably just means you're >> doing a lot of processing, I'm not particularly hopeful you'll be able >> to speed things up without either more shards or simplifying your >> schema. >> >> Best, >> Erick >> >> On Mon, Mar 13, 2017 at 6:58 AM, Mahmoud Almokadem >> <prog.mahm...@gmail.com> wrote: >> > Hi great community, >> > >> > I have a SolrCloud with the following configuration: >> > >> >- 2 nodes (r3.2xlarge 61GB RAM) >> >- 4 shards. >> >- The producer can produce 13,000+ docs per second >> >- The schema contains about 300+ fields and the document size is about >> >3KB. >> >- Using SolrJ and SolrCloudClient, each batch to solr contains 500 >> docs. >> > >> > When I start my bulk indexer program the CPU utilization is 100% on each >> > server but the rate of the indexer is about 1500 docs per second. >> > >> > I know that some solr benchmarks reached 70,000+ doc. per second. >> > >> > The question: What is the best way to determine the bottleneck on solr >> > indexing rate? >> > >> > Thanks, >> > Mahmoud >> -- Regards, Shalin Shekhar Mangar.
Re: Replicas fail immediately in new collection
SOLR-9739 changed the writeStr method to accept a CharSequence from String in 6.4 so my guess is that your classpath has a newer (6.4+) solrj version but an older solr-core jar that cannot find this new method. On Sat, Feb 18, 2017 at 5:16 AM, Walter Underwood <walter.r.underw...@gmail.com> wrote: > Any idea why I would be getting this on a brand new, empty collection on the > first update? > > HTTP ERROR 500 > Problem accessing /solr/tutors_shard1_replica9/update. Reason: > Server ErrorCaused > by:java.lang.NoSuchMethodError: > org.apache.solr.update.TransactionLog$LogCodec.writeStr(Ljava/lang/String;)V > at > org.apache.solr.update.TransactionLog.writeCommit(TransactionLog.java:457) > > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > -- Regards, Shalin Shekhar Mangar.
Re: ClusterStateMutator
Actually the state format defaults to 2 since many releases (all of 6.x at least). This default is enforced in CollectionsHandler much before the code in ClusterStateMutator is executed. On Wed, Jan 4, 2017 at 6:16 PM, Hendrik Haddorp <hendrik.hadd...@gmx.net> wrote: > Hi, > > in > solr-6.3.0/solr/core/src/java/org/apache/solr/cloud/overseer/ClusterStateMutator.java > there is the following code starting line 107: > > //TODO default to 2; but need to debug why BasicDistributedZk2Test fails > early on > String znode = message.getInt(DocCollection.STATE_FORMAT, 1) == 1 ? null > : ZkStateReader.getCollectionPath(cName); > > Any if that will be changed to default to version 2 anytime soon? > > thanks, > Hendrik -- Regards, Shalin Shekhar Mangar.
Re: CDCR: Help With Tlog Growth Issues
Even if buffer is enabled, the old tlogs should be remove once the updates in those tlogs have been replicated to the target. So the real question is why they haven't been removed automatically? On Thu, Dec 1, 2016 at 9:13 PM, Renaud Delbru <renaud@siren.solutions> wrote: > Hi Thomas, > > Looks like the buffer is enabled on the update log, and even if the updates > were replicated, they are not removed. > > What is the output of the command `cdcr?action=STATUS` on both cluster ? > > If you see in the response `enabled`, then the buffer > is enabled. > To disable it, you should run the command `/cdcr?action=DISABLEBUFFER`. > > Kind Regards > -- > Renaud Delbru > > On 10/11/2016 23:09, Thomas Tickle wrote: >> >> >> I am having an issue with cdcr that I could use some assistance in >> resolving. >> >> I followed the instructions found here: >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462 >> >> The CDCR is setup with a single source to a single target. Both the >> source and target cluster are identically setup as 3 machines, each running >> an external zookeeper and a solr instance. I’ve enabled the data >> replication and successfully seen the documents replicated from the source >> to the target with no errors in the log files. >> >> However, when examining the /cdcr?action=QUEUES command, I noticed that >> the tlogTotalSize and tlogTotalCount were alarmingly high. Checking the >> data directory for each shard, I was able to confirm that there was several >> thousand logs files of each 3-4 megs. It added up to almost 35 GBs of >> tlogs. Obviously, this amount of tlogs causes a serious issue when trying >> to restart a solr server after activities such as patch. >> >> *Is it normal for old tlogs to never get removed in a CDCR setup?* >> >> ** >> >> Thomas Tickle >> >> >> >> Nothing in this message is intended to constitute an electronic signature >> unless a specific statement to the contrary is included in this message. >> >> Confidentiality Note: This message is intended only for the person or >> entity to which it is addressed. It may contain confidential and/or >> privileged material. Any review, transmission, dissemination or other use, >> or taking of any action in reliance upon this message by persons or entities >> other than the intended recipient is prohibited and may be unlawful. If you >> received this message in error, please contact the sender and delete it from >> your computer. > > -- Regards, Shalin Shekhar Mangar.
Re: Using Solr CDCR with HdfsDirectoryFactory
At present, CDCR does not work with HDFS. The update log implementation is
different and incompatible at the moment. Please open a jira issue to
support cdcr under hdfs -- patches are always welcome!
On Tue, Nov 29, 2016 at 8:36 PM, ZHOU Ran (SAFRAN IDENTITY AND SECURITY) <
ran.z...@safrangroup.com> wrote:
> Hello all,
>
>
>
> could someone help on this?
>
>
>
> Best Regards
>
> Ran
>
>
>
> *Von:* ZHOU Ran (SAFRAN IDENTITY AND SECURITY)
> *Gesendet:* Freitag, 25. November 2016 15:37
> *An:* 'solr-user@lucene.apache.org' <solr-user@lucene.apache.org>
> *Betreff:* Using Solr CDCR with HdfsDirectoryFactory
>
>
>
> Hello
>
>
>
>
>
> Hi All,
>
>
>
> I have followed the guide „Cross Data Center Replication (CDCR)“ and get
> my source collection replicated to the target. And then I tried to use HDFS
> as storage for both Solr clusters, but failed with the following error
> message:
>
>
>
> ERROR: Failed to create collection 'collection11' due to:
> {192.168.5.95:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> from server at http://192.168.5.95:8983/solr: Error CREATEing SolrCore
> 'collection11_shard1_replica1': Unable to create core
> [collection11_shard1_replica1] Caused by: Solr instance is not configured
> with the cdcr update log.}
>
>
>
> Actually Solr with HDFS works for me. In the configuration for CDCR, there
> is one block:
>
>
>
>
>
>
>
> ${solr.ulog.dir:}
>
>
>
>
>
>
>
>
>
>
>
> And I know that if HdfsDirectoryFactory is used, then updateHandler will
> initialized updateLog with the class HdfsUpdateLog. Is this the problem
> that CDCR does not work with HDFS? Since the updateLog cannot be
> initialized with CdcrUpdateLog?
>
>
>
> Thanks in advance for your help!
>
>
>
> Best Regards
>
>
>
> *Ran ZHOU*
> Software Engineer
>
>
> *T* +49 (0) 234 97 87 59 <+49%20234%20978759>
>
> *E* ran.z...@safrangroup.com *NEW*
>
>
>
> L-1 Identity Solutions AG
>
> Universitätsstrasse 160 I 44801 BOCHUM - GERMANY
> *www.safran-identity-security.com*
> <http://www.safran-identity-security.com/?utm_source=signmail_medium=mailpro_campaign=signmailEN>
>
>
> <http://www.safran-identity-security.com/en/stay-in-touch-with-us?utm_source=subscribemail_medium=mailpro_campaign=subscribemailE>
> <http://www.safran-identity-security.com/en/stay-in-touch-with-us?utm_source=subscribemail_medium=mailpro_campaign=subscribemailE>
>
> <https://twitter.com/SafranIDSec>
>
> <https://www.youtube.com/user/SafranMorpho>
>
> <https://www.linkedin.com/company/safran-identity-&-security/>
>
>
>
>
> *Managing Board:* Dr. Martin Werner (Vors. / Chairman) I Christèle Jacqz
> I Francois Rieul* Supervisory Board: *Jean-Christophe Fondeur (Vors. /
> Chairman)
>
> *Register Court:* Amtsgericht Bochum I HRB 69 54 | *UST-ID / VAT ID:* DE
> 813124378
>
>
>
> #
> " Ce courriel et les documents qui lui sont joints peuvent contenir des
> informations confidentielles, être soumis aux règlementations relatives au
> contrôle des exportations ou ayant un caractère privé. S'ils ne vous sont
> pas destinés, nous vous signalons qu'il est strictement interdit de les
> divulguer, de les reproduire ou d'en utiliser de quelque manière que ce
> soit le contenu. Toute exportation ou réexportation non autorisée est
> interdite.Si ce message vous a été transmis par erreur, merci d'en informer
> l'expéditeur et de supprimer immédiatement de votre système informatique ce
> courriel ainsi que tous les documents qui y sont attachés."
> **
> " This e-mail and any attached documents may contain confidential or
> proprietary information and may be subject to export control laws and
> regulations. If you are not the intended recipient, you are notified that
> any dissemination, copying of this e-mail and any attachments thereto or
> use of their contents by any means whatsoever is strictly prohibited.
> Unauthorized export or re-export is prohibited. If you have received this
> e-mail in error, please advise the sender immediately and delete this
> e-mail and all attached documents from your computer system."
> #
>
--
Regards,
Shalin Shekhar Mangar.
Re: Documentation of Zookeeper's specific roles and functions in Solr Cloud?
On Tue, Nov 29, 2016 at 10:22 PM, Kevin Risden <compuwizard...@gmail.com> wrote: > If using CloudSolrClient or another zookeeper aware client, then a request > gets sent to Zookeeper to determine the live nodes. If indexing, > CloudSolrClient can find the leader and send documents directly there. The > client then uses that information to query the correct nodes directly. The live node information is watched and cached so it is re-fetched only when it changes. There is no direct request to ZooKeeper on either indexing or querying unless our cached live nodes and cluster state information has become stale. > > Zookeeper is not forwarding requests to Solr. The client requests from > Zookeeper and then the client uses that information to query Solr directly. > > Kevin Risden > > On Tue, Nov 29, 2016 at 10:49 AM, John Bickerstaff <j...@johnbickerstaff.com >> wrote: > >> All, >> >> I've thought I understood that Solr search requests are made to the Solr >> servers and NOT Zookeeper directly. (I.E. Zookeeper doesn't decide which >> Solr server responds to requests and requests are made directly to Solr) >> >> My new place tells me they're sending requests to Zookeeper - and those are >> getting sent on to Solr by Zookeeper - -- this is news to me if it's >> true... >> >> Is there any documentation of exactly the role(s) played by Zookeeper in a >> SolrCloud setup? >> -- Regards, Shalin Shekhar Mangar.
Re: Break up a supplier's documents (products) from dominating search result.
There is a related work done in Lucene land which hasn't been exposed in Solr yet. It is called DiversifiedTopDocsCollector. See https://issues.apache.org/jira/browse/LUCENE-6066 On Mon, Nov 28, 2016 at 2:39 PM, Derek Poh <d...@globalsources.com> wrote: > Hi > > We have a business requirement to breakupa supplier's products from > dominating search resultso as to allow othersuppliers' products in the > search result to have exposure. > Business users are open to implementing this for the first page of the > search resultif it is not possible to apply tothe entire search result. > > From the sample keywords users have provided, I also discovered thatmost of > the time a supplier's products that are listed consecutively in the result > all have the same score. > > Any advice/suggestions on how I cando it? > > Please let me know if more information is require. Thank you. > > Derek > > -- > CONFIDENTIALITY NOTICE > This e-mail (including any attachments) may contain confidential and/or > privileged information. If you are not the intended recipient or have > received this e-mail in error, please inform the sender immediately and > delete this e-mail (including any attachments) from your computer, and you > must not use, disclose to anyone else or copy this e-mail (including any > attachments), whether in whole or in part. > This e-mail and any reply to it may be monitored for security, legal, > regulatory compliance and/or other appropriate reasons. -- Regards, Shalin Shekhar Mangar.
[ANNOUNCE] Apache Solr 6.3.0 released
8 November 2016, Apache Solr 6.3.0 available Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search and analytics, rich document parsing, geospatial search, extensive REST APIs as well as parallel SQL. Solr is enterprise grade, secure and highly scalable, providing fault tolerant distributed search and indexing, and powers the search and navigation features of many of the world's largest internet sites. Solr 6.3.0 is available for immediate download at: . http://lucene.apache.org/solr/mirrors-solr-latest-redir.html Please read CHANGES.txt for a full list of new features and changes: . https://lucene.apache.org/solr/6_3_0/changes/Changes.html Solr 6.3 Release Highlights: DocValues, streaming, /export, machine learning: * Optimize, store and deploy AI models in Solr * Ability to add custom streaming expressions * New streaming expressions such as "fetch", "executor", and "commit" added. * Parallel SQL accepts <, >, =, etc., symbols. * Support facet scoring with the scoreNodes expression * Retrieving docValues as stored values was sped up by using the proper leaf reader rather than ask for a global view. In extreme cases, this leads to a 100x speedup. Faceting: * facet.method=enum can bypass exact counts calculation with facet.exists=true, it just returns 1 for terms which exists in result docset * Add "overrequest" parameter to JSON Facet API to control amount of overrequest on a distributed terms facet Logging: * You can now set Solr's log level through environment variable SOLR_LOG_LEVEL * GC logs are rotated by JVM to a max of 9 files, and backed up via bin/solr scripts * Solr's logging verbosity at the INFO level has been greatly reduced by moving much logging to DEBUG level * The solr-8983-console.log file now only logs STDOUT and STDERR output, not all log4j logs as before * Solr's main log file, solr.log, is now written to SOLR_LOGS_DIR without changing log4j.properties Start scripts: * Allow 180 seconds for shutdown before killing solr (configurable, old limit 5s) (Unix only) * Start scripts now exits with informative message if using wrong Java version * Fixed "bin/solr.cmd zk upconfig" command which was broken on windows * You can now ask for DEBUG logging simply with '-v' option, and for WARN logging with '-q' option SolrCloud: * The DELETEREPLICA API can accept a 'count' parameter and remove "count" number of replicas from each shard if the shard name is not provided * The config API shows expanded useParams for request handlers inline * Ability to create/delete/list snapshots at collection level * The modify collection API now waits for the modified properties to show up in the cluster state before returning * Many bug fixes related to SolrCloud recovery for data safety and faster recovery times. Security: * SolrJ now supports Kerberos delegation tokens * Pooled SSL connections were not being re-used. This is now fixed. * Fix for the blockUnknown property which made inter-node communication impossible * Support SOLR_AUTHENTICATION_OPTS and SOLR_AUTHENTICATION_CLIENT_CONFIGURER in windows bin/solr.cmd script * New parameter -u in bin/post to pass basicauth credentials Misc changes: * Optimizations to lower memory allocations when indexing JSON as well as for replication between solr cloud nodes. * A new Excel workbook (.xlsx) response writer has been added. Use 'wt=xlsx' request parameter on a query request to enable. Further details of changes are available in the change log available at: http://lucene.apache.org/solr/6_3_0/changes/Changes.html Please report any feedback to the mailing lists (http://lucene.apache.org/solr/discussion.html) Note: The Apache Software Foundation uses an extensive mirroring network for distributing releases. It is possible that the mirror you are using may not have replicated the release yet. If that is the case, please try another mirror. This also applies to Maven access. -- Regards, Shalin Shekhar Mangar.
Re: 'solr zk upconfig' etc not working on windows since 6.1 at least
Would you mind opening a jira issue and give a patch (diff)? 6.3 is coming out soon and we'd have to hurry if this fix has to go in. On Thu, Oct 27, 2016 at 6:32 PM, xavier jmlucjav <jmluc...@gmail.com> wrote: > Correcting myself here, I was wrong about the cause (I had already messed > with the script). > > I made it work by commenting out line 1261 (the number might be a bit off > as I have modified the script, but hopefully its easy to see where): > > ) ELSE IF "%1"=="/?" ( > goto zk_usage > ) ELSE IF "%1"=="-h" ( > goto zk_usage > ) ELSE IF "%1"=="-help" ( > goto zk_usage > ) ELSE IF "!ZK_SRC!"=="" ( > if not "%~1"=="" ( > goto set_zk_src > ) > * rem goto zk_usage* > ) ELSE IF "!ZK_DST!"=="" ( > IF "%ZK_OP%"=="cp" ( > goto set_zk_dst > ) > IF "%ZK_OP%"=="mv" ( > goto set_zk_dst > ) > set ZK_DST="_" > ) ELSE IF NOT "%1"=="" ( > set ERROR_MSG="Unrecognized or misplaced zk argument %1%" > > Now upconfig works! > > thanks > xavier > > > On Thu, Oct 27, 2016 at 2:43 PM, xavier jmlucjav <jmluc...@gmail.com> > wrote: > > > hi, > > > > Am I missing something or this is broken in windows? I cannot upconfig, > > the scripts keeps exiting immediately and showing usage, as if I use some > > wrong parameters. This is on win10, jdk8. But I am pretty sure I saw it > > also on win7 (don't have that around anymore to try) > > > > I think the issue is: there is a SHIFT too much in line 1276 of solr.cmd: > > > > :set_zk_op > > set ZK_OP=%~1 > > SHIFT > > goto parse_zk_args > > > > if this SHIFT is removed, then parse_zk_args works (and it does the shift > > itself). But the upconfig hangs, so still it does not work. > > > > this probably was introduced in a851d5f557aefd76c01ac23da076a14dc7576d8e > > by Erick (not sure which one :) ) on July 2nd. Master still has this > issue. > > Would be great if this was fixed in the incoming 6.3... > > > > My cmd scripting is not too strong and I did not go further. I searched > > Jira but found nothing. By the way is it not possible to open tickets in > > Jira anymore? > > > > xavier > > > -- Regards, Shalin Shekhar Mangar.
Re: solr 5 leaving tomcat, will I be the only one fearing about this?
As far as configuration is concerned -- everything Solr specific goes to zookeeper. The solr.xml can also be put into zookeeper. JNDI is not supported -- can you explain why you need it? Can cluster properties solve the problem? or replica properties? Both of those can go into zookeeper. Integration testing is possible using MiniSolrCloudCluster which is built to be used programatically. It starts a real zookeeper instance along with real solr instances on jetties. Most other things are by design e.g. whole cluster changes on configuration change otherwise sneaky bugs where one node misses a config change are easily possible and very very difficult to debug. But you can e.g. create a new collection with just one node, stage your changes, run health checks and update the configuration of the main collection. The old master/slave setup was very simple. The cloud is a whole new ball game. Having control of jetty has given Solr a lot of flexibility. At this time I discourage anyone from changing anything inside Jetty's config. If there are certain things that are not possible without then please let us know so figure out how to build such things in Solr itself. We want jetty to be a implementation detail and no more. If you have suggestions on how we can fix some of these problems, please speak up. On Sun, Oct 9, 2016 at 5:41 AM, Aristedes Maniatis <a...@maniatis.org> wrote: > On 9/10/16 2:09am, Shawn Heisey wrote: > > One of the historical challenges on this mailing list is that we were > > rarely aware of what steps the user had taken to install or start Solr, > > and we had to support pretty much any scenario. Since 5.0, the number > > of supported ways to deploy and start Solr is greatly reduced, and those > > ways were written by the project, so we tend to have a better > > understanding of what is happening when a user starts Solr. We also > > usually know the relative location of the logfiles and Solr's data. > > > This migration is causing a lot of grief for us as well, and we are still > struggling to get all the bits in place. Before: > > * gradle build script > * gradle project includes our own unit tests, run in jenkins > * generates war file > * relevant configuration is embedded into the build > * deployment specific variables (db uris, passwords, ip addresses) > conveniently contained in one context.xml file > > > Now: > > * Solr version is no longer bound to our tests or configuration > > * configuration is now scattered in three places: > - zookeeper > - solr.xml in the data directory > - jetty files as part of the solr install that you need to replace (for > example to set JNDI properties) > > * deployment is also scattered: > - Solr platform specific package manager (pkg in FreeBSD in my case, > which I've had to write myself since it didn't exist) > - updating config files above > - writing custom scripts to push Zookeeper configuration into production > - creating collections/cores using the API rather than in a config file > > * unit testing no longer possible since you can't run a mock zookeeper > instance > > * zookeeper is very hard to integrate with deployment processes (salt, > puppet, etc) since configuration is no longer a set of version controlled > files > > * you can't change the configuration of one node as a 'soft deployment': > the whole cluster needs to be changed at once > > If we didn't need a less broken replication solution, I'd stay on Solr4 > forever. > > > I really liked the old war deployment. It bound the solr version and > configuration management into our version controlled source repository > except for one context.xml file that contained server specific deployment > options. Nice. > > The new arrangement is a mess. > > > Ari > > > > -- > --> > Aristedes Maniatis > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A > -- Regards, Shalin Shekhar Mangar.
Re: Results not ordered by score and debug info is incorrect, crazy
Wow, it took me some time to realize what you were referring to :-) The manual (or the reference guide page) is under construction and I didn't finish that page. The sentence about faceting is just a copy/paste relic :) I'll go fix that asap. I remember a bunch of Jira issues w.r.t NPE and distrib.singlePass which have since been fixed. If you find something else, then please open a issue. The distrib.singlePass approach reads all the stored fields from all shards for all documents instead of just the top N docs. This can be advantageous if you are returning a small number of fields but may cause performance degradation if you are requesting all or a lot of fields. But coming back to this bug, it seems like debugQuery=true is behaving incorrectly for the normal two phase search? Please open a jira issue. This warrants more investigation. On Tue, Sep 27, 2016 at 6:26 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Shalin, that does the trick indeed! > > Also noted that the manual is incorrect, setting it to a blank or false > value, did not disable my facets: > If set to "true," this parameter enables facet counts in the query > response. If set to "false" to a blank or missing value, this parameter > disables faceting. None of the other parameters listed below will have any > effect unless this parameter is set to "true." The default value is blank. > > In the past, i do remember some discussions about this parameter, and > vaguely recall there were some drawbacks for this parameters, so i didn't > enable it. > > The manual is not clear about this though, any recommendations? > > Thanks, > Markus > > -Original message- > > From:Shalin Shekhar Mangar <shalinman...@gmail.com> > > Sent: Tuesday 27th September 2016 14:41 > > To: solr-user@lucene.apache.org > > Subject: Re: Results not ordered by score and debug info is incorrect, > crazy > > > > This may be relevant or not, I am not sure but one difference between > > fl=title_nl,score,id and fl=score,id is that the former executes a two > pass > > distributed search i.e. get ids, merge, get fields for top N docs but the > > latter skips the "get fields" phase because it already has all the right > > information. > > > > You can force the single pass for the first request as well by adding > > distrib.singlePass=true as a request parameter. It might be interesting > to > > get that output as well and compare it with the others. > > > > On Tue, Sep 27, 2016 at 5:39 PM, Markus Jelsma < > markus.jel...@openindex.io> > > wrote: > > > > > Hi, > > > > > > I just spotted something weird, again. A regular search popped up a > weird > > > candidate for first result, so i've reproduced it on our production > system. > > > > > > Digging deeper, it appears that the fl parameter has something to do > with > > > it. Not the order of results but the scores / explain in the debug > section > > > are different. This means i am not sure in what case the debugging > > > information is really correcty. > > > > > > Check this out. These are the top four results with > > > fl=title_nl,score,id=false=false=true& > > > wt=xml=true=misofonie=4 > > > > > > > > > > > > http://www.gezondheidenco.nl/179901/ > > > website-over-misofonie-online/ > > > Website over misofonie online > > > 153.99518 > > > > > > http://www.gezondheidenco.nl/tag/misofonie/ > > > misofonie Archives - Gezondheid Co > > > 149.88002 > > > > > > http://www.gezondheidenco.nl/263802/ > > > misofonie-psychische-aandoening-is-behandelen/ > > > Misofonie: een psychische aandoening. Is het > te > > > behandelen? | Gezondheid Co > > > 147.82176 > > > > > > http://www.gezondheidenco.nl/tag/ > > > ereniging-misofonie-nl/ > > > ereniging Misofonie NL Archives - Gezondheid > > > Co > > > 144.10309 > > > > > > > > > And here's the truncated explain section for above request, notice > that it > > > is not ordered correctly and the scores are different than above: > > > > > > http://www.gezondheidenco.nl/179901/ > > > website-over-misofonie-online/"> > > > 148.65146 = sum of: > > > 148.65146 = max plus 0.65 times others of: > > > http://www.gezondheidenco.nl/tag/misofonie/;> > > > 144.63885 = sum of: > > > 144.63885 = max plus 0.65 times others o
Re: Results not ordered by score and debug info is incorrect, crazy
This may be relevant or not, I am not sure but one difference between fl=title_nl,score,id and fl=score,id is that the former executes a two pass distributed search i.e. get ids, merge, get fields for top N docs but the latter skips the "get fields" phase because it already has all the right information. You can force the single pass for the first request as well by adding distrib.singlePass=true as a request parameter. It might be interesting to get that output as well and compare it with the others. On Tue, Sep 27, 2016 at 5:39 PM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Hi, > > I just spotted something weird, again. A regular search popped up a weird > candidate for first result, so i've reproduced it on our production system. > > Digging deeper, it appears that the fl parameter has something to do with > it. Not the order of results but the scores / explain in the debug section > are different. This means i am not sure in what case the debugging > information is really correcty. > > Check this out. These are the top four results with > fl=title_nl,score,id=false=false=true& > wt=xml=true=misofonie=4 > > > > http://www.gezondheidenco.nl/179901/ > website-over-misofonie-online/ > Website over misofonie online > 153.99518 > > http://www.gezondheidenco.nl/tag/misofonie/ > misofonie Archives - Gezondheid Co > 149.88002 > > http://www.gezondheidenco.nl/263802/ > misofonie-psychische-aandoening-is-behandelen/ > Misofonie: een psychische aandoening. Is het te > behandelen? | Gezondheid Co > 147.82176 > > http://www.gezondheidenco.nl/tag/ > ereniging-misofonie-nl/ > ereniging Misofonie NL Archives - Gezondheid > Co > 144.10309 > > > And here's the truncated explain section for above request, notice that it > is not ordered correctly and the scores are different than above: > > http://www.gezondheidenco.nl/179901/ > website-over-misofonie-online/"> > 148.65146 = sum of: > 148.65146 = max plus 0.65 times others of: > http://www.gezondheidenco.nl/tag/misofonie/;> > 144.63885 = sum of: > 144.63885 = max plus 0.65 times others of: > http://www.gezondheidenco.nl/263802/misofonie-psychische- > aandoening-is-behandelen/"> > 152.7493 = sum of: > 152.7493 = max plus 0.65 times others of: > http://www.gezondheidenco.nl/tag/ereniging-misofonie-nl/;> > 139.05501 = sum of: > 139.05501 = max plus 0.65 times others of: > > Here's the other request with only the fl-parameter having a different > value: fl=score,id=false=false=true=xml& > indent=true=misofonie=4 > > > > http://www.gezondheidenco.nl/179901/ > website-over-misofonie-online/ > 153.99518 > > http://www.gezondheidenco.nl/tag/misofonie/ > 149.88002 > > http://www.gezondheidenco.nl/263802/ > misofonie-psychische-aandoening-is-behandelen/ > 147.82176 > > http://www.gezondheidenco.nl/tag/ > ereniging-misofonie-nl/ > 144.10309 > > > And here are the scores for this query, notice these are ordered > descending, the ones above are not, also notice the scores are identical to > that of the result set itself. > > http://www.gezondheidenco.nl/179901/ > website-over-misofonie-online/"> > 153.9952 = sum of: > 153.9952 = max plus 0.65 times others of: > http://www.gezondheidenco.nl/tag/misofonie/;> > 149.88002 = sum of: > 149.88002 = max plus 0.65 times others of: > http://www.gezondheidenco.nl/263802/misofonie-psychische- > aandoening-is-behandelen/"> > 147.82175 = sum of: > 147.82175 = max plus 0.65 times others of: > http://www.gezondheidenco.nl/tag/ereniging-misofonie-nl/;> > 144.10309 = sum of: > 144.10309 = max plus 0.65 times others of: > > I am completely surprised to see this crazy behaviour. But i vaguely > remeber something about this before, but cannot find references. > > Any idea? > > Thanks, > Markus > -- Regards, Shalin Shekhar Mangar.
[ANNOUNCE] Apache Solr 6.2.1 released
20 September 2016, Apache Solr™ 6.2.1 available The Lucene PMC is pleased to announce the release of Apache Solr 6.2.1 Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search and analytics, rich document parsing, geospatial search, extensive REST APIs as well as parallel SQL. Solr is enterprise grade, secure and highly scalable, providing fault tolerant distributed search and indexing, and powers the search and navigation features of many of the world's largest internet sites. This release includes 11 bug fixes since the 6.2.0 release. Some of the major fixes are: - SOLR-9490: BoolField always returning false for non-DV fields when javabin involved (via solrj, or intra node communication) - SOLR-9188: BlockUnknown property makes inter-node communication impossible - SOLR-9389: HDFS Transaction logs stay open for writes which leaks Xceivers - SOLR-9438: Shard split can fail to write commit data on shutdown leading to data loss Furthermore, this release includes Apache Lucene 6.2.1 which includes 3 bug fixes since the 6.2.0 release. The release is available for immediate download at: - http://www.apache.org/dyn/closer.lua/lucene/solr/6.2.1 Please read CHANGES.txt for a detailed list of changes: - https://lucene.apache.org/solr/6_2_1/changes/Changes.html Please report any feedback to the mailing lists (http://lucene.apache.or g/solr/discussion.html) Note: The Apache Software Foundation uses an extensive mirroring network for distributing releases. It is possible that the mirror you are using may not have replicated the release yet. If that is the case, please try another mirror. This also goes for Maven access. -- Regards, Shalin Shekhar Mangar.
Searching Special charterer in solr behaving inconsistent
Problem: Searching Special charterer in solr behaving inconsistently
Example: search character ¢ , £ , ¤ , ¥ , ¦ , §
By searching this character by using particular field Like say title__v,
result is coming as expected.
*Query:*
solr/trucollection/select?q=title__v:(*%C2%A2%5C+%2C%5C+%C2%A3%5C+%2C%5C+%C2%A4%5C+%2C%5C+%C2%A5%5C+%2C%5C+%C2%A6%5C+%2C%5C+%C2%A7*
OR "%C2%A2+%2C+%C2%A3+%2C+%C2%A4+%2C+%C2%A5+%2C+%C2%A6+%2C+%C2%A7") AND
(isrestricted:false OR groupid:("07C90F45-AA61-4B2A-A446-BEA13C8409E2" OR
"23414BF3-046D-475A-9268-AA7BF5238DB3" OR
"50FC6521-DBCC-4097-A3BA-03AE6C202C2F"))=0=50
=owning_facility__v
desc=on=shard=owning_facility__v=status__v=type__v=classification__v
bulk_erp_code product_categorys owning_department__v docobjid erp_code
category_of_change shard groupid title__v author brands hassupportingdocs
last_periodic_review_date__vs technology__c document_language__c
effective_date__v reason_code__c owning_segments__c isrestricted region__c
document_number__v fileuniquename approved_date__vs impacted_departments__v
severity id owning_facility__v change_coordinator lifecycle_categories__c
status__v next_periodic_review_date__vs subtype__v version__v fg_erp_code
impacted_facilities__v type__v franchises country__v policy
legacy_document_number__c formula_number marketing_companys
impacted_segments__c=json
*
But when we use the same search criteria by using quick search(full text
search), It is not working*
*Query:*
solr/trucollection/select?q=(*%C2%A2+%2C+%C2%A3+%2C+%C2%A4+%2C+%C2%A5+%2C+%C2%A6+%2C+%C2%A7*
OR "%C2%A2+%2C+%C2%A3+%2C+%C2%A4+%2C+%C2%A5+%2C+%C2%A6+%2C+%C2%A7") AND
(isrestricted:false OR groupid="07C90F45-AA61-4B2A-A446-BEA13C8409E2" OR
"23414BF3-046D-475A-9268-AA7BF5238DB3" OR
"50FC6521-DBCC-4097-A3BA-03AE6C202C2F")=0=50
=owning_facility__v
desc=on=shard=owning_facility__v=status__v=type__v=classification__v
bulk_erp_code product_categorys owning_department__v docobjid erp_code
category_of_change shard groupid title__v author brands hassupportingdocs
last_periodic_review_date__vs technology__c document_language__c
effective_date__v reason_code__c owning_segments__c isrestricted region__c
document_number__v fileuniquename approved_date__vs impacted_departments__v
severity id owning_facility__v change_coordinator lifecycle_categories__c
status__v next_periodic_review_date__vs subtype__v version__v fg_erp_code
impacted_facilities__v type__v franchises country__v policy
legacy_document_number__c formula_number marketing_companys
impacted_segments__c=json
Can anyone help me out.
Best regards,
Shekhar
--
View this message in context:
http://lucene.472066.n3.nabble.com/Searching-Special-charterer-in-solr-behaving-inconsistent-tp4296493.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Unable to connect to correct port in solr 6.2.0
Good to know. Thank you! On Tue, Sep 13, 2016 at 4:19 PM, Preeti Bhat <preeti.b...@shoregrp.com> wrote: > Thanks Shekhar, Re install was successful. I had run on default port prior > to running on 2016. T > > > Thanks and Regards, > Preeti Bhat > > -Original Message- > From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] > Sent: Tuesday, September 13, 2016 1:01 AM > To: solr-user@lucene.apache.org > Subject: Re: Unable to connect to correct port in solr 6.2.0 > > I just tried this out on ubuntu (sorry I don't have access to a red hat > system) and it works fine. > > One thing that you have to take care of is that if you install the service > on the default 8983 port then, trying to upgrade with the same tar to a > different port does not work. So please ensure that you hadn't already > installed the service before already. > > On Tue, Sep 13, 2016 at 12:53 AM, Shalin Shekhar Mangar < > shalinman...@gmail.com> wrote: > > > Which version of red hat? Is lsof installed on this system? > > > > On Mon, Sep 12, 2016 at 4:30 PM, Preeti Bhat > > <preeti.b...@shoregrp.com> > > wrote: > > > >> HI All, > >> > >> I am trying to setup the solr in Redhat Linux, using the > >> install_solr_service.sh script of solr.6.2.0 tgz. The script runs > >> and starts the solr on port 8983 even when the port is specifically > >> specified as 2016. > >> > >> /root/install_solr_service.sh solr-6.2.0.tgz -i /opt -d /var/solr -u > >> root -s solr -p 2016 > >> > >> Is this correct way to setup solr in linux? Also, I have observed > >> that if I go to the /bin/solr and start with the port number its > >> working as expected but not as service. > >> > >> I would like to setup the SOLR in SOLRCloud mode with external > zookeepers. > >> > >> Could someone please advise on this? > >> > >> > >> > >> NOTICE TO RECIPIENTS: This communication may contain confidential > >> and/or privileged information. If you are not the intended recipient > >> (or have received this communication in error) please notify the > >> sender and it-supp...@shoregrp.com immediately, and destroy this > >> communication. Any unauthorized copying, disclosure or distribution > >> of the material in this communication is strictly forbidden. Any > >> views or opinions presented in this email are solely those of the > >> author and do not necessarily represent those of the company. > >> Finally, the recipient should check this email and any attachments > >> for the presence of viruses. The company accepts no liability for any > damage caused by any virus transmitted by this email. > >> > >> > >> > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > > > > -- > Regards, > Shalin Shekhar Mangar. > > NOTICE TO RECIPIENTS: This communication may contain confidential and/or > privileged information. If you are not the intended recipient (or have > received this communication in error) please notify the sender and > it-supp...@shoregrp.com immediately, and destroy this communication. Any > unauthorized copying, disclosure or distribution of the material in this > communication is strictly forbidden. Any views or opinions presented in > this email are solely those of the author and do not necessarily represent > those of the company. Finally, the recipient should check this email and > any attachments for the presence of viruses. The company accepts no > liability for any damage caused by any virus transmitted by this email. > > > -- Regards, Shalin Shekhar Mangar.
Re: Unable to connect to correct port in solr 6.2.0
I just tried this out on ubuntu (sorry I don't have access to a red hat system) and it works fine. One thing that you have to take care of is that if you install the service on the default 8983 port then, trying to upgrade with the same tar to a different port does not work. So please ensure that you hadn't already installed the service before already. On Tue, Sep 13, 2016 at 12:53 AM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > Which version of red hat? Is lsof installed on this system? > > On Mon, Sep 12, 2016 at 4:30 PM, Preeti Bhat <preeti.b...@shoregrp.com> > wrote: > >> HI All, >> >> I am trying to setup the solr in Redhat Linux, using the >> install_solr_service.sh script of solr.6.2.0 tgz. The script runs and >> starts the solr on port 8983 even when the port is specifically specified >> as 2016. >> >> /root/install_solr_service.sh solr-6.2.0.tgz -i /opt -d /var/solr -u root >> -s solr -p 2016 >> >> Is this correct way to setup solr in linux? Also, I have observed that if >> I go to the /bin/solr and start with the port number its working as >> expected but not as service. >> >> I would like to setup the SOLR in SOLRCloud mode with external zookeepers. >> >> Could someone please advise on this? >> >> >> >> NOTICE TO RECIPIENTS: This communication may contain confidential and/or >> privileged information. If you are not the intended recipient (or have >> received this communication in error) please notify the sender and >> it-supp...@shoregrp.com immediately, and destroy this communication. Any >> unauthorized copying, disclosure or distribution of the material in this >> communication is strictly forbidden. Any views or opinions presented in >> this email are solely those of the author and do not necessarily represent >> those of the company. Finally, the recipient should check this email and >> any attachments for the presence of viruses. The company accepts no >> liability for any damage caused by any virus transmitted by this email. >> >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > -- Regards, Shalin Shekhar Mangar.
Re: Unable to connect to correct port in solr 6.2.0
Which version of red hat? Is lsof installed on this system? On Mon, Sep 12, 2016 at 4:30 PM, Preeti Bhat <preeti.b...@shoregrp.com> wrote: > HI All, > > I am trying to setup the solr in Redhat Linux, using the > install_solr_service.sh script of solr.6.2.0 tgz. The script runs and > starts the solr on port 8983 even when the port is specifically specified > as 2016. > > /root/install_solr_service.sh solr-6.2.0.tgz -i /opt -d /var/solr -u root > -s solr -p 2016 > > Is this correct way to setup solr in linux? Also, I have observed that if > I go to the /bin/solr and start with the port number its working as > expected but not as service. > > I would like to setup the SOLR in SOLRCloud mode with external zookeepers. > > Could someone please advise on this? > > > > NOTICE TO RECIPIENTS: This communication may contain confidential and/or > privileged information. If you are not the intended recipient (or have > received this communication in error) please notify the sender and > it-supp...@shoregrp.com immediately, and destroy this communication. Any > unauthorized copying, disclosure or distribution of the material in this > communication is strictly forbidden. Any views or opinions presented in > this email are solely those of the author and do not necessarily represent > those of the company. Finally, the recipient should check this email and > any attachments for the presence of viruses. The company accepts no > liability for any damage caused by any virus transmitted by this email. > > > -- Regards, Shalin Shekhar Mangar.
Re: Replication Index fetch failed
On Thu, Sep 1, 2016 at 6:05 PM, Arkadi Colson <ark...@smartbit.be> wrote: > ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1 r:core_node5 > x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Index > fetch failed :org.apache.solr.common.SolrException: Unable to download > _6f46_cj.liv completely. Downloaded 0!=5596 > at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup( > IndexFetcher.java:1554) > at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile( > IndexFetcher.java:1437) > at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(Inde > xFetcher.java:852) > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexF > etcher.java:428) > at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexF > etcher.java:251) > There should be another exception in the logs that looks like the following: "Could not download file"... That one will have a more useful stack trace. Can you please find it and paste it on email? -- Regards, Shalin Shekhar Mangar.
Re: ShardDoc.sortFieldValues are not exposed in v5.2.1
This was made public in https://issues.apache.org/jira/browse/SOLR-7968 which is released already in 5.5 On Fri, Sep 2, 2016 at 12:01 AM, tedsolr <tsm...@sciquest.com> wrote: > I'm attempting to perform my own merge of IDs with a MergeStrategy in > v5.2.1. > I'm a bit hamstrung because the ShardFieldSortedHitQueue is not public. > When > trying to build my own priority queue I found out that the field > sortFieldValues in ShardDoc is package restricted. Now, in v6.1 I see that > both the HitQueue and the field are public. > > Would it be possible to patch 5.2.1, or maybe the latest v5, to expose > these > very useful objects? I can't upgrade to v6 due to the java 8 requirement. > > thanks, Ted > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/ShardDoc- > sortFieldValues-are-not-exposed-in-v5-2-1-tp4294336.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Regards, Shalin Shekhar Mangar.
Re: what to try next for replica that will not stay up.
;
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene54_0.dvd did not match. expected checksum is 762056409
> and actual is checksum 1514176651. expected length is 228 and actual length
> is 211
>
> 9/1/2016, 12:37:09 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
> replica2\data\tlog\tlog.005 refcount=2} active=true
> starting pos=20222
>
> 9/1/2016, 12:37:10 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Log replay finished. recoveryInfo=RecoveryInfo{adds=10 deletes=0
> deleteByQuery=1 errors=0 positionOfStart=20222}
>
> 9/1/2016, 12:37:10 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Could not publish as ACTIVE after succesful recovery
>
> 9/1/2016, 12:37:10 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Recovery failed - trying again... (0)
>
> 9/1/2016, 12:37:37 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
> replica2\data\tlog\tlog.006 refcount=2} active=true
> starting pos=0
>
> 9/1/2016, 12:37:38 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0
> deleteByQuery=0 errors=0 positionOfStart=0}
>
> 9/1/2016, 12:37:41 PM
>
>
>
> WARN false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Stopping recovery for core=[transcribedReports_shard1_replica2]
> coreNodeName=[core_node14]
>
> 9/1/2016, 12:42:13 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
> replica2\data\tlog\tlog.007 refcount=2} active=true
> starting pos=1748
>
> 9/1/2016, 12:42:14 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Log replay finished. recoveryInfo=RecoveryInfo{adds=12 deletes=0
> deleteByQuery=0 errors=0 positionOfStart=1748}
>
> 9/1/2016, 12:42:14 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Could not publish as ACTIVE after succesful recovery
>
> 9/1/2016, 12:42:14 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Recovery failed - trying again... (0)
>
> 9/1/2016, 12:42:43 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Could not publish as ACTIVE after succesful recovery
>
> 9/1/2016, 12:42:43 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Recovery failed - trying again... (0)
>
>
>
> *Jon Hawkesworth*
> Software Developer
>
>
>
>
>
> Hanley Road, Malvern, WR13 6NP. UK
>
> O: +44 (0) 1684 312313
>
> *jon.hawkeswo...@mmodal.com <jon.hawkeswo...@mmodal.com> www.mmodal.com
> <http://www.medquist.com/>*
>
>
>
> *This electronic mail transmission contains confidential information
> intended only for the person(s) named. Any use, distribution, copying or
> disclosure by another person is strictly prohibited. If you are not the
> intended recipient of this e-mail, promptly delete it and all attachments.*
>
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: ConcurrentModificationException due to high volume of requests
This is a bug. Please open a Jira issue. We need to dig where else this param map is being used. It is possible that there are more such bugs. On Fri, Aug 12, 2016 at 10:57 PM, Katherine Mora < km...@searchtechnologies.com> wrote: > Hi All, > > We are using Solr 5.2.1 in a production environment where we have a high > volume of requests. We are not having any timeouts or performance issues. > However, the logs are filled with this exception. > We've been researching and there isn't a lot of information about this > problem or why it happens. We have tried to reproduce it by sending queries > using JMeter and it only happens when multiple queries are sent at the same > time. > > Checking the code we found that this exception is within the core Solr > code and part of the exception caught our attention as it is logging debug > information (we are using INFO logging level) (org.apache.solr.search.stats. > LocalStatsCache.get(LocalStatsCache.java:40)). > > Has anyone seen this exception before? Would it be OK to generate a patch? > We were thinking about commenting the debug line or adding try/catch > statements. > > Thank you! > > > null:java.util.ConcurrentModificationException > at java.util.LinkedHashMap$LinkedHashIterator.nextEntry( > LinkedHashMap.java:394) > at java.util.LinkedHashMap$EntryIterator.next( > LinkedHashMap.java:413) > at java.util.LinkedHashMap$EntryIterator.next( > LinkedHashMap.java:412) > at org.apache.solr.common.params.ModifiableSolrParams.toString( > ModifiableSolrParams.java:201) > at java.lang.String.valueOf(String.java:2849) > at java.lang.StringBuilder.append(StringBuilder.java:128) > at org.apache.solr.request.SolrQueryRequestBase.toString( > SolrQueryRequestBase.java:165) > at org.apache.solr.search.stats.LocalStatsCache.get( > LocalStatsCache.java:40) > at org.apache.solr.handler.component.QueryComponent. > process(QueryComponent.java:366) > at org.apache.solr.handler.component.SearchHandler. > handleRequestBody(SearchHandler.java:255) > at org.apache.solr.handler.RequestHandlerBase.handleRequest( > RequestHandlerBase.java:143) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) > at org.apache.solr.servlet.HttpSolrCall.execute( > HttpSolrCall.java:654) > at org.apache.solr.servlet.HttpSolrCall.call( > HttpSolrCall.java:450) > at org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:227) > at org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:196) > at org.eclipse.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1652) > at org.eclipse.jetty.servlet.ServletHandler.doHandle( > ServletHandler.java:585) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:143) > at org.eclipse.jetty.security.SecurityHandler.handle( > SecurityHandler.java:577) > at org.eclipse.jetty.server.session.SessionHandler. > doHandle(SessionHandler.java:223) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:143) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:97) > at org.eclipse.jetty.server.handler.RequestLogHandler. > handle(RequestLogHandler.java:95) > at org.eclipse.jetty.server.handler.ContextHandler. > doHandle(ContextHandler.java:1129) > at org.eclipse.jetty.servlet.ServletHandler.doScope( > ServletHandler.java:515) > at org.eclipse.jetty.server.session.SessionHandler. > doScope(SessionHandler.java:185) > at org.eclipse.jetty.server.handler.ContextHandler. > doScope(ContextHandler.java:1061) > at org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:141) > at org.eclipse.jetty.server.handler.ContextHandlerCollection. > handle(ContextHandlerCollection.java:215) > at org.eclipse.jetty.server.handler.HandlerCollection. > handle(HandlerCollection.java:110) > at org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:497) > at org.eclipse.jetty.server.HttpChannel.handle( > HttpChannel.java:310) > at org.eclipse.jetty.server.HttpConnection.onFillable( > HttpConnection.java:257) > at org.eclipse.jetty.io.AbstractConnection$2.run( > AbstractConnection.java:540) > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > QueuedThreadPool.java:635) > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( > QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:745) > > > KATHERINE MORA > Senior Engineer > > -- Regards, Shalin Shekhar Mangar.
Re: Issue faced while re-starting solr 6.1.0 after cleaning zk data.
Hi Naveen, When you cleaned up the zk data, it also removed the configuration files that are stored in Zookeeper. These configuration files are used by Solr and therefore they can no longer be found. You need to upload those configuration files again to ZK using: bin/solr zk upconfig -d /path/to/your/conf/directory -n sample -z zkhost_connect_string On Sun, Aug 7, 2016 at 3:14 PM, Naveen Pajjuri <pajjuri.re...@myntra.com> wrote: > Here sample is the name of my collection. > > Thanks > > On Sun, Aug 7, 2016 at 3:10 PM, Naveen Pajjuri <pajjuri.re...@myntra.com> > wrote: > > > Hi, > > I'm trying to move to solr-6.1.0. it was working fine and i cleaned up zk > > data (version folder) and restarted solr and zookeeper. I started getting > > this error. > > > > > >- *sample_shard1_replica1:* org.apache.solr.common.cloud. > >ZooKeeperException:org.apache.solr.common.cloud.ZooKeeperException: > >Specified config does not exist in ZooKeeper: sample. > > > > > > Please let me know what i'm missing. > > > > Regards, > > Naveen Reddy. > > > -- Regards, Shalin Shekhar Mangar.
Re: problems with bulk indexing with concurrent DIH
Hi Bernd,
I think you are running into
https://issues.apache.org/jira/browse/LUCENE-6161. Can you upgrade to 5.1
or newer?
On Wed, Jul 27, 2016 at 7:29 PM, Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:
> After enhancing the server with SSDs I'm trying to speed up indexing.
>
> The server has 16 CPUs and more than 100G RAM.
> JAVA (1.8.0_92) has 24G.
> SOLR is 4.10.4.
> Plain XML data to load is 218G with about 96M records.
> This will result in a single index of 299G.
>
> I tried with 4, 8, 12 and 16 concurrent DIHs.
> 16 and 12 was to much because for 16 CPUs and my test continued with 8
> concurrent DIHs.
> Then i was trying different and settings but
> now I'm stuck.
> I can't figure out what is the best setting for bulk indexing.
> What I see is that the indexing is "falling asleep" after some time of
> indexing.
> It is only producing del-files, like _11_1.del, _w_2.del, _h_3.del,...
>
>
> 8
> 1024
> -1
>
> 8
> 100
> 512
>
> 8
> class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
> ${solr.lock.type:native}
> ...
>
>
>
> ### no autocommit at all
>
>${solr.autoSoftCommit.maxTime:-1}
>
>
>
>
>
> command=full-import=false=false=false=false
> After indexing finishes there is a final optimize.
>
> My idea is, if 8 DIHs use 8 CPUs then I have 8 CPUs left for merging
> (maxIndexingThreads/maxMergeAtOnce/mergeFactor).
> It should do no commit, no optimize.
> ramBufferSizeMB is high because I have plenty of RAM and I want make use
> the speed of RAM.
> segmentsPerTier is high to reduce merging.
>
> But somewhere is a misconfiguration because indexing gets stalled.
>
> Any idea what's going wrong?
>
>
> Bernd
>
>
>
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: File Descriptor/Memory Leak
I have myself seen this CLOSE_WAIT issue at a customer. I am running some tests with different versions trying to pinpoint the cause of this leak. Once I have some more information and a reproducible test, I'll open a jira issue. I'll keep you posted. On Thu, Jul 7, 2016 at 5:13 PM, Mads Tomasgård Bjørgan <m...@dips.no> wrote: > Hello there, > Our SolrCloud is experiencing a FD leak while running with SSL. This is > occurring on the one machine that our program is sending data too. We have > a total of three servers running as an ensemble. > > While running without SSL does the FD Count remain quite constant at > around 180 while indexing. Performing a garbage collection also clears > almost the entire JVM-memory. > > However - when indexing with SSL does the FDC grow polynomial. The count > increases with a few hundred every five seconds or so, but reaches easily > 50 000 within three to four minutes. Performing a GC swipes most of the > memory on the two machines our program isn't transmitting the data directly > to. The last machine is unaffected by the GC, and both memory nor FDC > doesn't reset before Solr is restarted on that machine. > > Performing a netstat reveals that the FDC mostly consists of > TCP-connections in the state of "CLOSE_WAIT". > > > -- Regards, Shalin Shekhar Mangar.
Re: Solr Merge Index
Why do you need the merged core? If the underlying data is changing then obviously the merged core won't automatically update it self. And another merge will introduce duplicate data. So this is a bad solution. You can just keep the two cores and do a distributed search across both of them? You can specify shards=http://localhost:8983/solr/ <http://localhost:8983/solr/admin/cores?action=mergeindexes=core3=sitecore_web_index=core2> core1,http://localhost:8983/solr/ <http://localhost:8983/solr/admin/cores?action=mergeindexes=core3=sitecore_web_index=core2>core2 as a parameter to your search requests. On Thu, Jul 7, 2016 at 4:32 AM, Kalpana <ksivanan...@portalsolutions.net> wrote: > Hello > > I have two sources - Sitecore web index (core 1) and a database table (core > 2). I have created core 3 which is a merge of core1 and core 2. > > > http://localhost:8983/solr/admin/cores?action=mergeindexes=core3=sitecore_web_index=core2 > > But when someone publishes a page on Sitecore, the sitecore web index gets > updated but not the merged core. How can get the real time data with the > merge? Is there a way? > > Thanks > Kalpana > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-Merge-Index-tp4286081.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Regards, Shalin Shekhar Mangar.
Re: sorlcloud connection issue
erver.request(HttpSolrServer.java:206) > at > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340) > at > > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301) > at > > org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > > please help me, look forward to your reply. Thanks a lot! > > Best Regards! > Kent > -- Regards, Shalin Shekhar Mangar.
Re: solrcloud goes down
You are asking many different questions. Lets take them up one by one: You say that you use the singleton pattern to create and get the Solr server connection. Yes, that is fine. The CloudSolrServer is thread safe and can be shared across requests/threads. When your application exits, you should shutdown that object. Once you shutdown ZooKeeper, Solr cannot process any updates. This is by design. A running and functional Zookeeper is necessary for SolrCloud. On Fri, Jul 1, 2016 at 2:15 PM, 穆 俊峰 <kent...@live.cn> wrote: > Dear Mr. Experts! > > > I came across a problem when use solrcloud, it goes down. The version is > 4.9.1. > > we got the java core log, it looks like the http connection pool is > exhausted and most threads are waiting to get a free connection. > > I have increased http connection defaults for the SolrJ client, and also > configed the connection defaults in solr.xml for all shard servers as below. > > class="HttpShardHandlerFactory"> > 6 > 3 > 1 > 500 > > > > > the java stack as below > > "httpShardExecutor-3-thread-541" prio=10 tid=0x7f7b1c02b000 nid=0x20af > waiting on condition [0x7f79fd49] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x000605710068> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133) > at > org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282) > at > org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) > at > org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177) > at > org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170) > at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102) > at > org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:208) > at > org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:422) > at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) > at > org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340) > at > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301) > at > org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:205) > at > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:162) > at > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > > > > > besides, we use the singleton pattern to create and get the solr server > connection, I wonder if this pattern is OK. Once I shutdown the zookeeper, > the application can not do the solr query, the error as "no live > SolrServers available to handle this request". > > so I need to restart our connection to reconnect solrcloud. > > as we use the singleton pattern, we do not use the method of "shutdown" to > release the solrserver connection, will this be a problem? or we need to > create the connection on demand every time? > > > please help me, look forward to your reply. Thanks a lot! > > > Best Regards! > Kent Mu > -- Regards, Shalin Shekhar Mangar.
Re: SimplePostTool: FATAL: IOException while posting data: java.io.IOException: too many bytes written
This was fixed in 5.5 and 6.0. You can upload files larger than 2GB with the simple post tool however I don't recommend it because it uses a single indexing thread. On Tue, Jun 28, 2016 at 3:55 PM, Toke Eskildsen <t...@statsbiblioteket.dk> wrote: > On Mon, 2016-06-27 at 13:24 +, Rajendran, Prabaharan wrote: > > I am trying to index a text file about 4.2 GB in size. [...] > > > > SimplePostTool: FATAL: IOException while posting data: > java.io.IOException: too many bytes written > > SimplePostTool uses > HttpUrlConnection.setFixedLengthStreamingMode(file_size) > where file_size is an integer. > > Unfortunately there is no check for overflow (which happens with files > > 2GB), so there is no sane error message up front and you only get the > error you pasted after some bytes has been sent. With a 4.2GB input > file, I would guess after about 200MB (4.2GB % 2GB). > > > Long story short: Keep your posts below 2GB. > > - Toke Eskildsen, State and University Library, Denmark > > > -- Regards, Shalin Shekhar Mangar.
Re: Nodes appear twice in state.json
Hmm not good. Definitely a new bug. Please open an issue.
Please look up the core node name in core.properties for that particular
core and remove the other one from state.json manually. Probably best to do
a cluster restart to avoid surprises. This is certainly uncharted territory.
On Wed, May 4, 2016 at 6:54 PM, Markus Jelsma <markus.jel...@openindex.io>
wrote:
> Hi - we've just upgraded a development environment from 5.5 to Solr 6.0.
> After the upgrade, which went fine, we see two replica's appear twice in
> the cloud view (see below), both being leader. We've seen this happen
> before on some older 5.x versions. Is there a Jira issue i am missing? An
> unknown issue?
>
> Also, how to fix this. How do we remove the double node from the
> state.json?
>
> Many thanks!
> Markus
>
> {"search":{
> "replicationFactor":"3",
> "shards":{
> "shard1":{
> "range":"8000-",
> "state":"active",
> "replicas":{
> "core_node6":{
> "core":"search_shard1_replica1",
> "base_url":"http://idx5.oi.dev:8983/solr;,
> "node_name":"idx5.oi.dev:8983_solr",
> "state":"down"},
> "core_node2":{
> "core":"search_shard1_replica2",
> "base_url":"http://idx2.oi.dev:8983/solr;,
> "node_name":"idx2.oi.dev:8983_solr",
> "state":"active",
> "leader":"true"},
> "core_node3":{
> "core":"search_shard1_replica2",
> "base_url":"http://idx2.oi.dev:8983/solr;,
> "node_name":"idx2.oi.dev:8983_solr",
> "state":"down",
> "leader":"true"},
> "core_node5":{
> "core":"search_shard1_replica3",
> "base_url":"http://idx3.oi.dev:8983/solr;,
> "node_name":"idx3.oi.dev:8983_solr",
> "state":"down"}}},
> "shard2":{
> "range":"0-7fff",
> "state":"active",
> "replicas":{
> "core_node1":{
> "core":"search_shard2_replica1",
> "base_url":"http://idx4.oi.dev:8983/solr;,
> "node_name":"idx4.oi.dev:8983_solr",
> "state":"down"},
> "core_node2":{
> "core":"search_shard2_replica2",
> "base_url":"http://idx6.oi.dev:8983/solr;,
> "node_name":"idx6.oi.dev:8983_solr",
> "state":"down"},
> "core_node4":{
> "core":"search_shard2_replica3",
> "base_url":"http://idx1.oi.dev:8983/solr;,
> "node_name":"idx1.oi.dev:8983_solr",
> "state":"active",
> "leader":"true",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"1",
> "autoAddReplicas":"false"}}
>
>
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Solr slave is doing full replication (entire index) of index after master restart
> > >> > This is exactly what happened in production when I restarted the > >> master. > >> > > >> > I attached the configurations files. > >> > > >> > Replication section: > >> > > >> > Master: > >> > > >> > > >> > > >> > commit > >> > > >> > > >> > > >> > Slave: > >> > > >> >> > >> > > >> > > >> http://solr01-isrl01.flr.local:8983/solr/replication-master/replication > < > >> http://solr01-isrl01.flr.local:8983/solr/replication-master/replication > >> > > >> > 00:00:60 > >> > > >> > > >> > > >> > > >> > > >> > Best, > >> > Lior > >> > > >> > On Thu, Apr 7, 2016 at 6:56 PM, Erick Erickson < > erickerick...@gmail.com > >> <mailto:erickerick...@gmail.com>> wrote: > >> > What does your configuration file look like for the replication > >> > handler? Does this happen whenever you restart a slave even if > >> > _nothing_ has changed on the master? > >> > > >> > And this will certainly happen if you're optimizing the master before > >> > you restart, although that doesn't sound likely. > >> > > >> > Best, > >> > Erick > >> > > >> > On Thu, Apr 7, 2016 at 6:54 AM, Lior Sapir <lior.sa...@gmail.com > >> <mailto:lior.sa...@gmail.com>> wrote: > >> > > Solr slave is doing full replication (entire index) of index after > >> master > >> > > restart > >> > > Using solr 5.3.1 not cloud (using maser slave architecture ) I see > >> that > >> > > slave replicates entire index after master restart even though the > >> index > >> > > version is the same > >> > > > >> > > This is bad for me since the slave which is doing serving replicates > >> 80gb > >> > > if I restart the server and our service is down > >> > > > >> > > I attached a file with some snippets of the slave log before and > >> after the > >> > > master restart. > >> > > > >> > > Is there some default configuration issue causing this problem? > >> > > Both indexes master and slave were not updated for sure before and > >> after the > >> > > master restart. > >> > > The index version stayed exactly the same. > >> > > > >> > > > >> > > > >> > > >> > > >> > >> > > > -- Regards, Shalin Shekhar Mangar.
Re: Curious case of DataSource.getConnection()
What is this Solr scheduler class? Is that your own custom code? None of
the information or code snippets in your email related to a Solr problem. I
guess you are looking to troubleshoot a DB connectivity problem and it
would be better to ask this on stackoverflow.
On Tue, Apr 12, 2016 at 4:01 PM, Srinivas Kashyap <
srini...@tradestonesoftware.com> wrote:
> Hi,
>
> In a Solr scheduler class which runs every 'n' interval of seconds, i'm
> polling a database table to do some custom job.
>
> I'm getting the connection to database, through context file as below:
>
> try {
> Context initContext = new InitialContext();
> DataSource ds = null;
> if ("tomcat".equals(p.getProperty("server.type")))
> {
>Context webContext =
> (Context)initContext.lookup("java:/comp/env");
>ds = (DataSource)
> webContext.lookup("");
> }
> else if ("ws".equals(p.getProperty("server.type")))
> //websphere
> {
>ds = (DataSource)
> initContext.lookup("");
> }
> }
>
> ds.getConnection();
>
>
> But the, connection is not being established. No Exception/error is being
> thrown in console.
>
> Context xml has been double checked to see all the datasource properties
> and attributes are set proper.
>
> Any reason, i'm not able to establish database connection?
>
> P.S: Normal IMPORT process is running unaffected i.e Data is being indexed
> into solr with the same datasource configuration in context xml.
>
>
> Thanks and Regards,
> Srinivas Kashyap
> Senior Software Engineer
> "GURUDAS HERITAGE"
> 'Block A' , No 59/2, 2nd Floor, 100 Feet Ring Road,
> Kadirenahalli, Padmanabhanagar
> Banashankari 2nd Stage,
> Bangalore-560070
> P: 973-986-6105
> Bamboo Rose
> The only B2B marketplace powered by proven trade engines.
> www.BambooRose.com<http://www.bamboorose.com/>
>
> Make Retail. Fun. Connected. Easier. Smarter. Together. Better.
>
>
> DISCLAIMER:
> E-mails and attachments from TradeStone Software, Inc. are confidential.
> If you are not the intended recipient, please notify the sender
> immediately by
> replying to the e-mail, and then delete it without making copies or using
> it
> in any way. No representation is made that this email or any attachments
> are
> free of viruses. Virus scanning is recommended and is the responsibility of
> the recipient.
--
Regards,
Shalin Shekhar Mangar.
Re: EmbeddedSolr for unit tests in Solr 6
Rohana, as I said earlier, the MiniSolrCloudCluster is specifically made for your use-case i.e. where you want to quickly setup a SolrCloud cluster in your own application for testing. It is available in the solr-test-framework artifact. On Tue, Apr 12, 2016 at 4:31 PM, Rohana Rajapakse < rohana.rajapa...@gossinteractive.com> wrote: > Please note that I am not writing unit tests for testing classes in Solr. > I need a temporary Solr index to test classes in my own application that > needs a Solr index. I would like to use classes that are available in > solr-core and solr-solrj jars. I could do this easily in solr-4.x versions > using EmbeddedSolrServer. I prefer not to extend SolrTestCaseJ4 class. Also > MiniSolrCloudCluster is not available in solr-core or solr-solrj jar. > > What is the best way of doing this in Solr-6.x / Solr-7.0 ? > > -Original Message- > From: Joe Lawson [mailto:jlaw...@opensourceconnections.com] > Sent: 11 April 2016 17:31 > To: solr-user@lucene.apache.org > Subject: Re: EmbeddedSolr for unit tests in Solr 6 > > Check for example tests here too: > > https://github.com/apache/lucene-solr/tree/master/solr/core/src/test/org/apache/solr > > On Mon, Apr 11, 2016 at 12:24 PM, Shalin Shekhar Mangar < > shalinman...@gmail.com> wrote: > > > Please use MiniSolrCloudCluster instead of EmbeddedSolrServer for > > unit/integration tests. > > > > On Mon, Apr 11, 2016 at 2:26 PM, Rohana Rajapakse < > > rohana.rajapa...@gossinteractive.com> wrote: > > > > > Thanks Shawn, > > > > > > I am now pointing solrHomeFolder to > > > lucene-solr-master\solr\server\solr > > > which contains the correct solr.xml file. > > > Tried the following two ways to create an EmbeddedSolrServer: > > > > > > > > > 1. CoreContainer corecon = > > > CoreContainer.createAndLoad(Paths.get(solrHomeFolder)); > > >corecon.load(); > > >SolrClient svr = new EmbeddedSolrServer(corecon, > > > corename); > > > > > > > > > 2. SolrClient svr = new EmbeddedSolrServer(Paths.get(solrHomeFolder), > > > corename); > > > > > > > > > They both throws the same exception (java.lang.NoClassDefFoundError: > > > Could not initialize class org.apache.solr.servlet.SolrRequestParsers). > > > org.apache.solr.servlet.SolrRequestParsers class is present in the > > > solr-core-7.0.0-SNAPSHOT.jar and this jar is present in the > > > WEB-INF\lib folder (in solr server) and also included as a > > > dependency jar in the pom.xml of the test project. > > > > > > Here is the full stack trace of the exception: > > > > > > java.lang.NoClassDefFoundError: Could not initialize class > > > org.apache.solr.servlet.SolrRequestParsers > > > at > > > > > org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.(Embedd > > edSolrServer.java:112) > > > at > > > > > org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.(Embedd > > edSolrServer.java:70) > > > at > > > > > com.gossinteractive.solr.DocPhraseUpdateProcessorTest.createEmbeddedSo > > lrServer(DocPhraseUpdateProcessorTest.java:141) > > > at > > > > > com.gossinteractive.solr.DocPhraseUpdateProcessorTest.setUp(DocPhraseU > > pdateProcessorTest.java:99) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > > ava:62) > > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > > orImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > at > > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkM > > ethod.java:44) > > > at > > > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCall > > able.java:15) > > > at > > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMet > > hod.java:41) > > > at > > > > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.j > > ava:27) > > > at > > > > > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.jav > > a:31) > > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunn > > er.java:
