This is probably due to SecurityManager rejecting an unknown path. Try starting Solr with:
-Dsolr.allowPaths=/netmnt/ or set env variable SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/netmnt/" Then try again. Alternatively you can try disabling SecMgr withenv.var SOLR_SECURITY_MANAGER_ENABLED=false https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html Jan > 13. aug. 2023 kl. 04:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C] > <craig.oak...@nih.gov.INVALID>: > > Thanks: I was forgetting the fact that (in order to get backups in the format > we expected) I added "incremental=false" to our backup script when we > upgraded from Solr8.5 to Solr8.11. > > It trying to test this functionality, however, I now find that backups do not > seem to be working for me *at all* in Solr9.2, with or without > "incremental=false" > > I keep getting a stack trace such as: > <?xml version="1.0" encoding="UTF-8"?> > <response> > > <lst name="responseHeader"> > <int name="status">500</int> > <int name="QTime">9</int> > </lst> > <lst name="error"> > <str name="msg">access denied ("java.io.FilePermission" > "/netmnt/[redacted-directory]" "read")</str> > <str name="trace">java.security.AccessControlException: access denied > ("java.io.FilePermission" "/netmnt/[redacted-directory]" "read") > at > java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) > at > java.base/java.security.AccessController.checkPermission(AccessController.java:897) > at > java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322) > at > java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:661) > at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) > at > java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525) > at java.base/java.nio.file.Files.exists(Files.java:2435) > at > org.apache.solr.core.backup.repository.LocalFileSystemRepository.exists(LocalFileSystemRepository.java:110) > at > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$32(CollectionsHandler.java:1404) > at > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1878) > at > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:355) > at > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:333) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224) > at > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:929) > at > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:877) > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:548) > at > org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:252) > at > org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:220) > at > org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:257) > at > org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:227) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:215) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) > at > org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) > at > org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) > at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) > at > org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) > at org.eclipse.jetty.server.Server.handle(Server.java:563) > at > org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) > at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) > at > org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) > at java.base/java.lang.Thread.run(Thread.java:829) > </str> > <int name="code">500</int> > </lst> > </response> > > > It should be noted that the Linux account which runs Solr has read and write > access to the specified location, which is a subdirectory of what is listed > in allowPaths in all the solr.xml files of this SolrCloud, and which is > mounted on all hosts of the SolrCloud. > > I have tried both V1 API and V2 API as found at > solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#backup > > Not being familiar with V2 API, I tried various permutations of > "file:///netmnt/" (various numbers of slashes, etc): they all seem to get the > same stack trace > > Does anyone have any example of a successful backup from Solr9.2? > > > > > -----Original Message----- > From: Dominique Bejean <dominique.bej...@eolya.fr> > Sent: Friday, August 11, 2023 9:32 AM > To: users@solr.apache.org > Subject: Re: [EXTERNAL] Re: authentication for Leader/Follower replication > > Hi Craig, > > Yes, starting with the 8.9 and 9.0 versions, Collection API allows > incremental backup and much more (corruption check, backup to Amazon S3 or > Google Cloud Storage) . > Take a look at this umbrella JIRA > https://issues.apache.org/jira/browse/SOLR-15086 > > Regards > > Dominique > > > > Le jeu. 10 août 2023 à 17:58, Oakley, Craig (NIH/NLM/NCBI) [C] > <craig.oak...@nih.gov.invalid> a écrit : > >> RE: a better option is to use the backup/restore functionality in the >> Collections API. >> >> My impression was that there is no facility for _incremental_ backup and >> retore in the collections API: is there? To do backup and scp and restore >> of terabytes of data every few minutes does not sound practical. >> >> What we have for our databases (MSSQL, Sybase, MongoDB, Postgres, MySQL as >> well as Solr) is redundancy (for failover) in our main Data Center, and >> redundancy in read-only reasonably-concurrent copies in our second Data >> Center. For Solr8.11 (and earlier), we have had SolrCloud for redundancy in >> our main Data Center, and Leader/Follower replication to the read-only >> SolrClouds in the second Data Center. At one time, we were hoping that CDCR >> would work better: but we never managed to get CDCR to work reliably (and >> perhaps others also found it unreliable, leading to it being deprecated >> rather than being fixed). We have found that SolrCloud does not work >> reliably when spread across Data Centers, so Leader/Follower replication >> (formerly known as Master/Slave replication) has been the only way we have >> found to keep our (read-only) copies in the second Data Center only a few >> minutes behind the data in the main Data Center (the only way to provide >> latency low enough to be comparable to MSSQL, Sybase, MongoDB, Postgres and >> MySQL). My supervisor was asking for clarification whether you are implying >> that Leader/Follower replication is being deprecated. >> >> >> In my continuing attempts to resolve these issues, I have come across a >> related question. The error message about SolrAuthV2 prompted me to wonder >> about the V1 and V2 syntax options (such as shown at >> https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html#reload); >> and I was wondering whether Leader/Follower replication changed from using >> syntax V1 to using syntax V2, and if that might contribute to the breaking >> of permissions. As an experiment in our test environment, I setup a >> permission in security.json to allow RELOAD collection without a password. >> After confirming that my V2 syntax does work _with_ a password, I then >> attempted RELOAD collection without a password using both syntax V1 and >> syntax V2. Syntax V1 succeeded and syntax V2 failed. I have tried several >> permutations in security.json to allow RELOAD without password in syntax >> V2, but have not yet found a successful permutation. Are there any >> clarifications what security.json changes are needed for syntax V2? Can it >> be confirmed whether Leader/Follower replication is using V2 (in other >> words, whether that may be contributing to the permission problem)? >> >> >> [11:51 dbh19850s 1152]$ curl -X POST "http://`cat >> /tmp/pswd230808`@localhost:$p/api/collections/helpdocs" -H 'Content-Type: >> application/json' -d '{"reload":{}}' >> { >> "responseHeader":{ >> "status":0, >> "QTime":255}, >> "success":{ >> "nosqltest21.be-md:9852_solr":{ >> "responseHeader":{ >> "status":0, >> "QTime":176}}, >> "nosqltest22.be-md:9852_solr":{ >> "responseHeader":{ >> "status":0, >> "QTime":219}}}} >> [11:51 dbh19850s 1152]$ curl >> "http://localhost:$p/solr/admin/collections?action=RELOAD&name=helpdocs"; >> >> { >> "responseHeader":{ >> "status":0, >> "QTime":237}, >> "success":{ >> "nosqltest21.be-md:9852_solr":{ >> "responseHeader":{ >> "status":0, >> "QTime":176}}, >> "nosqltest22.be-md:9852_solr":{ >> "responseHeader":{ >> "status":0, >> "QTime":216}}}} >> [11:51 dbh19850s 1153]$ curl -X POST >> "http://localhost:$p/api/collections/helpdocs"; >> -H 'Content-Type: application/json' -d '{"reload":{}}' >> <html> >> <head> >> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> >> <title>Error 401 Authentication failed, Response code: 401</title> >> </head> >> <body><h2>HTTP ERROR 401 Authentication failed, Response code: 401</h2> >> <table> >> <tr><th>URI:</th><td>/solr/____v2/collections/helpdocs</td></tr> >> <tr><th>STATUS:</th><td>401</td></tr> >> <tr><th>MESSAGE:</th><td>Authentication failed, Response code: >> 401</td></tr> >> <tr><th>SERVLET:</th><td>default</td></tr> >> </table> >> >> </body> >> </html> >> *[11:51 dbh19850s 1154]$ grep helpdocs 2023_08_10.request.log |tail -3 >> >> 127.0.0.1 - - [10/Aug/2023:15:51:23 +0000] "POST /api/collections/helpdocs >> HTTP/1.1" 200 280 >> 127.0.0.1 - - [10/Aug/2023:15:51:39 +0000] "GET >> /solr/admin/collections?action=RELOAD&name=helpdocs HTTP/1.1" 200 280 >> 127.0.0.1 - - [10/Aug/2023:15:51:47 +0000] "POST /api/collections/helpdocs >> HTTP/1.1" 401 491 >> [11:51 dbh19850s 1155]$ less solr.log >> ... >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{ >> "name":"openreload8", >> "path":"/admin/collections", >> "index":9, >> "collection":null, >> "role":null, >> "params":{ >> "action":["REGEX:(?i)RELOAD"], >> "name":["REGEX:(?i)helpdocs"]}}] to govern resource >> [/admin/collections] >> 2023-08-10 11:51:39.878 DEBUG (qtp1003693033-264) [ ] >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{ >> "name":"openreload8", >> "path":"/admin/collections", >> "index":9, >> "collection":null, >> "role":null, >> "params":{ >> "action":["REGEX:(?i)RELOAD"], >> "name":["REGEX:(?i)helpdocs"]}}] has no role; permitting access >> ... >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] >> o.a.s.s.RuleBasedAuthorizationPluginBase Found perm [{ >> "name":"catch-all-nocollection", >> "path":"/*", >> "index":24, >> "collection":null, >> "role":"allgen"}] to govern resource [/____v2/collections/helpdocs] >> 2023-08-10 11:51:47.731 DEBUG (qtp1003693033-249) [ ] >> o.a.s.s.RuleBasedAuthorizationPluginBase Governing permission [{ >> "name":"catch-all-nocollection", >> "path":"/*", >> "index":24, >> "collection":null, >> "role":"allgen"}] has role, but request principal cannot be identified; >> forbidding access >> >> >> As a side note, in our experience, the only thing that has been cluttering >> up solr.log with attempts to connect without a password has been >> Leader/Follower replication (formerly known as Master/Slave replication). >> >> >> -----Original Message----- >> From: Shawn Heisey <apa...@elyograg.org> >> Sent: Saturday, August 5, 2023 4:24 PM >> To: users@solr.apache.org >> Subject: [EXTERNAL] Re: authentication for Leader/Follower replication >> >> On 7/6/23 14:00, Oakley, Craig (NIH/NLM/NCBI) [C] wrote: >>> We are having problems transitioning Leader/Follower replication to >> Solr9.2.1 >>> >>> In Solr8.5 and below, what was then called Master/Slave replication had >> the annoying problem that, even though we specified httpBasicAuthUser and >> httpBasicAuthPassword, it would always attempt to connect first without a >> password before retrying with a password. This made solr.log noisy with >> lots of unnecessary login failures: but at least it worked. >> >> In general, this is how basic auth via http works. The client first >> attempts the request without any credentials, and receives a 401 response. >> >> At this point, a browser would see the 401 response and display a popup >> asking for username/password. >> >> Then on future requests to that server in the current session, the >> browser sends the supplied credentials on every request to that server. >> >> If you are supplying credentials in the replication config, it should >> NOT follow that pattern ... the credentials should be always used. >> >>> Now when we are preparing to upgrade to Solr9.2.1, we are having issues >> with the following: >>> 2023-07-06 15:46:53.315 INFO (indexFetcher-39-thread-1) [ ] >> o.a.s.h.IndexFetcher Last replication failed, so I'll force replication >>> 2023-07-06 15:46:53.320 WARN (indexFetcher-39-thread-1) [ ] >> o.a.s.h.IndexFetcher Leader at: >> http://[REDACTED]/solr/sequence2_shard1_replica_n1 >> is not available. >> >> The info above is valid for Solr running in standalone mode. But those >> core names indicate that you are running in SolrCloud mode. >> >> In cloud mode, Solr handles all replication. Don't attempt to actually >> configure the replication handler in cloud mode ... Solr will handle it >> all for you, and will even automatically take care of authenticating the >> requests. You don't need to configure the replication handler at all. >> >> If you are using the replication handler as a "back door" to copy >> indexes to a separate Solr install, a better option is to use the >> backup/restore functionality in the Collections API. >> >> Thanks, >> Shawn >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you recognize the sender and are >> confident the content is safe. >> >> > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and are confident > the content is safe. >
