Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/208/
No tests ran. Build Log: [...truncated 347 lines...] # ERROR: KeeperErrorCode = NoNode for /security.json # -- # # Last output: # WARN - 2026-05-14 03:48:07.494; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-14 03:48:07.497; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-14 03:48:07.941; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-14 03:48:07.941; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # # ERROR: KeeperErrorCode = NoNode for /security.json not ok 9 create, config, api, and delete with basic auth # in 1289 ms # (from function `setup' in test file test/test_basic_auth.bats, line 24) # `solr start -Dsolr.packages.enabled=true' failed # Starting Solr # # Port 42883 is already being used by another process (pid: 1113285) # Please choose a different port using the -p option. # # Please find the SOLR_HOME snapshot for failed test #1 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/9-1 # Last output: # WARN - 2026-05-14 03:48:09.091; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-14 03:48:09.094; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes ok 10 post, postlogs and export with basic auth # in 17551 ms ok 11 package with basic auth # in 11932 ms ok 12 admin UI static content is excepted from basic auth # in 10547 ms ok 13 nothing # in 10 ms ok 14 server does not compress response without Accept-Encoding header # in 158 ms ok 15 server compresses response when Accept-Encoding: gzip is requested # in 79 ms ok 16 compressed response can be decompressed and parsed # in 78 ms ok 17 setting property # in 10991 ms ok 18 short form of setting property # in 6349 ms ok 19 connecting to solr via various solr urls and zk hosts # in 9337 ms ok 20 create for non cloud mode # in 7817 ms ok 21 create for cloud mode # in 9761 ms ok 22 multiple connection options are prevented # in 860 ms ok 23 create collection # in 7644 ms ok 24 create collection using solrUrl # in 5221 ms ok 25 create collection using legacy solrUrl # in 5213 ms ok 26 create collection using Zookeeper # in 5247 ms ok 27 reject d option with invalid config dir # in 2656 ms ok 28 accept d option with builtin config # in 5158 ms ok 29 accept d option with explicit path to config # in 5257 ms ok 30 accept n option as config name # in 5290 ms ok 31 allow config reuse when n option specifies same config # in 8988 ms ok 32 create multisharded collections when s provided # in 5185 ms ok 33 create replicated collections when rf provided # in 6122 ms ok 34 can delete collections # in 10613 ms ok 35 can delete collections with solr-url # in 8294 ms ok 36 collection delete also deletes zk config # in 7284 ms ok 37 deletes accompanying zk config with nondefault name # in 7236 ms ok 38 delete-config option can opt to leave config in zk # in 7186 ms ok 39 start -e cloud works with --jvm-opts # in 21881 ms ok 40 start -e cloud works with --no-prompt # in 46125 ms ok 41 Check export command # in 19080 ms ok 42 export fails on non cloud mode # in 9415 ms Waiting for Tika Server to be ready on port 42888 WARNING: Tika Server did not become ready in time; Tika-dependent tests will be bypassed. WARNING: Docker not available; bypassing test. ok 43 using curl to extract a single pdf file # in 16602 ms WARNING: Docker not available; bypassing test. ok 44 using the bin/solr post tool to extract content from pdf # in 16550 ms WARNING: Docker not available; bypassing test. ok 45 using the bin/solr post tool to crawl web site # in 16573 ms Stopping Tika Server container ok 46 healthcheck on cloud solr # in 10174 ms ok 47 healthcheck errors on standalone solr # in 11209 ms ok 48 solr help flag prints help # in 728 ms ok 49 solr with no flags prints help # in 731 ms ok 50 start help flag prints help # in 142 ms ok 51 start h flag prints help # in 146 ms ok 52 stop help flag prints help # in 139 ms ok 53 restart help flag prints help # in 147 ms ok 54 status help flag prints help # in 803 ms ok 55 healthcheck help flag prints help # in 764 ms ok 56 create help flag prints help # in 772 ms ok 57 delete help flag prints help # in 742 ms ok 58 zk help flag prints help # in 750 ms ok 59 auth help flag prints help # in 767 ms ok 60 assert help flag prints help # in 760 ms ok 61 post help flag prints help # in 766 ms ok 62 SQL Module # in 15713 ms ok 63 icu collation in analysis-extras module # in 12276 ms ok 64 Check lifecycle of sentiment classification # in 23526 ms ok 65 lifecycle of package # in 8531 ms ok 66 deploying and undeploying a collection level package # in 13678 ms ok 67 deploying and undeploying a cluster level package # skip For developing package infra; requires a connection to github ok 68 Affinity placement plugin using sysprop # in 15563 ms ok 69 Random placement plugin using ENV # in 15311 ms ok 70 Check help command # in 2327 ms ok 71 basic post with a type specified # in 5394 ms ok 72 basic post with collection instead of url parameter # in 4127 ms ok 73 basic post with solr-url and collection # in 3902 ms ok 74 basic post WITHOUT a type specified # in 11385 ms ok 75 crawling a directory as a dry-run # in 790 ms ok 76 crawling a directory # in 4416 ms ok 77 crawling a web site # in 34882 ms ok 78 skipcommit and optimize and delete # in 4926 ms ok 79 args mode # in 9070 ms ok 80 stdin mode # in 3831 ms not ok 81 verbose echo the Solr response # in 11893 ms # (from function `refute_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 360, # in test file test/test_post.bats, line 216) # `refute_output --partial 'ERROR'' failed # # -- output should not contain substring -- # substring (1 lines): # ERROR # output (47 lines): # Posting files to [base] url http://localhost:42883/solr/monitors_verbose/update using content-type application/xml... # POSTing file monitor.xml to [base] # <?xml version="1.0" encoding="UTF-8"?> # <response> # # <lst name="responseHeader"> # <int name="rf">1</int> # <int name="status">0</int> # <int name="QTime">3108</int> # </lst> # </response> # 1 files indexed. # COMMITting Solr index changes to http://localhost:42883/solr/monitors_verbose/update... # # ERROR: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update # # org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update # at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:511) # at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302) # at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466) # at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442) # at org.apache.solr.cli.PostTool.commit(PostTool.java:778) # at org.apache.solr.cli.PostTool.execute(PostTool.java:346) # at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320) # at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81) # at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117) # Caused by: java.nio.channels.ClosedChannelException # at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2141) # at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1179) # at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:389) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:526) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:251) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196) # at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:216) # at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157) # at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:468) # at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) # at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) # at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.picRunTask(AdaptiveExecutionStrategy.java:473) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:399) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196) # at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408) # at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) # at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) # at java.base/java.lang.Thread.run(Thread.java:1583) # -- # # Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12 # Last output: # Posting files to [base] url http://localhost:42883/solr/monitors_verbose/update using content-type application/xml... # POSTing file monitor.xml to [base] # <?xml version="1.0" encoding="UTF-8"?> # <response> # # <lst name="responseHeader"> # <int name="rf">1</int> # <int name="status">0</int> # <int name="QTime">3108</int> # </lst> # </response> # 1 files indexed. # COMMITting Solr index changes to http://localhost:42883/solr/monitors_verbose/update... # # ERROR: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update # # org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://localhost:42883/solr/monitors_verbose/update # at org.apache.solr.client.solrj.jetty.HttpJettySolrClient.request(HttpJettySolrClient.java:511) # at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:302) # at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:466) # at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:442) # at org.apache.solr.cli.PostTool.commit(PostTool.java:778) # at org.apache.solr.cli.PostTool.execute(PostTool.java:346) # at org.apache.solr.cli.PostTool.runImpl(PostTool.java:320) # at org.apache.solr.cli.ToolBase.runTool(ToolBase.java:81) # at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:117) # Caused by: java.nio.channels.ClosedChannelException # at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:2141) # at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:1179) # at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:389) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:526) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:251) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196) # at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:216) # at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:157) # at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:468) # at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) # at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) # at org.eclipse.jetty.util.thread.Invocable.invokeNonBlocking(Invocable.java:224) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.picRunTask(AdaptiveExecutionStrategy.java:473) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:399) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255) # at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196) # at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:408) # at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) # at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) # at java.base/java.lang.Thread.run(Thread.java:1583) ok 82 post solr log into solr via cli # in 9482 ms # Test failed - capturing Docker diagnostics # === docker ps (summary) === NAMES STATUS IMAGE PORTS solr-node3 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node2 Up 10 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node1 Up 15 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp # === Docker logs for solr-node1 === 2026-05-14T03:58:52.489921273Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-14T03:58:52.489962680Z It should be set to 65000 to avoid operational disruption. 2026-05-14T03:58:52.489967439Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-14T03:58:52.490221965Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-14T03:58:52.490233893Z 2026-05-14T03:58:52.499451669Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-14T03:58:52.506915283Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-14T03:58:52.549277905Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-14T03:58:52.580171633Z WARNING: A command line option has enabled the Security Manager 2026-05-14T03:58:52.580193485Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-14T03:58:53.758239767Z 2026-05-14 03:58:53.754 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-14T03:58:54.030706437Z 2026-05-14 03:58:54.030 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-14T03:58:54.055421273Z 2026-05-14 03:58:54.054 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-14T03:58:54.062164720Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-14T03:58:54.062240329Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983 2026-05-14T03:58:54.062378234Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-14T03:58:54.062528962Z 2026-05-14 03:58:54.061 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-14T03:58:54.061953044Z 2026-05-14T03:58:54.065007446Z 2026-05-14 03:58:54.064 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-14T03:58:54.086374389Z 2026-05-14 03:58:54.085 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-14T03:58:54.099573459Z 2026-05-14 03:58:54.099 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-14T03:58:54.099635941Z 2026-05-14 03:58:54.099 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-14T03:58:54.133374425Z 2026-05-14 03:58:54.132 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-14T03:58:54.913841196Z 2026-05-14 03:58:54.913 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-14T03:58:55.154995007Z 2026-05-14 03:58:55.154 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-14T03:58:55.159953335Z 2026-05-14 03:58:55.159 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default 2026-05-14T03:58:55.160154173Z 2026-05-14 03:58:55.159 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg 2026-05-14T03:58:55.162376609Z 2026-05-14 03:58:55.162 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0 2026-05-14T03:58:55.162446736Z 2026-05-14 03:58:55.162 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details. 2026-05-14T03:58:55.198139991Z 2026-05-14 03:58:55.197 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0. 2026-05-14T03:58:55.666316018Z 2026-05-14 03:58:55.664 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983 2026-05-14T03:58:55.679562911Z 2026-05-14 03:58:55.679 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-14T03:58:55.739566107Z 2026-05-14 03:58:55.739 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-14T03:58:55.766486901Z 2026-05-14 03:58:55.765 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-14T03:58:55.766563759Z 2026-05-14 03:58:55.766 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-14T03:58:55.907897480Z 2026-05-14 03:58:55.906 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-14T03:58:55.932636689Z 2026-05-14 03:58:55.932 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-14T03:58:55.945466397Z 2026-05-14 03:58:55.944 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr 2026-05-14T03:58:55.957346057Z 2026-05-14 03:58:55.956 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72130694953107456-solr-node1:8983_solr-n_0000000000) starting 2026-05-14T03:58:56.027515852Z 2026-05-14 03:58:56.026 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr 2026-05-14T03:58:56.028311027Z 2026-05-14 03:58:56.027 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr 2026-05-14T03:58:56.038568226Z 2026-05-14 03:58:56.038 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-14T03:58:56.082100115Z 2026-05-14 03:58:56.081 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-14T03:58:56.441979105Z 2026-05-14 03:58:56.441 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-14T03:58:56.607074399Z 2026-05-14 03:58:56.606 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-14T03:58:56.868587264Z 2026-05-14 03:58:56.868 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-14T03:58:57.001656575Z 2026-05-14 03:58:57.000 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-14T03:58:57.007259820Z 2026-05-14 03:58:57.006 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_14.request.log 2026-05-14T03:58:57.011375645Z 2026-05-14 03:58:57.010 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2026-05-14T03:58:57.013161905Z 2026-05-14 03:58:57.012 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4507ms 2026-05-14T03:58:57.297774553Z 2026-05-14 03:58:57.296 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=138 2026-05-14T03:58:57.363851160Z 2026-05-14 03:58:57.363 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13 2026-05-14T03:59:00.773650676Z 2026-05-14 03:59:00.772 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-14T03:59:00.901867149Z 2026-05-14 03:59:00.901 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-14T03:59:05.132383656Z 2026-05-14 03:59:05.131 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8 2026-05-14T03:59:05.159289342Z 2026-05-14 03:59:05.158 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=8 2026-05-14T03:59:05.186129234Z 2026-05-14 03:59:05.185 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2 2026-05-14T03:59:05.466691142Z 2026-05-14 03:59:05.465 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection 2026-05-14T03:59:05.481844746Z 2026-05-14 03:59:05.481 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children 2026-05-14T03:59:05.487416836Z 2026-05-14 03:59:05.486 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:05.532519327Z 2026-05-14 03:59:05.531 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-14T03:59:05.532545612Z "core":"test-collection_shard3_replica_n1", 2026-05-14T03:59:05.532549289Z "node_name":"solr-node1:8983_solr", 2026-05-14T03:59:05.532551952Z "base_url":"http://solr-node1:8983/solr";, 2026-05-14T03:59:05.532554561Z "collection":"test-collection", 2026-05-14T03:59:05.532557249Z "shard":"shard3", 2026-05-14T03:59:05.532559752Z "state":"down", 2026-05-14T03:59:05.532562276Z "type":"NRT", 2026-05-14T03:59:05.532564790Z "operation":"ADDREPLICA", 2026-05-14T03:59:05.532567330Z "waitForFinalState":"false"} 2026-05-14T03:59:05.551053786Z 2026-05-14 03:59:05.547 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-14T03:59:05.551075935Z "core":"test-collection_shard2_replica_n2", 2026-05-14T03:59:05.551079163Z "node_name":"solr-node3:8985_solr", 2026-05-14T03:59:05.551081921Z "base_url":"http://solr-node3:8985/solr";, 2026-05-14T03:59:05.551084433Z "collection":"test-collection", 2026-05-14T03:59:05.551087039Z "shard":"shard2", 2026-05-14T03:59:05.551089435Z "state":"down", 2026-05-14T03:59:05.551091926Z "type":"NRT", 2026-05-14T03:59:05.551094324Z "operation":"ADDREPLICA", 2026-05-14T03:59:05.551096767Z "waitForFinalState":"false"} 2026-05-14T03:59:05.559382641Z 2026-05-14 03:59:05.556 INFO (OverseerStateUpdate-72130694953107456-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-14T03:59:05.559402678Z "core":"test-collection_shard1_replica_n4", 2026-05-14T03:59:05.559405924Z "node_name":"solr-node2:8984_solr", 2026-05-14T03:59:05.559420416Z "base_url":"http://solr-node2:8984/solr";, 2026-05-14T03:59:05.559422502Z "collection":"test-collection", 2026-05-14T03:59:05.559424402Z "shard":"shard1", 2026-05-14T03:59:05.559426286Z "state":"down", 2026-05-14T03:59:05.559428213Z "type":"NRT", 2026-05-14T03:59:05.559430163Z "operation":"ADDREPLICA", 2026-05-14T03:59:05.559432092Z "waitForFinalState":"false"} 2026-05-14T03:59:05.673572161Z 2026-05-14 03:59:05.672 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:05.805398475Z 2026-05-14 03:59:05.804 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin 2026-05-14T03:59:05.870185973Z 2026-05-14 03:59:05.869 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-14T03:59:05.933255968Z 2026-05-14 03:59:05.932 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-14T03:59:06.314755381Z 2026-05-14 03:59:06.314 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-14T03:59:06.314852410Z 2026-05-14 03:59:06.314 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true 2026-05-14T03:59:06.340912556Z 2026-05-14 03:59:06.340 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/] 2026-05-14T03:59:07.239268718Z 2026-05-14 03:59:07.237 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-14T03:59:07.286389304Z 2026-05-14 03:59:07.285 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-14T03:59:07.297650445Z 2026-05-14 03:59:07.297 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-14T03:59:07.299726615Z 2026-05-14 03:59:07.298 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-14T03:59:07.299822303Z 2026-05-14 03:59:07.299 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-14T03:59:07.382731565Z 2026-05-14 03:59:07.382 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-14T03:59:07.386339373Z 2026-05-14 03:59:07.385 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-14T03:59:07.388291920Z 2026-05-14 03:59:07.387 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-14T03:59:07.389765969Z 2026-05-14 03:59:07.389 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-14T03:59:07.404047565Z 2026-05-14 03:59:07.403 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-14T03:59:07.417302943Z 2026-05-14 03:59:07.416 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-14T03:59:07.428750930Z 2026-05-14 03:59:07.428 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-14T03:59:07.428786621Z 2026-05-14 03:59:07.428 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-14T03:59:07.443569826Z 2026-05-14 03:59:07.442 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm 2026-05-14T03:59:07.445238007Z 2026-05-14 03:59:07.444 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3 2026-05-14T03:59:07.449335883Z 2026-05-14 03:59:07.449 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-14T03:59:07.459895326Z 2026-05-14 03:59:07.459 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-14T03:59:07.459918610Z 2026-05-14 03:59:07.459 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-14T03:59:07.459921975Z 2026-05-14 03:59:07.459 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ 2026-05-14T03:59:07.461752618Z 2026-05-14 03:59:07.461 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-14T03:59:07.462268781Z 2026-05-14 03:59:07.462 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas 2026-05-14T03:59:07.465491326Z 2026-05-14 03:59:07.465 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72130694953107456-core_node3-n_0000000000 2026-05-14T03:59:07.471350709Z 2026-05-14 03:59:07.471 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3 2026-05-14T03:59:07.577413000Z 2026-05-14 03:59:07.576 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.582933708Z 2026-05-14 03:59:07.582 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-14T03:59:07.591684567Z 2026-05-14 03:59:07.591 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1790 2026-05-14T03:59:07.690288817Z 2026-05-14 03:59:07.689 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node1 === { "Name": "/solr-node1", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1144308, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-14T03:58:52.281986301Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node2 === 2026-05-14T03:58:57.663046158Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-14T03:58:57.663072603Z It should be set to 65000 to avoid operational disruption. 2026-05-14T03:58:57.663075675Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-14T03:58:57.663357177Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-14T03:58:57.663361038Z 2026-05-14T03:58:57.672714484Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-14T03:58:57.680730801Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-14T03:58:57.722660961Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-14T03:58:57.753609234Z WARNING: A command line option has enabled the Security Manager 2026-05-14T03:58:57.753626314Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-14T03:58:58.937812620Z 2026-05-14 03:58:58.934 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-14T03:58:59.215473566Z 2026-05-14 03:58:59.214 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-14T03:58:59.240008410Z 2026-05-14 03:58:59.239 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-14T03:58:59.246867919Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-14T03:58:59.246898592Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984 2026-05-14T03:58:59.247023152Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-14T03:58:59.247199007Z 2026-05-14 03:58:59.246 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-14T03:58:59.246490413Z 2026-05-14T03:58:59.249575262Z 2026-05-14 03:58:59.249 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-14T03:58:59.270734725Z 2026-05-14 03:58:59.270 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-14T03:58:59.293893426Z 2026-05-14 03:58:59.293 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-14T03:58:59.436852358Z 2026-05-14 03:58:59.436 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-14T03:58:59.455275618Z 2026-05-14 03:58:59.454 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-14T03:58:59.455440808Z 2026-05-14 03:58:59.454 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-14T03:58:59.455570771Z 2026-05-14 03:58:59.455 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-14T03:58:59.565974431Z 2026-05-14 03:58:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-14T03:58:59.566114392Z 2026-05-14 03:58:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-14T03:58:59.566251022Z 2026-05-14 03:58:59.565 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-14T03:58:59.605195612Z 2026-05-14 03:58:59.604 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-14T03:59:00.455697248Z 2026-05-14 03:59:00.455 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-14T03:59:00.613900062Z 2026-05-14 03:59:00.613 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-14T03:59:00.615223544Z 2026-05-14 03:59:00.613 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-14T03:59:00.624357395Z 2026-05-14 03:59:00.623 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-14T03:59:00.629593566Z 2026-05-14 03:59:00.629 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-14T03:59:00.633474073Z 2026-05-14 03:59:00.633 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-14T03:59:00.633564309Z 2026-05-14 03:59:00.633 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-14T03:59:00.667856564Z 2026-05-14 03:59:00.667 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-14T03:59:00.707931564Z 2026-05-14 03:59:00.706 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-14T03:59:00.738078283Z 2026-05-14 03:59:00.737 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-14T03:59:00.750287424Z 2026-05-14 03:59:00.749 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN 2026-05-14T03:59:00.761687616Z 2026-05-14 03:59:00.761 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr 2026-05-14T03:59:00.774708614Z 2026-05-14 03:59:00.774 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-14T03:59:00.818238563Z 2026-05-14 03:59:00.817 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-14T03:59:00.905193040Z 2026-05-14 03:59:00.904 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-14T03:59:01.173838006Z 2026-05-14 03:59:01.173 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-14T03:59:01.431267727Z 2026-05-14 03:59:01.430 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-14T03:59:01.772111607Z 2026-05-14 03:59:01.770 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-14T03:59:01.956932857Z 2026-05-14 03:59:01.956 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-14T03:59:01.966186996Z 2026-05-14 03:59:01.965 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_14.request.log 2026-05-14T03:59:01.970888066Z 2026-05-14 03:59:01.970 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984} 2026-05-14T03:59:01.973275011Z 2026-05-14 03:59:01.972 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4293ms 2026-05-14T03:59:02.242850000Z 2026-05-14 03:59:02.241 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=50 2026-05-14T03:59:02.302825473Z 2026-05-14 03:59:02.302 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=15 2026-05-14T03:59:05.796904644Z 2026-05-14 03:59:05.796 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin 2026-05-14T03:59:05.853438638Z 2026-05-14 03:59:05.852 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-14T03:59:05.932490996Z 2026-05-14 03:59:05.931 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-14T03:59:06.252588796Z 2026-05-14 03:59:06.252 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-14T03:59:06.252612757Z 2026-05-14 03:59:06.252 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true 2026-05-14T03:59:06.272293458Z 2026-05-14 03:59:06.271 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/] 2026-05-14T03:59:06.710306260Z 2026-05-14 03:59:06.709 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-14T03:59:06.733480895Z 2026-05-14 03:59:06.732 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-14T03:59:06.741139343Z 2026-05-14 03:59:06.740 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-14T03:59:06.742508861Z 2026-05-14 03:59:06.741 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-14T03:59:06.742525269Z 2026-05-14 03:59:06.742 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-14T03:59:06.789602393Z 2026-05-14 03:59:06.789 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-14T03:59:06.793397457Z 2026-05-14 03:59:06.792 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-14T03:59:06.795648537Z 2026-05-14 03:59:06.795 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-14T03:59:06.797449060Z 2026-05-14 03:59:06.797 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-14T03:59:06.811537755Z 2026-05-14 03:59:06.810 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-14T03:59:06.820661221Z 2026-05-14 03:59:06.820 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-14T03:59:06.830334920Z 2026-05-14 03:59:06.829 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-14T03:59:06.830360884Z 2026-05-14 03:59:06.829 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-14T03:59:06.838239664Z 2026-05-14 03:59:06.837 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-14T03:59:06.850490893Z 2026-05-14 03:59:06.849 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm 2026-05-14T03:59:06.852175272Z 2026-05-14 03:59:06.851 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1 2026-05-14T03:59:06.875630388Z 2026-05-14 03:59:06.875 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-14T03:59:06.875729038Z 2026-05-14 03:59:06.875 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-14T03:59:06.875748998Z 2026-05-14 03:59:06.875 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ 2026-05-14T03:59:06.877235155Z 2026-05-14 03:59:06.876 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-14T03:59:06.878069788Z 2026-05-14 03:59:06.877 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas 2026-05-14T03:59:06.879395487Z 2026-05-14 03:59:06.879 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72130694953107459-core_node6-n_0000000000 2026-05-14T03:59:06.890521427Z 2026-05-14 03:59:06.890 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1 2026-05-14T03:59:07.179875231Z 2026-05-14 03:59:07.179 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.184605447Z 2026-05-14 03:59:07.184 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-14T03:59:07.195095379Z 2026-05-14 03:59:07.194 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1399 2026-05-14T03:59:07.364093474Z 2026-05-14 03:59:07.363 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.577765148Z 2026-05-14 03:59:07.577 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.690689459Z 2026-05-14 03:59:07.690 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node2 === { "Name": "/solr-node2", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1144703, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-14T03:58:57.452471801Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node3 === 2026-05-14T03:58:57.890464787Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-14T03:58:57.890504791Z It should be set to 65000 to avoid operational disruption. 2026-05-14T03:58:57.890508388Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-14T03:58:57.890773289Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-14T03:58:57.890801572Z 2026-05-14T03:58:57.900536964Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-14T03:58:57.908953304Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-14T03:58:57.957767606Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-14T03:58:57.989945337Z WARNING: A command line option has enabled the Security Manager 2026-05-14T03:58:57.989970091Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-14T03:58:59.148239891Z 2026-05-14 03:58:59.144 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-14T03:58:59.417848969Z 2026-05-14 03:58:59.417 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-14T03:58:59.444020698Z 2026-05-14 03:58:59.443 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-14T03:58:59.451563297Z 2026-05-14 03:58:59.450 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-14T03:58:59.451585500Z 2026-05-14 03:58:59.451 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985 2026-05-14T03:58:59.451706926Z 2026-05-14 03:58:59.451 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-14T03:58:59.451844704Z 2026-05-14 03:58:59.451 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-14T03:58:59.451249098Z 2026-05-14T03:58:59.454754900Z 2026-05-14 03:58:59.454 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-14T03:58:59.477280340Z 2026-05-14 03:58:59.476 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-14T03:58:59.501312262Z 2026-05-14 03:58:59.500 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-14T03:58:59.646816369Z 2026-05-14 03:58:59.646 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-14T03:58:59.666792371Z 2026-05-14 03:58:59.666 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-14T03:58:59.666864765Z 2026-05-14 03:58:59.666 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-14T03:58:59.666996413Z 2026-05-14 03:58:59.666 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-14T03:58:59.776253466Z 2026-05-14 03:58:59.775 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-14T03:58:59.776561903Z 2026-05-14 03:58:59.776 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-14T03:58:59.776693015Z 2026-05-14 03:58:59.776 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-14T03:58:59.812192357Z 2026-05-14 03:58:59.811 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-14T03:59:00.647394937Z 2026-05-14 03:59:00.646 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-14T03:59:00.783123490Z 2026-05-14 03:59:00.782 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-14T03:59:00.784457380Z 2026-05-14 03:59:00.782 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-14T03:59:00.793454234Z 2026-05-14 03:59:00.793 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-14T03:59:00.798514987Z 2026-05-14 03:59:00.798 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-14T03:59:00.801873112Z 2026-05-14 03:59:00.801 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-14T03:59:00.802127529Z 2026-05-14 03:59:00.801 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-14T03:59:00.834871005Z 2026-05-14 03:59:00.834 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-14T03:59:00.844605617Z 2026-05-14 03:59:00.844 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) 2026-05-14T03:59:00.865676694Z 2026-05-14 03:59:00.865 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-14T03:59:00.878802944Z 2026-05-14 03:59:00.878 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN 2026-05-14T03:59:00.891674749Z 2026-05-14 03:59:00.891 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr 2026-05-14T03:59:00.903902935Z 2026-05-14 03:59:00.903 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-14T03:59:00.943988145Z 2026-05-14 03:59:00.943 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-14T03:59:01.228763534Z 2026-05-14 03:59:01.228 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-14T03:59:01.420570885Z 2026-05-14 03:59:01.419 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-14T03:59:01.728369034Z 2026-05-14 03:59:01.727 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-14T03:59:01.871283804Z 2026-05-14 03:59:01.870 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-14T03:59:01.878148635Z 2026-05-14 03:59:01.877 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_14.request.log 2026-05-14T03:59:01.882065836Z 2026-05-14 03:59:01.881 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985} 2026-05-14T03:59:01.883154553Z 2026-05-14 03:59:01.882 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3975ms 2026-05-14T03:59:03.974888370Z 2026-05-14 03:59:03.969 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=67 2026-05-14T03:59:04.042766822Z 2026-05-14 03:59:04.041 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=22 2026-05-14T03:59:05.431165535Z 2026-05-14 03:59:05.430 INFO (qtp985324122-58-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=9 2026-05-14T03:59:05.796910967Z 2026-05-14 03:59:05.796 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin 2026-05-14T03:59:05.857066010Z 2026-05-14 03:59:05.856 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-14T03:59:05.927027548Z 2026-05-14 03:59:05.926 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-14T03:59:06.392167340Z 2026-05-14 03:59:06.391 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-14T03:59:06.392384945Z 2026-05-14 03:59:06.391 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true 2026-05-14T03:59:06.410370259Z 2026-05-14 03:59:06.409 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/] 2026-05-14T03:59:06.865434296Z 2026-05-14 03:59:06.864 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-14T03:59:06.889861081Z 2026-05-14 03:59:06.889 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-14T03:59:06.899182839Z 2026-05-14 03:59:06.898 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-14T03:59:06.900259017Z 2026-05-14 03:59:06.899 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-14T03:59:06.900345354Z 2026-05-14 03:59:06.900 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-14T03:59:06.948168790Z 2026-05-14 03:59:06.947 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-14T03:59:06.951564003Z 2026-05-14 03:59:06.951 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-14T03:59:07.178102926Z 2026-05-14 03:59:07.176 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-14T03:59:07.181085263Z 2026-05-14 03:59:07.180 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-14T03:59:07.196040210Z 2026-05-14 03:59:07.195 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2026-05-14T03:59:07.205571992Z 2026-05-14 03:59:07.204 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-14T03:59:07.213411388Z 2026-05-14 03:59:07.212 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-14T03:59:07.213456655Z 2026-05-14 03:59:07.213 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-14T03:59:07.220350349Z 2026-05-14 03:59:07.219 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-14T03:59:07.228691484Z 2026-05-14 03:59:07.228 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2026-05-14T03:59:07.230231066Z 2026-05-14 03:59:07.229 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2 2026-05-14T03:59:07.242980867Z 2026-05-14 03:59:07.242 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-14T03:59:07.243003214Z 2026-05-14 03:59:07.242 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-14T03:59:07.243059336Z 2026-05-14 03:59:07.242 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ 2026-05-14T03:59:07.244457825Z 2026-05-14 03:59:07.244 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-14T03:59:07.245117314Z 2026-05-14 03:59:07.244 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas 2026-05-14T03:59:07.246374648Z 2026-05-14 03:59:07.246 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72130694953107460-core_node5-n_0000000000 2026-05-14T03:59:07.256887314Z 2026-05-14 03:59:07.256 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2 2026-05-14T03:59:07.364285761Z 2026-05-14 03:59:07.363 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.368562322Z 2026-05-14 03:59:07.368 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-14T03:59:07.382341669Z 2026-05-14 03:59:07.381 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1587 2026-05-14T03:59:07.577822128Z 2026-05-14 03:59:07.577 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.603827937Z 2026-05-14 03:59:07.603 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2026-05-14T03:59:07.690657707Z 2026-05-14 03:59:07.690 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-14T03:59:07.693407698Z 2026-05-14 03:59:07.693 INFO (qtp985324122-59-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2248 # === Docker inspect for solr-node3 === { "Name": "/solr-node3", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1144889, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-14T03:58:57.679867425Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker not ok 83 Docker SolrCloud rolling upgrade # in 26755 ms # (in test file test/test_rolling_upgrade.bats, line 118) # `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127 # 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # af6eca94c810: Pulling fs layer # cb0efb96dabd: Pulling fs layer # 3e9d91201f40: Pulling fs layer # 66b76b382631: Pulling fs layer # 601f2c23751f: Pulling fs layer # 0ed719aef393: Pulling fs layer # 69a09c96e8bf: Pulling fs layer # c937aee9a870: Pulling fs layer # 9cdfe5320de8: Pulling fs layer # 3aa9120e46f5: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # 66b76b382631: Waiting # 601f2c23751f: Waiting # 0ed719aef393: Waiting # 69a09c96e8bf: Waiting # c937aee9a870: Waiting # 9cdfe5320de8: Waiting # 3aa9120e46f5: Waiting # 4f4fb700ef54: Waiting # af6eca94c810: Verifying Checksum # af6eca94c810: Download complete # cb0efb96dabd: Verifying Checksum # cb0efb96dabd: Download complete # 66b76b382631: Verifying Checksum # 66b76b382631: Download complete # 3e9d91201f40: Verifying Checksum # 3e9d91201f40: Download complete # 601f2c23751f: Verifying Checksum # 601f2c23751f: Download complete # c937aee9a870: Verifying Checksum # c937aee9a870: Download complete # 69a09c96e8bf: Verifying Checksum # 69a09c96e8bf: Download complete # 3aa9120e46f5: Verifying Checksum # 3aa9120e46f5: Download complete # 9cdfe5320de8: Verifying Checksum # 9cdfe5320de8: Download complete # 4f4fb700ef54: Verifying Checksum # 4f4fb700ef54: Download complete # 0ed719aef393: Verifying Checksum # 0ed719aef393: Download complete # af6eca94c810: Pull complete # cb0efb96dabd: Pull complete # 3e9d91201f40: Pull complete # 66b76b382631: Pull complete # 601f2c23751f: Pull complete # 0ed719aef393: Pull complete # 69a09c96e8bf: Pull complete # c937aee9a870: Pull complete # 9cdfe5320de8: Pull complete # 3aa9120e46f5: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203 # Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim # 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # f63eb04151bc: Already exists # 87f3bb8db2f4: Already exists # 7c0b62ebdf15: Already exists # 5b95479e5f9b: Already exists # c913227527c9: Already exists # a34ba287db10: Pulling fs layer # d1122302140a: Pulling fs layer # e63de72c09be: Pulling fs layer # ddd33485f40b: Pulling fs layer # 67aedfcd29ab: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # ddd33485f40b: Waiting # 67aedfcd29ab: Waiting # 4f4fb700ef54: Waiting # d1122302140a: Verifying Checksum # d1122302140a: Download complete # e63de72c09be: Download complete # ddd33485f40b: Download complete # 4f4fb700ef54: Download complete # 67aedfcd29ab: Verifying Checksum # 67aedfcd29ab: Download complete # a34ba287db10: Verifying Checksum # a34ba287db10: Download complete # a34ba287db10: Pull complete # d1122302140a: Pull complete # e63de72c09be: Pull complete # ddd33485f40b: Pull complete # 67aedfcd29ab: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:ade140d83779bd4ea80626422a06cc925e99b05dd5c0df36ef88ceae30d3b74a # Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim # 5de6877746e1717837de99a07fbb11933201e4d11914e49878095ea74796c910 # solr-data1 # solr-data2 # solr-data3 # Starting solr-node1 with embedded ZooKeeper # 27821a8d01463d225b1b94a437f2b59def0356d9b6d91ad0b81d4a4ff7d77dea # Starting solr-node2 connected to first node's ZooKeeper # c16a39090f53bd3a261d9ea3ade361682ed9687e5afb8096a2d8cb7254ad8a9d # Started solr-node3 connected to first node's ZooKeeper # 5c76b1370d53d7cae07e72280f59f6699d1a4a049c4f0b2cc8a60e0abfb81c6d # Creating a Collection # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # WARN - 2026-05-14 03:59:05.225; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-14 03:59:05.257; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/]. # Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts' # Checking collection health # /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found # solr-node1 # solr-node2 # solr-node3 # solr-node1 # solr-node2 # solr-node3 # solr-data1 # solr-data2 # solr-data3 # solrcloud-test ok 84 allowPaths - backup # in 16409 ms ok 85 snapshot lifecycle # in 4845 ms ok 86 snapshot list # in 3355 ms ok 87 snapshot describe # in 3349 ms ok 88 start solr with ssl # in 16914 ms ok 89 use different hostname when not checking peer-name # in 25410 ms ok 90 start solr with ssl and auth # in 14784 ms ok 91 start solr with client truststore and security manager # in 16947 ms ok 92 start solr with mTLS needed # in 47502 ms ok 93 start solr with mTLS wanted # in 36514 ms ok 94 test keystore reload # in 60697 ms ok 95 SOLR-11740 check 'solr stop' connection # in 16188 ms ok 96 stop command for single port # in 18727 ms ok 97 check stop command doesn't hang # in 224 ms ok 98 SOLR-16976 solr starts with remote JMX enabled # in 8895 ms ok 99 deprecated system properties converted to modern properties # in 195316 ms ok 100 start with custom jetty options # in 8305 ms ok 101 bootstrapping a configset # in 9448 ms ok 102 status detects locally running solr # in 9654 ms ok 103 status with --solr-url from user # in 8456 ms ok 104 status with --port from user # in 8650 ms ok 105 multiple connection options are prevented # in 881 ms ok 106 status with invalid --solr-url from user # in 1286 ms ok 107 status with --short format # in 196108 ms ok 108 searching solr via locally executed streaming expression # in 1748 ms ok 109 searching solr via remotely executed streaming expression # in 1307 ms ok 110 variable interpolation # in 1698 ms ok 111 searching solr without credentials fails with error # in 1152 ms ok 112 --version returns Solr version # in 746 ms ok 113 version as direct tool call still runs # in 752 ms ok 114 short help # in 860 ms ok 115 short help is inferred # in 792 ms ok 116 long help # in 847 ms ok 117 running subcommands with zk is prevented # in 733 ms ok 118 listing out files # in 2184 ms ok 119 connecting to solr via various solr urls and zk hosts # in 8255 ms ok 120 copying files around # in 11170 ms ok 121 upconfig # in 2680 ms ok 122 SOLR-12429 test upconfig fails with symlink # in 1239 ms ok 123 downconfig # in 1214 ms ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1497 ms ok 125 env var ZK_HOST is honored # in 3312 ms ok 126 Cleanup # in 1890 ms > Task :solr:packaging:integrationTests FAILED FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339 * What went wrong: Execution failed for task ':solr:packaging:integrationTests'. > Process 'command > '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' > finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Get more help at https://help.gradle.org. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 30m 13s 128 actionable tasks: 128 executed Publishing Build Scan to Develocity... https://develocity.apache.org/s/4yaimmavwenku Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Archiving artifacts Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
