Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/202/
No tests ran. Build Log: [...truncated 425 lines...] # ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:40993/solr/admin/collections # -- # # Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11 # Last output: # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993. # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993. # WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is # NOT RECOMMENDED for production use. # # To turn it off: # curl http://localhost:40993/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}' # Or: # bin/solr config -c test_stdin -s http://localhost:40993 --action set-user-property --property update.autoCreateFields --value false # WARN - 2026-05-07 15:33:55.479; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-07 15:33:55.482; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # # ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:40993/solr/admin/collections not ok 81 verbose echo the Solr response # in 1065 ms # (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247, # in test file test/test_post.bats, line 210) # `assert_output --partial "Created collection 'monitors_verbose'"' failed # # -- output does not contain substring -- # substring (1 lines): # Created collection 'monitors_verbose' # output (3 lines): # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993. # # ERROR: Server refused connection at: http://localhost:40993/solr/admin/info/system # -- # # Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12 # Last output: # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:40993. # # ERROR: Server refused connection at: http://localhost:40993/solr/admin/info/system ok 82 post solr log into solr via cli # in 8788 ms # Test failed - capturing Docker diagnostics # === docker ps (summary) === NAMES STATUS IMAGE PORTS solr-node3 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node1 Up 22 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp # === Docker logs for solr-node1 === 2026-05-07T15:34:22.136581636Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-07T15:34:22.136612326Z It should be set to 65000 to avoid operational disruption. 2026-05-07T15:34:22.136615331Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-07T15:34:22.136880080Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-07T15:34:22.136890886Z 2026-05-07T15:34:22.171443068Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-07T15:34:22.179759880Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-07T15:34:22.229819487Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-07T15:34:22.264281706Z WARNING: A command line option has enabled the Security Manager 2026-05-07T15:34:22.264304653Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-07T15:34:24.075907719Z 2026-05-07 15:34:24.072 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-07T15:34:24.374379348Z 2026-05-07 15:34:24.373 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-07T15:34:24.428417968Z 2026-05-07 15:34:24.427 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-07T15:34:24.434805675Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-07T15:34:24.434888245Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983 2026-05-07T15:34:24.435000221Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-07T15:34:24.435126859Z 2026-05-07 15:34:24.434 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-07T15:34:24.434630016Z 2026-05-07T15:34:24.437572102Z 2026-05-07 15:34:24.437 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-07T15:34:24.458020535Z 2026-05-07 15:34:24.457 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-07T15:34:24.499251326Z 2026-05-07 15:34:24.498 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-07T15:34:24.499286581Z 2026-05-07 15:34:24.498 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-07T15:34:24.636708204Z 2026-05-07 15:34:24.635 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-07T15:34:25.506393075Z 2026-05-07 15:34:25.505 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-07T15:34:25.838075856Z 2026-05-07 15:34:25.837 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-07T15:34:25.859116745Z 2026-05-07 15:34:25.858 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default 2026-05-07T15:34:25.859219130Z 2026-05-07 15:34:25.858 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg 2026-05-07T15:34:25.865196285Z 2026-05-07 15:34:25.864 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0 2026-05-07T15:34:25.865276592Z 2026-05-07 15:34:25.864 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details. 2026-05-07T15:34:25.953568020Z 2026-05-07 15:34:25.953 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0. 2026-05-07T15:34:26.368901114Z 2026-05-07 15:34:26.368 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983 2026-05-07T15:34:26.382538175Z 2026-05-07 15:34:26.382 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-07T15:34:26.436388667Z 2026-05-07 15:34:26.435 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-07T15:34:26.468350930Z 2026-05-07 15:34:26.467 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-07T15:34:26.468461402Z 2026-05-07 15:34:26.468 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-07T15:34:26.601322202Z 2026-05-07 15:34:26.600 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-07T15:34:26.626818521Z 2026-05-07 15:34:26.626 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-07T15:34:26.640417378Z 2026-05-07 15:34:26.639 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr 2026-05-07T15:34:26.648711261Z 2026-05-07 15:34:26.648 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72093707922833408-solr-node1:8983_solr-n_0000000000) starting 2026-05-07T15:34:26.842939865Z 2026-05-07 15:34:26.842 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr 2026-05-07T15:34:26.844505598Z 2026-05-07 15:34:26.844 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr 2026-05-07T15:34:26.855519149Z 2026-05-07 15:34:26.854 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-07T15:34:26.904784200Z 2026-05-07 15:34:26.904 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-07T15:34:27.432672217Z 2026-05-07 15:34:27.432 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-07T15:34:27.685075571Z 2026-05-07 15:34:27.684 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-07T15:34:28.032954172Z 2026-05-07 15:34:28.032 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-07T15:34:28.331479711Z 2026-05-07 15:34:28.330 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-07T15:34:28.337221569Z 2026-05-07 15:34:28.336 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_07.request.log 2026-05-07T15:34:28.340345620Z 2026-05-07 15:34:28.339 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2026-05-07T15:34:28.341131817Z 2026-05-07 15:34:28.340 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @6162ms 2026-05-07T15:34:28.862961119Z 2026-05-07 15:34:28.862 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=48 2026-05-07T15:34:29.088906207Z 2026-05-07 15:34:29.088 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=21 2026-05-07T15:34:32.554017953Z 2026-05-07 15:34:32.553 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-07T15:34:36.208123944Z 2026-05-07 15:34:36.207 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-07T15:34:40.736944593Z 2026-05-07 15:34:40.735 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=76 2026-05-07T15:34:40.780331768Z 2026-05-07 15:34:40.768 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11 2026-05-07T15:34:40.794171467Z 2026-05-07 15:34:40.793 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2 2026-05-07T15:34:41.039682345Z 2026-05-07 15:34:41.038 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=28 2026-05-07T15:34:41.092024964Z 2026-05-07 15:34:41.091 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection 2026-05-07T15:34:41.118361480Z 2026-05-07 15:34:41.117 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children 2026-05-07T15:34:41.126431936Z 2026-05-07 15:34:41.126 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:41.180247976Z 2026-05-07 15:34:41.179 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-07T15:34:41.180268495Z "core":"test-collection_shard3_replica_n1", 2026-05-07T15:34:41.180271314Z "node_name":"solr-node1:8983_solr", 2026-05-07T15:34:41.180273445Z "base_url":"http://solr-node1:8983/solr";, 2026-05-07T15:34:41.180275439Z "collection":"test-collection", 2026-05-07T15:34:41.180277594Z "shard":"shard3", 2026-05-07T15:34:41.180279626Z "state":"down", 2026-05-07T15:34:41.180281549Z "type":"NRT", 2026-05-07T15:34:41.180283479Z "operation":"ADDREPLICA", 2026-05-07T15:34:41.180285452Z "waitForFinalState":"false"} 2026-05-07T15:34:41.186339396Z 2026-05-07 15:34:41.185 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-07T15:34:41.186356080Z "core":"test-collection_shard2_replica_n2", 2026-05-07T15:34:41.186359286Z "node_name":"solr-node3:8985_solr", 2026-05-07T15:34:41.186361533Z "base_url":"http://solr-node3:8985/solr";, 2026-05-07T15:34:41.186363496Z "collection":"test-collection", 2026-05-07T15:34:41.186365573Z "shard":"shard2", 2026-05-07T15:34:41.186367486Z "state":"down", 2026-05-07T15:34:41.186369391Z "type":"NRT", 2026-05-07T15:34:41.186371293Z "operation":"ADDREPLICA", 2026-05-07T15:34:41.186373469Z "waitForFinalState":"false"} 2026-05-07T15:34:41.191755874Z 2026-05-07 15:34:41.191 INFO (OverseerStateUpdate-72093707922833408-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-07T15:34:41.191777563Z "core":"test-collection_shard1_replica_n4", 2026-05-07T15:34:41.191780349Z "node_name":"solr-node2:8984_solr", 2026-05-07T15:34:41.191782368Z "base_url":"http://solr-node2:8984/solr";, 2026-05-07T15:34:41.191784426Z "collection":"test-collection", 2026-05-07T15:34:41.191786398Z "shard":"shard1", 2026-05-07T15:34:41.191788682Z "state":"down", 2026-05-07T15:34:41.191790892Z "type":"NRT", 2026-05-07T15:34:41.191792747Z "operation":"ADDREPLICA", 2026-05-07T15:34:41.191794679Z "waitForFinalState":"false"} 2026-05-07T15:34:41.301955403Z 2026-05-07 15:34:41.301 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:41.420069096Z 2026-05-07 15:34:41.419 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin 2026-05-07T15:34:41.583880776Z 2026-05-07 15:34:41.583 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-07T15:34:41.682133349Z 2026-05-07 15:34:41.681 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config 2026-05-07T15:34:42.063310301Z 2026-05-07 15:34:42.062 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-07T15:34:42.063342451Z 2026-05-07 15:34:42.062 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true 2026-05-07T15:34:42.094145357Z 2026-05-07 15:34:42.093 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/] 2026-05-07T15:34:43.279654807Z 2026-05-07 15:34:43.279 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-07T15:34:43.306611341Z 2026-05-07 15:34:43.305 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-07T15:34:43.309637045Z 2026-05-07 15:34:43.309 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-07T15:34:43.311063094Z 2026-05-07 15:34:43.310 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-07T15:34:43.311118824Z 2026-05-07 15:34:43.310 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-07T15:34:43.371527307Z 2026-05-07 15:34:43.370 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-07T15:34:43.373892043Z 2026-05-07 15:34:43.373 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-07T15:34:43.375720541Z 2026-05-07 15:34:43.375 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-07T15:34:43.382502051Z 2026-05-07 15:34:43.382 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-07T15:34:43.394839821Z 2026-05-07 15:34:43.394 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-07T15:34:43.402450776Z 2026-05-07 15:34:43.402 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-07T15:34:43.408271561Z 2026-05-07 15:34:43.408 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-07T15:34:43.408362253Z 2026-05-07 15:34:43.408 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-07T15:34:43.416563131Z 2026-05-07 15:34:43.416 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-07T15:34:43.421146685Z 2026-05-07 15:34:43.420 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm 2026-05-07T15:34:43.422282644Z 2026-05-07 15:34:43.422 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3 2026-05-07T15:34:43.433409592Z 2026-05-07 15:34:43.433 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-07T15:34:43.433422993Z 2026-05-07 15:34:43.433 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-07T15:34:43.433492750Z 2026-05-07 15:34:43.433 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ 2026-05-07T15:34:43.434464623Z 2026-05-07 15:34:43.434 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-07T15:34:43.434899128Z 2026-05-07 15:34:43.434 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas 2026-05-07T15:34:43.435730380Z 2026-05-07 15:34:43.435 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72093707922833408-core_node3-n_0000000000 2026-05-07T15:34:43.442036715Z 2026-05-07 15:34:43.441 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3 2026-05-07T15:34:43.548229083Z 2026-05-07 15:34:43.547 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:43.551238190Z 2026-05-07 15:34:43.550 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-07T15:34:43.557411727Z 2026-05-07 15:34:43.557 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2142 2026-05-07T15:34:43.570814527Z 2026-05-07 15:34:43.570 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2026-05-07T15:34:43.657384698Z 2026-05-07 15:34:43.656 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:43.659825130Z 2026-05-07 15:34:43.659 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=2587 # === Docker inspect for solr-node1 === { "Name": "/solr-node1", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1604526, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-07T15:34:21.220405061Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node2 === 2026-05-07T15:34:29.547988503Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-07T15:34:29.548013020Z It should be set to 65000 to avoid operational disruption. 2026-05-07T15:34:29.548016112Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-07T15:34:29.548320939Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-07T15:34:29.548326632Z 2026-05-07T15:34:29.557323203Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-07T15:34:29.565006122Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-07T15:34:29.630069622Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-07T15:34:29.664194626Z WARNING: A command line option has enabled the Security Manager 2026-05-07T15:34:29.664211234Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-07T15:34:34.694249454Z 2026-05-07 15:34:34.691 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-07T15:34:34.952894795Z 2026-05-07 15:34:34.952 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-07T15:34:34.978320809Z 2026-05-07 15:34:34.977 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-07T15:34:34.985059950Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-07T15:34:34.985152100Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984 2026-05-07T15:34:34.985283011Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-07T15:34:34.985416600Z 2026-05-07 15:34:34.984 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-07T15:34:34.984771319Z 2026-05-07T15:34:34.987918959Z 2026-05-07 15:34:34.987 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-07T15:34:35.009504908Z 2026-05-07 15:34:35.008 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-07T15:34:35.032606663Z 2026-05-07 15:34:35.031 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-07T15:34:35.165927949Z 2026-05-07 15:34:35.165 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-07T15:34:35.181517562Z 2026-05-07 15:34:35.181 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-07T15:34:35.181534294Z 2026-05-07 15:34:35.181 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-07T15:34:35.181619465Z 2026-05-07 15:34:35.181 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-07T15:34:35.290024969Z 2026-05-07 15:34:35.289 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-07T15:34:35.290112847Z 2026-05-07 15:34:35.289 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-07T15:34:35.290230207Z 2026-05-07 15:34:35.289 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-07T15:34:35.323776590Z 2026-05-07 15:34:35.323 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-07T15:34:35.970114196Z 2026-05-07 15:34:35.969 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-07T15:34:36.109429958Z 2026-05-07 15:34:36.108 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-07T15:34:36.111204221Z 2026-05-07 15:34:36.109 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-07T15:34:36.118785183Z 2026-05-07 15:34:36.118 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-07T15:34:36.123328353Z 2026-05-07 15:34:36.122 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-07T15:34:36.126882317Z 2026-05-07 15:34:36.126 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-07T15:34:36.126986192Z 2026-05-07 15:34:36.126 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-07T15:34:36.153995511Z 2026-05-07 15:34:36.153 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-07T15:34:36.161572653Z 2026-05-07 15:34:36.161 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) 2026-05-07T15:34:36.179750583Z 2026-05-07 15:34:36.179 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-07T15:34:36.190586816Z 2026-05-07 15:34:36.190 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN 2026-05-07T15:34:36.200320952Z 2026-05-07 15:34:36.200 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr 2026-05-07T15:34:36.299874944Z 2026-05-07 15:34:36.299 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-07T15:34:36.340887144Z 2026-05-07 15:34:36.340 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-07T15:34:36.619868257Z 2026-05-07 15:34:36.618 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-07T15:34:36.790705921Z 2026-05-07 15:34:36.789 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-07T15:34:37.050641211Z 2026-05-07 15:34:37.050 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-07T15:34:37.177343741Z 2026-05-07 15:34:37.176 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-07T15:34:37.183109916Z 2026-05-07 15:34:37.182 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_07.request.log 2026-05-07T15:34:37.186370044Z 2026-05-07 15:34:37.185 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984} 2026-05-07T15:34:37.187039605Z 2026-05-07 15:34:37.186 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @7623ms 2026-05-07T15:34:37.348510890Z 2026-05-07 15:34:37.347 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=46 2026-05-07T15:34:37.406163297Z 2026-05-07 15:34:37.405 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12 2026-05-07T15:34:41.416447933Z 2026-05-07 15:34:41.415 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin 2026-05-07T15:34:41.473217091Z 2026-05-07 15:34:41.472 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-07T15:34:41.533769025Z 2026-05-07 15:34:41.533 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config 2026-05-07T15:34:41.831365838Z 2026-05-07 15:34:41.830 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-07T15:34:41.831391287Z 2026-05-07 15:34:41.830 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true 2026-05-07T15:34:41.851173829Z 2026-05-07 15:34:41.850 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/] 2026-05-07T15:34:42.314735273Z 2026-05-07 15:34:42.314 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-07T15:34:42.337211868Z 2026-05-07 15:34:42.336 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-07T15:34:42.344920717Z 2026-05-07 15:34:42.344 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-07T15:34:42.346208235Z 2026-05-07 15:34:42.345 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-07T15:34:42.346221350Z 2026-05-07 15:34:42.345 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-07T15:34:42.393008493Z 2026-05-07 15:34:42.392 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-07T15:34:42.395969705Z 2026-05-07 15:34:42.395 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-07T15:34:42.397840006Z 2026-05-07 15:34:42.397 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-07T15:34:42.399456887Z 2026-05-07 15:34:42.399 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-07T15:34:42.414707900Z 2026-05-07 15:34:42.414 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2026-05-07T15:34:42.422262926Z 2026-05-07 15:34:42.421 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-07T15:34:42.427841127Z 2026-05-07 15:34:42.427 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-07T15:34:42.427948587Z 2026-05-07 15:34:42.427 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-07T15:34:42.435544609Z 2026-05-07 15:34:42.435 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-07T15:34:42.443986820Z 2026-05-07 15:34:42.443 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm 2026-05-07T15:34:42.445606038Z 2026-05-07 15:34:42.445 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1 2026-05-07T15:34:42.462153833Z 2026-05-07 15:34:42.461 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-07T15:34:42.462171455Z 2026-05-07 15:34:42.461 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-07T15:34:42.462174395Z 2026-05-07 15:34:42.461 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ 2026-05-07T15:34:42.463441171Z 2026-05-07 15:34:42.463 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-07T15:34:42.465130922Z 2026-05-07 15:34:42.463 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas 2026-05-07T15:34:42.465150820Z 2026-05-07 15:34:42.464 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72093707922833412-core_node6-n_0000000000 2026-05-07T15:34:42.478622293Z 2026-05-07 15:34:42.478 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1 2026-05-07T15:34:42.589422309Z 2026-05-07 15:34:42.588 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:42.593704173Z 2026-05-07 15:34:42.593 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-07T15:34:42.603637594Z 2026-05-07 15:34:42.603 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1189 2026-05-07T15:34:42.770598780Z 2026-05-07 15:34:42.770 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:43.548385139Z 2026-05-07 15:34:43.547 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:43.657730984Z 2026-05-07 15:34:43.657 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node2 === { "Name": "/solr-node2", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1604934, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-07T15:34:29.196509578Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node3 === 2026-05-07T15:34:29.610489764Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-07T15:34:29.610532413Z It should be set to 65000 to avoid operational disruption. 2026-05-07T15:34:29.610536716Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-07T15:34:29.610883268Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-07T15:34:29.610895101Z 2026-05-07T15:34:29.619746480Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-07T15:34:29.627425497Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-07T15:34:29.668373419Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-07T15:34:29.698487806Z WARNING: A command line option has enabled the Security Manager 2026-05-07T15:34:29.698506095Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-07T15:34:30.885204692Z 2026-05-07 15:34:30.882 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-07T15:34:31.157728580Z 2026-05-07 15:34:31.157 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-07T15:34:31.184957477Z 2026-05-07 15:34:31.184 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-07T15:34:31.192040229Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-07T15:34:31.192158366Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985 2026-05-07T15:34:31.192274265Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-07T15:34:31.192457352Z 2026-05-07 15:34:31.191 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-07T15:34:31.191868775Z 2026-05-07T15:34:31.194919642Z 2026-05-07 15:34:31.194 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-07T15:34:31.217067104Z 2026-05-07 15:34:31.216 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-07T15:34:31.245535611Z 2026-05-07 15:34:31.245 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-07T15:34:31.382569662Z 2026-05-07 15:34:31.381 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-07T15:34:31.403012176Z 2026-05-07 15:34:31.402 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-07T15:34:31.403084638Z 2026-05-07 15:34:31.402 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-07T15:34:31.403208257Z 2026-05-07 15:34:31.402 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-07T15:34:31.523370387Z 2026-05-07 15:34:31.522 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-07T15:34:31.523499263Z 2026-05-07 15:34:31.523 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-07T15:34:31.523589646Z 2026-05-07 15:34:31.523 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-07T15:34:31.562453878Z 2026-05-07 15:34:31.561 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-07T15:34:32.312555004Z 2026-05-07 15:34:32.311 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-07T15:34:32.456948546Z 2026-05-07 15:34:32.456 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-07T15:34:32.458146432Z 2026-05-07 15:34:32.456 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-07T15:34:32.465636800Z 2026-05-07 15:34:32.465 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-07T15:34:32.470065252Z 2026-05-07 15:34:32.469 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-07T15:34:32.473400781Z 2026-05-07 15:34:32.473 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-07T15:34:32.473544795Z 2026-05-07 15:34:32.473 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-07T15:34:32.500461243Z 2026-05-07 15:34:32.499 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-07T15:34:32.507621331Z 2026-05-07 15:34:32.507 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-07T15:34:32.526375462Z 2026-05-07 15:34:32.525 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-07T15:34:32.536011184Z 2026-05-07 15:34:32.535 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN 2026-05-07T15:34:32.544458799Z 2026-05-07 15:34:32.544 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr 2026-05-07T15:34:32.554730250Z 2026-05-07 15:34:32.554 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-07T15:34:32.593841437Z 2026-05-07 15:34:32.593 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-07T15:34:32.861458454Z 2026-05-07 15:34:32.860 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-07T15:34:33.034794603Z 2026-05-07 15:34:33.034 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-07T15:34:33.291012430Z 2026-05-07 15:34:33.290 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-07T15:34:33.435019440Z 2026-05-07 15:34:33.434 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-07T15:34:33.440365212Z 2026-05-07 15:34:33.439 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_07.request.log 2026-05-07T15:34:33.443480642Z 2026-05-07 15:34:33.443 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985} 2026-05-07T15:34:33.444253144Z 2026-05-07 15:34:33.444 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3818ms 2026-05-07T15:34:36.208932394Z 2026-05-07 15:34:36.208 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-07T15:34:39.248710859Z 2026-05-07 15:34:39.244 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=77 2026-05-07T15:34:39.317798087Z 2026-05-07 15:34:39.317 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16 2026-05-07T15:34:41.416479195Z 2026-05-07 15:34:41.415 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin 2026-05-07T15:34:41.491868979Z 2026-05-07 15:34:41.491 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-07T15:34:41.557701473Z 2026-05-07 15:34:41.557 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config 2026-05-07T15:34:41.878394054Z 2026-05-07 15:34:41.877 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-07T15:34:41.878417657Z 2026-05-07 15:34:41.878 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true 2026-05-07T15:34:41.896881775Z 2026-05-07 15:34:41.896 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/] 2026-05-07T15:34:42.318457834Z 2026-05-07 15:34:42.317 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-07T15:34:42.340397487Z 2026-05-07 15:34:42.339 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-07T15:34:42.345830046Z 2026-05-07 15:34:42.345 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-07T15:34:42.346854540Z 2026-05-07 15:34:42.346 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-07T15:34:42.346902199Z 2026-05-07 15:34:42.346 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-07T15:34:42.393155336Z 2026-05-07 15:34:42.392 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-07T15:34:42.396135835Z 2026-05-07 15:34:42.395 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-07T15:34:42.399573394Z 2026-05-07 15:34:42.397 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-07T15:34:42.400913245Z 2026-05-07 15:34:42.400 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-07T15:34:42.416228284Z 2026-05-07 15:34:42.415 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-07T15:34:42.424601836Z 2026-05-07 15:34:42.424 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-07T15:34:42.431400137Z 2026-05-07 15:34:42.430 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-07T15:34:42.431423238Z 2026-05-07 15:34:42.430 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-07T15:34:42.437108790Z 2026-05-07 15:34:42.436 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-07T15:34:42.445737216Z 2026-05-07 15:34:42.445 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2026-05-07T15:34:42.447060304Z 2026-05-07 15:34:42.446 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2 2026-05-07T15:34:42.462535580Z 2026-05-07 15:34:42.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-07T15:34:42.462557198Z 2026-05-07 15:34:42.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-07T15:34:42.462607826Z 2026-05-07 15:34:42.462 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ 2026-05-07T15:34:42.464041554Z 2026-05-07 15:34:42.463 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-07T15:34:42.464646535Z 2026-05-07 15:34:42.464 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas 2026-05-07T15:34:42.465820094Z 2026-05-07 15:34:42.465 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72093707922833410-core_node5-n_0000000000 2026-05-07T15:34:42.478926348Z 2026-05-07 15:34:42.478 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2 2026-05-07T15:34:42.589219864Z 2026-05-07 15:34:42.588 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 219] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:42.593086471Z 2026-05-07 15:34:42.592 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-07T15:34:42.601884490Z 2026-05-07 15:34:42.601 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1187 2026-05-07T15:34:42.770631488Z 2026-05-07 15:34:42.770 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 223] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:43.548697968Z 2026-05-07 15:34:43.548 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 238] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-07T15:34:43.657589629Z 2026-05-07 15:34:43.657 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node3 === { "Name": "/solr-node3", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1605064, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-07T15:34:29.406068335Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker not ok 83 Docker SolrCloud rolling upgrade # in 29440 ms # (in test file test/test_rolling_upgrade.bats, line 118) # `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127 # 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203 # Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim # 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # f63eb04151bc: Already exists # a1b8f7509ae1: Already exists # e434fc527435: Already exists # 3acf851d0cdf: Already exists # 142ed9e09943: Already exists # f71bdaed5253: Pulling fs layer # 4b0d3fb3d62d: Pulling fs layer # 651268e22bd5: Pulling fs layer # ed01378be80d: Pulling fs layer # df4326c4e3ea: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # ed01378be80d: Waiting # df4326c4e3ea: Waiting # 4f4fb700ef54: Waiting # 651268e22bd5: Verifying Checksum # 651268e22bd5: Download complete # 4b0d3fb3d62d: Verifying Checksum # 4b0d3fb3d62d: Download complete # ed01378be80d: Download complete # 4f4fb700ef54: Verifying Checksum # 4f4fb700ef54: Download complete # df4326c4e3ea: Verifying Checksum # df4326c4e3ea: Download complete # f71bdaed5253: Verifying Checksum # f71bdaed5253: Download complete # f71bdaed5253: Pull complete # 4b0d3fb3d62d: Pull complete # 651268e22bd5: Pull complete # ed01378be80d: Pull complete # df4326c4e3ea: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:c596878100cb3e6f257b2bf7137884cf5710f73b2d05f24e05be9fa69347feb9 # Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim # b25f78e2ffe088e2d6243e61a0705ea92b2690e6856ecaaef2ef814c2b7af237 # solr-data1 # solr-data2 # solr-data3 # Starting solr-node1 with embedded ZooKeeper # e6a5b5569c2a6e07a0d65af2c20e4ba09b09ca39a32d7a61b6850976395ac542 # Starting solr-node2 connected to first node's ZooKeeper # d1b98ffdc547f4c840f7949f3226dbb05f4f181fdbefd9f63ff802c0d543457c # Started solr-node3 connected to first node's ZooKeeper # 1eca627bc7e335429cf814f04317c41effd6bed666bb3a26b16cc2b639548cf4 # Creating a Collection # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # WARN - 2026-05-07 15:34:40.825; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-07 15:34:40.860; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/]. # Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts' # Checking collection health # /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found # solr-node1 # solr-node2 # solr-node3 # solr-node1 # solr-node2 # solr-node3 # solr-data1 # solr-data2 # solr-data3 # solrcloud-test ok 84 allowPaths - backup # in 14900 ms ok 85 snapshot lifecycle # in 4591 ms ok 86 snapshot list # in 3208 ms ok 87 snapshot describe # in 3222 ms ok 88 start solr with ssl # in 15356 ms ok 89 use different hostname when not checking peer-name # in 23758 ms ok 90 start solr with ssl and auth # in 14505 ms ok 91 start solr with client truststore and security manager # in 15921 ms ok 92 start solr with mTLS needed # in 44841 ms ok 93 start solr with mTLS wanted # in 34315 ms ok 94 test keystore reload # in 56779 ms ok 95 SOLR-11740 check 'solr stop' connection # in 15482 ms ok 96 stop command for single port # in 17848 ms ok 97 check stop command doesn't hang # in 207 ms ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7974 ms ok 99 deprecated system properties converted to modern properties # in 195275 ms ok 100 start with custom jetty options # in 7804 ms ok 101 bootstrapping a configset # in 8862 ms ok 102 status detects locally running solr # in 9151 ms ok 103 status with --solr-url from user # in 7757 ms ok 104 status with --port from user # in 7849 ms ok 105 multiple connection options are prevented # in 819 ms ok 106 status with invalid --solr-url from user # in 1258 ms ok 107 status with --short format # in 196020 ms ok 108 searching solr via locally executed streaming expression # in 1612 ms ok 109 searching solr via remotely executed streaming expression # in 1193 ms ok 110 variable interpolation # in 1584 ms ok 111 searching solr without credentials fails with error # in 1057 ms ok 112 --version returns Solr version # in 683 ms ok 113 version as direct tool call still runs # in 703 ms ok 114 short help # in 742 ms ok 115 short help is inferred # in 722 ms ok 116 long help # in 746 ms ok 117 running subcommands with zk is prevented # in 690 ms ok 118 listing out files # in 2087 ms ok 119 connecting to solr via various solr urls and zk hosts # in 7534 ms ok 120 copying files around # in 10586 ms ok 121 upconfig # in 2469 ms ok 122 SOLR-12429 test upconfig fails with symlink # in 1059 ms ok 123 downconfig # in 1124 ms ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1362 ms ok 125 env var ZK_HOST is honored # in 3075 ms ok 126 Cleanup # in 1804 ms > Task :solr:packaging:integrationTests FAILED FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339 * What went wrong: Execution failed for task ':solr:packaging:integrationTests'. > Process 'command > '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' > finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Get more help at https://help.gradle.org. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 29m 42s 128 actionable tasks: 128 executed Publishing Build Scan to Develocity... https://develocity.apache.org/s/lm5yz75eduq52 Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Archiving artifacts Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
