Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/207/
No tests ran. Build Log: [...truncated 433 lines...] # ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:39281/solr/admin/collections # -- # # Please find the SOLR_HOME snapshot for failed test #11 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/80-11 # Last output: # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281. # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281. # WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is # NOT RECOMMENDED for production use. # # To turn it off: # curl http://localhost:39281/solr/test_stdin/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}' # Or: # bin/solr config -c test_stdin -s http://localhost:39281 --action set-user-property --property update.autoCreateFields --value false # WARN - 2026-05-12 12:35:49.819; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-12 12:35:49.822; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # # ERROR: Failed to create collection 'test_stdin' due to: IOException occurred when talking to server at: http://localhost:39281/solr/admin/collections not ok 81 verbose echo the Solr response # in 1133 ms # (from function `assert_output' in file .gradle/node/node_modules/bats-assert/src/assert.bash, line 247, # in test file test/test_post.bats, line 210) # `assert_output --partial "Created collection 'monitors_verbose'"' failed # # -- output does not contain substring -- # substring (1 lines): # Created collection 'monitors_verbose' # output (3 lines): # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281. # # ERROR: Server refused connection at: http://localhost:39281/solr/admin/info/system # -- # # Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12 # Last output: # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:39281. # # ERROR: Server refused connection at: http://localhost:39281/solr/admin/info/system ok 82 post solr log into solr via cli # in 8546 ms # Test failed - capturing Docker diagnostics # === docker ps (summary) === NAMES STATUS IMAGE PORTS solr-node3 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node2 Up 11 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node1 Up 16 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp # === Docker logs for solr-node1 === 2026-05-12T12:36:18.108407718Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-12T12:36:18.108460959Z It should be set to 65000 to avoid operational disruption. 2026-05-12T12:36:18.108464809Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-12T12:36:18.108785180Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-12T12:36:18.108797263Z 2026-05-12T12:36:18.118890777Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-12T12:36:18.128399232Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-12T12:36:18.177363477Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-12T12:36:18.213141502Z WARNING: A command line option has enabled the Security Manager 2026-05-12T12:36:18.213174625Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-12T12:36:19.453575062Z 2026-05-12 12:36:19.449 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-12T12:36:19.736179385Z 2026-05-12 12:36:19.735 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-12T12:36:19.764142387Z 2026-05-12 12:36:19.763 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-12T12:36:19.771825704Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-12T12:36:19.772025343Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983 2026-05-12T12:36:19.772205538Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-12T12:36:19.772410250Z 2026-05-12 12:36:19.771 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-12T12:36:19.771430816Z 2026-05-12T12:36:19.774908300Z 2026-05-12 12:36:19.774 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-12T12:36:19.797465170Z 2026-05-12 12:36:19.796 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-12T12:36:19.823970621Z 2026-05-12 12:36:19.823 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-12T12:36:19.823999137Z 2026-05-12 12:36:19.823 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-12T12:36:19.863679664Z 2026-05-12 12:36:19.862 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-12T12:36:20.901301838Z 2026-05-12 12:36:20.900 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-12T12:36:21.227682836Z 2026-05-12 12:36:21.226 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-12T12:36:21.234386858Z 2026-05-12 12:36:21.233 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default 2026-05-12T12:36:21.234529192Z 2026-05-12 12:36:21.233 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg 2026-05-12T12:36:21.238628296Z 2026-05-12 12:36:21.238 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0 2026-05-12T12:36:21.238719690Z 2026-05-12 12:36:21.238 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details. 2026-05-12T12:36:21.311739936Z 2026-05-12 12:36:21.311 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0. 2026-05-12T12:36:21.741465614Z 2026-05-12 12:36:21.740 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983 2026-05-12T12:36:21.754015857Z 2026-05-12 12:36:21.753 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-12T12:36:21.839683110Z 2026-05-12 12:36:21.838 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-12T12:36:21.861759991Z 2026-05-12 12:36:21.861 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-12T12:36:21.861895044Z 2026-05-12 12:36:21.861 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-12T12:36:21.987536224Z 2026-05-12 12:36:21.987 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-12T12:36:22.011542156Z 2026-05-12 12:36:22.011 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-12T12:36:22.028150035Z 2026-05-12 12:36:22.027 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr 2026-05-12T12:36:22.038259406Z 2026-05-12 12:36:22.037 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72121319242924032-solr-node1:8983_solr-n_0000000000) starting 2026-05-12T12:36:22.103526695Z 2026-05-12 12:36:22.102 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr 2026-05-12T12:36:22.103618455Z 2026-05-12 12:36:22.103 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr 2026-05-12T12:36:22.112606041Z 2026-05-12 12:36:22.112 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-12T12:36:22.150367884Z 2026-05-12 12:36:22.149 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-12T12:36:22.573225895Z 2026-05-12 12:36:22.571 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-12T12:36:22.734980585Z 2026-05-12 12:36:22.734 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-12T12:36:22.980314954Z 2026-05-12 12:36:22.979 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-12T12:36:23.104035652Z 2026-05-12 12:36:23.103 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-12T12:36:23.109442883Z 2026-05-12 12:36:23.109 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_12.request.log 2026-05-12T12:36:23.145617495Z 2026-05-12 12:36:23.113 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2026-05-12T12:36:23.147416717Z 2026-05-12 12:36:23.147 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5020ms 2026-05-12T12:36:23.340377012Z 2026-05-12 12:36:23.339 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=63 2026-05-12T12:36:23.397355505Z 2026-05-12 12:36:23.396 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=12 2026-05-12T12:36:26.576112169Z 2026-05-12 12:36:26.575 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-12T12:36:26.914300419Z 2026-05-12 12:36:26.913 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-12T12:36:30.914073714Z 2026-05-12 12:36:30.913 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=36 2026-05-12T12:36:30.939800643Z 2026-05-12 12:36:30.939 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=10 2026-05-12T12:36:30.956902933Z 2026-05-12 12:36:30.956 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=2 2026-05-12T12:36:31.332529879Z 2026-05-12 12:36:31.331 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection 2026-05-12T12:36:31.371939271Z 2026-05-12 12:36:31.371 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children 2026-05-12T12:36:31.400301079Z 2026-05-12 12:36:31.399 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 174] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:31.475919003Z 2026-05-12 12:36:31.475 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-12T12:36:31.475941424Z "core":"test-collection_shard1_replica_n1", 2026-05-12T12:36:31.475944632Z "node_name":"solr-node2:8984_solr", 2026-05-12T12:36:31.475947269Z "base_url":"http://solr-node2:8984/solr";, 2026-05-12T12:36:31.475950275Z "collection":"test-collection", 2026-05-12T12:36:31.475952892Z "shard":"shard1", 2026-05-12T12:36:31.475955394Z "state":"down", 2026-05-12T12:36:31.475958016Z "type":"NRT", 2026-05-12T12:36:31.475960439Z "operation":"ADDREPLICA", 2026-05-12T12:36:31.475963016Z "waitForFinalState":"false"} 2026-05-12T12:36:31.488029416Z 2026-05-12 12:36:31.487 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-12T12:36:31.488051374Z "core":"test-collection_shard2_replica_n3", 2026-05-12T12:36:31.488054930Z "node_name":"solr-node3:8985_solr", 2026-05-12T12:36:31.488057989Z "base_url":"http://solr-node3:8985/solr";, 2026-05-12T12:36:31.488060838Z "collection":"test-collection", 2026-05-12T12:36:31.488063313Z "shard":"shard2", 2026-05-12T12:36:31.488065787Z "state":"down", 2026-05-12T12:36:31.488068274Z "type":"NRT", 2026-05-12T12:36:31.488070652Z "operation":"ADDREPLICA", 2026-05-12T12:36:31.488073083Z "waitForFinalState":"false"} 2026-05-12T12:36:31.492181156Z 2026-05-12 12:36:31.491 INFO (OverseerStateUpdate-72121319242924032-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-12T12:36:31.492197999Z "core":"test-collection_shard3_replica_n4", 2026-05-12T12:36:31.492201068Z "node_name":"solr-node1:8983_solr", 2026-05-12T12:36:31.492214122Z "base_url":"http://solr-node1:8983/solr";, 2026-05-12T12:36:31.492216279Z "collection":"test-collection", 2026-05-12T12:36:31.492218222Z "shard":"shard3", 2026-05-12T12:36:31.492220100Z "state":"down", 2026-05-12T12:36:31.492222065Z "type":"NRT", 2026-05-12T12:36:31.492223993Z "operation":"ADDREPLICA", 2026-05-12T12:36:31.492225896Z "waitForFinalState":"false"} 2026-05-12T12:36:31.606620249Z 2026-05-12 12:36:31.606 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 186] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:31.744648544Z 2026-05-12 12:36:31.744 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&shard=shard3&wt=javabin 2026-05-12T12:36:31.864612087Z 2026-05-12 12:36:31.864 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-12T12:36:31.962501729Z 2026-05-12 12:36:31.961 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-12T12:36:32.517215377Z 2026-05-12 12:36:32.516 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-12T12:36:32.517240403Z 2026-05-12 12:36:32.516 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n4' using configuration from configset techproducts, trusted=true 2026-05-12T12:36:32.554565925Z 2026-05-12 12:36:32.554 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n4], dataDir=[/var/solr/data/test-collection_shard3_replica_n4/data/] 2026-05-12T12:36:34.065270171Z 2026-05-12 12:36:34.064 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-12T12:36:34.128986710Z 2026-05-12 12:36:34.128 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-12T12:36:34.140046876Z 2026-05-12 12:36:34.139 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-12T12:36:34.141041822Z 2026-05-12 12:36:34.140 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-12T12:36:34.141120674Z 2026-05-12 12:36:34.140 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-12T12:36:34.254091526Z 2026-05-12 12:36:34.253 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-12T12:36:34.258448961Z 2026-05-12 12:36:34.258 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-12T12:36:34.261489058Z 2026-05-12 12:36:34.261 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-12T12:36:34.263254309Z 2026-05-12 12:36:34.263 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-12T12:36:34.284533250Z 2026-05-12 12:36:34.284 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-12T12:36:34.292608168Z 2026-05-12 12:36:34.292 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-12T12:36:34.296896735Z 2026-05-12 12:36:34.296 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-12T12:36:34.296998001Z 2026-05-12 12:36:34.296 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-12T12:36:34.308953882Z 2026-05-12 12:36:34.308 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n4 solr-node1-6 core_node6 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-12T12:36:34.315752978Z 2026-05-12 12:36:34.315 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node6=0}, version=0} for registerTerm 2026-05-12T12:36:34.316996841Z 2026-05-12 12:36:34.316 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3 2026-05-12T12:36:34.335370097Z 2026-05-12 12:36:34.335 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-12T12:36:34.335389875Z 2026-05-12 12:36:34.335 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-12T12:36:34.335462531Z 2026-05-12 12:36:34.335 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ 2026-05-12T12:36:34.336373071Z 2026-05-12 12:36:34.336 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-12T12:36:34.336796168Z 2026-05-12 12:36:34.336 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ has no replicas 2026-05-12T12:36:34.337706242Z 2026-05-12 12:36:34.337 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72121319242924032-core_node6-n_0000000000 2026-05-12T12:36:34.342478826Z 2026-05-12 12:36:34.342 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n4/ shard3 2026-05-12T12:36:34.447654165Z 2026-05-12 12:36:34.447 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:34.465192025Z 2026-05-12 12:36:34.464 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node6 x:test-collection_shard3_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-12T12:36:34.471878160Z 2026-05-12 12:36:34.471 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2730 2026-05-12T12:36:34.571583497Z 2026-05-12 12:36:34.571 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node1 === { "Name": "/solr-node1", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1699922, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-12T12:36:17.876237366Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node2 === 2026-05-12T12:36:23.670757896Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-12T12:36:23.670785354Z It should be set to 65000 to avoid operational disruption. 2026-05-12T12:36:23.670788579Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-12T12:36:23.671106032Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-12T12:36:23.671121907Z 2026-05-12T12:36:23.679783264Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-12T12:36:23.687178999Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-12T12:36:23.730390851Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-12T12:36:23.763789395Z WARNING: A command line option has enabled the Security Manager 2026-05-12T12:36:23.763810673Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-12T12:36:24.810398178Z 2026-05-12 12:36:24.807 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-12T12:36:25.073214439Z 2026-05-12 12:36:25.072 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-12T12:36:25.097751282Z 2026-05-12 12:36:25.097 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-12T12:36:25.104878875Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-12T12:36:25.104900681Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984 2026-05-12T12:36:25.105072145Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-12T12:36:25.105221114Z 2026-05-12 12:36:25.104 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-12T12:36:25.104494308Z 2026-05-12T12:36:25.107663101Z 2026-05-12 12:36:25.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-12T12:36:25.129828719Z 2026-05-12 12:36:25.129 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-12T12:36:25.154787827Z 2026-05-12 12:36:25.154 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-12T12:36:25.289638762Z 2026-05-12 12:36:25.289 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-12T12:36:25.306143344Z 2026-05-12 12:36:25.305 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-12T12:36:25.306203858Z 2026-05-12 12:36:25.305 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-12T12:36:25.306328436Z 2026-05-12 12:36:25.305 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-12T12:36:25.416130470Z 2026-05-12 12:36:25.415 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-12T12:36:25.416148135Z 2026-05-12 12:36:25.415 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-12T12:36:25.416211060Z 2026-05-12 12:36:25.415 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-12T12:36:25.448794830Z 2026-05-12 12:36:25.448 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-12T12:36:26.261644186Z 2026-05-12 12:36:26.260 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-12T12:36:26.414033022Z 2026-05-12 12:36:26.411 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-12T12:36:26.414061064Z 2026-05-12 12:36:26.412 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-12T12:36:26.423710030Z 2026-05-12 12:36:26.423 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-12T12:36:26.428886466Z 2026-05-12 12:36:26.428 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-12T12:36:26.434270270Z 2026-05-12 12:36:26.433 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-12T12:36:26.434288310Z 2026-05-12 12:36:26.433 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-12T12:36:26.481539403Z 2026-05-12 12:36:26.480 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-12T12:36:26.490999626Z 2026-05-12 12:36:26.490 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-12T12:36:26.514551793Z 2026-05-12 12:36:26.513 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-12T12:36:26.549680420Z 2026-05-12 12:36:26.548 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN 2026-05-12T12:36:26.562182362Z 2026-05-12 12:36:26.561 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr 2026-05-12T12:36:26.577302606Z 2026-05-12 12:36:26.576 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-12T12:36:26.627496562Z 2026-05-12 12:36:26.626 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-12T12:36:26.915124349Z 2026-05-12 12:36:26.913 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-12T12:36:26.957674800Z 2026-05-12 12:36:26.957 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-12T12:36:27.156005864Z 2026-05-12 12:36:27.155 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-12T12:36:27.495654640Z 2026-05-12 12:36:27.494 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-12T12:36:27.657563934Z 2026-05-12 12:36:27.656 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-12T12:36:27.664674752Z 2026-05-12 12:36:27.664 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_12.request.log 2026-05-12T12:36:27.670156840Z 2026-05-12 12:36:27.668 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984} 2026-05-12T12:36:27.671910269Z 2026-05-12 12:36:27.671 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @3985ms 2026-05-12T12:36:27.878717051Z 2026-05-12 12:36:27.878 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=67 2026-05-12T12:36:27.957115637Z 2026-05-12 12:36:27.956 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=18 2026-05-12T12:36:31.742701263Z 2026-05-12 12:36:31.742 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node2&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&shard=shard1&wt=javabin 2026-05-12T12:36:31.790912156Z 2026-05-12 12:36:31.790 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-12T12:36:31.870106004Z 2026-05-12 12:36:31.869 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-12T12:36:32.177629854Z 2026-05-12 12:36:32.177 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-12T12:36:32.177657374Z 2026-05-12 12:36:32.177 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n1' using configuration from configset techproducts, trusted=true 2026-05-12T12:36:32.195749540Z 2026-05-12 12:36:32.195 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n1], dataDir=[/var/solr/data/test-collection_shard1_replica_n1/data/] 2026-05-12T12:36:32.615671583Z 2026-05-12 12:36:32.615 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-12T12:36:32.633365566Z 2026-05-12 12:36:32.632 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-12T12:36:32.636618509Z 2026-05-12 12:36:32.636 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-12T12:36:32.637717232Z 2026-05-12 12:36:32.637 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-12T12:36:32.637731005Z 2026-05-12 12:36:32.637 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-12T12:36:32.674722513Z 2026-05-12 12:36:32.674 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-12T12:36:32.677271755Z 2026-05-12 12:36:32.676 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-12T12:36:32.678897080Z 2026-05-12 12:36:32.678 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-12T12:36:32.680452063Z 2026-05-12 12:36:32.680 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-12T12:36:32.691239539Z 2026-05-12 12:36:32.690 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-12T12:36:32.698908919Z 2026-05-12 12:36:32.698 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-12T12:36:32.703785268Z 2026-05-12 12:36:32.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node2 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-12T12:36:32.703799978Z 2026-05-12 12:36:32.703 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node2 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-12T12:36:32.710720821Z 2026-05-12 12:36:32.710 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n1 solr-node1-6 core_node2 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-12T12:36:32.723147803Z 2026-05-12 12:36:32.722 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node2=0}, version=0} for registerTerm 2026-05-12T12:36:32.724551622Z 2026-05-12 12:36:32.724 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1 2026-05-12T12:36:32.739181333Z 2026-05-12 12:36:32.738 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-12T12:36:32.739200291Z 2026-05-12 12:36:32.738 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-12T12:36:32.739258401Z 2026-05-12 12:36:32.739 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ 2026-05-12T12:36:32.740498426Z 2026-05-12 12:36:32.740 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-12T12:36:32.740962232Z 2026-05-12 12:36:32.740 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ has no replicas 2026-05-12T12:36:32.742016463Z 2026-05-12 12:36:32.741 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72121319242924035-core_node2-n_0000000000 2026-05-12T12:36:32.751431065Z 2026-05-12 12:36:32.751 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n1/ shard1 2026-05-12T12:36:32.858185570Z 2026-05-12 12:36:32.857 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 206] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:32.862313141Z 2026-05-12 12:36:32.861 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard1 r:core_node2 x:test-collection_shard1_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-12T12:36:32.871494080Z 2026-05-12 12:36:32.871 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1130 2026-05-12T12:36:32.974735723Z 2026-05-12 12:36:32.974 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 209] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:34.448090962Z 2026-05-12 12:36:34.447 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:34.571769447Z 2026-05-12 12:36:34.571 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node2 === { "Name": "/solr-node2", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1700321, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-12T12:36:23.479358828Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node3 === 2026-05-12T12:36:23.871027852Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-12T12:36:23.871063615Z It should be set to 65000 to avoid operational disruption. 2026-05-12T12:36:23.871066681Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-12T12:36:23.871389768Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-12T12:36:23.871410750Z 2026-05-12T12:36:23.880304696Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-12T12:36:23.887440312Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-12T12:36:23.935593917Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-12T12:36:23.964801246Z WARNING: A command line option has enabled the Security Manager 2026-05-12T12:36:23.964824845Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-12T12:36:25.154319612Z 2026-05-12 12:36:25.151 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-12T12:36:25.416072940Z 2026-05-12 12:36:25.415 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-12T12:36:25.440127540Z 2026-05-12 12:36:25.439 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-12T12:36:25.446740551Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-12T12:36:25.446968677Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985 2026-05-12T12:36:25.446978516Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-12T12:36:25.447122290Z 2026-05-12 12:36:25.446 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-12T12:36:25.446375032Z 2026-05-12T12:36:25.449344238Z 2026-05-12 12:36:25.448 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-12T12:36:25.470385368Z 2026-05-12 12:36:25.469 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-12T12:36:25.493425874Z 2026-05-12 12:36:25.492 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-12T12:36:25.626933533Z 2026-05-12 12:36:25.626 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-12T12:36:25.643330286Z 2026-05-12 12:36:25.642 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-12T12:36:25.643430229Z 2026-05-12 12:36:25.643 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-12T12:36:25.643577573Z 2026-05-12 12:36:25.643 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-12T12:36:25.753075747Z 2026-05-12 12:36:25.752 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-12T12:36:25.753111945Z 2026-05-12 12:36:25.752 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-12T12:36:25.753245898Z 2026-05-12 12:36:25.752 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-12T12:36:25.792519167Z 2026-05-12 12:36:25.791 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-12T12:36:26.612309676Z 2026-05-12 12:36:26.611 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-12T12:36:26.764358825Z 2026-05-12 12:36:26.763 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-12T12:36:26.766238694Z 2026-05-12 12:36:26.763 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-12T12:36:26.776183731Z 2026-05-12 12:36:26.775 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-12T12:36:26.783087023Z 2026-05-12 12:36:26.781 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-12T12:36:26.788487649Z 2026-05-12 12:36:26.786 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-12T12:36:26.788541329Z 2026-05-12 12:36:26.787 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-12T12:36:26.837665460Z 2026-05-12 12:36:26.836 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-12T12:36:26.848177990Z 2026-05-12 12:36:26.847 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) 2026-05-12T12:36:26.874459526Z 2026-05-12 12:36:26.873 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-12T12:36:26.888605846Z 2026-05-12 12:36:26.887 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN 2026-05-12T12:36:26.901444929Z 2026-05-12 12:36:26.900 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr 2026-05-12T12:36:26.915165390Z 2026-05-12 12:36:26.914 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-12T12:36:26.962292269Z 2026-05-12 12:36:26.961 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-12T12:36:27.274543425Z 2026-05-12 12:36:27.273 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-12T12:36:27.468202655Z 2026-05-12 12:36:27.467 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-12T12:36:27.762629796Z 2026-05-12 12:36:27.761 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-12T12:36:27.892985180Z 2026-05-12 12:36:27.892 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-12T12:36:27.899003210Z 2026-05-12 12:36:27.898 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_12.request.log 2026-05-12T12:36:27.902748828Z 2026-05-12 12:36:27.902 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985} 2026-05-12T12:36:27.903636090Z 2026-05-12 12:36:27.903 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @4017ms 2026-05-12T12:36:29.762508230Z 2026-05-12 12:36:29.718 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=187 2026-05-12T12:36:29.841461507Z 2026-05-12 12:36:29.840 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=34 2026-05-12T12:36:31.263595054Z 2026-05-12 12:36:31.263 INFO (qtp985324122-59-solr-node3-3) [c: s: r: x: t:solr-node3-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=61 2026-05-12T12:36:31.743519840Z 2026-05-12 12:36:31.741 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n3&action=CREATE&numShards=3&shard=shard2&wt=javabin 2026-05-12T12:36:32.017814167Z 2026-05-12 12:36:32.017 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-12T12:36:32.149275961Z 2026-05-12 12:36:32.148 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-12T12:36:32.821134919Z 2026-05-12 12:36:32.820 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-12T12:36:32.821269006Z 2026-05-12 12:36:32.820 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n3' using configuration from configset techproducts, trusted=true 2026-05-12T12:36:32.926194140Z 2026-05-12 12:36:32.925 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n3], dataDir=[/var/solr/data/test-collection_shard2_replica_n3/data/] 2026-05-12T12:36:34.092381438Z 2026-05-12 12:36:34.091 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-12T12:36:34.115750010Z 2026-05-12 12:36:34.115 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-12T12:36:34.118867320Z 2026-05-12 12:36:34.118 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-12T12:36:34.119714417Z 2026-05-12 12:36:34.119 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-12T12:36:34.119770608Z 2026-05-12 12:36:34.119 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-12T12:36:34.174689866Z 2026-05-12 12:36:34.174 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-12T12:36:34.181543300Z 2026-05-12 12:36:34.181 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-12T12:36:34.184241179Z 2026-05-12 12:36:34.183 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-12T12:36:34.187351061Z 2026-05-12 12:36:34.187 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-12T12:36:34.199324745Z 2026-05-12 12:36:34.199 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-12T12:36:34.209013628Z 2026-05-12 12:36:34.208 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-12T12:36:34.216748223Z 2026-05-12 12:36:34.216 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n3 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-12T12:36:34.216820135Z 2026-05-12 12:36:34.216 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n3 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-12T12:36:34.225496538Z 2026-05-12 12:36:34.225 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n3 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-12T12:36:34.258848084Z 2026-05-12 12:36:34.258 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2026-05-12T12:36:34.260173373Z 2026-05-12 12:36:34.259 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2 2026-05-12T12:36:34.272071614Z 2026-05-12 12:36:34.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-12T12:36:34.272089358Z 2026-05-12 12:36:34.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-12T12:36:34.272109385Z 2026-05-12 12:36:34.271 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n3/ 2026-05-12T12:36:34.273636893Z 2026-05-12 12:36:34.273 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-12T12:36:34.273802717Z 2026-05-12 12:36:34.273 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n3/ has no replicas 2026-05-12T12:36:34.274826116Z 2026-05-12 12:36:34.274 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72121319242924036-core_node5-n_0000000000 2026-05-12T12:36:34.287304398Z 2026-05-12 12:36:34.287 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n3/ shard2 2026-05-12T12:36:34.448906373Z 2026-05-12 12:36:34.447 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:34.451630642Z 2026-05-12 12:36:34.451 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n3 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-12T12:36:34.459123030Z 2026-05-12 12:36:34.458 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n3&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2718 2026-05-12T12:36:34.479539442Z 2026-05-12 12:36:34.479 INFO (qtp985324122-58-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2026-05-12T12:36:34.571581427Z 2026-05-12 12:36:34.571 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 244] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-12T12:36:34.574114091Z 2026-05-12 12:36:34.573 INFO (qtp985324122-58-solr-node3-4) [c:test-collection s: r: x: t:solr-node3-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3298 # === Docker inspect for solr-node3 === { "Name": "/solr-node3", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1700506, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-12T12:36:23.66430058Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker not ok 83 Docker SolrCloud rolling upgrade # in 26751 ms # (in test file test/test_rolling_upgrade.bats, line 118) # `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127 # 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # af6eca94c810: Pulling fs layer # cb0efb96dabd: Pulling fs layer # 3e9d91201f40: Pulling fs layer # 66b76b382631: Pulling fs layer # 601f2c23751f: Pulling fs layer # 0ed719aef393: Pulling fs layer # 69a09c96e8bf: Pulling fs layer # c937aee9a870: Pulling fs layer # 9cdfe5320de8: Pulling fs layer # 3aa9120e46f5: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # 66b76b382631: Waiting # 601f2c23751f: Waiting # 0ed719aef393: Waiting # 69a09c96e8bf: Waiting # c937aee9a870: Waiting # 3aa9120e46f5: Waiting # 9cdfe5320de8: Waiting # 4f4fb700ef54: Waiting # af6eca94c810: Verifying Checksum # af6eca94c810: Download complete # cb0efb96dabd: Verifying Checksum # cb0efb96dabd: Download complete # 66b76b382631: Verifying Checksum # 66b76b382631: Download complete # 601f2c23751f: Verifying Checksum # 601f2c23751f: Download complete # 3e9d91201f40: Verifying Checksum # 3e9d91201f40: Download complete # 69a09c96e8bf: Download complete # c937aee9a870: Verifying Checksum # c937aee9a870: Download complete # 9cdfe5320de8: Verifying Checksum # 9cdfe5320de8: Download complete # 4f4fb700ef54: Verifying Checksum # 4f4fb700ef54: Download complete # 3aa9120e46f5: Download complete # 0ed719aef393: Verifying Checksum # 0ed719aef393: Download complete # af6eca94c810: Pull complete # cb0efb96dabd: Pull complete # 3e9d91201f40: Pull complete # 66b76b382631: Pull complete # 601f2c23751f: Pull complete # 0ed719aef393: Pull complete # 69a09c96e8bf: Pull complete # c937aee9a870: Pull complete # 9cdfe5320de8: Pull complete # 3aa9120e46f5: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203 # Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim # 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # f63eb04151bc: Already exists # 87f3bb8db2f4: Already exists # 7c0b62ebdf15: Already exists # 5b95479e5f9b: Already exists # c913227527c9: Already exists # 44d992948799: Pulling fs layer # 716ed2882a63: Pulling fs layer # 8372919fc868: Pulling fs layer # 2198107b532d: Pulling fs layer # 49f4cafecda6: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # 49f4cafecda6: Waiting # 2198107b532d: Waiting # 4f4fb700ef54: Waiting # 8372919fc868: Verifying Checksum # 8372919fc868: Download complete # 716ed2882a63: Verifying Checksum # 716ed2882a63: Download complete # 49f4cafecda6: Verifying Checksum # 49f4cafecda6: Download complete # 4f4fb700ef54: Verifying Checksum # 4f4fb700ef54: Download complete # 2198107b532d: Download complete # 44d992948799: Verifying Checksum # 44d992948799: Download complete # 44d992948799: Pull complete # 716ed2882a63: Pull complete # 8372919fc868: Pull complete # 2198107b532d: Pull complete # 49f4cafecda6: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:f0544343d632eb05fdba9371a0f48b98c03354d7394cfdb0e648434d09fc5498 # Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim # 7e574ffef0df92be69c0f77afb64fa39b94b1e1e31cfa4f2c22bdca8e55c2763 # solr-data1 # solr-data2 # solr-data3 # Starting solr-node1 with embedded ZooKeeper # 06ba64ef120cc3b6885db0ea762c77450652d827542183dfd09476117e972361 # Starting solr-node2 connected to first node's ZooKeeper # c9abb6e08d4e65815e07cca5260256e687071ab7350e96fbf8847b1f541787e3 # Started solr-node3 connected to first node's ZooKeeper # 27e6607e5186255f4113777682fda5d54ad824629f3ff655105c10ff930b9037 # Creating a Collection # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # WARN - 2026-05-12 12:36:31.008; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-12 12:36:31.043; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node3:8985/solr] to [http://solr-node3:8985/]. # Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts' # Checking collection health # /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found # solr-node1 # solr-node2 # solr-node3 # solr-node1 # solr-node2 # solr-node3 # solr-data1 # solr-data2 # solr-data3 # solrcloud-test ok 84 allowPaths - backup # in 14469 ms ok 85 snapshot lifecycle # in 4455 ms ok 86 snapshot list # in 3075 ms ok 87 snapshot describe # in 3097 ms ok 88 start solr with ssl # in 15173 ms ok 89 use different hostname when not checking peer-name # in 23178 ms ok 90 start solr with ssl and auth # in 13714 ms ok 91 start solr with client truststore and security manager # in 15319 ms ok 92 start solr with mTLS needed # in 44460 ms ok 93 start solr with mTLS wanted # in 34139 ms ok 94 test keystore reload # in 56067 ms ok 95 SOLR-11740 check 'solr stop' connection # in 15445 ms ok 96 stop command for single port # in 17631 ms ok 97 check stop command doesn't hang # in 204 ms ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7742 ms ok 99 deprecated system properties converted to modern properties # in 195312 ms ok 100 start with custom jetty options # in 7715 ms ok 101 bootstrapping a configset # in 8781 ms ok 102 status detects locally running solr # in 9090 ms ok 103 status with --solr-url from user # in 7824 ms ok 104 status with --port from user # in 7696 ms ok 105 multiple connection options are prevented # in 799 ms ok 106 status with invalid --solr-url from user # in 1203 ms ok 107 status with --short format # in 196011 ms ok 108 searching solr via locally executed streaming expression # in 1643 ms ok 109 searching solr via remotely executed streaming expression # in 1203 ms ok 110 variable interpolation # in 1606 ms ok 111 searching solr without credentials fails with error # in 1092 ms ok 112 --version returns Solr version # in 688 ms ok 113 version as direct tool call still runs # in 711 ms ok 114 short help # in 730 ms ok 115 short help is inferred # in 725 ms ok 116 long help # in 734 ms ok 117 running subcommands with zk is prevented # in 675 ms ok 118 listing out files # in 2080 ms ok 119 connecting to solr via various solr urls and zk hosts # in 7465 ms ok 120 copying files around # in 10492 ms ok 121 upconfig # in 2447 ms ok 122 SOLR-12429 test upconfig fails with symlink # in 1053 ms ok 123 downconfig # in 1100 ms ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1369 ms ok 125 env var ZK_HOST is honored # in 3091 ms ok 126 Cleanup # in 1823 ms > Task :solr:packaging:integrationTests FAILED FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339 * What went wrong: Execution failed for task ':solr:packaging:integrationTests'. > Process 'command > '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' > finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Get more help at https://help.gradle.org. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 31m 128 actionable tasks: 128 executed Publishing Build Scan to Develocity... https://develocity.apache.org/s/tg6vr5opsgi4y Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Archiving artifacts Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
