Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/204/
No tests ran. Build Log: [...truncated 426 lines...] # ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:44059/solr/admin/collections # -- # # Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12 # Last output: # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44059. # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:44059. # WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is # NOT RECOMMENDED for production use. # # To turn it off: # curl http://localhost:44059/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}' # Or: # bin/solr config -c monitors_verbose -s http://localhost:44059 --action set-user-property --property update.autoCreateFields --value false # WARN - 2026-05-09 17:21:22.250; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-09 17:21:22.253; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # # ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:44059/solr/admin/collections ok 82 post solr log into solr via cli # in 8514 ms # Test failed - capturing Docker diagnostics # === docker ps (summary) === NAMES STATUS IMAGE PORTS solr-node3 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node2 Up 17 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node1 Up 28 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp # === Docker logs for solr-node1 === 2026-05-09T17:21:47.244765886Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-09T17:21:47.244802289Z It should be set to 65000 to avoid operational disruption. 2026-05-09T17:21:47.244805346Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-09T17:21:47.245035669Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-09T17:21:47.245047400Z 2026-05-09T17:21:47.258139040Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-09T17:21:47.268229031Z [0.001s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-09T17:21:47.315809982Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-09T17:21:47.359332362Z WARNING: A command line option has enabled the Security Manager 2026-05-09T17:21:47.359355032Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-09T17:21:49.901596423Z 2026-05-09 17:21:49.898 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-09T17:21:50.362391813Z 2026-05-09 17:21:50.361 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-09T17:21:50.470469106Z 2026-05-09 17:21:50.470 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-09T17:21:50.501158118Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-09T17:21:50.501217962Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983 2026-05-09T17:21:50.501331085Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-09T17:21:50.501439724Z 2026-05-09 17:21:50.500 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-09T17:21:50.500878884Z 2026-05-09T17:21:50.504718090Z 2026-05-09 17:21:50.504 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-09T17:21:50.550331830Z 2026-05-09 17:21:50.549 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-09T17:21:50.648063645Z 2026-05-09 17:21:50.647 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-09T17:21:50.648150473Z 2026-05-09 17:21:50.647 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-09T17:21:50.750226046Z 2026-05-09 17:21:50.749 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-09T17:21:52.290944355Z 2026-05-09 17:21:52.289 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-09T17:21:52.778341899Z 2026-05-09 17:21:52.777 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-09T17:21:52.793206897Z 2026-05-09 17:21:52.792 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default 2026-05-09T17:21:52.793235859Z 2026-05-09 17:21:52.792 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg 2026-05-09T17:21:52.798239849Z 2026-05-09 17:21:52.797 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0 2026-05-09T17:21:52.798295826Z 2026-05-09 17:21:52.797 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details. 2026-05-09T17:21:52.876122153Z 2026-05-09 17:21:52.875 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0. 2026-05-09T17:21:53.300511441Z 2026-05-09 17:21:53.299 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983 2026-05-09T17:21:53.374559304Z 2026-05-09 17:21:53.374 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-09T17:21:53.428242749Z 2026-05-09 17:21:53.427 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-09T17:21:53.451520468Z 2026-05-09 17:21:53.451 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-09T17:21:53.451604890Z 2026-05-09 17:21:53.451 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-09T17:21:53.584349939Z 2026-05-09 17:21:53.583 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-09T17:21:53.617629378Z 2026-05-09 17:21:53.617 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-09T17:21:53.627964351Z 2026-05-09 17:21:53.627 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr 2026-05-09T17:21:53.634917471Z 2026-05-09 17:21:53.634 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72105455046819840-solr-node1:8983_solr-n_0000000000) starting 2026-05-09T17:21:53.722813210Z 2026-05-09 17:21:53.722 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr 2026-05-09T17:21:53.727311567Z 2026-05-09 17:21:53.726 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr 2026-05-09T17:21:53.738353427Z 2026-05-09 17:21:53.737 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-09T17:21:53.885898980Z 2026-05-09 17:21:53.885 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-09T17:21:54.508831382Z 2026-05-09 17:21:54.508 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-09T17:21:54.809808876Z 2026-05-09 17:21:54.809 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-09T17:21:55.181052912Z 2026-05-09 17:21:55.180 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-09T17:21:55.657918531Z 2026-05-09 17:21:55.657 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-09T17:21:55.844313993Z 2026-05-09 17:21:55.843 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_09.request.log 2026-05-09T17:21:55.853538527Z 2026-05-09 17:21:55.852 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2026-05-09T17:21:55.856850402Z 2026-05-09 17:21:55.856 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @8589ms 2026-05-09T17:21:57.179331055Z 2026-05-09 17:21:57.178 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=143 2026-05-09T17:21:57.360482862Z 2026-05-09 17:21:57.359 INFO (qtp985324122-97-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=48 2026-05-09T17:22:02.496205229Z 2026-05-09 17:22:02.495 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-09T17:22:02.595294073Z 2026-05-09 17:22:02.594 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-09T17:22:09.214213030Z 2026-05-09 17:22:09.213 INFO (qtp985324122-100-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=18 2026-05-09T17:22:09.275869232Z 2026-05-09 17:22:09.275 INFO (qtp985324122-100-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=15 2026-05-09T17:22:09.303532787Z 2026-05-09 17:22:09.303 INFO (qtp985324122-99-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=1 2026-05-09T17:22:09.700935238Z 2026-05-09 17:22:09.700 INFO (qtp985324122-100-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=22 2026-05-09T17:22:09.755049630Z 2026-05-09 17:22:09.754 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection 2026-05-09T17:22:09.799425504Z 2026-05-09 17:22:09.798 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children 2026-05-09T17:22:09.822441446Z 2026-05-09 17:22:09.822 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:09.924633700Z 2026-05-09 17:22:09.924 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-09T17:22:09.924651689Z "core":"test-collection_shard3_replica_n1", 2026-05-09T17:22:09.924654934Z "node_name":"solr-node1:8983_solr", 2026-05-09T17:22:09.924657628Z "base_url":"http://solr-node1:8983/solr";, 2026-05-09T17:22:09.924660204Z "collection":"test-collection", 2026-05-09T17:22:09.924662817Z "shard":"shard3", 2026-05-09T17:22:09.924665434Z "state":"down", 2026-05-09T17:22:09.924667873Z "type":"NRT", 2026-05-09T17:22:09.924670455Z "operation":"ADDREPLICA", 2026-05-09T17:22:09.924673058Z "waitForFinalState":"false"} 2026-05-09T17:22:09.933014815Z 2026-05-09 17:22:09.932 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-09T17:22:09.933033594Z "core":"test-collection_shard2_replica_n2", 2026-05-09T17:22:09.933037071Z "node_name":"solr-node3:8985_solr", 2026-05-09T17:22:09.933039891Z "base_url":"http://solr-node3:8985/solr";, 2026-05-09T17:22:09.933042480Z "collection":"test-collection", 2026-05-09T17:22:09.933049785Z "shard":"shard2", 2026-05-09T17:22:09.933052343Z "state":"down", 2026-05-09T17:22:09.933055002Z "type":"NRT", 2026-05-09T17:22:09.933057461Z "operation":"ADDREPLICA", 2026-05-09T17:22:09.933060216Z "waitForFinalState":"false"} 2026-05-09T17:22:09.940944635Z 2026-05-09 17:22:09.940 INFO (OverseerStateUpdate-72105455046819840-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-09T17:22:09.940970691Z "core":"test-collection_shard1_replica_n4", 2026-05-09T17:22:09.940973543Z "node_name":"solr-node2:8984_solr", 2026-05-09T17:22:09.940975671Z "base_url":"http://solr-node2:8984/solr";, 2026-05-09T17:22:09.940978035Z "collection":"test-collection", 2026-05-09T17:22:09.940980084Z "shard":"shard1", 2026-05-09T17:22:09.940982011Z "state":"down", 2026-05-09T17:22:09.940984053Z "type":"NRT", 2026-05-09T17:22:09.940985920Z "operation":"ADDREPLICA", 2026-05-09T17:22:09.940988022Z "waitForFinalState":"false"} 2026-05-09T17:22:10.058789426Z 2026-05-09 17:22:10.057 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:10.249220305Z 2026-05-09 17:22:10.248 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin 2026-05-09T17:22:10.374353582Z 2026-05-09 17:22:10.373 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-09T17:22:10.498229447Z 2026-05-09 17:22:10.497 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config 2026-05-09T17:22:11.377438544Z 2026-05-09 17:22:11.376 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-09T17:22:11.377462198Z 2026-05-09 17:22:11.377 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true 2026-05-09T17:22:11.494632323Z 2026-05-09 17:22:11.494 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/] 2026-05-09T17:22:13.671191366Z 2026-05-09 17:22:13.670 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-09T17:22:13.754958721Z 2026-05-09 17:22:13.754 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-09T17:22:13.759261563Z 2026-05-09 17:22:13.758 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-09T17:22:13.760153565Z 2026-05-09 17:22:13.759 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-09T17:22:13.760205775Z 2026-05-09 17:22:13.759 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-09T17:22:13.881926253Z 2026-05-09 17:22:13.881 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-09T17:22:13.885151060Z 2026-05-09 17:22:13.884 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-09T17:22:13.889028810Z 2026-05-09 17:22:13.888 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-09T17:22:13.891950147Z 2026-05-09 17:22:13.891 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-09T17:22:13.919231474Z 2026-05-09 17:22:13.918 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4} 2026-05-09T17:22:13.970197424Z 2026-05-09 17:22:13.969 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-09T17:22:13.983423312Z 2026-05-09 17:22:13.982 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-09T17:22:13.983441374Z 2026-05-09 17:22:13.983 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-09T17:22:14.003281728Z 2026-05-09 17:22:14.002 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-8 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-09T17:22:14.025142229Z 2026-05-09 17:22:14.024 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm 2026-05-09T17:22:14.030222647Z 2026-05-09 17:22:14.029 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3 2026-05-09T17:22:14.044435410Z 2026-05-09 17:22:14.044 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-09T17:22:14.044474457Z 2026-05-09 17:22:14.044 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-09T17:22:14.044538176Z 2026-05-09 17:22:14.044 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ 2026-05-09T17:22:14.045428560Z 2026-05-09 17:22:14.045 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-09T17:22:14.045866878Z 2026-05-09 17:22:14.045 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas 2026-05-09T17:22:14.046970175Z 2026-05-09 17:22:14.046 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72105455046819840-core_node3-n_0000000000 2026-05-09T17:22:14.052627241Z 2026-05-09 17:22:14.052 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3 2026-05-09T17:22:14.159630856Z 2026-05-09 17:22:14.159 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:14.162238559Z 2026-05-09 17:22:14.161 INFO (qtp985324122-29-solr-node1-8) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-09T17:22:14.168256751Z 2026-05-09 17:22:14.168 INFO (qtp985324122-29-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3922 2026-05-09T17:22:14.185166181Z 2026-05-09 17:22:14.184 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2026-05-09T17:22:14.268265079Z 2026-05-09 17:22:14.267 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:14.270219505Z 2026-05-09 17:22:14.269 INFO (qtp985324122-101-solr-node1-7) [c:test-collection s: r: x: t:solr-node1-7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=4550 # === Docker inspect for solr-node1 === { "Name": "/solr-node1", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 2808717, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-09T17:21:46.209323555Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node2 === 2026-05-09T17:21:58.060767412Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-09T17:21:58.060813827Z It should be set to 65000 to avoid operational disruption. 2026-05-09T17:21:58.060817074Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-09T17:21:58.061101957Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-09T17:21:58.061120961Z 2026-05-09T17:21:58.072948163Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-09T17:21:58.082923109Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-09T17:21:58.143784679Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-09T17:21:58.182580349Z WARNING: A command line option has enabled the Security Manager 2026-05-09T17:21:58.182613161Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-09T17:21:59.862456637Z 2026-05-09 17:21:59.858 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-09T17:22:00.358957284Z 2026-05-09 17:22:00.358 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-09T17:22:00.401632248Z 2026-05-09 17:22:00.401 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-09T17:22:00.414823657Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-09T17:22:00.414842397Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984 2026-05-09T17:22:00.414845434Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-09T17:22:00.414848525Z 2026-05-09 17:22:00.414 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-09T17:22:00.414058605Z 2026-05-09T17:22:00.419249204Z 2026-05-09 17:22:00.418 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-09T17:22:00.463447684Z 2026-05-09 17:22:00.462 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-09T17:22:00.518855749Z 2026-05-09 17:22:00.518 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-09T17:22:00.849167268Z 2026-05-09 17:22:00.848 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-09T17:22:00.883506769Z 2026-05-09 17:22:00.882 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-09T17:22:00.883516815Z 2026-05-09 17:22:00.882 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-09T17:22:00.883519418Z 2026-05-09 17:22:00.882 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-09T17:22:01.004061718Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-09T17:22:01.004163033Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-09T17:22:01.004222727Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-09T17:22:01.067746189Z 2026-05-09 17:22:01.067 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-09T17:22:02.347119939Z 2026-05-09 17:22:02.346 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-09T17:22:02.481789040Z 2026-05-09 17:22:02.481 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-09T17:22:02.483147949Z 2026-05-09 17:22:02.481 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-09T17:22:02.491934626Z 2026-05-09 17:22:02.491 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-09T17:22:02.496991865Z 2026-05-09 17:22:02.496 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-09T17:22:02.499917452Z 2026-05-09 17:22:02.499 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-09T17:22:02.499982098Z 2026-05-09 17:22:02.499 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-09T17:22:02.541638890Z 2026-05-09 17:22:02.541 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-09T17:22:02.549508662Z 2026-05-09 17:22:02.548 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) 2026-05-09T17:22:02.569126235Z 2026-05-09 17:22:02.568 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-09T17:22:02.579462985Z 2026-05-09 17:22:02.579 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN 2026-05-09T17:22:02.588240073Z 2026-05-09 17:22:02.587 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr 2026-05-09T17:22:02.596370961Z 2026-05-09 17:22:02.596 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-09T17:22:02.642227955Z 2026-05-09 17:22:02.641 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-09T17:22:02.922779848Z 2026-05-09 17:22:02.922 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-09T17:22:03.171206540Z 2026-05-09 17:22:03.170 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-09T17:22:03.445247590Z 2026-05-09 17:22:03.444 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-09T17:22:03.585783261Z 2026-05-09 17:22:03.585 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-09T17:22:03.593820080Z 2026-05-09 17:22:03.592 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_09.request.log 2026-05-09T17:22:03.598875059Z 2026-05-09 17:22:03.597 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984} 2026-05-09T17:22:03.599040532Z 2026-05-09 17:22:03.598 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5517ms 2026-05-09T17:22:04.529866050Z 2026-05-09 17:22:04.529 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=41 2026-05-09T17:22:04.670682234Z 2026-05-09 17:22:04.669 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=13 2026-05-09T17:22:10.241786615Z 2026-05-09 17:22:10.241 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin 2026-05-09T17:22:10.301501086Z 2026-05-09 17:22:10.301 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-09T17:22:10.385924678Z 2026-05-09 17:22:10.385 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config 2026-05-09T17:22:10.835084962Z 2026-05-09 17:22:10.834 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-09T17:22:10.835133792Z 2026-05-09 17:22:10.834 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true 2026-05-09T17:22:10.880478773Z 2026-05-09 17:22:10.879 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/] 2026-05-09T17:22:11.836884826Z 2026-05-09 17:22:11.836 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-09T17:22:11.860615369Z 2026-05-09 17:22:11.860 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-09T17:22:11.875909087Z 2026-05-09 17:22:11.875 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-09T17:22:11.876911037Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-09T17:22:11.876966403Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-09T17:22:11.933485189Z 2026-05-09 17:22:11.933 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-09T17:22:11.936830690Z 2026-05-09 17:22:11.936 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-09T17:22:11.938775110Z 2026-05-09 17:22:11.938 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-09T17:22:11.940999492Z 2026-05-09 17:22:11.940 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-09T17:22:11.958151033Z 2026-05-09 17:22:11.957 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2026-05-09T17:22:11.965536461Z 2026-05-09 17:22:11.965 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-09T17:22:11.972141520Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-09T17:22:11.972186539Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-09T17:22:11.978693399Z 2026-05-09 17:22:11.978 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-8 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-09T17:22:11.998497949Z 2026-05-09 17:22:11.997 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm 2026-05-09T17:22:11.999746354Z 2026-05-09 17:22:11.999 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1 2026-05-09T17:22:12.042878887Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-09T17:22:12.042939115Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-09T17:22:12.042984822Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ 2026-05-09T17:22:12.044105160Z 2026-05-09 17:22:12.043 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-09T17:22:12.045322041Z 2026-05-09 17:22:12.045 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas 2026-05-09T17:22:12.046368083Z 2026-05-09 17:22:12.046 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72105455046819844-core_node6-n_0000000000 2026-05-09T17:22:12.080886064Z 2026-05-09 17:22:12.080 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1 2026-05-09T17:22:12.188577723Z 2026-05-09 17:22:12.188 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:12.193125083Z 2026-05-09 17:22:12.192 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-09T17:22:12.219068820Z 2026-05-09 17:22:12.218 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1978 2026-05-09T17:22:12.320889773Z 2026-05-09 17:22:12.320 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:14.160112346Z 2026-05-09 17:22:14.159 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:14.268402126Z 2026-05-09 17:22:14.267 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node2 === { "Name": "/solr-node2", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 2809130, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-09T17:21:57.528714979Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node3 === 2026-05-09T17:21:58.062026244Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-09T17:21:58.062050171Z It should be set to 65000 to avoid operational disruption. 2026-05-09T17:21:58.062052873Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-09T17:21:58.062427747Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-09T17:21:58.062444526Z 2026-05-09T17:21:58.072348444Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-09T17:21:58.083107812Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-09T17:21:58.144202222Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-09T17:21:58.182626741Z WARNING: A command line option has enabled the Security Manager 2026-05-09T17:21:58.182655288Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-09T17:21:59.862692123Z 2026-05-09 17:21:59.858 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-09T17:22:00.362091778Z 2026-05-09 17:22:00.361 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-09T17:22:00.401457539Z 2026-05-09 17:22:00.400 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-09T17:22:00.414608273Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-09T17:22:00.414641262Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985 2026-05-09T17:22:00.414831539Z 2026-05-09 17:22:00.413 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-09T17:22:00.414947708Z 2026-05-09 17:22:00.414 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-09T17:22:00.414052416Z 2026-05-09T17:22:00.419220716Z 2026-05-09 17:22:00.418 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-09T17:22:00.463517772Z 2026-05-09 17:22:00.462 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-09T17:22:00.518855837Z 2026-05-09 17:22:00.518 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-09T17:22:00.849224760Z 2026-05-09 17:22:00.848 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-09T17:22:00.883460686Z 2026-05-09 17:22:00.882 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-09T17:22:00.883653909Z 2026-05-09 17:22:00.882 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-09T17:22:00.883807066Z 2026-05-09 17:22:00.883 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-09T17:22:01.004064279Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-09T17:22:01.004207476Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-09T17:22:01.004218491Z 2026-05-09 17:22:01.003 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-09T17:22:01.067600864Z 2026-05-09 17:22:01.067 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-09T17:22:02.143629889Z 2026-05-09 17:22:02.143 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-09T17:22:02.355722186Z 2026-05-09 17:22:02.355 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-09T17:22:02.357996220Z 2026-05-09 17:22:02.355 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-09T17:22:02.370797979Z 2026-05-09 17:22:02.370 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-09T17:22:02.379675267Z 2026-05-09 17:22:02.379 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-09T17:22:02.382428548Z 2026-05-09 17:22:02.382 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-09T17:22:02.382535488Z 2026-05-09 17:22:02.382 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-09T17:22:02.421336307Z 2026-05-09 17:22:02.420 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-09T17:22:02.433914003Z 2026-05-09 17:22:02.433 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-09T17:22:02.461270247Z 2026-05-09 17:22:02.460 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-09T17:22:02.472461808Z 2026-05-09 17:22:02.472 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN 2026-05-09T17:22:02.484536471Z 2026-05-09 17:22:02.484 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr 2026-05-09T17:22:02.497433202Z 2026-05-09 17:22:02.497 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 2026-05-09T17:22:02.558465149Z 2026-05-09 17:22:02.557 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-09T17:22:02.595658418Z 2026-05-09 17:22:02.595 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) 2026-05-09T17:22:02.922738025Z 2026-05-09 17:22:02.922 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-09T17:22:03.170810279Z 2026-05-09 17:22:03.170 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-09T17:22:03.445179630Z 2026-05-09 17:22:03.444 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-09T17:22:03.585814153Z 2026-05-09 17:22:03.585 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-09T17:22:03.593232263Z 2026-05-09 17:22:03.592 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_09.request.log 2026-05-09T17:22:03.596834927Z 2026-05-09 17:22:03.596 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985} 2026-05-09T17:22:03.598606796Z 2026-05-09 17:22:03.598 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5517ms 2026-05-09T17:22:07.280438000Z 2026-05-09 17:22:07.272 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=65 2026-05-09T17:22:07.338440988Z 2026-05-09 17:22:07.338 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=16 2026-05-09T17:22:10.245320031Z 2026-05-09 17:22:10.243 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin 2026-05-09T17:22:10.310449013Z 2026-05-09 17:22:10.310 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-09T17:22:10.428727648Z 2026-05-09 17:22:10.428 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Schema name=default-config 2026-05-09T17:22:10.838315333Z 2026-05-09 17:22:10.837 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-09T17:22:10.838368004Z 2026-05-09 17:22:10.838 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true 2026-05-09T17:22:10.880438713Z 2026-05-09 17:22:10.879 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/] 2026-05-09T17:22:11.836796217Z 2026-05-09 17:22:11.836 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-09T17:22:11.860577028Z 2026-05-09 17:22:11.860 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-09T17:22:11.875874490Z 2026-05-09 17:22:11.875 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-09T17:22:11.876854821Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-09T17:22:11.876928625Z 2026-05-09 17:22:11.876 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-09T17:22:11.932877912Z 2026-05-09 17:22:11.932 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-09T17:22:11.935445774Z 2026-05-09 17:22:11.935 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-09T17:22:11.937485115Z 2026-05-09 17:22:11.937 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-09T17:22:11.939799676Z 2026-05-09 17:22:11.939 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-09T17:22:11.958163044Z 2026-05-09 17:22:11.957 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.s.DirectSolrSpellChecker init: {maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2026-05-09T17:22:11.965266816Z 2026-05-09 17:22:11.964 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-09T17:22:11.971593500Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-09T17:22:11.971605961Z 2026-05-09 17:22:11.971 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-09T17:22:11.978869761Z 2026-05-09 17:22:11.978 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-8 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-09T17:22:11.998346115Z 2026-05-09 17:22:11.997 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2026-05-09T17:22:11.999790603Z 2026-05-09 17:22:11.999 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2 2026-05-09T17:22:12.042478579Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-09T17:22:12.042501228Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-09T17:22:12.042559992Z 2026-05-09 17:22:12.042 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ 2026-05-09T17:22:12.043653267Z 2026-05-09 17:22:12.043 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-09T17:22:12.044139018Z 2026-05-09 17:22:12.043 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas 2026-05-09T17:22:12.045563472Z 2026-05-09 17:22:12.045 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72105455046819843-core_node5-n_0000000000 2026-05-09T17:22:12.070727527Z 2026-05-09 17:22:12.070 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2 2026-05-09T17:22:12.188479453Z 2026-05-09 17:22:12.187 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 218] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:12.193132508Z 2026-05-09 17:22:12.192 INFO (qtp985324122-58-solr-node1-8) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-8] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-09T17:22:12.219125609Z 2026-05-09 17:22:12.218 INFO (qtp985324122-58-solr-node1-8) [c: s: r: x: t:solr-node1-8] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1976 2026-05-09T17:22:12.321329653Z 2026-05-09 17:22:12.320 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 222] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:14.159900809Z 2026-05-09 17:22:14.159 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 237] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-09T17:22:14.268353550Z 2026-05-09 17:22:14.267 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 243] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node3 === { "Name": "/solr-node3", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 2809258, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-09T17:21:57.746182154Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker not ok 83 Docker SolrCloud rolling upgrade # in 35275 ms # (in test file test/test_rolling_upgrade.bats, line 118) # `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127 # 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203 # Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim # 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # f63eb04151bc: Already exists # 87f3bb8db2f4: Already exists # 7c0b62ebdf15: Already exists # 5b95479e5f9b: Already exists # c913227527c9: Already exists # 7405f766627f: Pulling fs layer # 32c4816d11cd: Pulling fs layer # d9e949a1f2d6: Pulling fs layer # 996d9212abb2: Pulling fs layer # 5d0c482f99f0: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # 996d9212abb2: Waiting # 4f4fb700ef54: Waiting # 5d0c482f99f0: Waiting # d9e949a1f2d6: Verifying Checksum # d9e949a1f2d6: Download complete # 32c4816d11cd: Verifying Checksum # 32c4816d11cd: Download complete # 996d9212abb2: Verifying Checksum # 996d9212abb2: Download complete # 4f4fb700ef54: Verifying Checksum # 4f4fb700ef54: Download complete # 5d0c482f99f0: Verifying Checksum # 5d0c482f99f0: Download complete # 7405f766627f: Verifying Checksum # 7405f766627f: Download complete # 7405f766627f: Pull complete # 32c4816d11cd: Pull complete # d9e949a1f2d6: Pull complete # 996d9212abb2: Pull complete # 5d0c482f99f0: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:d6c725a06cee8371811b8e37f896f7d170274f5190cfe455e37f141e463eb072 # Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim # 70788eaa4adf3b34d6858f1cc9dd87c247e3c985d4448e35589c939a0299b364 # solr-data1 # solr-data2 # solr-data3 # Starting solr-node1 with embedded ZooKeeper # 01761ee5c7015a053c8d2654bed84691d439dfd407475ff5e331463c97f35a91 # Starting solr-node2 connected to first node's ZooKeeper # 488049f0e6a4072a1c3e826defdd5b49c4177d6a2a310a618ddc3bf63d26b776 # Started solr-node3 connected to first node's ZooKeeper # 13d9be7eeee6d32274c152caaf150ce17cc5f578226c6ff3367acbfdf0865045 # Creating a Collection # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # WARN - 2026-05-09 17:22:09.365; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-09 17:22:09.438; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node1:8983/solr] to [http://solr-node1:8983/]. # Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts' # Checking collection health # /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found # solr-node1 # solr-node2 # solr-node3 # solr-node1 # solr-node2 # solr-node3 # solr-data1 # solr-data2 # solr-data3 # solrcloud-test ok 84 allowPaths - backup # in 14529 ms ok 85 snapshot lifecycle # in 4582 ms ok 86 snapshot list # in 3121 ms ok 87 snapshot describe # in 3132 ms ok 88 start solr with ssl # in 15540 ms ok 89 use different hostname when not checking peer-name # in 23621 ms ok 90 start solr with ssl and auth # in 13876 ms ok 91 start solr with client truststore and security manager # in 15748 ms ok 92 start solr with mTLS needed # in 44820 ms ok 93 start solr with mTLS wanted # in 34898 ms ok 94 test keystore reload # in 55966 ms ok 95 SOLR-11740 check 'solr stop' connection # in 15459 ms ok 96 stop command for single port # in 17755 ms ok 97 check stop command doesn't hang # in 205 ms ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7766 ms ok 99 deprecated system properties converted to modern properties # in 195244 ms ok 100 start with custom jetty options # in 7982 ms ok 101 bootstrapping a configset # in 8929 ms ok 102 status detects locally running solr # in 9064 ms ok 103 status with --solr-url from user # in 7757 ms ok 104 status with --port from user # in 7776 ms ok 105 multiple connection options are prevented # in 807 ms ok 106 status with invalid --solr-url from user # in 1249 ms ok 107 status with --short format # in 196160 ms ok 108 searching solr via locally executed streaming expression # in 1608 ms ok 109 searching solr via remotely executed streaming expression # in 1202 ms ok 110 variable interpolation # in 1579 ms ok 111 searching solr without credentials fails with error # in 1083 ms ok 112 --version returns Solr version # in 690 ms ok 113 version as direct tool call still runs # in 687 ms ok 114 short help # in 721 ms ok 115 short help is inferred # in 726 ms ok 116 long help # in 731 ms ok 117 running subcommands with zk is prevented # in 679 ms ok 118 listing out files # in 2082 ms ok 119 connecting to solr via various solr urls and zk hosts # in 7427 ms ok 120 copying files around # in 10671 ms ok 121 upconfig # in 2516 ms ok 122 SOLR-12429 test upconfig fails with symlink # in 1043 ms ok 123 downconfig # in 1129 ms ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1362 ms ok 125 env var ZK_HOST is honored # in 3090 ms ok 126 Cleanup # in 1808 ms > Task :solr:packaging:integrationTests FAILED FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339 * What went wrong: Execution failed for task ':solr:packaging:integrationTests'. > Process 'command > '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' > finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Get more help at https://help.gradle.org. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 29m 33s 128 actionable tasks: 128 executed Publishing Build Scan to Develocity... https://develocity.apache.org/s/njgt67jbkhsys Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Archiving artifacts Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
