Build: https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/200/
No tests ran. Build Log: [...truncated 426 lines...] # ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:41435/solr/admin/collections # -- # # Please find the SOLR_HOME snapshot for failed test #12 at: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/failure-snapshots/81-12 # Last output: # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:41435. # Neither --zk-host or --solr-url parameters, nor ZK_HOST env var provided, so assuming solr url is http://localhost:41435. # WARNING: Using _default configset. Data driven schema functionality is enabled by default, which is # NOT RECOMMENDED for production use. # # To turn it off: # curl http://localhost:41435/solr/monitors_verbose/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}' # Or: # bin/solr config -c monitors_verbose -s http://localhost:41435 --action set-user-property --property update.autoCreateFields --value false # WARN - 2026-05-05 12:33:02.584; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-05 12:33:02.588; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # # ERROR: Failed to create collection 'monitors_verbose' due to: IOException occurred when talking to server at: http://localhost:41435/solr/admin/collections ok 82 post solr log into solr via cli # in 9659 ms # Test failed - capturing Docker diagnostics # === docker ps (summary) === NAMES STATUS IMAGE PORTS solr-node3 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node2 Up 14 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp solr-node1 Up 27 seconds apache/solr-nightly:9.10.0-SNAPSHOT-slim 8983/tcp # === Docker logs for solr-node1 === 2026-05-05T12:33:31.584009819Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-05T12:33:31.584051678Z It should be set to 65000 to avoid operational disruption. 2026-05-05T12:33:31.584055345Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-05T12:33:31.584288437Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-05T12:33:31.584301693Z 2026-05-05T12:33:31.596881588Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-05T12:33:31.607232669Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-05T12:33:31.659564167Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-05T12:33:31.837140008Z WARNING: A command line option has enabled the Security Manager 2026-05-05T12:33:31.837165413Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-05T12:33:34.284986308Z 2026-05-05 12:33:34.281 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-05T12:33:34.820110624Z 2026-05-05 12:33:34.819 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-05T12:33:35.048203344Z 2026-05-05 12:33:35.047 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-05T12:33:35.108143329Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-05T12:33:35.108230315Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8983 2026-05-05T12:33:35.108364184Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-05T12:33:35.108499321Z 2026-05-05 12:33:35.107 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-05T12:33:35.107843696Z 2026-05-05T12:33:35.113076744Z 2026-05-05 12:33:35.112 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-05T12:33:35.198852266Z 2026-05-05 12:33:35.198 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-05T12:33:35.561594691Z 2026-05-05 12:33:35.560 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-05T12:33:35.561929919Z 2026-05-05 12:33:35.561 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-05T12:33:35.763093977Z 2026-05-05 12:33:35.762 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-05T12:33:37.485265547Z 2026-05-05 12:33:37.484 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-05T12:33:38.014655472Z 2026-05-05 12:33:38.013 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-05T12:33:38.026803667Z 2026-05-05 12:33:38.026 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Zookeeper configuration not found in /var/solr/data, using built-in default 2026-05-05T12:33:38.026918473Z 2026-05-05 12:33:38.026 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServerProps Reading configuration from: /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/zoo.cfg 2026-05-05T12:33:38.035185110Z 2026-05-05 12:33:38.034 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER at port 9983, listening on host 0.0.0.0 2026-05-05T12:33:38.035200400Z 2026-05-05 12:33:38.034 WARN (main) [c: s: r: x: t:] o.a.s.c.SolrZkServer Embedded Zookeeper is not recommended in production environments. See Reference Guide for details. 2026-05-05T12:33:38.138093548Z 2026-05-05 12:33:38.137 WARN (embeddedZkServer) [c: s: r: x: t:] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0. 2026-05-05T12:33:38.537712151Z 2026-05-05 12:33:38.536 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=172.18.0.2:9983 2026-05-05T12:33:38.604193287Z 2026-05-05 12:33:38.603 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-05T12:33:38.657979787Z 2026-05-05 12:33:38.657 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-05T12:33:38.686606510Z 2026-05-05 12:33:38.686 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-05T12:33:38.686710978Z 2026-05-05 12:33:38.686 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-05T12:33:38.808375606Z 2026-05-05 12:33:38.807 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-05T12:33:38.921267884Z 2026-05-05 12:33:38.920 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-05T12:33:38.933814622Z 2026-05-05 12:33:38.933 INFO (main) [c: s: r: x: t:] o.a.s.c.OverseerElectionContext I am going to be the leader solr-node1:8983_solr 2026-05-05T12:33:38.940183893Z 2026-05-05 12:33:38.939 INFO (main) [c: s: r: x: t:] o.a.s.c.Overseer Overseer (id=72081758103601152-solr-node1:8983_solr-n_0000000000) starting 2026-05-05T12:33:39.031452641Z 2026-05-05 12:33:39.030 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node1:8983_solr 2026-05-05T12:33:39.034514967Z 2026-05-05 12:33:39.034 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.Overseer Starting to work on the main queue : solr-node1:8983_solr 2026-05-05T12:33:39.046680457Z 2026-05-05 12:33:39.046 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-05T12:33:39.149493064Z 2026-05-05 12:33:39.148 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-05T12:33:39.719179922Z 2026-05-05 12:33:39.718 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-05T12:33:39.974797535Z 2026-05-05 12:33:39.974 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-05T12:33:40.339821131Z 2026-05-05 12:33:40.339 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-05T12:33:40.854766401Z 2026-05-05 12:33:40.853 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-05T12:33:40.913402195Z 2026-05-05 12:33:40.912 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_05.request.log 2026-05-05T12:33:40.921489803Z 2026-05-05 12:33:40.921 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8983} 2026-05-05T12:33:40.923694266Z 2026-05-05 12:33:40.923 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @9317ms 2026-05-05T12:33:41.252879376Z 2026-05-05 12:33:41.251 INFO (qtp985324122-27-solr-node1-1) [c: s: r: x: t:solr-node1-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=106 2026-05-05T12:33:41.440707261Z 2026-05-05 12:33:41.440 INFO (qtp985324122-96-solr-node1-2) [c: s: r: x: t:solr-node1-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=17 2026-05-05T12:33:46.117922103Z 2026-05-05 12:33:46.117 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3) 2026-05-05T12:33:51.772947255Z 2026-05-05 12:33:51.771 INFO (qtp985324122-99-solr-node1-3) [c: s: r: x: t:solr-node1-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=10 2026-05-05T12:33:51.805334290Z 2026-05-05 12:33:51.804 INFO (qtp985324122-99-solr-node1-4) [c: s: r: x: t:solr-node1-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=11 2026-05-05T12:33:51.849074753Z 2026-05-05 12:33:51.848 INFO (qtp985324122-27-solr-node1-5) [c: s: r: x: t:solr-node1-5] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=6 2026-05-05T12:33:52.174003526Z 2026-05-05 12:33:52.173 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection test-collection 2026-05-05T12:33:52.208303391Z 2026-05-05 12:33:52.205 INFO (OverseerThreadFactory-17-thread-1) [c:test-collection s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/test-collection/state.json is deleted, stop watching children 2026-05-05T12:33:52.234602153Z 2026-05-05 12:33:52.234 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/test-collection/state.json zxid: 173] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:52.307293213Z 2026-05-05 12:33:52.305 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-05T12:33:52.307311854Z "core":"test-collection_shard3_replica_n1", 2026-05-05T12:33:52.307314683Z "node_name":"solr-node1:8983_solr", 2026-05-05T12:33:52.307316969Z "base_url":"http://solr-node1:8983/solr";, 2026-05-05T12:33:52.307319095Z "collection":"test-collection", 2026-05-05T12:33:52.307321506Z "shard":"shard3", 2026-05-05T12:33:52.307323573Z "state":"down", 2026-05-05T12:33:52.307325567Z "type":"NRT", 2026-05-05T12:33:52.307327598Z "operation":"ADDREPLICA", 2026-05-05T12:33:52.307329632Z "waitForFinalState":"false"} 2026-05-05T12:33:52.316177217Z 2026-05-05 12:33:52.315 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-05T12:33:52.316193449Z "core":"test-collection_shard2_replica_n2", 2026-05-05T12:33:52.316196115Z "node_name":"solr-node3:8985_solr", 2026-05-05T12:33:52.316198633Z "base_url":"http://solr-node3:8985/solr";, 2026-05-05T12:33:52.316201089Z "collection":"test-collection", 2026-05-05T12:33:52.316203686Z "shard":"shard2", 2026-05-05T12:33:52.316205698Z "state":"down", 2026-05-05T12:33:52.316207649Z "type":"NRT", 2026-05-05T12:33:52.316209505Z "operation":"ADDREPLICA", 2026-05-05T12:33:52.316211460Z "waitForFinalState":"false"} 2026-05-05T12:33:52.324382016Z 2026-05-05 12:33:52.323 INFO (OverseerStateUpdate-72081758103601152-solr-node1:8983_solr-n_0000000000) [c: s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { 2026-05-05T12:33:52.324400737Z "core":"test-collection_shard1_replica_n4", 2026-05-05T12:33:52.324403851Z "node_name":"solr-node2:8984_solr", 2026-05-05T12:33:52.324406120Z "base_url":"http://solr-node2:8984/solr";, 2026-05-05T12:33:52.324408265Z "collection":"test-collection", 2026-05-05T12:33:52.324420114Z "shard":"shard1", 2026-05-05T12:33:52.324422104Z "state":"down", 2026-05-05T12:33:52.324424090Z "type":"NRT", 2026-05-05T12:33:52.324426108Z "operation":"ADDREPLICA", 2026-05-05T12:33:52.324428059Z "waitForFinalState":"false"} 2026-05-05T12:33:52.436013239Z 2026-05-05 12:33:52.435 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 185] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:52.564764138Z 2026-05-05 12:33:52.564 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node3&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&shard=shard3&wt=javabin 2026-05-05T12:33:52.655290669Z 2026-05-05 12:33:52.654 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-05T12:33:52.793151910Z 2026-05-05 12:33:52.792 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-05T12:33:53.537925071Z 2026-05-05 12:33:53.536 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-05T12:33:53.537954742Z 2026-05-05 12:33:53.536 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard3_replica_n1' using configuration from configset techproducts, trusted=true 2026-05-05T12:33:53.598729449Z 2026-05-05 12:33:53.598 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard3_replica_n1], dataDir=[/var/solr/data/test-collection_shard3_replica_n1/data/] 2026-05-05T12:33:55.187998235Z 2026-05-05 12:33:55.186 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-05T12:33:55.266272537Z 2026-05-05 12:33:55.265 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-05T12:33:55.273034463Z 2026-05-05 12:33:55.272 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-05T12:33:55.274378800Z 2026-05-05 12:33:55.273 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-05T12:33:55.274404730Z 2026-05-05 12:33:55.273 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-05T12:33:55.399590219Z 2026-05-05 12:33:55.399 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-05T12:33:55.402620353Z 2026-05-05 12:33:55.402 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-05T12:33:55.404343586Z 2026-05-05 12:33:55.404 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-05T12:33:55.406135688Z 2026-05-05 12:33:55.405 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-05T12:33:55.426147214Z 2026-05-05 12:33:55.425 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01} 2026-05-05T12:33:55.434407127Z 2026-05-05 12:33:55.434 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-05T12:33:55.443324287Z 2026-05-05 12:33:55.442 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-05T12:33:55.443344798Z 2026-05-05 12:33:55.442 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-05T12:33:55.494324615Z 2026-05-05 12:33:55.493 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard3_replica_n1 solr-node1-6 core_node3 solr-node1:8983_solr test-collection shard3) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-05T12:33:55.528246605Z 2026-05-05 12:33:55.527 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard3 to Terms{values={core_node3=0}, version=0} for registerTerm 2026-05-05T12:33:55.530536143Z 2026-05-05 12:33:55.530 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard3 2026-05-05T12:33:55.545696728Z 2026-05-05 12:33:55.545 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-05T12:33:55.545929918Z 2026-05-05 12:33:55.545 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-05T12:33:55.545938906Z 2026-05-05 12:33:55.545 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ 2026-05-05T12:33:55.547579392Z 2026-05-05 12:33:55.547 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-05T12:33:55.548108132Z 2026-05-05 12:33:55.547 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ has no replicas 2026-05-05T12:33:55.549355596Z 2026-05-05 12:33:55.549 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard3/leader after winning as /collections/test-collection/leader_elect/shard3/election/72081758103601152-core_node3-n_0000000000 2026-05-05T12:33:55.571274648Z 2026-05-05 12:33:55.570 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node1:8983/solr/test-collection_shard3_replica_n1/ shard3 2026-05-05T12:33:55.676205900Z 2026-05-05 12:33:55.675 INFO (zkCallback-11-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.679559102Z 2026-05-05 12:33:55.679 INFO (qtp985324122-99-solr-node1-6) [c:test-collection s:shard3 r:core_node3 x:test-collection_shard3_replica_n1 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-05T12:33:55.689795030Z 2026-05-05 12:33:55.688 INFO (qtp985324122-99-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=techproducts&newCollection=true&name=test-collection_shard3_replica_n1&action=CREATE&numShards=3&collection=test-collection&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3126 2026-05-05T12:33:55.787826190Z 2026-05-05 12:33:55.787 INFO (zkCallback-11-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node1 === { "Name": "/solr-node1", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1774551, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-05T12:33:28.430541021Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node2 === 2026-05-05T12:33:42.050242126Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-05T12:33:42.050269073Z It should be set to 65000 to avoid operational disruption. 2026-05-05T12:33:42.050272404Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-05T12:33:42.050561345Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-05T12:33:42.050568024Z 2026-05-05T12:33:42.059568683Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-05T12:33:42.068446824Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-05T12:33:42.123848174Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-05T12:33:42.166272549Z WARNING: A command line option has enabled the Security Manager 2026-05-05T12:33:42.166296744Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-05T12:33:43.667658403Z 2026-05-05 12:33:43.663 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-05T12:33:44.114279930Z 2026-05-05 12:33:44.113 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-05T12:33:44.159707748Z 2026-05-05 12:33:44.159 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-05T12:33:44.169476025Z 2026-05-05 12:33:44.168 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-05T12:33:44.169642551Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8984 2026-05-05T12:33:44.169647367Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-05T12:33:44.169751825Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-05T12:33:44.169261315Z 2026-05-05T12:33:44.173641100Z 2026-05-05 12:33:44.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-05T12:33:44.207057916Z 2026-05-05 12:33:44.206 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-05T12:33:44.262042883Z 2026-05-05 12:33:44.261 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-05T12:33:44.511928875Z 2026-05-05 12:33:44.511 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-05T12:33:44.533367973Z 2026-05-05 12:33:44.532 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-05T12:33:44.533495490Z 2026-05-05 12:33:44.533 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-05T12:33:44.533606250Z 2026-05-05 12:33:44.533 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-05T12:33:44.659168940Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-05T12:33:44.659431281Z 2026-05-05 12:33:44.659 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-05T12:33:44.659657691Z 2026-05-05 12:33:44.659 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-05T12:33:44.716405761Z 2026-05-05 12:33:44.715 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-05T12:33:45.698245189Z 2026-05-05 12:33:45.696 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-05T12:33:45.988241865Z 2026-05-05 12:33:45.987 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-05T12:33:45.990121656Z 2026-05-05 12:33:45.987 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-05T12:33:46.005123064Z 2026-05-05 12:33:46.004 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-05T12:33:46.011261908Z 2026-05-05 12:33:46.010 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-05T12:33:46.014708837Z 2026-05-05 12:33:46.014 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-05T12:33:46.014797870Z 2026-05-05 12:33:46.014 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-05T12:33:46.049860664Z 2026-05-05 12:33:46.049 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-05T12:33:46.058765823Z 2026-05-05 12:33:46.058 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-05T12:33:46.082590393Z 2026-05-05 12:33:46.082 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-05T12:33:46.096758528Z 2026-05-05 12:33:46.096 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node2:8984_solr as DOWN 2026-05-05T12:33:46.107902383Z 2026-05-05 12:33:46.107 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node2:8984_solr 2026-05-05T12:33:46.119121981Z 2026-05-05 12:33:46.118 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3) 2026-05-05T12:33:46.184165428Z 2026-05-05 12:33:46.183 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-05T12:33:46.518706581Z 2026-05-05 12:33:46.518 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-05T12:33:46.765864920Z 2026-05-05 12:33:46.765 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-05T12:33:47.052887320Z 2026-05-05 12:33:47.052 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-05T12:33:47.208481243Z 2026-05-05 12:33:47.207 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-05T12:33:47.215611530Z 2026-05-05 12:33:47.215 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_05.request.log 2026-05-05T12:33:47.220410906Z 2026-05-05 12:33:47.219 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8984} 2026-05-05T12:33:47.222091548Z 2026-05-05 12:33:47.221 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5154ms 2026-05-05T12:33:47.445218144Z 2026-05-05 12:33:47.444 INFO (qtp985324122-27-solr-node2-1) [c: s: r: x: t:solr-node2-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=52 2026-05-05T12:33:47.519414694Z 2026-05-05 12:33:47.518 INFO (qtp985324122-55-solr-node2-2) [c: s: r: x: t:solr-node2-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19 2026-05-05T12:33:52.134005651Z 2026-05-05 12:33:52.133 INFO (qtp985324122-58-solr-node2-3) [c: s: r: x: t:solr-node2-3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=7 2026-05-05T12:33:52.559835004Z 2026-05-05 12:33:52.559 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node6&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&shard=shard1&wt=javabin 2026-05-05T12:33:52.638101261Z 2026-05-05 12:33:52.637 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-05T12:33:52.790386958Z 2026-05-05 12:33:52.787 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-05T12:33:53.291138006Z 2026-05-05 12:33:53.290 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-05T12:33:53.291240239Z 2026-05-05 12:33:53.290 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard1_replica_n4' using configuration from configset techproducts, trusted=true 2026-05-05T12:33:53.322661246Z 2026-05-05 12:33:53.322 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard1_replica_n4], dataDir=[/var/solr/data/test-collection_shard1_replica_n4/data/] 2026-05-05T12:33:54.534884249Z 2026-05-05 12:33:54.534 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-05T12:33:54.556816460Z 2026-05-05 12:33:54.556 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-05T12:33:54.582171311Z 2026-05-05 12:33:54.581 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-05T12:33:54.583463926Z 2026-05-05 12:33:54.582 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-05T12:33:54.583476696Z 2026-05-05 12:33:54.583 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-05T12:33:54.662219240Z 2026-05-05 12:33:54.661 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-05T12:33:54.665143701Z 2026-05-05 12:33:54.664 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-05T12:33:54.667641873Z 2026-05-05 12:33:54.667 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-05T12:33:54.669590871Z 2026-05-05 12:33:54.669 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-05T12:33:54.689571198Z 2026-05-05 12:33:54.689 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2026-05-05T12:33:54.697359941Z 2026-05-05 12:33:54.696 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-05T12:33:54.742416168Z 2026-05-05 12:33:54.741 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-05T12:33:54.742438811Z 2026-05-05 12:33:54.742 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-05T12:33:54.751871798Z 2026-05-05 12:33:54.751 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard1_replica_n4 solr-node1-6 core_node6 solr-node2:8984_solr test-collection shard1) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-05T12:33:54.770093445Z 2026-05-05 12:33:54.769 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard1 to Terms{values={core_node6=0}, version=0} for registerTerm 2026-05-05T12:33:54.771495727Z 2026-05-05 12:33:54.771 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard1 2026-05-05T12:33:54.902634049Z 2026-05-05 12:33:54.902 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-05T12:33:54.902649720Z 2026-05-05 12:33:54.902 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-05T12:33:54.902653630Z 2026-05-05 12:33:54.902 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ 2026-05-05T12:33:54.903874832Z 2026-05-05 12:33:54.903 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-05T12:33:54.904379426Z 2026-05-05 12:33:54.904 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ has no replicas 2026-05-05T12:33:54.905487751Z 2026-05-05 12:33:54.905 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard1/leader after winning as /collections/test-collection/leader_elect/shard1/election/72081758103601156-core_node6-n_0000000000 2026-05-05T12:33:54.921554244Z 2026-05-05 12:33:54.921 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node2:8984/solr/test-collection_shard1_replica_n4/ shard1 2026-05-05T12:33:54.945126821Z 2026-05-05 12:33:54.944 INFO (qtp985324122-60-solr-node1-6) [c:test-collection s:shard1 r:core_node6 x:test-collection_shard1_replica_n4 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-05T12:33:54.954917005Z 2026-05-05 12:33:54.954 INFO (qtp985324122-60-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=techproducts&newCollection=true&name=test-collection_shard1_replica_n4&action=CREATE&numShards=3&collection=test-collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2396 2026-05-05T12:33:55.056460824Z 2026-05-05 12:33:55.055 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.171882391Z 2026-05-05 12:33:55.171 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.676285910Z 2026-05-05 12:33:55.675 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.703247528Z 2026-05-05 12:33:55.702 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 2026-05-05T12:33:55.787954961Z 2026-05-05 12:33:55.787 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.790582113Z 2026-05-05 12:33:55.790 INFO (qtp985324122-59-solr-node2-4) [c:test-collection s: r: x: t:solr-node2-4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=techproducts&name=test-collection&nrtReplicas=1&action=CREATE&numShards=3&wt=json&version=2.2} status=0 QTime=3638 # === Docker inspect for solr-node2 === { "Name": "/solr-node2", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1774970, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-05T12:33:41.692958252Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # === Docker logs for solr-node3 === 2026-05-05T12:33:42.112913653Z *** [WARN] *** Your open file limit is currently 1024. 2026-05-05T12:33:42.112950164Z It should be set to 65000 to avoid operational disruption. 2026-05-05T12:33:42.112953653Z If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh 2026-05-05T12:33:42.112956195Z Solr will start in SolrCloud mode by default in version 10, and you will no longer need to pass in -c or --cloud flag. 2026-05-05T12:33:42.112958333Z 2026-05-05T12:33:42.121349576Z Java 17 detected. Enabled workaround for SOLR-16463 2026-05-05T12:33:42.129334172Z [0.002s][warning][pagesize] UseLargePages disabled, no large pages configured and available on the system. 2026-05-05T12:33:42.178136899Z CompileCommand: exclude com/github/benmanes/caffeine/cache/BoundedLocalCache.put bool exclude = true 2026-05-05T12:33:42.212407487Z WARNING: A command line option has enabled the Security Manager 2026-05-05T12:33:42.212452904Z WARNING: The Security Manager is deprecated and will be removed in a future release 2026-05-05T12:33:43.687805801Z 2026-05-05 12:33:43.684 INFO (main) [c: s: r: x: t:] o.e.j.s.Server jetty-10.0.26; built: 2025-08-14T15:56:56.454Z; git: 65a55c55a68c56d5f2e8232164439bd6e76e53d1; jvm 17.0.16+8 2026-05-05T12:33:44.114235071Z 2026-05-05 12:33:44.113 WARN (main) [c: s: r: x: t:] o.e.j.u.DeprecationWarning Using @Deprecated Class org.eclipse.jetty.servlet.listener.ELContextCleaner 2026-05-05T12:33:44.159605082Z 2026-05-05 12:33:44.159 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 2026-05-05T12:33:44.169512666Z 2026-05-05 12:33:44.168 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.10.0-SNAPSHOT 1edeb3ac6a97186b4ae7de511196a1769de7960e [snapshot build, details omitted] 2026-05-05T12:33:44.169575922Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port 8985 2026-05-05T12:33:44.169689265Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /opt/solr-9.10.0-SNAPSHOT-slim 2026-05-05T12:33:44.169817107Z 2026-05-05 12:33:44.169 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2026-05-05T12:33:44.169248944Z 2026-05-05T12:33:44.173474193Z 2026-05-05 12:33:44.173 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr started with "-XX:+CrashOnOutOfMemoryError" that will crash on any OutOfMemoryError exception. The cause of the OOME will be logged in the crash file at the following path: /var/solr/logs/jvm_crash_1.log 2026-05-05T12:33:44.207011432Z 2026-05-05 12:33:44.206 INFO (main) [c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /var/solr/data (source: system property: solr.solr.home) 2026-05-05T12:33:44.262053630Z 2026-05-05 12:33:44.261 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-05T12:33:44.511614367Z 2026-05-05 12:33:44.510 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 2026-05-05T12:33:44.533052384Z 2026-05-05 12:33:44.532 INFO (zkCallback-1-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-05T12:33:44.533548974Z 2026-05-05 12:33:44.532 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-05T12:33:44.533556272Z 2026-05-05 12:33:44.532 WARN (main) [c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 2026-05-05T12:33:44.659090295Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper) 2026-05-05T12:33:44.659457860Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig solr.xml not found in SOLR_HOME, using built-in default 2026-05-05T12:33:44.659676594Z 2026-05-05 12:33:44.658 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrXmlConfig Loading solr.xml from /opt/solr-9.10.0-SNAPSHOT-slim/server/solr/solr.xml 2026-05-05T12:33:44.716283785Z 2026-05-05 12:33:44.715 INFO (main) [c: s: r: x: t:] o.a.s.c.SolrResourceLoader Added 1 libs to classloader, from paths: [/opt/solr-9.10.0-SNAPSHOT-slim/lib] 2026-05-05T12:33:45.697339877Z 2026-05-05 12:33:45.696 INFO (main) [c: s: r: x: t:] o.a.s.u.t.SimplePropagator Always-on trace id generation enabled. 2026-05-05T12:33:45.988016327Z 2026-05-05 12:33:45.987 WARN (main) [c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Jetty request logging enabled. Will retain logs for last 3 days. See chapter "Configuring Logging" in reference guide for how to configure. 2026-05-05T12:33:45.990184358Z 2026-05-05 12:33:45.987 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=solr-node1:9983 2026-05-05T12:33:46.003711004Z 2026-05-05 12:33:46.003 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-05T12:33:46.011235436Z 2026-05-05 12:33:46.010 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 2026-05-05T12:33:46.014165625Z 2026-05-05 12:33:46.013 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 2026-05-05T12:33:46.014232035Z 2026-05-05 12:33:46.013 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2026-05-05T12:33:46.049897924Z 2026-05-05 12:33:46.049 WARN (main) [c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 2026-05-05T12:33:46.058610595Z 2026-05-05 12:33:46.058 INFO (main) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2026-05-05T12:33:46.082590395Z 2026-05-05 12:33:46.082 INFO (main) [c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates. 2026-05-05T12:33:46.097708882Z 2026-05-05 12:33:46.097 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Publish node=solr-node3:8985_solr as DOWN 2026-05-05T12:33:46.108722831Z 2026-05-05 12:33:46.108 INFO (main) [c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper: /live_nodes/solr-node3:8985_solr 2026-05-05T12:33:46.119119057Z 2026-05-05 12:33:46.118 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3) 2026-05-05T12:33:46.184229536Z 2026-05-05 12:33:46.183 WARN (main) [c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 2026-05-05T12:33:46.518656135Z 2026-05-05 12:33:46.518 INFO (main) [c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /var/solr/data 2026-05-05T12:33:46.766608686Z 2026-05-05 12:33:46.766 WARN (main) [c: s: r: x: t:] o.g.j.m.i.MessagingBinders A class jakarta.activation.DataSource for a default provider MessageBodyWriter<jakarta.activation.DataSource> was not found. The provider is not available. 2026-05-05T12:33:47.052892106Z 2026-05-05 12:33:47.052 INFO (main) [c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-05T12:33:47.208522297Z 2026-05-05 12:33:47.207 INFO (main) [c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@71f67a79{solr-jetty-context.xml,/solr,file:///opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp/,AVAILABLE}{/opt/solr-9.10.0-SNAPSHOT-slim/server/solr-webapp/webapp} 2026-05-05T12:33:47.215613078Z 2026-05-05 12:33:47.215 INFO (main) [c: s: r: x: t:] o.e.j.s.RequestLogWriter Opened /var/solr/logs/2026_05_05.request.log 2026-05-05T12:33:47.220409921Z 2026-05-05 12:33:47.219 INFO (main) [c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@61884cb1{HTTP/1.1, (http/1.1, h2c)}{0.0.0.0:8985} 2026-05-05T12:33:47.221678756Z 2026-05-05 12:33:47.221 INFO (main) [c: s: r: x: t:] o.e.j.s.Server Started Server@350ec41e{STARTING}[10.0.26,sto=0] @5093ms 2026-05-05T12:33:49.876359989Z 2026-05-05 12:33:49.867 INFO (qtp985324122-27-solr-node3-1) [c: s: r: x: t:solr-node3-1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=javabin&version=2} status=0 QTime=112 2026-05-05T12:33:49.974867288Z 2026-05-05 12:33:49.974 INFO (qtp985324122-56-solr-node3-2) [c: s: r: x: t:solr-node3-2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=javabin&version=2} status=0 QTime=19 2026-05-05T12:33:52.561046321Z 2026-05-05 12:33:52.560 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=techproducts&newCollection=true&collection=test-collection&version=2&replicaType=NRT&coreNodeName=core_node5&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&shard=shard2&wt=javabin 2026-05-05T12:33:52.647671509Z 2026-05-05 12:33:52.647 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.12.0 2026-05-05T12:33:52.790974166Z 2026-05-05 12:33:52.790 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Schema name=default-config 2026-05-05T12:33:53.293331040Z 2026-05-05 12:33:53.292 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.IndexSchema Loaded schema default-config/1.7 with uniqueid field id 2026-05-05T12:33:53.293389425Z 2026-05-05 12:33:53.293 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.CoreContainer Creating SolrCore 'test-collection_shard2_replica_n2' using configuration from configset techproducts, trusted=true 2026-05-05T12:33:53.322700609Z 2026-05-05 12:33:53.322 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Opening new SolrCore at [/var/solr/data/test-collection_shard2_replica_n2], dataDir=[/var/solr/data/test-collection_shard2_replica_n2/data/] 2026-05-05T12:33:54.535078961Z 2026-05-05 12:33:54.534 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 2026-05-05T12:33:54.557772146Z 2026-05-05 12:33:54.557 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 2026-05-05T12:33:54.582252463Z 2026-05-05 12:33:54.581 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 2026-05-05T12:33:54.583529351Z 2026-05-05 12:33:54.582 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 2026-05-05T12:33:54.583633944Z 2026-05-05 12:33:54.583 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.u.CommitTracker Soft AutoCommit: if uncommitted for 3000ms; 2026-05-05T12:33:54.662546932Z 2026-05-05 12:33:54.662 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/techproducts 2026-05-05T12:33:54.665884116Z 2026-05-05 12:33:54.665 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/techproducts 2026-05-05T12:33:54.667901185Z 2026-05-05 12:33:54.667 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/techproducts/managed-schema.xml 2026-05-05T12:33:54.669615933Z 2026-05-05 12:33:54.669 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest 2026-05-05T12:33:54.689633448Z 2026-05-05 12:33:54.689 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.s.DirectSolrSpellChecker init: {accuracy=0.5, maxQueryFrequency=0.01, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, name=default, field=_text_, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal} 2026-05-05T12:33:54.697667836Z 2026-05-05 12:33:54.697 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 2026-05-05T12:33:54.742952247Z 2026-05-05 12:33:54.742 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.QuerySenderListener QuerySenderListener done. 2026-05-05T12:33:54.743024874Z 2026-05-05 12:33:54.742 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default 2026-05-05T12:33:54.753287812Z 2026-05-05 12:33:54.752 INFO (searcherExecutor-22-thread-1-processing-test-collection_shard2_replica_n2 solr-node1-6 core_node5 solr-node3:8985_solr test-collection shard2) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 2026-05-05T12:33:54.877866818Z 2026-05-05 12:33:54.877 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test-collection/terms/shard2 to Terms{values={core_node5=0}, version=0} for registerTerm 2026-05-05T12:33:54.879383191Z 2026-05-05 12:33:54.879 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test-collection/leaders/shard2 2026-05-05T12:33:54.900867084Z 2026-05-05 12:33:54.900 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 2026-05-05T12:33:54.900888973Z 2026-05-05 12:33:54.900 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 2026-05-05T12:33:54.900988089Z 2026-05-05 12:33:54.900 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync replicas to http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ 2026-05-05T12:33:54.902220486Z 2026-05-05 12:33:54.901 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 2026-05-05T12:33:54.902803347Z 2026-05-05 12:33:54.902 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.SyncStrategy http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ has no replicas 2026-05-05T12:33:54.903994841Z 2026-05-05 12:33:54.903 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test-collection/leaders/shard2/leader after winning as /collections/test-collection/leader_elect/shard2/election/72081758103601155-core_node5-n_0000000000 2026-05-05T12:33:54.922396562Z 2026-05-05 12:33:54.921 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://solr-node3:8985/solr/test-collection_shard2_replica_n2/ shard2 2026-05-05T12:33:55.056433902Z 2026-05-05 12:33:55.055 INFO (zkCallback-13-thread-1) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 221] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.062912516Z 2026-05-05 12:33:55.062 INFO (qtp985324122-58-solr-node1-6) [c:test-collection s:shard2 r:core_node5 x:test-collection_shard2_replica_n2 t:solr-node1-6] o.a.s.c.ZkController I am the leader, no recovery necessary 2026-05-05T12:33:55.071403935Z 2026-05-05 12:33:55.070 INFO (qtp985324122-58-solr-node1-6) [c: s: r: x: t:solr-node1-6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=techproducts&newCollection=true&name=test-collection_shard2_replica_n2&action=CREATE&numShards=3&collection=test-collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2512 2026-05-05T12:33:55.172324084Z 2026-05-05 12:33:55.171 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 224] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.676366589Z 2026-05-05 12:33:55.675 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 239] for collection [test-collection] has occurred - updating... (live nodes size: [3]) 2026-05-05T12:33:55.787975505Z 2026-05-05 12:33:55.787 INFO (zkCallback-13-thread-2) [c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test-collection/state.json zxid: 245] for collection [test-collection] has occurred - updating... (live nodes size: [3]) # === Docker inspect for solr-node3 === { "Name": "/solr-node3", "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1775100, "ExitCode": 0, "Error": "", "StartedAt": "2026-05-05T12:33:41.916923496Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Ports": { "8983/tcp": null } } # Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker not ok 83 Docker SolrCloud rolling upgrade # in 34097 ms # (in test file test/test_rolling_upgrade.bats, line 118) # `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127 # 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203 # Status: Image is up to date for apache/solr-nightly:9.10.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:9.10.0-SNAPSHOT-slim # 10.0.0-SNAPSHOT-slim: Pulling from apache/solr-nightly # f63eb04151bc: Already exists # a1b8f7509ae1: Already exists # e434fc527435: Already exists # 3acf851d0cdf: Already exists # 142ed9e09943: Already exists # 1cdb71aca53d: Pulling fs layer # 0d8eec1b6c24: Pulling fs layer # 7e69c308213e: Pulling fs layer # c8f61c25585a: Pulling fs layer # d959d69295ca: Pulling fs layer # 4f4fb700ef54: Pulling fs layer # c8f61c25585a: Waiting # d959d69295ca: Waiting # 4f4fb700ef54: Waiting # 0d8eec1b6c24: Download complete # 7e69c308213e: Download complete # d959d69295ca: Verifying Checksum # d959d69295ca: Download complete # c8f61c25585a: Verifying Checksum # c8f61c25585a: Download complete # 4f4fb700ef54: Verifying Checksum # 4f4fb700ef54: Download complete # 1cdb71aca53d: Verifying Checksum # 1cdb71aca53d: Download complete # 1cdb71aca53d: Pull complete # 0d8eec1b6c24: Pull complete # 7e69c308213e: Pull complete # c8f61c25585a: Pull complete # d959d69295ca: Pull complete # 4f4fb700ef54: Pull complete # Digest: sha256:dd9c7a5a0d1a56548540e76fb46ce879d58198257ba5caf6de6f48bb614155a7 # Status: Downloaded newer image for apache/solr-nightly:10.0.0-SNAPSHOT-slim # docker.io/apache/solr-nightly:10.0.0-SNAPSHOT-slim # a9bd7bdedbc5b8c1102990d9bf526d83db6e5fa98bc415b5d72dc5b454ed444c # solr-data1 # solr-data2 # solr-data3 # Starting solr-node1 with embedded ZooKeeper # 71bd2354e93a912a99d6f4ae12368e6ecd861f38776f1bb8511737bdd1126e9d # Starting solr-node2 connected to first node's ZooKeeper # a39e7337abc206741b456c80f667aa2ec0c1e5014063d5538cd922713424a112 # Started solr-node3 connected to first node's ZooKeeper # e7efbdb0ecaa3c6c41c1aec46d30b38782608fb23e362966d848ba53fd0c36c1 # Creating a Collection # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # Neither --zk-host or --solr-url parameters provided so assuming solr url is http://localhost:8983. # WARN - 2026-05-05 12:33:51.901; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARN - 2026-05-05 12:33:51.952; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes # WARNING: URLs provided to this tool needn't include Solr's context-root (e.g. "/solr"). Such URLs are deprecated and support for them will be removed in a future release. Correcting from [http://solr-node2:8984/solr] to [http://solr-node2:8984/]. # Created collection 'test-collection' with 3 shard(s), 1 replica(s) with config-set 'techproducts' # Checking collection health # /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/test/test_rolling_upgrade.bats: line 118: wait_for: command not found # solr-node1 # solr-node2 # solr-node3 # solr-node1 # solr-node2 # solr-node3 # solr-data1 # solr-data2 # solr-data3 # solrcloud-test ok 84 allowPaths - backup # in 15669 ms ok 85 snapshot lifecycle # in 4839 ms ok 86 snapshot list # in 3241 ms ok 87 snapshot describe # in 3391 ms ok 88 start solr with ssl # in 16152 ms ok 89 use different hostname when not checking peer-name # in 24719 ms ok 90 start solr with ssl and auth # in 14579 ms ok 91 start solr with client truststore and security manager # in 16260 ms ok 92 start solr with mTLS needed # in 46613 ms ok 93 start solr with mTLS wanted # in 34118 ms ok 94 test keystore reload # in 58704 ms ok 95 SOLR-11740 check 'solr stop' connection # in 15882 ms ok 96 stop command for single port # in 18371 ms ok 97 check stop command doesn't hang # in 218 ms ok 98 SOLR-16976 solr starts with remote JMX enabled # in 7952 ms ok 99 deprecated system properties converted to modern properties # in 195160 ms ok 100 start with custom jetty options # in 8033 ms ok 101 bootstrapping a configset # in 9206 ms ok 102 status detects locally running solr # in 9583 ms ok 103 status with --solr-url from user # in 8064 ms ok 104 status with --port from user # in 7924 ms ok 105 multiple connection options are prevented # in 841 ms ok 106 status with invalid --solr-url from user # in 1258 ms ok 107 status with --short format # in 195843 ms ok 108 searching solr via locally executed streaming expression # in 1743 ms ok 109 searching solr via remotely executed streaming expression # in 1314 ms ok 110 variable interpolation # in 1689 ms ok 111 searching solr without credentials fails with error # in 1135 ms ok 112 --version returns Solr version # in 759 ms ok 113 version as direct tool call still runs # in 743 ms ok 114 short help # in 821 ms ok 115 short help is inferred # in 815 ms ok 116 long help # in 791 ms ok 117 running subcommands with zk is prevented # in 749 ms ok 118 listing out files # in 2210 ms ok 119 connecting to solr via various solr urls and zk hosts # in 7861 ms ok 120 copying files around # in 11207 ms ok 121 upconfig # in 2537 ms ok 122 SOLR-12429 test upconfig fails with symlink # in 1209 ms ok 123 downconfig # in 1179 ms ok 124 bin/solr zk cp gets 'solrhome' from '--solr-home' command line option # in 1450 ms ok 125 env var ZK_HOST is honored # in 3168 ms ok 126 Cleanup # in 1863 ms > Task :solr:packaging:integrationTests FAILED FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build.gradle' line: 339 * What went wrong: Execution failed for task ':solr:packaging:integrationTests'. > Process 'command > '/home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/.gradle/node/node_modules/bats/bin/bats'' > finished with non-zero exit value 1 * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. > Get more help at https://help.gradle.org. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 31m 38s 128 actionable tasks: 128 executed Publishing Build Scan to Develocity... https://develocity.apache.org/s/yf4oio33rw32e Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Archiving artifacts Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
