This is likely the same situation described in the upgrade notes: https://solr.apache.org/guide/8_11/solr-upgrade-notes.html#solr-8-5
Solr 8.5 introduces a change in the format used for the elements in the Overseer queues and maps Please let us know if that matches and helps Mike On Sun, Sep 25, 2022 at 7:21 PM Richard <[email protected]> wrote: > Hello again Solr gurus! > > We've been slowly progressing on our project to move our deployment from > v7 to v8 and I've hit a snag while doing some validation. I wanted to > see if this was an expected failure or a peculiarity of our setup. > > tl;dr: Solr8, when retrieving responses from a solr7 overseer throws > exceptions because they (seem to) use different codecs to save different > objects. Is this an expected incompatibility in a heterogeneous fleet, > are there others I may need to be wary of? Are we doing something odd? > > Details: > > I did an initial deployment of our build and ran one of our > orchestration tools. It essentially: > > 1. issues an async deletereplica call to the local solr instance (v8) > 2. polls requeststatus for that async_id until it completes > > The solr8 node logs contained the following while servicing > requeststatus: > > org.apache.solr.common.SolrException: Exception deserializing response > from Javabin > at > > org.apache.solr.cloud.OverseerSolrResponseSerializer.deserialize(OverseerSolrResponseSerializer.java:72) > ~[?:?] > at > > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$21(CollectionsHandler.java:908) > ~[?:?] > at > > org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1463) > ~[?:?] > at > > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:285) > ~[?:?] > at > > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:257) > ~[?:?] > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:216) > ~[?:?] > at > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:833) > ~[?:?] > at > > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:797) > ~[?:?] > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542) > ~[?:?] > ... > Caused by: java.lang.RuntimeException: Invalid version (expected 2, but > -84) or the data in not in 'javabin' format > at > org.apache.solr.common.util.JavaBinCodec._init(JavaBinCodec.java:213) > ~[?:?] > at > org.apache.solr.common.util.JavaBinCodec.initRead(JavaBinCodec.java:207) > ~[?:?] > at > org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:191) > ~[?:?] > at org.apache.solr.common.util.Utils.fromJavabin(Utils.java:188) > ~[?:?] > at > > org.apache.solr.cloud.OverseerSolrResponseSerializer.deserialize(OverseerSolrResponseSerializer.java:66) > ~[?:?] > ... 49 more > > Looking at JavaBinCodec on 8.11.2 I see that, sure enough, we require > 0x02 as the first byte of the serialized data at > /overseer/collection-map-completed. However the same is true on the > 7.7.3 JavaBinCodec. Initially I thought that this might mean we made a > change that resulted changing how the data there got serialized. > > My next step was to take an essentially clean release from the web for > 8.11.2 and 7.7.3 and issue an async deletereplica to see what got > stored: > > localhost $ ./zk.sh get > /overseer/collection-map-completed/mn-solr8-async-deletereplica | xxd > 00000000: 02c2 e027 7375 6363 6573 73a2 e037 3139 ...'success..719 > 00000010: 322e 3136 382e 312e 3137 393a 3839 3833 2.168.1.179:8983 > 00000020: 5f73 6f6c 72a1 e02e 7265 7370 6f6e 7365 _solr...response > 00000030: 4865 6164 6572 a2e0 2673 7461 7475 7306 Header..&status. > > localhost $ ./zk.sh get > /overseer/collection-map-completed/mn-solr7-async-deletereplica | xxd > 00000000: aced 0005 7372 002a 6f72 672e 6170 6163 ....sr.*org.apac > 00000010: 6865 2e73 6f6c 722e 636c 6f75 642e 4f76 he.solr.cloud.Ov > 00000020: 6572 7365 6572 536f 6c72 5265 7370 6f6e erseerSolrRespon > 00000030: 7365 4186 ae6d 5e29 0df0 0200 024a 000b seA..m^).....J.. > > For reference: 0xACED -> the magic number for Java's Serializeable > output and 0xAC -> 172 (uint8) and -84 (int8) which matches the > RuntimeException we originally saw. > > In addition to the codec change (Solr7 looks to use straight Java > Serialization (0xaced) vs JavaBinCodec (0x02) for Solr8) it also looks > like a different object entirely is getting stored. Solr7 keeping the > OverseerSolrResponse container and Solr8 having only the NamedList > responseData. > > Is this an expected failure mode / version incompatibility and is it > documented in someplace that I might not have seen? Maybe this is all > surprising and we've clearly configured something unexpectedly or are > doing something surprising? Happy to answer any questions if i've missed > some important details. > > Appreciate any guidance or insight folks might have here. > -r
