Hey Steffen, I can reproduce the issue on 8.8.2 using the steps you laid out in your original email. (With one deviation - I had to enable auth to get around an error about "Can't create a configset with an unauthenticated request from a trusted baseConfigSet")
The steps you're using to trigger this are a bit odd. Do you really edit the configset stored with the backup as a part of your production workflow, or is this just a step you introduced to simplify the reproduction? The /schema API is the recommended way to handle configuration changes, followed by editing the configset files and uploading them to ZooKeeper. None of that quirkiness makes this any less of a bug though - it seems very much like something's going wrong here. I can't imagine any reason why Solr would intentionally not pick up fields until a restart. To further the reproduction a bit, it's possible to reproduce this without backup/restore in the picture at all. The key piece seems to be that the new (post-schema-edit) collection and config set have the same name as their now-deleted counterparts. 1. Start Solr: bin/solr start -c -p 7574 2. Turn on auth (this was necessary for me to create a configset that extended "_default"): bin/solr auth enable -type basicAuth -credentials solr:solrRocks -z localhost:8574 3. Create a config based on "_default": curl -ilk -X GET -u solr:solrRocks "http://localhost:7574/solr/admin/configs?action=CREATE&name=testconfigset&baseConfigSet=_default"; 4. Create a collection using this configset: curl -ilk -X GET -u solr:solrRocks "http://localhost:7574/solr/admin/collections?action=CREATE&name=coll1&collection.configName=testconfigset&numShards=1"; 5. Download the configset locally: bin/solr zk downconfig -d downloaded_test_config -n testconfig2 -z localhost:8574 6. Drop a new <field> definition into downloaded_test_config/conf/managed-schema 7. Delete the existing collection: curl -ilk -X GET -u solr:solrRocks "http://localhost:7574/solr/admin/collections?action=DELETE&name=coll1"; 8. Delete the configset from ZK: curl -ilk -X GET -u solr:solrRocks "http://localhost:7574/solr/admin/configs?action=DELETE&name=testconfigset"; 9. Create the config from the local copy: bin/solr zk upconfig -d downloaded_test_config -n testconfigset -z localhost:8574 10. Create a new collection with the same name as the previous collection: curl -ilk -X GET -u solr:solrRocks "http://localhost:7574/solr/admin/collections?action=CREATE&name=coll1&collection.configName=testconfigset&numShards=1"; 11. List the fields and show that the added field is missing: curl -ilk -X GET -u solr:solrRocks "http://localhost:7574/solr/coll1/schema/fields"; My best guess is that Solr is caching the configset (or the schemas parsed out of them) somewhere that sticks around even after the corresponding configset/collection is deleted. But I'm not very familiar with our configset code, so I can't dig in further than that with the time I have. Hopefully someone else can pick it up from here. In the meantime, you might be able to workaround this by specifying a "collection.configName" param on your RESTORE call - this will cause the backed up configset to be uploaded to ZK under a different name - hopefully avoiding whatever caching is causing the trouble here. Best, Jason On Fri, Jun 11, 2021 at 8:38 AM Jason Gerlowski <[email protected]> wrote: > > Hey Steffen, > > I took a quick look at the backup/restore codepath involved here - > surprisingly the restore code itself hasn't changed between 8.6.3 and > 8.8.2. In both 8.6.3 and 8.8.2, if the configset mentioned in the > backup has the same name as a config currently in ZooKeeper, the > version in ZK is used. So fields added to the schema after the backup > _should_ be visible. Obviously per your report this isn't happening > though. To me that says that there must be some change of behavior in > collection-creation or in schema-reading. > > I'm hoping to do a bit more digging to get to the bottom of this. In > the meantime, would you mind filing a JIRA ticket regarding this > behavior (if you haven't already)? > > One last question: you mentioned above that the field becomes visible > after a restart - does a reload > (/admin/collections?action=RELOAD&name=testcollection) have the same > effect? If so that may be a more palatable work-around for you in the > short-term... > > Best, > > Jason > > On Thu, Jun 3, 2021 at 9:32 AM Steffen Moldenhauer > <[email protected]> wrote: > > > > we tried to upgrade from 8.6.3 to 8.8 but had trouble with collection > > restore and schema changes. > > If there were changes to the schema - they are not visible in the Schema > > API. > > We are using the backup/restore to transfer pre-built collections to > > production systems and this issue prevents us from using the latest version > > 8.8.2 > > > > These are the basic steps to reproduce the issue on a "solr start -c" : > > > > For preparation create a config set, collection and backup: > > http://localhost:8983/solr/admin/configs?action=CREATE&name=testcollection&baseConfigSet=_default > > http://localhost:8983/solr/admin/collections?action=CREATE&collection.configName=testcollection&name=testcollection&numShards=1 > > http://localhost:8983/solr/admin/collections?action=BACKUP&location=/var/solr_backup&name=testcollection_backup&collection=testcollection > > > > Check the list of fields with the schema API: > > http://localhost:8983/solr/testcollection/schema/fields > > Note the fields listed. > > > > Simulate a schema change - add a field "test1"to the managed-schema: > > /var/solr_backup/testcollection_backup/zk_backup/configs/testcollection/managed-schema > > <field name="test1" type="string" indexed="true" stored="true" /> > > > > Delete the collection and config set and restore it from the backup: > > http://localhost:8983/solr/admin/collections?action=DELETE&name=testcollection > > http://localhost:8983/solr/admin/configs?action=DELETE&name=testcollection > > http://localhost:8983/solr/admin/collections?action=RESTORE&location=/var/solr_backup&name=testcollection_backup&collection=testcollection > > > > Check the list of fields with the schema API again: > > http://localhost:8983/solr/testcollection/schema/fields > > The added test1 field is missing. > > > > The field is immediately visible in 8.6.3 In 8.8.2 a server restart is > > necessary to see the new field. > > Is this a known issue? > > > > Regards > > Steffen Moldenhauer
