[jira] [Commented] (SOLR-17460) Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files and Shard Restoration Failures
[ https://issues.apache.org/jira/browse/SOLR-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17897305#comment-17897305 ] Arda commented on SOLR-17460: - [~epugh] Thank you for asking! That’s a great question. There were two main reasons behind the decision to migrate from Solr 7 to Solr 8 before moving to Solr 9: *1. Dependency Considerations:* Some of the other products that collaborate with the Solr environment have recommended using Solr 8 for compatibility reasons, even though it’s not strictly necessary, but preferred. Because of this, I think, sticking with Solr 8 ensures better alignment and helps maintain stability across these integrations, which minimizes any potential interruptions. *2. Risk Mitigation:* Jumping straight from Solr 7 to Solr 9 means skipping over a major version (Solr 8), and with that comes major changes. I think, it may provide serious risks to a production environment. By upgrading step-by-step, I can manage any compatibility and configuration changes gradually, reducing the risk of serious issues along the way. Before moving on to Solr 9, I can fully test and resolve any issues by upgrading to Solr 8. > Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files > and Shard Restoration Failures > - > > Key: SOLR-17460 > URL: https://issues.apache.org/jira/browse/SOLR-17460 > Project: Solr > Issue Type: Bug > Components: hdfs, SolrCloud >Affects Versions: 7.0, 8.4 >Reporter: Arda >Priority: Minor > Labels: backup, restore > > I was attempting to migrate a collection with 3 shards from a Solr 7.0 > cluster to a Solr 8.4 cluster. The data is stored in HDFS. I followed the > backup-restore process but encountered issues with two of the shards during > the restoration. > h1. *Migration Process:* > *1-* *Backup Command:* To avoid timeouts, I initiated the backup with an > async parameter: > curl -k --negotiate -u : > 'https://:/solr/admin/collections?action=BACKUP&name=&collection=x&location=& > async=12346' > *2- Copy Backup to Local:* After the backup, I copied the data from HDFS to > the local filesystem: > hdfs dfs --copyToLocal > *3- Transfer Backup to New Cluster:* I then copied the backup files from the > older Solr node to the newer one: > scp -pr @: > *4- Prepare New HDFS Path:* On the new Solr cluster, I created a new > directory in HDFS and adjusted ownership: > hdfs dfs -mkdir > hdfs dfs -chown solr:solr > *5- Copy Backup to New HDFS Location:* I transferred the backup data from > local to the new HDFS path. Before that, I deleted > "queryDocAuthorization" parts from solrconfig.xml file to become > compatible with the newer version. > hdfs dfs --copyFromLocal > *6- Restore Collection:* Finally, I ran the restore command: > curl -k --negotiate -u : > 'https://:/solr/admin/collections?action=RESTORE&name=&collection=x&location=& > async=12345' > h1. > *Issue:* > After the restore process completed, I found that two of the shards could not > be restored. The logs displayed the following errors: > *Error During Shard Restoration:* > ERROR [c: s: shard2 r:core_node5 x: : > _shard2_replica_n4] o.a.s.h.RequestHandlerBase > org.apache.solr.common. SolrException: Error CREATEing SolrCore > '_shard2_replica_n4': Unable to create core > [:_shard2_replica_n4] Caused by: > org.apache.solr.handler.component.QueryDocAuthorizationComponent. > *FileNotFoundException and Index Corruption:* > WARN > (parallelCoreAdminExecutor-6-thread-7-processing-n::_solrx:_shard2 > _replica_n1 RESTORECORE) [x:_shard2_replica_n1] > o.a.s.h. RestoreCore Could not switch to restored index. Rolling back to the > current index => org.apache.lucene.index.CorruptindexException: Unexpected > file read error while reading index. > (resource=BufferedChecksumIndexInput(segments_1g9dk)) > Caused by: java.io. FileNotFoundException: File does not exist: > hdfs:///core_node2/data/restore/ > It appears that Solr is looking for a file in HDFS that doesn't exist, > despite no manual deletions being made. I cannot determine why these specific > shards failed to restore, or why the system is unable to locate the required > files. > *Expected Behavior:* > The backup and restore process should complete without errors, and all shards > should be restored successfully to the new cluster. > *Actual Behavior:* > Two shards failed to restore, with errors related to missing files and index > corruption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-17460) Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files and Shard Restoration Failures
[ https://issues.apache.org/jira/browse/SOLR-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17897273#comment-17897273 ] Eric Pugh commented on SOLR-17460: -- [~ardate] out of curiosity, what made you go to Solr 8 instead of going straight to Solr 9? > Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files > and Shard Restoration Failures > - > > Key: SOLR-17460 > URL: https://issues.apache.org/jira/browse/SOLR-17460 > Project: Solr > Issue Type: Bug > Components: hdfs, SolrCloud >Affects Versions: 7.0, 8.4 >Reporter: Arda >Priority: Minor > Labels: backup, restore > > I was attempting to migrate a collection with 3 shards from a Solr 7.0 > cluster to a Solr 8.4 cluster. The data is stored in HDFS. I followed the > backup-restore process but encountered issues with two of the shards during > the restoration. > h1. *Migration Process:* > *1-* *Backup Command:* To avoid timeouts, I initiated the backup with an > async parameter: > curl -k --negotiate -u : > 'https://:/solr/admin/collections?action=BACKUP&name=&collection=x&location=& > async=12346' > *2- Copy Backup to Local:* After the backup, I copied the data from HDFS to > the local filesystem: > hdfs dfs --copyToLocal > *3- Transfer Backup to New Cluster:* I then copied the backup files from the > older Solr node to the newer one: > scp -pr @: > *4- Prepare New HDFS Path:* On the new Solr cluster, I created a new > directory in HDFS and adjusted ownership: > hdfs dfs -mkdir > hdfs dfs -chown solr:solr > *5- Copy Backup to New HDFS Location:* I transferred the backup data from > local to the new HDFS path. Before that, I deleted > "queryDocAuthorization" parts from solrconfig.xml file to become > compatible with the newer version. > hdfs dfs --copyFromLocal > *6- Restore Collection:* Finally, I ran the restore command: > curl -k --negotiate -u : > 'https://:/solr/admin/collections?action=RESTORE&name=&collection=x&location=& > async=12345' > h1. > *Issue:* > After the restore process completed, I found that two of the shards could not > be restored. The logs displayed the following errors: > *Error During Shard Restoration:* > ERROR [c: s: shard2 r:core_node5 x: : > _shard2_replica_n4] o.a.s.h.RequestHandlerBase > org.apache.solr.common. SolrException: Error CREATEing SolrCore > '_shard2_replica_n4': Unable to create core > [:_shard2_replica_n4] Caused by: > org.apache.solr.handler.component.QueryDocAuthorizationComponent. > *FileNotFoundException and Index Corruption:* > WARN > (parallelCoreAdminExecutor-6-thread-7-processing-n::_solrx:_shard2 > _replica_n1 RESTORECORE) [x:_shard2_replica_n1] > o.a.s.h. RestoreCore Could not switch to restored index. Rolling back to the > current index => org.apache.lucene.index.CorruptindexException: Unexpected > file read error while reading index. > (resource=BufferedChecksumIndexInput(segments_1g9dk)) > Caused by: java.io. FileNotFoundException: File does not exist: > hdfs:///core_node2/data/restore/ > It appears that Solr is looking for a file in HDFS that doesn't exist, > despite no manual deletions being made. I cannot determine why these specific > shards failed to restore, or why the system is unable to locate the required > files. > *Expected Behavior:* > The backup and restore process should complete without errors, and all shards > should be restored successfully to the new cluster. > *Actual Behavior:* > Two shards failed to restore, with errors related to missing files and index > corruption. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-17460) Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files and Shard Restoration Failures
[ https://issues.apache.org/jira/browse/SOLR-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884426#comment-17884426 ] Arda commented on SOLR-17460: - To successfully migrate the collection with all data intact, I followed these steps: *1. Prepare the Collection Configuration:* First, I retrieved the Solr collection's configuration files from the Solr 7.0 cluster and modified them to be compatible with Solr 8.4. I then created the new collection on the Solr 8.4 cluster without any data, using the updated configuration. *2. Copy Shard Core Nodes Locally:* I copied each shard’s core nodes from the HDFS path of the Solr 7.0 cluster to a local directory. Using the Solr Web UI, I identified which nodes corresponded to each shard and replica for the collection. Note that the core node paths may differ for each collection. Here’s an example of the commands used: _# Shard 1, 2, and 3_ hdfs dfs --copyToLocal /core_node1 _# Shard 1_ hdfs dfs --copyToLocal /core_node2 _# Shard 2_ hdfs dfs --copyToLocal /core_node3 _# Shard 3_ _# Replica nodes_ hdfs dfs --copyToLocal /core_node6 _# Replica 1_ hdfs dfs --copyToLocal /core_node8 _# Replica 2_ hdfs dfs --copyToLocal /core_node10 _# Replica 3_ After copying the core node files locally, I transferred them to the Solr 8.4 cluster. *3. Copy Shard Core Nodes to HDFS on the New Cluster:* I copied the shard core node files into the appropriate HDFS directory on the new Solr 8.4 cluster. *Important:* The shard core nodes must be copied into the exact corresponding location. For example, if shard 1 was stored in *_core_node1_* in the old cluster and is now assigned to *_core_node5_* in the new cluster, you must copy the data from *_core_node1_* to *_core_node5._* Example commands: _# Copy shard 1, 2, and 3 core nodes_ hdfs dfs -put /core_node1 /core_node5 _# Shard 1_ hdfs dfs -put /core_node2 /core_node6 _# Shard 2_ hdfs dfs -put /core_node3 /core_node7 _# Shard 3_ _# Copy replica core nodes_ hdfs dfs -put /core_node6 /core_node11 _# Replica 1_ hdfs dfs -put /core_node8 /core_node12 _# Replica 2_ hdfs dfs -put /core_node10 /core_node9 _# Replica 3_ *4. Adjust Ownership in HDFS:* I changed the ownership of the collection’s HDFS path on the new cluster to ensure Solr had the necessary permissions to access the data. hdfs dfs -chown -R solr:solr To verify that the files were correctly copied, I compared the file sizes on both clusters using the following command: hdfs dfs -du -s -h -v -x *5. Reload the Collection:* Finally, I reloaded the collection via the Solr Web UI on the new Solr 8.4 cluster. To confirm the successful migration of all data, I queried the collection and verified that the document count matched the original. With these steps, I was able to migrate the entire collection along with all data. You can check the document count by running a simple query to verify that the migration was successful. > Error During Collection Migration from Solr 7.0 to Solr 8.4: Missing Files > and Shard Restoration Failures > - > > Key: SOLR-17460 > URL: https://issues.apache.org/jira/browse/SOLR-17460 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: hdfs, SolrCloud >Affects Versions: 7.0, 8.4 >Reporter: Arda >Priority: Minor > Labels: backup, restore > > I was attempting to migrate a collection with 3 shards from a Solr 7.0 > cluster to a Solr 8.4 cluster. The data is stored in HDFS. I followed the > backup-restore process but encountered issues with two of the shards during > the restoration. > h1. *Migration Process:* > *1-* *Backup Command:* To avoid timeouts, I initiated the backup with an > async parameter: > curl -k --negotiate -u : > 'https://:/solr/admin/collections?action=BACKUP&name=&collection=x&location=& > async=12346' > *2- Copy Backup to Local:* After the backup, I copied the data from HDFS to > the local filesystem: > hdfs dfs --copyToLocal > *3- Transfer Backup to New Cluster:* I then copied the backup files from the > older Solr node to the newer one: > scp -pr @: > *4- Prepare New HDFS Path:* On the new Solr cluster, I created a new > directory in HDFS and adjusted ownership: > hdfs dfs -mkdir > hdfs dfs -chown solr:solr > *5- Copy Backup to New HDFS Location:* I transferred the backup data from > local to the new HDFS path. Before that, I deleted > "queryDocAuthorization" parts from solrconfig.xml file to become > compatible with the newer version. > hdfs dfs --copyFromLocal > *6- Restore Collection:* Finally, I ran the restore command: > curl -k --negotiate -u : >
