I've opened a PR to make the documentation much clearer for this: https://github.com/apache/solr/pull/652
Please leave any notes if you think it can be improved! - Houston On Tue, Feb 8, 2022 at 12:18 PM Eric Charles <[email protected]> wrote: > I was not pre-creating the folder in the S3 bucket in my previous tests. > When I create that `temp1` folder (without `/` at the start), the following > 4 variants Houston just pasted work fine > > I think the confusion (and I have been fooled again just now) is that the > location=s3://temp1 does not contain the bucket name (it is just s3 prefix > and the folder) > > On 2022/02/07 17:59:09 Houston Putman wrote: > > Sorry for the back and forth everyone, I'm trying to understand the > issue, > > but I absolutely cannot replicate this locally. > > > > In the S3 console, I create a directory named "temp1/" *(Please note, > this > > is not the same as "/temp1/", the actual folder in the bucket CANNOT > start > > with a "/")* under the bucket that I am using. > > When I use the following Solr APIs, they ALL work for me: > > > > > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=temp1 > > > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=/temp1 > > > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=s3://temp1 > > > http://localhost:8983/solr/admin/collections?action=BACKUP&name=test-s3-test&repository=default&collection=test&location=s3:///temp1 > > > > The only possible reason I can think of for so many people to be seeing > > errors is that they are pre-creating directories in S3 that have "/" > > prepended to the name. > > This is already documented in the ref guide > > < > https://solr.apache.org/guide/8_11/making-and-restoring-backups.html#s3backuprepository > >, > > but we can think of verbiage to make it more clear. > > > > I understand that we can make it so that "location" is not a required > > property and default it to "/", but that doesn't make sense for all > backup > > repository types, such as a fileSystem backup. > > > > - Houston > > > > On Mon, Feb 7, 2022 at 8:54 AM Michael Conrad <[email protected]> > wrote: > > > > > The restore process allows restoring to a collection with a different > > > name than the original. So I would expect the requirement to have a > > > collection already existing to be legit. > > > > > > The documentation does say (or at least implies) that to restore the > > > number of shards in the destination collection has to match the backup. > > > > > > On 2/5/22 12:02, Eric Charles wrote: > > > > ... and thx Houston and other participants to this thread. > > > > > > > > I have just restored the backup, and it works fine. The only point is > > > that I had to create the collection before restoring (I had dropped > it). I > > > wonder if I have to recreate the collection with the same number of > > > shards... ? > > > > > > > > On 2022/02/05 16:55:26 Eric Charles wrote: > > > >> I have faced similar questions to get it working. I have first > opened > > > an issue on the solr-operatorhttps:// > > > github.com/apache/solr-operator/issues/404 and then was able to > > > replicate the issue on my laptop, so came to the conclusion the issue > was > > > at solr level, and not solr-operator. > > > >> > > > >> Without Sergio reply, I would not have been able to make sense of > the > > > parameters and get is working. Thx! > > > >> > > > >> On 2022/01/27 14:01:46 Sergio García Maroto wrote: > > > >>> Hi, > > > >>> > > > >>> Correct. Using "location=backupfolder" didn't work for me. > > > >>> > > > >>> The only way I made it work is with location=s3:/ > > > >>> Below my sample url which works well. > > > >>> > > > > http://servername:8983/solr/admin/collections?action=BACKUP&name=personbackup&collection=person&repository=s3&location=s3:/ > > > >>> > > > >>> > > > >>> On Tue, 25 Jan 2022 at 17:23, Houston Putman<[email protected]> > > > wrote: > > > >>> > > > >>>> Thanks for all of the information everybody. I want to determine > if > > > this is > > > >>>> actually a bug before we release 9.0 > > > >>>> > > > >>>> First, I want to clear up the usage of the "location" parameter: > > > >>>> > > > >>>> - It is required, but you can provide "/" as an "empty" > > > directory, much > > > >>>> like "s3:/". > > > >>>> - You don't have to include "s3:/" or "s3://". You can you > "/dir", > > > >>>> "dir", "s3:/dir" or "s3://dir". All of these options will > > > eventually be > > > >>>> converted to the "dir/" directory in your bucket. > > > >>>> - The s3 repository does not allow for directory names > starting > > > with > > > >>>> "/". In general this is to allow all of the above ^ examples > to > > > compute > > > >>>> to > > > >>>> the same thing without users being confused how many '/'s they > > > need > > > >>>> after > > > >>>> "s3:". Now I see that this may have contributed to the > confusion, > > > but we > > > >>>> can always improve the documentation in the ref-guide. > > > >>>> > > > >>>> Now with these points cleared up, you said that you were unable to > > > use the > > > >>>> collections API calls with "location=backupfolder" did not work, > even > > > >>>> though you had a directory in your bucket named "backupfolder". > Just > > > to be > > > >>>> clear, in S3 the folder did not have a "/" prefix correct? If so, > > > this is > > > >>>> indeed a bug and something that we should fix. I'll do more > thorough > > > >>>> testing once you confirm that the directory is following the > > > expectation. > > > >>>> > > > >>>> - Houston > > > >>>> > > > >>>> On Sat, Jan 22, 2022 at 3:32 AM Shawn Heisey<[email protected]> > > > wrote: > > > >>>> > > > >>>>> On 1/22/2022 12:38 AM, Sergio García Maroto wrote: > > > >>>>>> The only difference I can see i am using collection API trying > to > > > >>>> backup > > > >>>>>> solr cloud collection. > > > >>>>> That could do it. I have done very little with SolrCloud beyond > > > >>>>> occasionally firing up the cloud example. > > > >>>>> > > > >>>>> The Collections API backup is most likely a different code path > than > > > the > > > >>>>> replication handler backup. I wonder if the behavior you're > seeing > > > >>>>> would be considered a bug. > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Shawn > > > >>>>> > > > > > >
