> Announce your intention to proceed with lazy consensus in two business days.
I intend to proceed then. : )
I'm going to file JIRAs for the SIP today and start pushing up PRs.
Based on how this discussion went I also plan on updating the "SIP"
process page to say that lazy consensus can be used
I think embrace lazy consensus -- no formal vote. Announce your intention
to proceed with lazy consensus in two business days.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Fri, Jan 15, 2021 at 9:32 AM Jason Gerlowski
wrote:
> Jan - I've
Jan - I've modified the file-structure page to include support for
storing multiple collections within the same "location" per your
suggestion.
I've also clarified the file-structure page to state that _all_ files
required to restore a shard are recorded in the shard-metadata file
(regardless of
Yeah, that's a good point. I think the only change we'd need to make
to the file-structure to support multi-collection down the road would
be to introduce a top-level directories for each collection. I'll
experiment with that and tweak the described file structure to handle
that (assuming the
> It's tough to decide between /v2/cluster/backups and
> /v2/collections//backups as alternatives until you
> figure out whether we currently support multi-collection backup, or
> want to in the near future.
I suppose multi-collection / TRA support cold be expanded on later
by supporting e.g.
Hey all,
Two follow ups on recent discussion.
I reviewed the gc/ref-counting part of the BlobDirectory proposal on
SOLR-15051 that David mentioned. We talked about it a bit offline and
agreed that while an automatic gc mechanism is really needed for what
he's trying to do, the requirements of
Hey all,
I've put replies to everyone's questions below. Hope they help!
> Do the shard metadata files list all of the segments that make up the backup,
> or only the segments that were uploaded in this incremental update?
Mike: The former - they're intended to hold metadata about all of the
Jason, Shalin and Dat, thanks for the thorough work. This is an example for
other SIPs to follow!
> I've also amended the backcompat/migration section to mention Jan's
> suggestion that the "incremental" features be exposed in the v2 API
> only. Though it's unclear to me whether that's still
Jason, I know you've seen SOLR-15051 (Shared storage -- BlobDirectory), but
perhaps not a part of the linked document:
https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit#
"ZK Data: Listings" that discusses an aspect of the design that is very
similar to a part of
Do the shard metadata files list all of the segments that make up the
backup, or only the segments that were uploaded in this incremental update?
On Fri, Jan 8, 2021 at 3:19 PM Jason Gerlowski
wrote:
> Sure thing. I put together a writeup on the file layout and formats
> here:
>
Sure thing. I put together a writeup on the file layout and formats
here:
https://cwiki.apache.org/confluence/display/SOLR/Incremental+Backup+File+Format
The details get a little verbose, so I made it a subpage that the
SIP-proper calls out to.
Let me know what you think when you get a chance
Thanks Jason! This is great, and a very much needed feature.
> This helps to avoid confusion that would
> otherwise arise between identically named files when e.g. a shard
> leader changes between two incremental backups. (I'll try to expand
> on this in the SIP, as it's a bit hard to give the
Thanks for the feedback Mike. I've gotta give any credit to Shalin
though, he wrote most of it before the holiday. He and Dat wrote much
of the code involved as well. I haven't done more than steward things
along so far. As you suggested, I've updated the SIP to mention the
related SOLR-13608
This is a very thorough SIP, thank you for spending the time on it, Jason!
I have a few minor questions about points that are unclear to me.
1) If we assume that we cannot overwrite files, how does the manifest file
stay current for incremental backup operations to the same directory?
2) How is
Can you explicitly call out in the SIP how it relates to the work done in
SOLR-13608?
On Tue, Jan 5, 2021 at 8:55 AM Jason Gerlowski
wrote:
> Hey, Happy New Year everybody.
>
> Some SIP updates based on the discussion above:
>
> I added v2 examples for each API to the SIP. Feedback welcome,
>
Hey, Happy New Year everybody.
Some SIP updates based on the discussion above:
I added v2 examples for each API to the SIP. Feedback welcome,
especially on the v2 APIs that are net-new to this proposal (namely:
"list backups" and "delete backup").
I've also amended the backcompat/migration
Ok, that’s the one I was looking for, it’s not documented in the backup chapter
of ref-guide :(
Jan Høydahl
> 23. des. 2020 kl. 17:10 skrev Jason Gerlowski :
>
>
>>
>> We have a path alias to the old API ... but we don’t have a true v2 API spec
>> for it, do we?
>
> Tbh I'm not yet
> We have a path alias to the old API ... but we don’t have a true v2 API spec
> for it, do we?
Tbh I'm not yet familiar enough with the v2 APIs to understand the
distinction you're making. (Do you have a pointer to something that'd
fill me in?)
To zoom in on "backup" as an example, the v2 API
Actually, don’t think we do have a v2 Backup/Restore API. We have a path alias
to the old API which takes GET ...=backup... but we don’t have a true v2
API spec for it, do we? Where is that documented?
Jan Høydahl
> 22. des. 2020 kl. 18:04 skrev Jason Gerlowski :
>
> Hey guys,
>
> Following
Thanks for taking on this effort, Jason. I'll review and suggest more next
year. My initial impression is that any non core functionality should
remain outside Solr core as much as possible. I hope we can leverage
modularity wherever possible.
On Tue, 22 Dec, 2020, 10:34 pm Jason Gerlowski,
Hey guys,
Following up to make sure I understand the specifics you're
suggesting. You're proposing that:
1. The brand new backup-related APIs (list-backups and delete-backup)
be added in v2-form only.
2. Tweaks to existing backup-related APIs (create-backup, restore) be
made in V2-form only.
3.
>, and implement the new imporved version as a V2-api only, and then deprecate
>the v1 API?
V2 only please
On Tue, Dec 22, 2020 at 1:34 AM Jason Gerlowski wrote:
>
> Hey Jan, thanks for the review.
>
> I hadn't thought about the V2 API in connection to this work. You're
> right though I
Hey Jan, thanks for the review.
I hadn't thought about the V2 API in connection to this work. You're
right though I think - the SIP proposes net-new APIs, so it should add
V2 equivalents at the very least. I'll draft tentative details for
these APIs on the SIP and we can refine things from
Much needed! Thanks for initiating this Jason!
As we want to move away from v1 APIs where a HTTP GET is used for creation and
deletion, would it be an idea to leave the old backup/resotre APIs as-is, and
implement the new imporved version as a V2-api only, and then deprecate the v1
API? Then
Hey all,
This morning I published SIP-12, which proposes an overhaul of Solr's
backup and restore functionality. While the "headline" improvement in
this SIP is a change to do backups incrementally, it bundles in a
number of other improvements as well, including the addition of
corruption
25 matches
Mail list logo
