Re: [DISCUSS] Mango indexes on FDB

[Alex Miller]

> On Mar 24, 2020, at 05:51, Garren Smith  wrote:
> On Tue, Mar 24, 2020 at 1:30 AM Joan Touzet  > wrote:
> 
>> Question: Imagine a node that's been offline for a bit and is just
>> coming back on. (I'm not 100% sure how this works in FDB land.) If
>> there's a (stale) index on disk, and the index is being updated, and the
>> index on disk is kind of stale...what happens?
>> 
> 
> With couchdb_layer this can't happen as each CouchDB node is stateless and
> doesn't actually keep any indexes. Everything would be in FoundationDB. So
> if the index is built then it is built and ready for all couch_layer nodes.
> 
> FoundationDB storage servers could fall behind the Tlogs. I'm not 100% sure
> what would happen in this case. But it would be consistent for all
> couch_layer nodes.

When a client gets a read version to begin a transaction in FDB, it is promised 
that this was the most recent version at some point in time between issuing the 
request and receiving the reply.  When it issues reads, those reads must 
include the version, and must get back the most recently written value for that 
key as of the included version.  FDB is not allowed to break this contract 
during faults.

The cluster will continue advancing in versions, as it does not throttle if 
only one server in a shard falls behind (or is offline).  When the server comes 
back online, it will pull the stream of mutations from the transaction logs to 
catch up.  In the meantime, it will continue to be unavailable for reads until 
it catches up, as clients send read requests for a specific (recent) version 
that the lagging storage server knows that it does not have.  After 1s, it will 
reply with a `future_version` error to tell the client it won’t be getting an 
answer soon.  The client will then make a decision based upon either the error 
or observed latency to re-issue the read to a different replica of that shard 
so that it may get an answer, and will continue doing so until it notices that 
the lagged storage server has caught up and is responding successfully.

If you’re interested in more details around the operational side of a storage 
server failure, I’d suggest reading the threads that Kyle Snavely started on 
the FDB Forums:
https://forums.foundationdb.org/t/quick-question-on-tlog-disk-space-for-large-clusters/1962
 

https://forums.foundationdb.org/t/questions-regarding-maintenance-for-multiple-storage-oriented-machines-in-a-data-hall/2010
 

Re: [DISCUSS] Mango indexes on FDB

[Robert Samuel Newson]

Hi,

We had a long discussion on the CouchDB Slack on this topic, which I will 
brutally summarise;

While we intend to update json indexes in the same transaction as the 
associated document update, there is a concern that this won't always be 
possible (large document and large number of indexes to update at once) and 
that it will not apply to javascript map indexes or search indexes.

It was therefore felt that having an immediate "Not ready" signal for just 
_some_ calls to _find, based on the type of backing index, was a bad and 
confusing api.

We also discussed _find calls where the user does not specify an index, and 
concluded that we would be free to choose between using the _all_docs index 
(which is always up to date but rarely the best index for a given selector) or 
blocking to update a better but stale index.

Summary-ing my summarisation;

1) if you specify an index, we'll use it even if we have to update it, no 
matter how long that takes.
2) if you don't specify an index, it's the dealers choice. The details here may 
change in point releases.

No new status code is therefore needed.

B.

> On 24 Mar 2020, at 12:51, Garren Smith  wrote:
> 
> On Tue, Mar 24, 2020 at 1:30 AM Joan Touzet  wrote:
> 
>> 
>> 
>> On 2020-03-23 4:46 p.m., Mike Rhodes wrote:
>>> Garren,
>>> 
>>> Very much +1 on this suggestion, as it is, at least for me, what I'd
>> expect to happen if I were leaving the system to select an index -- as you
>> imply, the build process almost certainly takes longer than using the
>> _all_docs index. In addition, for the common case where there is a less
>> optimal but still useful index available, one might expect that index to be
>> used in preference to the "better" but unbuilt one.
>> 
>> I agree.
>> 
>>> But I do think this is important:
>>> 
 We can amend the warning message
 to let them know that they have an index that is building that could
 service the index when it's ready.
>>> 
>>> Otherwise it's a bit too easy to get confused when trying to understand
>> the reason why an index you were _sure_ should've been used in fact was not.
>> 
>> Question: Imagine a node that's been offline for a bit and is just
>> coming back on. (I'm not 100% sure how this works in FDB land.) If
>> there's a (stale) index on disk, and the index is being updated, and the
>> index on disk is kind of stale...what happens?
>> 
> 
> With couchdb_layer this can't happen as each CouchDB node is stateless and
> doesn't actually keep any indexes. Everything would be in FoundationDB. So
> if the index is built then it is built and ready for all couch_layer nodes.
> 
> FoundationDB storage servers could fall behind the Tlogs. I'm not 100% sure
> what would happen in this case. But it would be consistent for all
> couch_layer nodes.
> 
> 
> 
>> 
>> -Joan

Re: [DISCUSS] Mango indexes on FDB

[Garren Smith]

On Tue, Mar 24, 2020 at 1:30 AM Joan Touzet  wrote:

>
>
> On 2020-03-23 4:46 p.m., Mike Rhodes wrote:
> > Garren,
> >
> > Very much +1 on this suggestion, as it is, at least for me, what I'd
> expect to happen if I were leaving the system to select an index -- as you
> imply, the build process almost certainly takes longer than using the
> _all_docs index. In addition, for the common case where there is a less
> optimal but still useful index available, one might expect that index to be
> used in preference to the "better" but unbuilt one.
>
> I agree.
>
> > But I do think this is important:
> >
> >> We can amend the warning message
> >> to let them know that they have an index that is building that could
> >> service the index when it's ready.
> >
> > Otherwise it's a bit too easy to get confused when trying to understand
> the reason why an index you were _sure_ should've been used in fact was not.
>
> Question: Imagine a node that's been offline for a bit and is just
> coming back on. (I'm not 100% sure how this works in FDB land.) If
> there's a (stale) index on disk, and the index is being updated, and the
> index on disk is kind of stale...what happens?
>

With couchdb_layer this can't happen as each CouchDB node is stateless and
doesn't actually keep any indexes. Everything would be in FoundationDB. So
if the index is built then it is built and ready for all couch_layer nodes.

FoundationDB storage servers could fall behind the Tlogs. I'm not 100% sure
what would happen in this case. But it would be consistent for all
couch_layer nodes.



>
> -Joan
>

Re: [DISCUSS] Mango indexes on FDB

[Robert Newson]

No, 425 is something specific

A 503 Service Unavailable seems the only suitable standard code. 

B. 

> On 24 Mar 2020, at 08:48, Glynn Bird  wrote:
> 
> If a user didn't specify the index they wanted to use, leaving the choice
> of index up to CouchDB, I would expect Couch would ignore the partially
> built index and fall back on _all_docs. so +1 on this.
> 
> But we need also consider the API response if a user *specifies* an index
> during a query (with use_index) when that index is not built yet, I think I
> would prefer an instant 4** response indicating that the requested
> resource isn't ready yet, rather than performing a very slow,
> _all_docs-powered search. Is "425 Too Early" a suitable response?
> 
> 
> 
> 
>> On Mon, 23 Mar 2020 at 23:30, Joan Touzet  wrote:
>> 
>> 
>> 
>>> On 2020-03-23 4:46 p.m., Mike Rhodes wrote:
>>> Garren,
>>> 
>>> Very much +1 on this suggestion, as it is, at least for me, what I'd
>> expect to happen if I were leaving the system to select an index -- as you
>> imply, the build process almost certainly takes longer than using the
>> _all_docs index. In addition, for the common case where there is a less
>> optimal but still useful index available, one might expect that index to be
>> used in preference to the "better" but unbuilt one.
>> 
>> I agree.
>> 
>>> But I do think this is important:
>>> 
 We can amend the warning message
 to let them know that they have an index that is building that could
 service the index when it's ready.
>>> 
>>> Otherwise it's a bit too easy to get confused when trying to understand
>> the reason why an index you were _sure_ should've been used in fact was not.
>> 
>> Question: Imagine a node that's been offline for a bit and is just
>> coming back on. (I'm not 100% sure how this works in FDB land.) If
>> there's a (stale) index on disk, and the index is being updated, and the
>> index on disk is kind of stale...what happens?
>> 
>> -Joan
>> 

Re: [DISCUSS] Mango indexes on FDB

[Glynn Bird]

If a user didn't specify the index they wanted to use, leaving the choice
of index up to CouchDB, I would expect Couch would ignore the partially
built index and fall back on _all_docs. so +1 on this.

But we need also consider the API response if a user *specifies* an index
during a query (with use_index) when that index is not built yet, I think I
would prefer an instant 4** response indicating that the requested
resource isn't ready yet, rather than performing a very slow,
_all_docs-powered search. Is "425 Too Early" a suitable response?




On Mon, 23 Mar 2020 at 23:30, Joan Touzet  wrote:

>
>
> On 2020-03-23 4:46 p.m., Mike Rhodes wrote:
> > Garren,
> >
> > Very much +1 on this suggestion, as it is, at least for me, what I'd
> expect to happen if I were leaving the system to select an index -- as you
> imply, the build process almost certainly takes longer than using the
> _all_docs index. In addition, for the common case where there is a less
> optimal but still useful index available, one might expect that index to be
> used in preference to the "better" but unbuilt one.
>
> I agree.
>
> > But I do think this is important:
> >
> >> We can amend the warning message
> >> to let them know that they have an index that is building that could
> >> service the index when it's ready.
> >
> > Otherwise it's a bit too easy to get confused when trying to understand
> the reason why an index you were _sure_ should've been used in fact was not.
>
> Question: Imagine a node that's been offline for a bit and is just
> coming back on. (I'm not 100% sure how this works in FDB land.) If
> there's a (stale) index on disk, and the index is being updated, and the
> index on disk is kind of stale...what happens?
>
> -Joan
>