Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
COngrats Donat. Awesome news. On Thu, Jan 14, 2021 at 11:56 PM Joan Touzet wrote: > Cheers Donat - thanks for the info! And welcome again. > > -Joan > > On 2021-01-14 4:26 p.m., Bessenyei Balázs Donát wrote: > > Dear All, > > > > Thank you for the invitation and the welcome! It's a great honour. > > > > @Joan: my username is bessbd where I can pick and it isn't already in > > use. People usually call me Donat. > > > > > > Donat > > > > > > On Thu, Jan 14, 2021 at 10:09 PM Nick Vatamaniuc > wrote: > >> > >> Congrats Donat and welcome! > >> > >> -Nick > >> > >> > >> On Thu, Jan 14, 2021 at 3:55 PM Joan Touzet wrote: > >>> > >>> Congratulations Bessenyei! Do you have a nickname (other than bessbd)? > >>> > >>> -Joan > >>> > >>> On 2021-01-14 3:48 p.m., Robert Newson wrote: > Dear community, > > I am pleased to announce that the CouchDB Project Management > Committee has elected Bessenyei Balázs Donát as a CouchDB committer. > > Apache ID: bessbd > > Committers are given a binding vote in certain project decisions, as > well as write access to public project infrastructure. > > This election was made in recognition of Donát's commitment to the > project. We mean this in the sense of being loyal to the project and its > interests. > > Please join me in extending a warm welcome to Donát! > > On behalf of the CouchDB PMC, > > Robert Newson > >
Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Cheers Donat - thanks for the info! And welcome again. -Joan On 2021-01-14 4:26 p.m., Bessenyei Balázs Donát wrote: Dear All, Thank you for the invitation and the welcome! It's a great honour. @Joan: my username is bessbd where I can pick and it isn't already in use. People usually call me Donat. Donat On Thu, Jan 14, 2021 at 10:09 PM Nick Vatamaniuc wrote: Congrats Donat and welcome! -Nick On Thu, Jan 14, 2021 at 3:55 PM Joan Touzet wrote: Congratulations Bessenyei! Do you have a nickname (other than bessbd)? -Joan On 2021-01-14 3:48 p.m., Robert Newson wrote: Dear community, I am pleased to announce that the CouchDB Project Management Committee has elected Bessenyei Balázs Donát as a CouchDB committer. Apache ID: bessbd Committers are given a binding vote in certain project decisions, as well as write access to public project infrastructure. This election was made in recognition of Donát's commitment to the project. We mean this in the sense of being loyal to the project and its interests. Please join me in extending a warm welcome to Donát! On behalf of the CouchDB PMC, Robert Newson
Re: [DISCUSS] Replicator scheduler improvement
On 2021-01-12 12:55 p.m., Nick Vatamaniuc wrote: Hello everyone Hi (Dr.) Nick! ;) I wanted to see what we thought about adding a scheduling improvement to the replicator. Specifically adding round-robin fairness amongst different _replicator dbs We've needed some sort of priority system for a while -- critically, in some cases, our failure to offer something has forced my hand into recommending use of external replication scheduling techniques. I'd never thought of using multiple databases as a way of handling this problem, though. Currently, the scheduler runs all the jobs in the system fairly. It does it by using the jobs' "last started" timestamp to select which jobs to stop and start next. So each scheduling cycle, the jobs which ran the longest (have the lowest start time) get stopped, then, from the list of pending jobs we also pick those with the lowest start times (since they waited the longest) to run next. There is also some difference between continuous and one-shot replication jobs, isn't there? I seem to recall one-shot jobs get priority over continuous ones. However, this algorithm can be unfair among replication jobs from different _replicator dbs. For example, if max_jobs=10 and one _replicator db has a 1000 jobs and another has only one doc then, that one job might have to wait for quite a while to get its turn to run. If we picked fairly amongst _replicator dbs, then for example, the one job would get at least one of the 10 max_jobs run "slots", and the rest of the 9 slots would go to the replicator db with 1000 docs. If there would be 11 _replicator dbs, then a docs from the ones with the lowest start times amongst them would be picked first. That does sound unfair. I wasn't aware of this "gotcha" with multiple replicator DBs (in the field I rarely see >1 replicator DB) but it intuitively makes sense - all replicator DBs are unioned in the scheduler's queue and treated like a single, big DB. (There is basically no advantage to multiple _replicator DBs today then, other than perhaps security / obscurity reasons, is there?) This feature would also allow running some quick replication jobs, when there is already a full queue main _replicator db jobs by creating a "quickjobs/_replicator" db, and insert these new jobs there. With this new scheme they would start running right away even if the queue is full with jobs from the main _replicator db. Another, related idea I had, was to add per job user-configurable priorities: high/default/low or numeric. However, that scheme is not as good as it could lead to permanent starvation of jobs while the round-robin db scheduling scheme still guarantees all jobs will eventually run. I think this proposal can be improved. I like the idea of leveraging multiple replicator DBs to provide some sort of priority tweak, but I am not sure I can predict what will happen in the case of e.g. 10 different replicator DBs fighting with each other in the scheduler as you've laid it out. Would a fair-share scheduler [1] make sense here? We can't borrow many of the OS-level scheduling ideas because we don't have the same insight into the runtime characteristics of our jobs, but fair share might be a good match. Here's one way it could be implemented: * Each replicator DB could represent a logical grouping of jobs, e.g. background continuous backup jobs, one-shot migration jobs, high-priority user requests, etc. * An INI-level config could determine how these groups are handled: * 100% fair (default) -- total available job slots are divided by the number of replicator DBs, then allocated round-robin within that DB. The difference from today's setup is that each DB gets its own round-robin scheduler. I *think* this might be what Nick was suggesting, but I can't be 100% sure. * Specific priority -- A list of DBs and their fixed percentages. Example: _replicator:30,high/_replicator:50,low/_replicator:20 If a configured database is not present, or has no active jobs in it, its fixed proportion of cycles are "given up" to the other replicator DBs. * Jobs are then allocated based on these breakdowns, and we continue to use round-robin within each database to maintain the current approach. This means no visible change for upgrading users with a single _replicator DB. Example, with the numbers picked to make the math turn out nice: ``` [replicator] db_priorities = _replicator:30,high/_replicator:50,low/_replicator:20 max_jobs = 100 ``` The % allocation of jobs across DBs is, by default: _replicator: 30 / (30+20+50) = 30% high/_replicator: 20 / (30+20+50) = 20% low/_replicator: 50 / (30+20+50) = 50% First, put 100 jobs into _replicator. Because the other two DBs have no jobs, all 100 free jobs are allocated to those jobs: _replicator: 30 / (30) = 100% * 100 jobs = 100 Now, we add another 100 jobs into the high DB. Since low is empty, we recalculate calculate the dist
Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Congratulations, Donat! Well done! On Thu, Jan 14, 2021 at 4:26 PM Bessenyei Balázs Donát wrote: > Dear All, > > Thank you for the invitation and the welcome! It's a great honour. > > @Joan: my username is bessbd where I can pick and it isn't already in > use. People usually call me Donat. > > > Donat > > > On Thu, Jan 14, 2021 at 10:09 PM Nick Vatamaniuc > wrote: > > > > Congrats Donat and welcome! > > > > -Nick > > > > > > On Thu, Jan 14, 2021 at 3:55 PM Joan Touzet wrote: > > > > > > Congratulations Bessenyei! Do you have a nickname (other than bessbd)? > > > > > > -Joan > > > > > > On 2021-01-14 3:48 p.m., Robert Newson wrote: > > > > Dear community, > > > > > > > > I am pleased to announce that the CouchDB Project Management > Committee has elected Bessenyei Balázs Donát as a CouchDB committer. > > > > > > > > Apache ID: bessbd > > > > > > > > Committers are given a binding vote in certain project decisions, as > well as write access to public project infrastructure. > > > > > > > > This election was made in recognition of Donát's commitment to the > project. We mean this in the sense of being loyal to the project and its > interests. > > > > > > > > Please join me in extending a warm welcome to Donát! > > > > > > > > On behalf of the CouchDB PMC, > > > > > > > > Robert Newson > > > > >
Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Dear All, Thank you for the invitation and the welcome! It's a great honour. @Joan: my username is bessbd where I can pick and it isn't already in use. People usually call me Donat. Donat On Thu, Jan 14, 2021 at 10:09 PM Nick Vatamaniuc wrote: > > Congrats Donat and welcome! > > -Nick > > > On Thu, Jan 14, 2021 at 3:55 PM Joan Touzet wrote: > > > > Congratulations Bessenyei! Do you have a nickname (other than bessbd)? > > > > -Joan > > > > On 2021-01-14 3:48 p.m., Robert Newson wrote: > > > Dear community, > > > > > > I am pleased to announce that the CouchDB Project Management Committee > > > has elected Bessenyei Balázs Donát as a CouchDB committer. > > > > > > Apache ID: bessbd > > > > > > Committers are given a binding vote in certain project decisions, as well > > > as write access to public project infrastructure. > > > > > > This election was made in recognition of Donát's commitment to the > > > project. We mean this in the sense of being loyal to the project and its > > > interests. > > > > > > Please join me in extending a warm welcome to Donát! > > > > > > On behalf of the CouchDB PMC, > > > > > > Robert Newson > > >
Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Congrats Donat and welcome! -Nick On Thu, Jan 14, 2021 at 3:55 PM Joan Touzet wrote: > > Congratulations Bessenyei! Do you have a nickname (other than bessbd)? > > -Joan > > On 2021-01-14 3:48 p.m., Robert Newson wrote: > > Dear community, > > > > I am pleased to announce that the CouchDB Project Management Committee has > > elected Bessenyei Balázs Donát as a CouchDB committer. > > > > Apache ID: bessbd > > > > Committers are given a binding vote in certain project decisions, as well > > as write access to public project infrastructure. > > > > This election was made in recognition of Donát's commitment to the project. > > We mean this in the sense of being loyal to the project and its interests. > > > > Please join me in extending a warm welcome to Donát! > > > > On behalf of the CouchDB PMC, > > > > Robert Newson > >
Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Congratulations Bessenyei! Do you have a nickname (other than bessbd)? -Joan On 2021-01-14 3:48 p.m., Robert Newson wrote: Dear community, I am pleased to announce that the CouchDB Project Management Committee has elected Bessenyei Balázs Donát as a CouchDB committer. Apache ID: bessbd Committers are given a binding vote in certain project decisions, as well as write access to public project infrastructure. This election was made in recognition of Donát's commitment to the project. We mean this in the sense of being loyal to the project and its interests. Please join me in extending a warm welcome to Donát! On behalf of the CouchDB PMC, Robert Newson
[ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Dear community, I am pleased to announce that the CouchDB Project Management Committee has elected Bessenyei Balázs Donát as a CouchDB committer. Apache ID: bessbd Committers are given a binding vote in certain project decisions, as well as write access to public project infrastructure. This election was made in recognition of Donát's commitment to the project. We mean this in the sense of being loyal to the project and its interests. Please join me in extending a warm welcome to Donát! On behalf of the CouchDB PMC, Robert Newson
