[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Marostegui]

Marostegui added a comment.


  In T219145#5088371 , 
@alaa_wmde wrote:
  
  > In T219145#5087634 , 
@Marostegui wrote:
  >
  > > In T219145#5084305 , 
@alaa_wmde wrote:
  > >
  > > > @Marostegui thanks for your update!
  > > >
  > > > Postponing until the new master is up and running is very unlikely to 
be the adjustment we would go for.
  > > >
  > > > The more likely one would be to change to migration plan 2 (take from 
bonfire slides 
):
  > >
  > >
  > > I don't have access to that. Just requested it.
  >
  >
  > For some reason I can't see a request from you in my mailbox .. but here's 
the slide screenshot that describes the plan
  >  F28584563: image.png 
  
  
  @Addshore gave me access a few hours ago :)
  
  > 
  > 
  >>> **on write**: write to new schema and delete from old one
  >>>  **on read** read from both schemas
  >>>  when old wb_terms is empty, read only from new schema and drop wb_terms 
table eventually
  >>> 
  >>> This plan will actually not require extra space (not significant increase 
at least) and will result in reducing the total space usage over time.
  >> 
  >> You've got any estimations on how much it would require?
  > 
  > I think it won't require anything significant, I mean nothing other than 
usual dbms needs for schema/meta info on tables and indexes.
  > 
  >>> It is hard to estimate the footprint of this plan on performance, which 
is likely to be a little higher than the original one. If I can think of a way 
to estimate I will share some numbers soon.
  >> 
  >> This would be useful to know if you can come with some figures.
  >>  I assume it is possible to stop the migration right away if we see 
performance issues and then resume once we are on a better shape (ie: we 
discover the performance degradation is an issue and we decide to fully wait 
for the new master), right?
  > 
  > Good question.. I don't think stop here would be possible as in "we stop 
dealing with new schema immediately and just switch back to write and read from 
old wb_terms", because we would have already deleted some data from `wb_terms` 
and moved them over to new schema. So stopping here would mean to "migrate 
back" in which we: **on write** we write to old schema and delete from new 
schema, and **on read** we read from both until new schema is empty. That would 
not release us immediately from the degradation in performance as we would wish.
  > 
  >  
  > 
  > I think we'll need to wait until we have written some of the necessary code 
for migration and run some tests to get some numbers.
  
  Yes, we definitely need some re-thinking on how to stop the migration or 
rollback if something arises, which can be a possibility on such complex 
process. Specially performance-wise, things can go different than original 
planned. 
  Also, we might have the new master in place if this will take some more time, 
which is also a good thing in general: more disk space, faster disks, more 
memory... :-)

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Marostegui
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[alaa_wmde]

alaa_wmde added a comment.


  In T219145#5087634 , 
@Marostegui wrote:
  
  > In T219145#5084305 , 
@alaa_wmde wrote:
  >
  > > @Marostegui thanks for your update!
  > >
  > > Postponing until the new master is up and running is very unlikely to be 
the adjustment we would go for.
  > >
  > > The more likely one would be to change to migration plan 2 (take from 
bonfire slides 
):
  >
  >
  > I don't have access to that. Just requested it.
  
  
  For some reason I can't see a request from you in my mailbox .. but here's 
the slide screenshot that describes the plan
  F28584563: image.png 
  
  >> **on write**: write to new schema and delete from old one
  >>  **on read** read from both schemas
  >>  when old wb_terms is empty, read only from new schema and drop wb_terms 
table eventually
  >> 
  >> This plan will actually not require extra space (not significant increase 
at least) and will result in reducing the total space usage over time.
  > 
  > You've got any estimations on how much it would require?
  
  I think it won't require anything significant, I mean nothing other than 
usual dbms needs for schema/meta info on tables and indexes.
  
  >> It is hard to estimate the footprint of this plan on performance, which is 
likely to be a little higher than the original one. If I can think of a way to 
estimate I will share some numbers soon.
  > 
  > This would be useful to know if you can come with some figures.
  >  I assume it is possible to stop the migration right away if we see 
performance issues and then resume once we are on a better shape (ie: we 
discover the performance degradation is an issue and we decide to fully wait 
for the new master), right?
  
  Good question.. I don't think stop here would be possible as in "we stop 
dealing with new schema immediately and just switch back to write and read from 
old wb_terms", because we would have already deleted some data from `wb_terms` 
and moved them over to new schema. So stopping here would mean to "migrate 
back" in which we: **on write** we write to old schema and delete from new 
schema, and **on read** we read from both until new schema is empty. That would 
not release us immediately from the degradation in performance as we would wish.
  
  
  
  I think we'll need to wait until we have written some of the necessary code 
for migration and run some tests to get some numbers.

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Marostegui]

Marostegui added a comment.


  In T219145#5084305 , 
@alaa_wmde wrote:
  
  > @Marostegui thanks for your update!
  >
  > Postponing until the new master is up and running is very unlikely to be 
the adjustment we would go for.
  >
  > The more likely one would be to change to migration plan 2 (take from 
bonfire slides 
):
  
  
  I don't have access to that. Just requested it.
  
  > **on write**: write to new schema and delete from old one
  >  **on read** read from both schemas
  >  when old wb_terms is empty, read only from new schema and drop wb_terms 
table eventually
  > 
  > This plan will actually not require extra space (not significant increase 
at least) and will result in reducing the total space usage over time.
  
  You've got any estimations on how much it would require?
  
  > It is hard to estimate the footprint of this plan on performance, which is 
likely to be a little higher than the original one. If I can think of a way to 
estimate I will share some numbers soon.
  
  This would be useful to know if you can come with some figures.
  I assume it is possible to stop the migration right away if we see 
performance issues and then resume once we are on a better shape (ie: we 
discover the performance degradation is an issue and we decide to fully wait 
for the new master), right?

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Marostegui
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[alaa_wmde]

alaa_wmde added a comment.


  @Marostegui thanks for your update!
  
  Postponing until the new master is up and running is very unlikely to be the 
adjustment we would go to.
  
  The more likely one would be to change to migration plan 2 (take from bonfire 
slides 
):
  **on write**: write to new schema and delete from old one
  **on read** read from both schemas
  when old wb_terms is empty, read only from new schema and drop wb_terms table 
eventually
  
  This plan will actually not require extra space (not significant increase at 
least) and will result in reducing the total space usage over time gradually 
too.
  It is hard to estimate the footprint of this plan on performance, which is 
likely to be a little higher than the original one. If I can think of a way to 
estimate I will share some numbers soon.
  
  We have yet to discuss it a bit amongst us .. but thought of sharing it with 
you already in case you can already spot any concerns with it.

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Marostegui]

Marostegui added a comment.


  Yeah, I just read a notification from Jaime on a different ticket about it. 
Looks like he wasn't aware either of that actually being created, that is a bit 
odd but should be discussed on that other ticket not here I guess :-/
  If it is disabled and with minimal data (`-rw-rw 1 mysql mysql 192K Jan  
9 23:47 wb_terms.ibd`), it shouldn't be a blocker for this.
  I will do some math on s8 with those estimations and get back to you

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Marostegui
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Ladsgroup]

Ladsgroup added a comment.


  In T219145#5078233 , 
@Marostegui wrote:
  
  > wb_terms is in commonswiki (s4)? What? When did that happen?
  
  
  commons turned to a Wikibase repo as part of SDoC then it started to have 
some data but it got disabled later. I'm not sure dropping the table would be 
good idea but it's empty so not much harm I guess.

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Ladsgroup
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Marostegui]

Marostegui added a comment.


  wb_terms is in commonswiki (s4)? What? When did that happen?

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Marostegui
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[alaa_wmde]

alaa_wmde added a comment.


  @Marostegui we can ignore s5 .. and probably s3 is many orders of magnitude 
smaller than the estimations above .. not sure if that tells you something

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Ladsgroup]

Ladsgroup added a comment.


  In T219145#5077988 , 
@alaa_wmde wrote:
  
  > @Marostegui I think so as it is only needed where we have `wb_terms` table 
for now. @Ladsgroup am I correct ?
  
  
  We have testwikidatawiki in s3 + wb_terns exists in commonswiki (s5) but it's 
empty

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Ladsgroup
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[alaa_wmde]

alaa_wmde added a comment.


  @Marostegui I think so as it is only needed where we have `wb_terms` table 
for now. @Ladsgroup am I correct ?

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T219145: Pass migration plan to DBAs for review

[Marostegui]

Marostegui added a comment.


  @alaa_wmde quick question before I read everything, those extra requirements 
will only affect s8  (wikidatawiki), right?

TASK DETAIL
  https://phabricator.wikimedia.org/T219145

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: alaa_wmde, Marostegui
Cc: jcrespo, Marostegui, JeroenDeDauw, Addshore, Ladsgroup, Aklapper, 
alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, 
_jensen, rosalieper, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs