[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Aklapper]

Aklapper added a parent task: T212282: Timeout in Special Page Constraints 
check for France (Q142).

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Aklapper
Cc: Epidosis, Lectrician1, Manuel, Naike, Ottomata, JeroenDeDauw, Michael, 
WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Ullasoff, Danny_Benjafield_WMDE, S8321414, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Demian, Eihel, 
DannyS712, Nandana, kostajh, Lahi, Gq86, GoranSMilovanovic, RazeSoldier, 
QZanden, KimKelting, Esc3300, merbst, LawExplorer, _jensen, rosalieper, 
xSavitar, Scott_WUaS, Izno, SBisson, Uzume, Perhelion, Wikidata-bugs, Base, 
aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, 
Addshore, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[VladimirAlexiev]

VladimirAlexiev added a subtask: T290961: rewrite KrBot to publish Constraint 
Violation pages.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: VladimirAlexiev
Cc: Manuel, Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
Invadibot, maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, 
Gq86, GoranSMilovanovic, RazeSoldier, QZanden, Esc3300, merbst, LawExplorer, 
_jensen, rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, 
Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, 
Jdforrester-WMF, Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Manuel]

Manuel added a comment.


  Hi @WMDE-leszek, did you have time to look at this already?

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Manuel
Cc: Manuel, Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
Invadibot, maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, 
Gq86, GoranSMilovanovic, RazeSoldier, QZanden, Esc3300, merbst, LawExplorer, 
_jensen, rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, 
Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, 
Jdforrester-WMF, Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore added a comment.


  @WMDE-leszek will be tackling the topic of this RFC again shortly.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
Invadibot, maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, 
Gq86, GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, 
rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, 
Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, 
Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
Invadibot, maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, 
Gq86, GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, 
rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, 
Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, 
Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Ottomata]

Ottomata added a comment.


  @Addshore I've quickly read the task description but I have to admit I don't 
fully understand it yet (what gets stored where, etc.).  Find me on IRC? or set 
up quick little meeting? :)

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ottomata
Cc: Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, Gq86, 
GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, 
rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, 
Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, 
Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore added a comment.


  @Lucas_Werkmeister_WMDE would you be able to come up with some sort of 
estimation regarding the disk this would likely end up using?

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, Gq86, 
GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, 
rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, 
Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, 
Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[daniel]

daniel added a subscriber: Naike.
daniel added a comment.


  In T214362#6921322 , 
@Addshore wrote:
  
  > It looks like we would be able to use the ParserCache for this one T270710: 
Allow values other than ParserOutput to be stored in a ParserCache instance 
 is done?
  
  Yes, but that's not on any roadmap currently. If you want it prioritized, 
please reach out to @Naike so she can slot it into our process.
  
  > So I guess the scope of this RFC would then switch to, is it okay to use 
the parser cache for this?
  
  Conceptually I think this is the correct thing to do. But there are 
oprational considerations - if this adds a considerable amount of data to the 
storage that backs the parser cache, this needs to be planned. Scaling the 
parser cache backend storage is currently a very manual and slow process.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Naike, Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, 
CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, 
Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, 
maantietaja, Akuckartz, Demian, DannyS712, Nandana, kostajh, Lahi, Gq86, 
GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, 
rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, 
Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, 
Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore added a subscriber: Ottomata.
Addshore added a comment.


  @Ottomata This could also be a good candidate for storage in some persistent 
kafka stream in production?
  Though we would likely also want to be able to back such a stream by SQL for 
3rd party users of this extension.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Ottomata, JeroenDeDauw, Michael, WMDE-leszek, eprodromou, CCicalese_WMF, 
kchapman, Krinkle, mobrovac, abian, Lydia_Pintscher, Lucas_Werkmeister_WMDE, 
Marostegui, Joe, daniel, Agabi10, Aklapper, Addshore, maantietaja, Akuckartz, 
Demian, DannyS712, Nandana, kostajh, Lahi, Gq86, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, GWicke, 
Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, Jay8g, 
Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore added a comment.


  It looks like we would be able to use the ParserCache for this one T270710: 
Allow values other than ParserOutput to be stored in a ParserCache instance 
 is done?

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: JeroenDeDauw, Michael, WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, 
Krinkle, mobrovac, abian, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, 
Joe, daniel, Agabi10, Aklapper, Addshore, maantietaja, Akuckartz, Demian, 
DannyS712, Nandana, kostajh, Lahi, Gq86, GoranSMilovanovic, RazeSoldier, 
QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, Scott_WUaS, Izno, 
SBisson, Perhelion, Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, 
santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[CCicalese_WMF]

CCicalese_WMF removed a project: Platform Engineering.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: CCicalese_WMF
Cc: Michael, WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, Krinkle, 
mobrovac, abian, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, 
daniel, Agabi10, Aklapper, Addshore, Akuckartz, Demian, DannyS712, Nandana, 
kostajh, Lahi, Gq86, GoranSMilovanovic, RazeSoldier, QZanden, merbst, 
LawExplorer, _jensen, rosalieper, xSavitar, Scott_WUaS, Izno, SBisson, 
Perhelion, Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, 
Jdforrester-WMF, Ladsgroup, Mbch331, Rxy, Jay8g, Ltrlg, bd808, Legoktm, 
WDoranWMF, holger.knust, EvanProdromou, Pchelolo
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  If it would help I'm more than happy to talk this through from the product 
side in a call and I'm sure Lucas and Adam would be as well.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, WDoranWMF, holger.knust, EvanProdromou, 
DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, 
GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, 
Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  In T214362#6680770 , 
@Krinkle wrote:
  
  > - The authoritive source for describing items is Wikidata.org.
  > - The authoritive source for describing the constraint checks is also on 
Wikidata.org.
  >
  > What is the authoritive source for executing a constraint check if all 
caches and secondary services were empty? I believe this is currently in 
MediaWiki (WBQC extension), which may consult WDQS as part of the constraint 
check, where WDQS in this context is mainly used as a way to query the 
relational data of Wikidata items, which we can't do efficiently within 
MediaWiki so we rely on WDQS for that. This means to run a constraint check, 
WDQS needs to be fairly up to date with all the items, which happens through 
some sync process that is not related to this RFC. Does that sound right?
  
  Yes, though I would add that WDQS is only used for certain kinds of 
constraint checks – mainly those that have to traverse the subclass of 
 hierarchy. Constraint checks that 
only use the data of one or two items (this item should also have property X; 
the value of property Y should be an item that has property Z) don’t use the 
query service.
  
  Or put differently – when checking constraints, we use the query service for 
data of some other items, but not for data of the item itself; that’s why it’s 
okay if the query service has slightly stale data when we check constraints of 
an item immediately after it has been edited. (I realized while writing this 
that that’s not entirely true, but hopefully T269859 
 won’t be too hard to fix.)
  
  > Speaking of caches and secondary services, where do we currently expose or 
store the result of constraint checks? As I understand it, they are:
  >
  > - Saved in Memcached for 1 day after a computation happens.
  > - Exposed via action=rdf, which is best-effort only. Returns cache hit or 
nothing. It's not clear to me when one would use this, and what higher-level 
requirements this needs to meet. I'll assume for now there are cases somewhere 
where a JS gadget can't affort to wait to generate it and is fine with results 
just being missing if they weren't recently computed by something unrelated.
  
  It’s mainly used by the query service updater, which adds constraint check 
results to the query service, best-effort as you say.
  
  > - Exposed via Special:ConstraintReport/Q123, which ignores the cache and 
always computes it fresh.
  > - Exposed via API action=wbcheckconstraints, which is the main and reliable 
way to access this data from the outside. Considers cache and re-generates on 
the fly as needed, so it might be slow.
  >
  > It's not clear to me why Special:ConstraintReport exists in this way.
  
  Historically, the special page predates the API (and also any form of 
caching); the main reason it //still// exists this way is just that we haven’t 
removed it yet. You can see on Grafana 
 that 
it’s barely used, a dozen requests a day or so.
  
  I’ll skip quoting the rest of the comment, but regarding using WDQS as the 
store itself, I guess it’s important that WDQS isn’t really one store – there 
are (I believe) about a dozen instances, distributed across eqiad and codfw, 
and they’re fairly independent from one another. They’re supposed to all 
contain the same data, since they update themselves from the same data source, 
but over time the number of triples 
 
still drifts apart slightly. So to use WDQS as the store, we would have to push 
the data to each instance. The store which this RFC asks for, I believe, should 
instead store the data only once, and then each of the query service updaters 
(each instance has its own update) can pull the data from there, and so can 
action=wbcheckconstraints.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, WDoranWMF, holger.knust, EvanProdromou, 
DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, 
GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, 
Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Krinkle]

Krinkle added a comment.


  I still find myself very confused by this task and imagine that others might 
be struggling as well.
  
  I'll try to pick up the thread from before and continue to ask clarifying 
questions.
  
  - The authoritive source for describing items is Wikidata.org.
  - The authoritive source for describing the constraint checks is also on 
Wikidata.org.
  
  What is the authoritive source for executing a constraint check if all caches 
and secondary services were empty? I believe this is currently in MediaWiki 
(WBQC extension), which may consult WDQS as part of the constraint check, where 
WDQS in this context is mainly used as a way to query the relational data of 
Wikidata items, which we can't do efficiently within MediaWiki so we rely on 
WDQS for that. This means to run a constraint check, WDQS needs to be fairly up 
to date with all the items, which happens through some sync process that is not 
related to this RFC. Does that sound right?
  
  Speaking of caches and secondary services, where do we currently expose or 
store the result of constraint checks? As I understand it, they are:
  
  - Saved in Memcached for 1 day after a computation happens.
  - Exposed via action=rdf, which is best-effort only. Returns cache hit or 
nothing. It's not clear to me when one would use this, and what higher-level 
requirements this needs to meet. I'll assume for now there are cases somewhere 
where a JS gadget can't affort to wait to generate it and is fine with results 
just being missing if they weren't recently computed by something unrelated.
  - Exposed via Special:ConstraintReport/Q123, which ignores the cache and 
always computes it fresh.
  - Exposed via API action=wbcheckconstraints, which is the main and reliable 
way to access this data from the outside. Considers cache and re-generates on 
the fly as needed, so it might be slow.
  
  It's not clear to me why Special:ConstraintReport exists in this way. I 
suspect maybe it is to allow for an external service to be a cache or storage 
of constraint check results without having to worry about stale caches, so it's 
bascially exposing the computation end-point directly. That seems fine. What 
are those external stores? I think that's WDQS right? So WDQS is used for 
storing relational items, but also for storing constraint data. If so, why not 
use that as the store for this task? (Also, how is that currently backfilled? 
Are we happy with that? Would a different outcome to this task result in WDQS 
no longer doing this this way?)
  
  I suspect the reason we don't want to use WDQS for this is that you want to 
regularly clear that out and repopulate it from scratch, and ideally in a way 
that doesn't require running all non-memcached checks again which presumably 
would take a very long time. How long would that be? And how do we load these 
results currently into WDQS?
  
  Having public dumps of these seems valuable indeed. Is this something that 
could be dumped from WDQS?
  
  Responding to the main RFC question - using a permanent store logically owned 
by MediaWiki and populated progressively seems like a better direction indeed 
and would make sense. I'm not seriously proposing WDQS be used for this, rather 
I'm trying to better understand the needs through asking what it isn't serving 
right now.
  
  Assuming a store will be needed, how big would the canonical data be in 
gigabytes? What would be the expected writes per second from the job queue for 
this. And the expected reads per second from the various end points?
  
  Could it have a natural eviction strategy where MW takes care of replacing or 
removing things that are no longer needed, or would it need TTL-based eviction?
  
  Depending on the answers to this, using Main Stash might work for this. Which 
is essentially a persisted and replicated cache without LRU/pressure eviction, 
which sounds like it would fit. This is currently backed by Redis and was until 
recently used for sessions. This is now being migrated to a simple external 
MariaDB cluster (T212129 ).

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Krinkle
Cc: WMDE-leszek, eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, WDoranWMF, holger.knust, EvanProdromou, 
DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, 
GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, 
Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mail

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[daniel]

daniel added a comment.


  In T214362#6653148 , 
@Addshore wrote:
  
  > It's a shame that this has made its way into the "icebox" of the 
#platform_engineering_roadmap_decision_making 

 board.
  
  To clarify: It's unfortunately not very clear, but "idebox" doesn't mean "we 
definitely won't do it". It just means "not currently on our roadmap". If it's 
important to someone, it can be taken out of the icebox. It's basically a 
matter of discussing priorities and collaboration between teams.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, WDoranWMF, holger.knust, EvanProdromou, 
DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, 
GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, 
Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore added a comment.


  It's a shame that this has made its way into the "icebox" of the 
#platform_engineering_roadmap_decision_making 

 board.
  We (WMDE) are seemingly not in a place to make this decision ourselves and 
need collaboration from #techcom 
 and I imagine 
#platform_engineering 
 on this matter.
  Potentially this could make for a good candidate for the new 
https://www.mediawiki.org/wiki/Proposal_for_a_Technical_Decision_Making_Process

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, WDoranWMF, holger.knust, EvanProdromou, 
DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, 
GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, 
Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Addshore]

Addshore added a comment.


  T227776: General ParserCache service class for large "current" page-derived 
data  has now been abandoned though 
T268848: Allow ParserCache to store information other than ParserOutput 
 could still be interesting for this 
RFC.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, WDoranWMF, holger.knust, EvanProdromou, 
DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, 
RazeSoldier, QZanden, merbst, LawExplorer, _jensen, rosalieper, xSavitar, 
Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, Wikidata-bugs, Base, aude, 
GWicke, Bawolff, jayvdb, fbstj, santhosh, Jdforrester-WMF, Ladsgroup, Mbch331, 
Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T214362: RFC: Store WikibaseQualityConstraint check data in persistent storage

[Naike]

Naike added a project: Platform Engineering Roadmap Decision Making.

TASK DETAIL
  https://phabricator.wikimedia.org/T214362

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Naike
Cc: eprodromou, CCicalese_WMF, kchapman, Krinkle, mobrovac, abian, 
Lydia_Pintscher, Lucas_Werkmeister_WMDE, Marostegui, Joe, daniel, Agabi10, 
Aklapper, Addshore, Akuckartz, Demian, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, DannyS712, Nandana, kostajh, Lahi, Gq86, Pablo-WMDE, 
GoranSMilovanovic, RazeSoldier, QZanden, merbst, LawExplorer, _jensen, 
rosalieper, xSavitar, Scott_WUaS, Pchelolo, Izno, SBisson, Perhelion, 
Wikidata-bugs, Base, aude, GWicke, Bawolff, jayvdb, fbstj, santhosh, 
Jdforrester-WMF, Ladsgroup, Mbch331, Rxy, Jay8g, Ltrlg, bd808, Legoktm
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs