[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2019-03-11 Thread elukey 
elukey added a comment.


  I found another occurrence of timeouts in mcrouter with SETs for this key, it 
might be due to TTL expiring or similar. Reducing the traffic volume to 
memcached is surely a good thing.

TASK DETAIL
  https://phabricator.wikimedia.org/T97368

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore, elukey
Cc: elukey, Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, 
mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, 
daniel, Aklapper, hoo, alaa_wmde, Legado_Shulgin, CucyNoiD, Nandana, 
NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, 
Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, 
Vali.matei, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Wong128hk, 
Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2019-03-10 Thread elukey 
elukey added a comment.


  Correction: the last bursts of mcrouter's timeouts seems to match a high set 
rate for slab 140 on mc1022, that is exactly where the key is:
  
  
https://grafana.wikimedia.org/d/00317/memcache-slabs?orgId=1=1552231823147=1552232692113=eqiad%20prometheus%2Fops=memcached=mc1022=All=49
  
  Will add more data tomorrow :)

TASK DETAIL
  https://phabricator.wikimedia.org/T97368

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore, elukey
Cc: elukey, Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, 
mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, 
daniel, Aklapper, hoo, alaa_wmde, Legado_Shulgin, CucyNoiD, Nandana, 
NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, 
Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, 
Vali.matei, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Wong128hk, 
Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2019-03-10 Thread elukey 
elukey added a comment.


  Quick comment: I noticed while checking some memcached metrics that 
`wikibase_shared/1_33_0-wmf_20-wikidatawiki-hhvm:CacheAwarePropertyInfoStore` 
generates 30~60MB/s of GET traffic since the last deployment of mediawiki:
  
  
https://grafana.wikimedia.org/d/00317/memcache-slabs?orgId=1=now-7d=now=eqiad%20prometheus%2Fops=memcached=mc1022=140=60
  
  Confirmed also via memkeys. Given what happened in T204083 
, is this meant to generate this 
kind of traffic volume? If so, will it increase over time, stay steady, etc..? 
I don't see impact on the network interface or correlation with timeouts, but 
better safe and ask :)

TASK DETAIL
  https://phabricator.wikimedia.org/T97368

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore, elukey
Cc: elukey, Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, 
mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, 
daniel, Aklapper, hoo, alaa_wmde, Legado_Shulgin, CucyNoiD, Nandana, 
NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, 
Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, 
Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, 
Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, 
Vali.matei, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, Wong128hk, 
Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2019-01-07 Thread gerritbot 
gerritbot added a comment.
Change 480731 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add docs/adr/0002-CacheAwarePropertyInfoStore-caching.md

https://gerrit.wikimedia.org/r/480731TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, _jensen, D3r1ck01, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-12-19 Thread gerritbot 
gerritbot added a comment.
Change 480731 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Add docs/adr/0002-CacheAwarePropertyInfoStore-caching.md

https://gerrit.wikimedia.org/r/480731TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, _jensen, D3r1ck01, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-12-18 Thread Addshore 
Addshore added a comment.
Picked back up by the campsite to write an ADR for what was done and why & how we might need to change this in the future.TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, _jensen, D3r1ck01, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-11-09 Thread Lydia_Pintscher 
Lydia_Pintscher added a comment.
@Addshore can you close this?TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, Lydia_PintscherCc: Lydia_Pintscher, Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, D3r1ck01, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-11-01 Thread gerritbot 
gerritbot added a comment.
Change 470567 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] CachingPropertyInfoLookup local server cache 10s -> 15s

https://gerrit.wikimedia.org/r/470567TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-10-30 Thread gerritbot 
gerritbot added a comment.
Change 470567 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] CachingPropertyInfoLookup local server cache 10s -> 15s

https://gerrit.wikimedia.org/r/470567TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-10-03 Thread Addshore 
Addshore added a comment.
The 10 second on server cache of the data resulted in a pretty significant drop in traffic:

F26296567: image.pngTASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, CucyNoiD, Nandana, NebulousIris, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-10-02 Thread Addshore 
Addshore added a comment.
This patch should go live on wikidata.org with the train on Wednesday 3rd OctTASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Jonas, Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, CucyNoiD, Nandana, NebulousIris, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-25 Thread gerritbot 
gerritbot added a comment.
Change 461371 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] SqlStore wrap PropertyInfo classes in an APC cache too

https://gerrit.wikimedia.org/r/461371TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, NebulousIris, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Jonas, Wong128hk, Wikidata-bugs, ArielGlenn, Lydia_Pintscher, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-19 Thread gerritbot 
gerritbot added a comment.
Change 461371 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] SqlStore wrap PropertyInfo classes in an APC cache too

https://gerrit.wikimedia.org/r/461371TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread Addshore 
Addshore added a comment.

In T97368#4576510, @daniel wrote:
Another option would be to force this to go into APC instead of memcached.


Even with just a short life this would probably massively reduce the traffic hitting this one key.TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread daniel 
daniel added a comment.
Another option would be to force this to go into APC instead of memcached.TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread Addshore 
Addshore added a comment.
So the logging I added only ran for 7 minutes, but already gives us some idea of the usage of this thing:

getPropertyInfoForDataType was not called at all. After a quick look through the code the only place this seems to be called is on https://www.wikidata.org/wiki/Special:ListProperties

addshore@mwlog1001:/srv/mw-log$ cat AdHocDebug.log |grep "dataType" | wc -l
0

getPropertyInfo (by ID) is far more heavily used. Some requests can apparently call this many thousands of times. The below request involves http://wikidata.org/wiki/Q56596767 which is pretty large:

addshore@mwlog1001:/srv/mw-log$ cat AdHocDebug.log |grep W5jEGwpAAK4AAKd9eigV |grep "for property" | wc -l
23100

It might make sense to have cache keys per property instead. That would result in around cache entries instead.
We would still have to cover the case of wanting to retrieve all property info at once, but the main key holding all data could remain, but the traffic to it would be greatly decreased.

The request above would have result in 11 separate calls to the cache:

addshore@mwlog1001:/srv/mw-log$ cat AdHocDebug.log |grep W5jEGwpAAK4AAKd9eigV |grep "for property" | cut -c 130- | sort | uniq -c
 10 P1433
 10 P1476
   9170 P1545
 91 P1932
  13636 P2093
 10 P31
 10 P356
 10 P433
 10 P478
133 P50
 10 P577

The number of calls to memcached essentially matches up to the number of properties referenced on the entity being touched.
I guess in the worst situation this would be ~6000, if an entity touched every single property, but this case would essentially never happen.
I'll try and get some numbers for the most number of properties used on a single entity.
If we were cleverer we could have some modifiable threshold to choose between hitting the cache of all property info vs the caches per property (if we go down this route)TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-09-12T07:45:39Z]  Synchronized php-1.32.0-wmf.20/extensions/Wikibase/lib/includes/Store/: REVERT: Debug logging for T97368 [[gerrit:459905]] (duration: 00m 51s)TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-09-12T07:38:06Z]  Synchronized php-1.32.0-wmf.20/extensions/Wikibase/lib/includes/Store/: Debug logging for T97368 [[gerrit:459905]] (duration: 00m 53s)TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot, gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread gerritbot 
gerritbot added a comment.
Change 459905 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.20] PropertyInfoStore cache usage debug logging for wmf.20

https://gerrit.wikimedia.org/r/459905TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread gerritbot 
gerritbot added a comment.
Change 459906 abandoned by Addshore:
Debug logging for T97368

Reason:
using AdHocDebug instead

https://gerrit.wikimedia.org/r/459906TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread gerritbot 
gerritbot added a comment.
Change 459906 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] Debug logging for T97368

https://gerrit.wikimedia.org/r/459906TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Zppix, Maathavan, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread gerritbot 
gerritbot added a comment.
Change 459905 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.20] PropertyInfoStore cache usage debug logging for wmf.20

https://gerrit.wikimedia.org/r/459905TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Davinaclare77, Qtn1293, Imarlier, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Vali.matei, Zppix, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T97368: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore

2018-09-12 Thread Addshore 
Addshore added a comment.

In T97368#1241094, @daniel wrote:
One thin we can do is to split the cache by "type" of info, i.e. have the data types cached separate from the cache fore base URIs, etc. We can further shard the cache, so base URIs would be cached in one of n random shards. This should give good performance for cases that need info about a lot of properties, while reducing the amount of data that needs to be loaded for a single lookup.


Sounds like it could be a good idea.


First lets figure out how the data being cached is actually used / which methods in CachingPropertyInfoLookup are called and how much.
Also we don't really need to rotate the cache key by mw version, the serialization code hasn't changed in some time (created by PropertyInfoBuilder) so we could switch to some other constant / versioning of the serialization for use in the cache key instead.


I'll file some sub tickets and get these brought into the #wikidata-campsite this week.TASK DETAILhttps://phabricator.wikimedia.org/T97368EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Jdforrester-WMF, Joe, mark, Addshore, jijiki, akosiaris, Krinkle, JanZerebecki, thiemowmde, aude, daniel, Aklapper, hoo, AndyTan, Davinaclare77, Qtn1293, Imarlier, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Vali.matei, Zppix, Wong128hk, Wikidata-bugs, ArielGlenn, faidon, He7d3r, Mbch331, Jay8g, fgiunchedi, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs