[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[ItamarWMDE]

ItamarWMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ItamarWMDE
Cc: Lucas_Werkmeister_WMDE, toan, hoo, Lydia_Pintscher, RolandUnger, Aklapper, 
Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, 
SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[ItamarWMDE]

ItamarWMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ItamarWMDE
Cc: Lucas_Werkmeister_WMDE, toan, hoo, Lydia_Pintscher, RolandUnger, Aklapper, 
Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, 
SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a comment.


  The Wikivoyage template vCard  
can use up to 50 statements per template (see table ParWD at 
Module:VCard/Params , too). 
About 5 `addStatementUsage` calls per template on average are not really much 
even for an entity which is already loaded (loading takes only about 2.5 ms). 
Within the last weeks we added only //one// statement (P3025 
: open days). Therefore, the number of 
`addStatementUsage` calls increased only by a few percent.
  
  Some days ago a single `addStatementUsage` call costs // only about 1 ms // 
but now // 6 ms //. The question arose: why this drastic change: more than 500 
%? I have no idea why an access to an entity table already loaded is // now // 
as much expensive.
  
  If we cannot solve this problem then Wikidata will become useless for 
Wikimedia projects especially for list articles. An optimized Wikidata access 
for Wikivoyage and other Wikimedia projects is a mission-critical goal. We need 
the data at Wikidata to easily maintain them for all Wikivoyage projects. The 
main problem at Wikivoyage is to use data not only in a single information box 
but at many places in text, too. And why we cannot use this data any longer as 
we did it within the last four years?
  
  When I started to work on a Wikidata usage in vCard template four years ago I 
was really astonished by the huge computing time: about one minute per article 
(of course, the parsing was stopped before). That's why I am using a lot of 
helper tables to // prevent // additional Wikidata access for instance to get 
labels and country-specific data.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: Lucas_Werkmeister_WMDE, toan, hoo, Lydia_Pintscher, RolandUnger, Aklapper, 
Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, 
SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.


  Halle (Saale) currently has some seven hundred statement usages:
  
MariaDB [dewikivoyage]> SELECT COUNT(*) FROM wbc_entity_usage WHERE 
eu_page_id = (SELECT page_id FROM page WHERE page_namespace = 0 AND page_title 
= 'Halle_(Saale)') AND eu_aspect LIKE 'C%';
+--+
| COUNT(*) |
+--+
|  702 |
+--+
  
  If each of those corresponds to one `addStatementUsage` call, then they only 
cost ca. 6 ms per call.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, toan, hoo, Lydia_Pintscher, RolandUnger, Aklapper, 
Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, 
SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[WMDE-leszek]

WMDE-leszek updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: WMDE-leszek
Cc: Lucas_Werkmeister_WMDE, toan, hoo, Lydia_Pintscher, RolandUnger, Aklapper, 
Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, 
SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added subscribers: toan, Lucas_Werkmeister_WMDE.
Lucas_Werkmeister_WMDE added a comment.


  @toan suggested getting old profiling reports from the Internet Archive; they 
have one Halle (Salle) snapshot from August 2020 
,
 where the Lua profile looks like this:
  
Lua Profile:
Scribunto_LuaSandboxCallback::callParserFunction2120 ms 
  41.7%
Scribunto_LuaSandboxCallback::getEntity  800 ms 
  15.7%
Scribunto_LuaSandboxCallback::addStatementUsage  580 ms 
  11.4%
Scribunto_LuaSandboxCallback::getEntityStatements360 ms 
   7.1%
?180 ms 
   3.5%
Scribunto_LuaSandboxCallback::gsub   140 ms 
   2.8%
recursiveClone120 ms 
   2.4%
init80 ms 
   1.6%
Scribunto_LuaSandboxCallback::addSiteLinksUsage   60 ms 
   1.2%
(for generator)   60 ms 
   1.2%
[others] 580 ms 
  11.4%
Number of Wikibase entities loaded: 157/400

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, toan, hoo, Lydia_Pintscher, RolandUnger, Aklapper, 
Akuckartz, Iflorez, darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, 
Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, 
SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, 
Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lydia_Pintscher]

Lydia_Pintscher moved this task from Prioritized Product to 
Wikidata-Campsite-Iteration-∞ on the Wikidata-Campsite board.
Lydia_Pintscher edited projects, added Wikidata-Campsite 
(Wikidata-Campsite-Iteration-∞); removed Wikidata-Campsite.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3402/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, Iflorez, 
darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, SundanceRaphael, _jensen, 
rosalieper, Scott_WUaS, Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, 
GWicke, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lydia_Pintscher]

Lydia_Pintscher added a comment.


  Mike_Peel and RexxS might be other people to ask if they've seen similar 
changes in their Lua usage.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lydia_Pintscher]

Lydia_Pintscher updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a comment.


  If the bug is caused by cache limitations: Maybe a simple destructor 
`entity:remove()` would help. People told me that all entities are cached until 
the end of article parsing. But most of the entities are used in only one 
template and must not be cached any longer.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a comment.


  I tried to remove the museum section from 
https://de.wikivoyage.org/wiki/Halle_(Saale) and to stop using qualifiers and 
references from entities but the bug remained.
  
  But it seems that this bug depends on the number of entities loaded:
  
  - https://de.wikivoyage.org/wiki/Wernigerode: 33 entities: addStatementUsage 
not mentioned in Lua profile
  - https://de.wikivoyage.org/wiki/El_Gouna: 28 entities: addStatementUsage 180 
ms (12 % of Lua computing time)
  - https://de.wikivoyage.org/wiki/St._Gallen: 62 entities: addStatementUsage 
400 ms (14 %)
  - https://de.wikivoyage.org/wiki/Bremen: 104 entities: addStatementUsage 640 
ms (16 %)
  - https://de.wikivoyage.org/wiki/Benutzer:Balou46/Testseite6: 124 entities: 
addStatementUsage 3020 ms (48 %)
  - https://de.wikivoyage.org/wiki/Benutzer:Balou46/Testseite5: 127 entities: 
addStatementUsage 2780 ms (47 %)
  - https://de.wikivoyage.org/wiki/Eisenbahnmuseen_in_Europa: 144 entities: 
addStatementUsage 6900 ms (69 %)
  - https://de.wikivoyage.org/wiki/Halle_(Saale): 166 entities: 
addStatementUsage 4220 ms (51 %)
  
  A week ago, the addStatementUsage computing time for the last case was about 
600 ms (less than 10 %).
  
  All statements are retrieved with `entity:getBestStatements( p )` function in 
`Module:Wikidata_utilities`. The article entity is untouched.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a comment.


  See also T189409 .

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Anomie, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added projects: MediaWiki-extensions-WikibaseClient, 
MediaWiki-extensions-Scribunto.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, 
Jonas, Volker_E, alex-mashin, Wikidata-bugs, aude, GWicke, Dinoguy1000, jayvdb, 
MrStradivarius, Anomie, Jackmcbarn, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a comment.


  It seems that it is a sudden spike. The computing time only for 
`addStatementUsage` jumped from about 600 ms to 5000 ms (all Lua stuff from 
about 4.5 sec to about 9 sec). Within the last two weeks I added and optimized 
the analysis of opening hours to our vCard template. As in the past I used the 
same technique to //prevent// retrieving data from Wikidata. We are using 
several tables like this: https://de.wikivoyage.org/wiki/Modul:Hours/i18n with 
prepared data. On this way there is usually only one getEntity call per 
template. The number of entities used increased within the last two weeks only 
by about 1. Of course I added some statements to the Wikidata datasets (mainly 
of museums) to test the display of opening hours. Within the last two weeks 
there was no significant change. The core functions were added already on Sep. 
26, 2020.
  
  There is another article with the same behavior: 
https://de.wikivoyage.org/wiki/Eisenbahnmuseen_in_Europa. For this article, I 
got today in the morning a runtime error that there was not enough time.
  
Lua Profile:
Scribunto_LuaSandboxCallback::addStatementUsage 5580 ms 
  68.7%
Scribunto_LuaSandboxCallback::addSiteLinksUsage  800 ms 
   9.9%
Scribunto_LuaSandboxCallback::callParserFunction 720 ms 
   8.9%
Scribunto_LuaSandboxCallback::getEntity  320 ms 
   3.9%
Scribunto_LuaSandboxCallback::addLabelUsage  120 ms 
   1.5%
Scribunto_LuaSandboxCallback::incrementStatsKey  120 ms 
   1.5%
Scribunto_LuaSandboxCallback::fullUrl 80 ms 
   1.0%
Scribunto_LuaSandboxCallback::gsub40 ms 
   0.5%
init 40 ms 
   0.5%
? 40 ms 
   0.5%
[others] 
  
  I think there is a cache problem (see also T236749 
). And I assume that we exceeded a 
threshold with only minor additions. As already stated I think this bug is the 
same like T236485 . The caching bug 
was not really solved but only downplayed. Therefore it was only a question of 
time that this bug reoccured.
  
  There are only 166 entity calls (one per template) in the article of Halle 
(Saale) which is clearly less than the limit of 400. And I hope that we can use 
up to 250 Wikidata calls in future. Of course we use a lot of properties (in 
article mentioned about 50, maximally about 100) but we do not retrieve 
additional ones because of the tables mentioned. (I remember that you stated 
that Wikivoyage is the Wikimedia site which uses Wikidata in most extreme 
extent -- more than Commons. That is really true.)

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[Lydia_Pintscher]

Lydia_Pintscher edited projects, added Wikidata-Campsite; removed 
Wikidata-Campsite (Wikidata-Campsite-Iteration-∞).
Lydia_Pintscher added a comment.


  Thank you, Roland.
  
  Some questions:
  
  - Was this a sudden spike or a slow increase?
  - When did you see the change?
  - Is this only happening on this article or also others?
  
  Some initial investigation I've done:
  
  - The changes since the beginning of July on the Item for Halle 

 don't seem very significant so should not be the cause.
  - The article uses data from a lot of other entities 
. There 
is a chance that one of those has been significantly changed. I have not 
checked yet.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, darthmon_wmde, 
Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Volker_E, 
Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, Rxy, Jay8g, Iflorez, 
alaa_wmde
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a project: MediaWiki-Cache.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: hoo, Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, Iflorez, 
darthmon_wmde, alaa_wmde, Nandana, lucamauri, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, 
Scott_WUaS, Jonas, Volker_E, Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, 
Rxy, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger added a comment.


  Maybe it is the same like T236485 .

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, Iflorez, darthmon_wmde, 
alaa_wmde, Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, 
Volker_E, Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T265562: Wikibase Lua: addStatementUsage seems unjustifiably expensive

[RolandUnger]

RolandUnger created this task.
RolandUnger added projects: Wikidata, Wikibase-Lua, Wikidata-Campsite 
(Wikidata-Campsite-Iteration-∞), Performance Issue.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  The computing times increased drastically by about 100 %, mainly produced by 
a drastic increase of the computing time of the Lua function addStatementUsage 
(increase of about 500 %).
  
  From the Lua profile of https://de.wikivoyage.org/wiki/Halle_(Saale) (see 
html source code):
  
Lua Profile:
Scribunto_LuaSandboxCallback::addStatementUsage 4540 ms 
  55.0%
Scribunto_LuaSandboxCallback::callParserFunction1640 ms 
  19.9%
Scribunto_LuaSandboxCallback::getEntity  540 ms 
   6.5%
Scribunto_LuaSandboxCallback::addSiteLinksUsage  420 ms 
   5.1%
Scribunto_LuaSandboxCallback::getEntityStatements320 ms 
   3.9%
init100 ms 
   1.2%
recursiveClone100 ms 
   1.2%
?100 ms 
   1.2%
Scribunto_LuaSandboxCallback::gsub80 ms 
   1.0%
init80 ms 
   1.0%
[others] 340 ms 
   4.1%

Maybe it is the same like T236485.

TASK DETAIL
  https://phabricator.wikimedia.org/T265562

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RolandUnger
Cc: Lydia_Pintscher, RolandUnger, Aklapper, Akuckartz, Iflorez, darthmon_wmde, 
alaa_wmde, Nandana, lucamauri, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, 
Volker_E, Wikidata-bugs, aude, GWicke, Dinoguy1000, Mbch331, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs