Addshore added a comment.
In T154601#6202918 <https://phabricator.wikimedia.org/T154601#6202918>, @Lucas_Werkmeister_WMDE wrote: > That said, I tried to run the queries against stat1007. The properties query completes in 18 seconds, that’s fine. But the items query just kept running, so I killed it after one hour or so (I don’t know how long exactly). This might need to be optimized further. Yup, with the old wb_terms storage this script was disabled due to the query never really finishing/causing too much load. Hence the desire to switch to Hadoop rather than using SQL. In T154601#2919187 <https://phabricator.wikimedia.org/T154601#2919187>, @Addshore wrote: > PHP Fatal error: Uncaught exception 'RuntimeException' with message 'Something went wrong with the db query' in /a/analytics-wmde/src/scripts/src/wikidata/datamodel/terms_by_language.php:24 > Stack trace: > #0 /a/analytics-wmde/src/scripts/src/wikidata/datamodel/terms_by_language.php(12): WikidataTermsByLanguage->execute() > #1 {main} > thrown in /a/analytics-wmde/src/scripts/src/wikidata/datamodel/terms_by_language.php on line 24 > + date '+%F %T daily.03.sh Ended!' > 2017-01-05 05:00:07 daily.03.sh Ended! > > Probably due to the query taking too long now as reported in https://lists.wikimedia.org/pipermail/analytics/2016-December/005631.html > > I'll disable the script to stop it running repeatedly while failing and work on a dump based replacement. So we either need to use the dump, or use hadoop. AFAIK we can't read from Hadoop using the wmde analytics scripts stuff. If we were to do this in a dump then we could do it as part of wmde scripts. Although that would be massively inefficient compared with hadoop TASK DETAIL https://phabricator.wikimedia.org/T154601 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Lucas_Werkmeister_WMDE, Ladsgroup, Bugreporter, WMDE-leszek, Aleksey_WMDE, Ivanhercaz, VIGNERON, Lydia_Pintscher, GoranSMilovanovic, gerritbot, Addshore, Sjoerddebruin, Aklapper, Blissjay007, Oblanco79, Alter-paule, Beast1978, Un1tY, Hook696, Daryl-TTMG, RomaAmorRoma, E.S.A-Sheild, Iflorez, darthmon_wmde, Kent7301, alaa_wmde, Meekrab2012, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Af420, Darkminds3113, Bsandipan, Lordiis, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
