date:20171003

[Wikidata-bugs] [Maniphest] [Commented On] T176312: Don’t check format constraint via SPARQL (safely evaluating user-provided regular expressions)

2017-10-03 Thread Halfak

Halfak added a comment.
I'm not sure we'd make use of an external service.  In our case, I think a more robust timeout and some testing gets us what we need.TASK DETAILhttps://phabricator.wikimedia.org/T176312EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: HalfakCc: Halfak, Anomie, Smalyshev, tstarling, daniel, GWicke, Joe, Lucas_Werkmeister_WMDE, Krinkle, Aklapper, GoranSMilovanovic, QZanden, Agabi10, Izno, SBisson, Wikidata-bugs, aude, jayvdb, fbstj, RobLa-WMF, santhosh, Jdforrester-WMF, Mbch331, Rxy, Jay8g, Ltrlg, bd808, Legoktm___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T177257: ArticlePlaceholder hit counts from bnwiki seem bogus

2017-10-03 Thread hoo

hoo updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION...What's going on here? All metrics seem overly high for `bnwiki` and I can't make any sense of that.TASK DETAILhttps://phabricator.wikimedia.org/T177257EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, hooCc: Aklapper, Lucie, Lydia_Pintscher, Nuria, Addshore, hoo, GoranSMilovanovic, QZanden, cmadeo, Izno, Wikidata-bugs, aude, jayvdb, Ricordisamoa, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

2017-10-03 Thread Gehel

Gehel added a comment.
It would be interesting to see if allocation rate goes up when we see the JVM locking up. gceasy does not graph allocation rate over time, but I think that JClarity Censum might do that (closed source, but demo license available).TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Gq86, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T177311: 2% of /w/load.php spent in WebStart/onBeforeCreateEchoEvent for Wikibase

2017-10-03 Thread Krinkle

Krinkle created this task.Krinkle added projects: Performance-Team, MediaWiki-extensions-WikibaseClient, Wikidata.Herald added a subscriber: Aklapper.
TASK DESCRIPTION/includes/WebStart.php


EchoHooks::initEchoExtension (3.05%)
Wikibase\Client\Hooks\EchoNotificationsHandlers::onBeforeCreateEchoEvent
Wikibase\Client\Hooks\EchoNotificationsHandlers::newFromGlobalState  (2.63%)



	public static function onBeforeCreateEchoEvent(
		array &$notifications,
		array &$notificationCategories,
		array &$icons
	) {
		$self = self::newFromGlobalState();
		$self->doBeforeCreateEchoEvent( $notifications, $notificationCategories, $icons );
	}

For example: https://performance.wikimedia.org/xenon/svgs/daily/2017-10-01.load.svgz
F9984501: Screen Shot 2017-10-03 at 19.24.12.png

The doBeforeCreateEchoEvent() method call itself is trivial. Most time is spent in newFromGlobalState(), which requires numerous services that are not used.


NamespaceChecker (unused)
RepoLinker (unused)
WikibaseClient (mostly unused, except for Settings)
WikibaseClient//SiteLookup (unused)
WikibaseClient//DataTypeDefinitions (unused)
WikibaseClient//Settings (used)



The most expensive here is SiteLookup and NamespaceChecker, which are both created here for the unrelated hook onWikibaseHandleChange - which is not called on most requests and not worth blocking WebStart on all web requests.

From a brief look it seems that the main problem is that all hook handlers share the same singleton for injection (instead of obtaining their own services as needed), which means calling one hook ends up also requiring services for other hooks.

A deeper problem is that these services have expensive constructors, which is unexpected, and likely the reason why this approach was assumed to be unproblematic.



See also: T160678: Reduce ChronologyProtector init cost for load.phpTASK DETAILhttps://phabricator.wikimedia.org/T177311EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: KrinkleCc: aude, thiemowmde, Krinkle, Aklapper, GoranSMilovanovic, QZanden, Vali.matei, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T168041: Assign different favicons to query.wikidata.org and test.wikidata.org

2017-10-03 Thread Smalyshev

Smalyshev closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T168041EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Hanna_Petruschat_WMDE, gerritbot, Esc3300, PokestarFan, Smalyshev, James_Budday, Jonas, Jan_Dittrich, Charlie_WMDE, Lydia_Pintscher, Aklapper, MisterSynergy, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Gehel, JGirault, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

2017-10-03 Thread Gehel

Gehel added a comment.
Comments inline. Keep in mind that my understanding of GC is limited, I am most probably wrong in a lot of what I write below. And my understanding of G1 is even more limited...

In T175919#3647588, @Smalyshev wrote:
So I took a look at our logs from Sep 29 with http://gceasy.io/ and I got the following conclusions (could be totally misguided, please tell me if so):

The log file covers about 5 to 6 hours. I think this should be enough but maybe if there are daily patterns it could be useful to have a larger data set to look integratively on. We could do this probably by bumping rotated file size to ~100M from 20M.

gceasy allows you to upload multiple files in a .tar.gz archive, so no need to change the config

GC consumes 7-8% of CPU time. Have no idea if it's normal for Java service or not.

No, it is not. Anything above 5% is suspicious. For comparison, elastic1023 is at a little bit over 1%.

There's four long pause GC events I found in the logs for the last day. One took 40s. From what I see, three of them were in quick succession (around 3:08 am) - which indicates the same failure scenario I alluded to before - instead of failing the allocation, JVM starts obsessively GCing, trying to eke out that last byte. This is the failure mode I am concerned the most about. Not sure what's the alternative here - as I understand, Java has no safe out of memory failure mode? It's either slow death by GC or fast death by OOM?

Actually, there are many failure modes (what fun would it be without them).

When no heap is available for allocation (either not enough space, or too much fragmentation), GC is run. If after a certain amount of effort not enough memory can be freed to do the allocation, then an OOME is raised. That's the easy scenario.

As long as the GC can free up memory for allocation, no OOME occurs. But if the application continues to ask for allocation, GC will start again. Trouble begins when the GC rate is close to the allocation rate, basically, the GC can just keep up, throughput decrease as the GC pressure raises, keeping things mostly in balance.

Consecutive full GC is still another problem. The main reason for full GC (as far as I understand) is when old gen needs to be collected. Full GC implies a longer stop-the-world phase. The cause is usually too early promotion, or a too large (and thus more expensive to collect) old gen. It does not necessarily means that no heap is available, but more that it isn't available in the right place. To minimize this, we should ensure that only long lived objects make it to the old gen and that no premature promotion occurs. Note that too much allocation can result in the same symptoms, with new allocation taking all the new gen space and pushing too much to old gen.

Cross-referencing these events with WDQS dashboards shows that these events blocked updates, causing several minute lag in updates. Which means the server was essentially in coma for several minutes, but then recovered. Good news is it did recover, bad news is it should never happen in the first place.

This makes me think that we have an increased allocation rate for some time (a few queries allocating much more, or allocating objects more expensive to collect). Once the workload is back to normal, the JVM recovers. That's only an assumption, and I'm not sure how to invalidate it...

Heap usage after GC is reported just under 6G. I think that means making heap size less than 6G is asking for trouble. There are several peaks reaching 12G.

What is important is the minima of heap after GC, which on the logs I sampled seems to be fairly consistently between 2 and 3 GB. The minima are important, because in most cases, GC will only free what it needs, leaving a lot of garbage in place. The minima is when we are the closest of not having any garbage left... By slowly minimizing the overall size of heap, we will see better what the reasonable limit is.

GC times mostly stay under 5sec, but there are many over 1sec, it probably would be a good idea to keep them down.

I'm used to GC taking less than 100ms in most cases. Again, elastic1023 as comparison: 99.849% in less than 1 sec, avg GC time: 50ms, longest GC ~3 seconds.

Old Gen size stays mostly between 5-6G, which seems to indicate that is what long-term Blazegraph data structures need (unless I misunderstand what Old Gen means). There are several spikes, which I suspect are due to long-running queries that take so much time that normally transient structures get promoted to old gen, but I may be wrong here. This conclusion also is supported by the fact that the peaks are generally short-lived (which I interpret as query finally finishing or timing out and its memory allocation being released).

Again, what is interesting are the minima of old gen after GC, which I see between 2 and 3 GB.

Young gen size goes from 6-7G before GC to under 1G after GC. That means there's probably some space for reducing the

[Wikidata-bugs] [Maniphest] [Retitled] T174519: [epic] Determine baseline for metrics on Commons

2017-10-03 Thread debt

debt renamed this task from "Determine baseline for metrics on Commons" to "[epic] Determine baseline for metrics on Commons".
TASK DETAILhttps://phabricator.wikimedia.org/T174519EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: debtCc: Capt_Swing, Ramsey-WMF, SandraF_WMF, Abit, chelsyx, mpopov, debt, Aklapper, E1presidente, Jmmuguerza, GoranSMilovanovic, QZanden, EBjune, Acer, Avner, Gehel, FloNight, Susannaanas, Izno, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Retitled] T174519: [epic] SDoC: Determine baseline for metrics

2017-10-03 Thread debt

debt renamed this task from "[epic] Determine baseline for metrics on Commons" to "[epic] SDoC: Determine baseline for metrics".debt raised the priority of this task from "Normal" to "High".debt edited projects, added Discovery-Analysis (Current work); removed Discovery-Analysis.debt updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTIONAs the #structured-data-commons project ramps up, we'll need to figure out a baseline for metrics on Commons in order to measure future successes. This ticket will be used as a base for the needs/wants for this endeavor. 

> This is scheduled to take place in Q2, as per goals: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q2#Segment_2:_Search_integration_and_exposure

= Let's do this by:...** is eventlogging setup only on Commons or is it on other projects
*** Multi-projects at once?
** can eventlogging from various sources be combined into a meaningful set of metrics/dashboard*** Search satisfaction schema
* How do users generally use Commons...** Is there a way to tell what the current zero results rate is for searches on Commons



> This is scheduled to take place in Q2,* How many search "hits" are based on a match in the file name vs. description vs. category
* Better analysis of how many files may be currently "unfindable" because of lack of categorization, unhelpful file name, no description (or poor description)
* Analysis of how many contributions are made by individuals vs. as per goals: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q2#Segment_2:_Search_integration_and_exposuremass-tools/institutions

---

= Things to think about / keep in mind:
* Zero results rate (ZRR)
* More relevant results
** What is ‘relevant’
* Clickthroughs from cross-project searches
* API usage
* User engagement
** ‘People were able to learn more’
* User satisfaction
* Effort users spend finding something
* Time users spend finding something
* Tracking downloads of media
* Steps to unique queries
* How many embeds on other projects
* How many times a specific file/image has shown up in searches
* How many files/images never show up in searches
* How many searches are by exclusions 
** ‘Pictures of cats but not calicos’
TASK DETAILhttps://phabricator.wikimedia.org/T174519EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: debtCc: Capt_Swing, Ramsey-WMF, SandraF_WMF, Abit, chelsyx, mpopov, debt, Aklapper, E1presidente, Jmmuguerza, GoranSMilovanovic, QZanden, EBjune, Acer, Avner, Gehel, FloNight, Susannaanas, Izno, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176515: More flexible user experience for monitoring Wikidata changes from client, using new filters

2017-10-03 Thread Mattflaschen-WMF

Mattflaschen-WMF added a comment.

In T176515#3644935, @eranroz wrote:
I think one edit in Wikidata can change both description, label and sitelink (at least the API expose edit entity), so assuming it is one to many connection, I would stick to  tag_summary.


To solve that, we could evaluate splitting one repository edit (which affected e.g. all of those) into multiple separate client edits (each with only one rc_source, e.g. one for wb.description, one for wb.sitelinks, etc.).

However we implement this, we should evaluate the performance implications so it doesn't worsen T171027 .TASK DETAILhttps://phabricator.wikimedia.org/T176515EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Mattflaschen-WMFCc: Noella94, hoo, Trizek-WMF, Lea_Lacroix_WMDE, Elitre, RP88, Catrope, SBisson, Mattflaschen-WMF, Aklapper, eranroz, GoranSMilovanovic, QZanden, Iniquity, jmatazzoni, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

59 matches

Mail list logo