[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #19 from Max Semenik maxsem.w...@gmail.com --- No, read above. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #20 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 123811 had a related patch set uploaded by MaxSem: Don't request page images separately for search results https://gerrit.wikimedia.org/r/123811 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #21 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 123811 merged by jenkins-bot: Don't request page images separately for search results https://gerrit.wikimedia.org/r/123811 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 Jon jrob...@wikimedia.org changed: What|Removed |Added Status|PATCH_TO_REVIEW |RESOLVED Resolution|--- |FIXED --- Comment #22 from Jon jrob...@wikimedia.org --- Looks like this should be taken care of now... -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #18 from Jon jrob...@wikimedia.org --- Is this now resolved? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #17 from Max Semenik maxsem.w...@gmail.com --- (In reply to Max Semenik from comment #7) (In reply to Jon from comment #6) Ideally the search results API would return these results. This would remove the need for the or query and the additional get itself. Could the results of the search api query be piped into page images? Currently it's not possible as prefix search is available only with the non-query opensearch module. It should be trivial to add a similar generator module, however - I wonder why it hasn't been done yet:) https://gerrit.wikimedia.org/r/123118 adds an API module needed for this, after it's merged this will need some FE work to use this module. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #8 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 121930 merged by jenkins-bot: Don't request pageimages for 0 pages https://gerrit.wikimedia.org/r/121930 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #9 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122582 had a related patch set uploaded by MaxSem: Don't request pageimages for 0 pages https://gerrit.wikimedia.org/r/122582 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #10 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122583 had a related patch set uploaded by MaxSem: Don't request pageimages for 0 pages https://gerrit.wikimedia.org/r/122583 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #11 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122582 merged by jenkins-bot: Don't request pageimages for 0 pages https://gerrit.wikimedia.org/r/122582 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #12 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122583 merged by jenkins-bot: Don't request pageimages for 0 pages https://gerrit.wikimedia.org/r/122583 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #13 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122594 had a related patch set uploaded by MaxSem: Update MobileFrontend, PageImages and TextExtracts for bug 63248 https://gerrit.wikimedia.org/r/122594 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #14 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122596 had a related patch set uploaded by MaxSem: Update MobileFrontend, PageImages and TextExtracts for bug 63248 https://gerrit.wikimedia.org/r/122596 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #16 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122594 merged by jenkins-bot: Update MobileFrontend, PageImages and TextExtracts for bug 63248 https://gerrit.wikimedia.org/r/122594 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #15 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 122596 merged by jenkins-bot: Update MobileFrontend, PageImages and TextExtracts for bug 63248 https://gerrit.wikimedia.org/r/122596 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 Gerrit Notification Bot gerritad...@wikimedia.org changed: What|Removed |Added Status|NEW |PATCH_TO_REVIEW -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #2 from Gerrit Notification Bot gerritad...@wikimedia.org --- Change 121930 had a related patch set uploaded by MaxSem: Don't request pageimages for 0 pages https://gerrit.wikimedia.org/r/121930 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #3 from Max Semenik maxsem.w...@gmail.com --- ^^^ kills the most popular cache miss. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #4 from Max Semenik maxsem.w...@gmail.com --- https://gerrit.wikimedia.org/r/121932 makes PI results cacheable. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #5 from Faidon Liambotis fai...@wikimedia.org --- I don't think the cacheable bit will help much. The pageimages API parameter will be for a more-or-less unique combination of articles OR'ed with each other, as they resulted from the search query. I'm not sure of the distribution of our search queries, but I'm guessing caching those combinations won't make a huge difference (and it will be a waste of cache memory/disk). It won't help client performance either, as clients will still do 17 extra requests on every search for this feature. I think the design of this whole feature honestly sounds a bit naive to me. We should really go a step backwards and rethink the best way to do this with server- and client-side performance in mind. Assuming images on search results is something that is not hurtful from a performance PoV and a requirement for where mobile is going, then we should probably have one single search endpoint in the API which returns the JSON result set enriched with image thumbnails URLs. There is no reason for this to be in an extension or a separate request, AIUI. The dozen upload.wikimedia.org requests to fetch a 1-2KB thumb eachs is still going to kill client-side performance though. There's a tremendous overhead of data for the request and round-trips that will delay the total page load time from a viewer's perspective. It's also high-latency requests, as these 50px images in most cases will need to be scaled on the fly. Unfortunately our architecture makes it difficult to return this directly in the result set as e.g. data URIs but if that'd work maybe we could find a way. Let's think of other ways to improve this feature in general. I'd be happy to discuss performance with you, as well as help in case some deeper infrastructure change (Swift/Varnish) could help make this feature better for everyone. I'm sure Ori could be of help here as well, with his client-side performance expert hat on. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #6 from Jon jrob...@wikimedia.org --- Ideally the search results API would return these results. This would remove the need for the or query and the additional get itself. Could the results of the search api query be piped into page images? As an interim would it make any difference if we just fetched the top 5 results and threw in some client side caching so that additional images are not retrieved? If we need to completely turn this off we will have to quickly check with design what is preferable and whether we can have a placeholder image there so it doesn't look strange. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #7 from Max Semenik maxsem.w...@gmail.com --- (In reply to Faidon Liambotis from comment #5) I don't think the cacheable bit will help much. The pageimages API parameter will be for a more-or-less unique combination of articles OR'ed with each other, as they resulted from the search query. I'm not sure of the distribution of our search queries, but I'm guessing caching those combinations won't make a huge difference The frequency dump says otherwise: * The top line would have been cached (but we're killing it with 121930 anyway). * From line 5 onwards there are a bunch of popular requests with titles from prefix search for r, ri, m, k, john etc - these are not random fluctuations and will always be present with some noticeable frequency so caching them will be highly beneficial. (and it will be a waste of cache memory/disk). API results with cache mode = 'public' have a lifetime of 12 hours, this should be enough to prevent infinite hoarding of cache objects. Anyway, most query modules' results are public so this will not change the situation much. It won't help client performance either, as clients will still do 17 extra requests on every search for this feature. I think the design of this whole feature honestly sounds a bit naive to me. We should really go a step backwards and rethink the best way to do this with server- and client-side performance in mind. The dozen upload.wikimedia.org requests to fetch a 1-2KB thumb eachs is still going to kill client-side performance though. There's a tremendous overhead of data for the request and round-trips that will delay the total page load time from a viewer's perspective. It's also high-latency requests, as these 50px images in most cases will need to be scaled on the fly. Unfortunately our architecture makes it difficult to return this directly in the result set as e.g. data URIs but if that'd work maybe we could find a way. The current implementation tries to work around this by waiting for 500ms after search results are outputted before displaying the images. The problem with it is that doesn't retrieve the page images information in the same request as search, resulting in extra API requests. (In reply to Jon from comment #6) Ideally the search results API would return these results. This would remove the need for the or query and the additional get itself. Could the results of the search api query be piped into page images? Currently it's not possible as prefix search is available only with the non-query opensearch module. It should be trivial to add a similar generator module, however - I wonder why it hasn't been done yet:) As an interim would it make any difference if we just fetched the top 5 results and threw in some client side caching so that additional images are not retrieved? How reliable would it be to detect which list items are actually visible on screen? This would increase cache fragmentation though because currently you're always requesting as many results as you have. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 --- Comment #1 from Bingle bingle-ad...@wikimedia.org --- Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/mobile/cards/1820 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 Faidon Liambotis fai...@wikimedia.org changed: What|Removed |Added Priority|Unprioritized |Highest CC||o...@wikimedia.org -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 63248] search pageimages feature causes significant extra infrastructure load
https://bugzilla.wikimedia.org/show_bug.cgi?id=63248 Greg Grossmeier g...@wikimedia.org changed: What|Removed |Added CC||g...@wikimedia.org Severity|normal |critical -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
