[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #19 from Aaron Schulz aschulz4...@gmail.com 2012-05-22 15:56:42 UTC --- (In reply to comment #18) Asher and I our still tracking down additional problems. I've made https://gerrit.wikimedia.org/r/8006 to add more profiling. The problem is tracked down to LocalFile::upload. For upload profiling: The edit portions tends to be around =1 sec at all times. p90 for purging from swift goes up to 11 sec for much of the day. As a workaround, perhaps the thumbnails purging could be skipped for brand-new uploads, since there should be nothing to purge. The problem (same as bug 34717), is that the Swift container DBs have extremely poor performance and while likely need to be mapped to SSD devices instead of HDD. SELECT queries go fast when the sqlite3 db file is paged in memory, but slow otherwise (much of the time). The slowness just comes from the listings, the actual concurrent DELETEs are very fast (1 second even for p90). -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #20 from Aaron Schulz aschulz4...@gmail.com 2012-05-22 15:57:28 UTC --- (In reply to comment #19) The problem (same as bug 34717), is that the Swift container DBs have extremely poor performance and while likely need to be mapped to SSD devices instead of HDD. *will likely* -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #21 from Max Semenik maxsem.w...@gmail.com 2012-05-22 16:18:51 UTC --- They're using SQLite with no other options? That's stupid cause even if you put the DB on the fastest drive on the Earth, it will still suck because SQLite had never been intended for any serious level of concurrency, and it's not a bug but a design decision. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #22 from Aaron Schulz aschulz4...@gmail.com 2012-05-22 20:03:25 UTC --- Per the LocalFile graphs at https://graphite.wikimedia.org, this was fixed for brand new uploads. Re-uploads will still have potential slowness. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #23 from Aaron Schulz aschulz4...@gmail.com 2012-05-22 20:12:09 UTC --- (In reply to comment #21) They're using SQLite with no other options? That's stupid cause even if you put the DB on the fastest drive on the Earth, it will still suck because SQLite had never been intended for any serious level of concurrency, and it's not a bug but a design decision. The concurrency limitations are probably dealt with by the fact that we shard the containers for the large wikis. This also mitigates the B-tree index update slowness (proportional to the no. of items in the container object list table). Since the container service and object service are running on the same boxes (and the later gets used way more and on way more data), no DB files can stay in the page cache. The random seeks can't be well handled by SATA drives. Asher also floating trying some hacks with http://www.kernel.org/doc/man-pages/online/pages/man2/posix_fadvise.2.html for the sake of investigation. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 Erik Moeller e...@wikimedia.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Comment #24 from Erik Moeller e...@wikimedia.org 2012-05-22 21:08:36 UTC --- Confirmed that new file uploads are now zippy, yay. Closing this bug as fixed; additional improvements should be discussed in separate bugs. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #17 from Erik Moeller e...@wikimedia.org 2012-05-19 00:30:08 UTC --- $wgSiteStatsAsyncFactor is now set to 1 for Wikimedia Commons. I'm no longer able to get 15-30 second requests, although I did catch a 12 second call for a 400K filekey upload that was part of a large batch (most of which was processed very quickly). So it's possible this issue is not (fully) resolved yet. Let's keep the bug open for now -- if you see the issue in the wild (manifests as last step of Upload Wizard uploads taking a very long time), please report it here. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #18 from Aaron Schulz aschulz4...@gmail.com 2012-05-19 01:03:29 UTC --- Asher and I our still tracking down additional problems. I've made https://gerrit.wikimedia.org/r/8006 to add more profiling. The problem is tracked down to LocalFile::upload. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 Erik Moeller e...@wikimedia.org changed: What|Removed |Added CC||bryan.tongm...@gmail.com Component|UploadWizard|Uploading AssignedTo|roan.katt...@gmail.com |wikibugs-l@lists.wikimedia. ||org Product|MediaWiki extensions|MediaWiki Summary|Internal api error reported |Some API action=upload |( |filekey requests take 30 |UploadStashFileNotFoundExce |seconds to complete |ption ) | --- Comment #14 from Erik Moeller e...@wikimedia.org 2012-05-16 23:11:23 UTC --- Updating bug description. This is not inherently an UploadWizard issue, although UploadWizard may exacerbate contention issues because it does batch uploading. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 Rob Lanphier ro...@wikimedia.org changed: What|Removed |Added AssignedTo|wikibugs-l@lists.wikimedia. |aschulz4...@gmail.com |org | --- Comment #15 from Rob Lanphier ro...@wikimedia.org 2012-05-17 01:50:27 UTC --- Latest status on this: we're currently testing a change that should fix this problem. Aaron has enabled this on mediawiki.org, and we'll likely make this change on commons tomorrow. If our theory is correct, the problem is caused when page editing or image uploading requires an update to the site statistics. The code that fixes this is deployed, but needs to be configured. Tech minutia follows: The fix we're relying on is here: https://gerrit.wikimedia.org/r/#/c/7724 The configuration change needs to be made, changing $wgSiteStatsAsyncFactor. You can see which wikis this has been deployed to by looking for wgSiteStatsAsyncFactor here: http://wikitech.wikimedia.org/view/Server_admin_log -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 34785] Some API action=upload filekey requests take 30 seconds to complete
https://bugzilla.wikimedia.org/show_bug.cgi?id=34785 --- Comment #16 from Rob Lanphier ro...@wikimedia.org 2012-05-17 01:55:09 UTC --- Gah, copied the wrong link. Two relevant revs: https://gerrit.wikimedia.org/r/#/c/7136/ https://gerrit.wikimedia.org/r/#/c/7244/ -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
