On Mon, Jul 15, 2013 at 7:57 PM, Ilya Grigorik igrigo...@google.com wrote:
+asher (woops, forgot to cc :))
On Mon, Jul 15, 2013 at 7:54 PM, Ilya Grigorik igrigo...@google.comwrote:
Anyway, I've already started working on something I noticed in
mod_pagespeed - a much better JS minification,
[cc'ing Joshua Marantz who leads the mod_pagespeed effort at Google, and Ilya
Grigorik, their developer advocate for page performance]
The principals behind mod_pagespeed, especially as they related to mobile
page load performance as outlined in http://bit.ly/mobilecrp could
themselves be
Welcome, Sean! It's great to have you on board.
On Monday, June 24, 2013, Ct Woo wrote:
Hi All,
The Technical Operations team is pleased to announce Sean Pringle joined us
today ( 24th June, 2013). Among his duties, Sean will be attending to all
aspects of the database layer including
https://twitter.com/ioerror/status/342922052841377793
Why not - would the patrol cost really be too high?
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Ah, thanks Sumana!
On Friday, June 7, 2013, Sumana Harihareswara wrote:
On 06/07/2013 08:43 AM, Asher Feldman wrote:
https://twitter.com/ioerror/status/342922052841377793
Why not - would the patrol cost really be too high?
Discussion from December:
http://www.gossamer-threads.com/lists
Faidon - thanks for the more accurate trackdown, and fix!
On Sunday, May 5, 2013, Faidon Liambotis wrote:
On Fri, May 03, 2013 at 03:19:13PM -0700, Asher Feldman wrote:
1) Our multicast purge stream is very busy and isn't split up by cache
type, so it includes lots of purge requests
The problem is due to recent changes that were made to how mobile caching
works. I just flushed cache on all of the frontend varnish instances which
indeed appears to have fixed the problem but it isn't actually fixed.
Note, the frontend instances just have 1GB of cache, so only very popular
This sounds like a great plan. Thank you!
On Fri, Mar 29, 2013 at 2:45 AM, Max Semenik maxsem.w...@gmail.com wrote:
Hi, we at the mobile team are currently working on improving our
current hit rate, publishing the half-implemented plan here for review:
== Current status ==
* X-Device
Welcome Brandon!!
On Fri, Mar 29, 2013 at 3:00 PM, Ct Woo ct...@wikimedia.org wrote:
All,
We are excited to announce that Brandon Black will join us this Monday
(2013-04-01) as a full-time member of the Operations Engineering team.
Brandon comes with deep and wide technical experience.
Why don't we continue to use the bits cache for all things resourceloader.
Can you provide a different path for these requests, such as instead of:
http://bits.wikimedia.org/en.wikipedia.org/load.php?..
use something like:
http://bits.wikimedia.org/m/en.wikipedia.org/load.php?..
Then we can if
. I do
think another benchmark Ops features would be a set of
latency-to-datacenter values, but I know that is a much harder taks.
Thanks
for putting this together.
On Thu, Mar 21, 2013 at 6:40 PM, Asher Feldman afeld...@wikimedia.org
wrote:
I'd like to push for a codified set
and a
consistent location would help to define the pass/fail of each test. I
do
think another benchmark Ops features would be a set of
latency-to-datacenter values, but I know that is a much harder taks.
Thanks
for putting this together.
On Thu, Mar 21, 2013 at 6:40 PM, Asher Feldman afeld
I'd like to push for a codified set of minimum performance standards that
new mediawiki features must meet before they can be deployed to larger
wikimedia sites such as English Wikipedia, or be considered complete.
These would look like (numbers pulled out of a hat, not actual
suggestions):
-
On Thu, Mar 7, 2013 at 3:57 PM, Tim Starling tstarl...@wikimedia.orgwrote:
On 07/03/13 12:12, Asher Feldman wrote:
Ori - I think this has been discussed but automated xhprof configuration
as
part of the vagrant dev env setup would be amazing :)
I don't think xhprof is the best technology
that
environment to profile or tune queries as they would actually run in
production.
When you need to ask for a performance review, you can check out
https://www.mediawiki.org/wiki/Developers/Maintainers#Other_Areas_of_Focus
which suggests Tim Starling, Asher Feldman, and Ori Livneh. I also
I don't think a custom daemon would actually be needed.
http://redis.io/topics/pubsub
While I was at flickr, we implemented a pubsub based system to push
notifications of all photo uploads and metadata changes to google using
redis as the backend. The rate of uploads and edits at flickr in 2010
. Doing something a little more modern
might attract different uses. It might not, but I have no idea.
On Fri, Mar 1, 2013 at 5:46 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:;
wrote:
I don't think a custom daemon would actually be needed.
http://redis.io/topics/pubsub
While I
On Friday, March 1, 2013, Tyler Romeo wrote:
On Fri, Mar 1, 2013 at 11:46 AM, Asher Feldman
afeld...@wikimedia.orgjavascript:;
wrote:
don't think a custom daemon would actually be needed.
http://redis.io/topics/pubsub
While I was at flickr, we implemented a pubsub based system
a new generic
mediawiki http response header dedicated to logging containing key value
pairs.
-Asher
On Tue, Feb 12, 2013 at 9:56 AM, Asher Feldman afeld...@wikimedia.orgwrote:
On Tuesday, February 12, 2013, Diederik van Liere wrote:
It does still seem to me that the data to determine secondary
, Petr Bena wrote:
thanks for updates.
Can you tell me what is a difference between maria db you are using and
the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:_e({}, 'cvml', 'afeld...@wikimedia.org');
wrote
, or it
should
On Thu, Feb 14, 2013 at 4:47 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:_e({}, 'cvml', 'afeld...@wikimedia.org');
wrote:
For most projects, I recommend using the official packages available via
the MariaDB projects own apt repo.
The official packages are based on the Debian
I would much rather abandon using debs than use what the debian project has
done to mysql packaging in any production environment. If the discussion
has come down to this, I did WMF a disservice by drifting away from Domas'
optimized make ; make install ; rsync unstripped binaries to prod
The production migration to MariaDB was paused for a time by the EQIAD
datacenter migration and issues involving other projects that took up my
time, but the trial production roll-out will resume this month. All signs
still point to our using it in production.
I did a lot of query testing on an
It looks like Daniel's change to log implicit commits went live on the wmf
cluster with the release of 1.21wmf9.
Unfortunately, it doesn't appear to be as useful as hoped for tracking down
nested callers of Database::begin, the majority of log entries just look
like:
Wed Feb 13 22:07:21 UTC 2013
On Tuesday, February 12, 2013, Diederik van Liere wrote:
It does still seem to me that the data to determine secondary api
requests
should already be present in the existing log line. If the value of the
page param in an action=mobileview api request matches the page in the
referrer
Max - good answers re: caching concerns. That leaves studying if the bytes
transferred on average mobile article view increases or decreases with lazy
section loading. If it increases, I'd say this isn't a positive direction
to go in and stop there. If it decreases, then we should look at the
caching.
Let us know which method is preferred. From my perspective
implementation of either is easy.
[1] http://www.mediawiki.org/wiki/MobileFrontend/Dynamic_Sections
On Mon, Feb 11, 2013 at 12:50 PM, Asher Feldman afeld...@wikimedia.org
wrote:
Max - good answers re: caching concerns
On Thu, Feb 7, 2013 at 4:32 AM, Mark Bergsma m...@wikimedia.org wrote:
- Since we're repurposing X-CS, should we perhaps rename it to something
more apt to address concerns about cryptic non-standard headers flying
about?
I'd like to propose to define *one* request header to be used for
On Wednesday, February 6, 2013, David Schoonover wrote:
Just want to summarize and make sure I've got the right conclusions, as
this thread has wandered a bit.
*1. X-MF-Mode: Alpha/Beta Site Usage*
*
*
We'll roll this into the X-CS header, which will now be KV-pairs (using
normal URL
On Wednesday, February 6, 2013, David Schoonover wrote:
That all sounds fine to me so long as we're all agreed.
Lol. RFC closed.
--
David Schoonover
d...@wikimedia.org javascript:;
On Wed, Feb 6, 2013 at 12:59 PM, Asher Feldman
afeld...@wikimedia.orgjavascript:;
wrote
On Mon, Feb 4, 2013 at 4:24 PM, Arthur Richards aricha...@wikimedia.orgwrote:
Asher, I understand your hesitation about using HTTP header fields, but
there are a couple problems I'm seeing with using query string parameters.
Perhaps you or others have some ideas how to get around these:
* We
On Mon, Feb 4, 2013 at 4:59 PM, Arthur Richards aricha...@wikimedia.orgwrote:
In the case of the cookie, the header would actually get set by the backend
response (from Apache) and I believe Dave cooked up or was planning on
cooking some magic to somehow make that information discernable when
On Mon, Feb 4, 2013 at 5:21 PM, Asher Feldman afeld...@wikimedia.orgwrote:
On Mon, Feb 4, 2013 at 4:59 PM, Arthur Richards
aricha...@wikimedia.orgwrote:
In the case of the cookie, the header would actually get set by the
backend
response (from Apache) and I believe Dave cooked up
If you want to differentiate categories of API requests in logs, add
descriptive noop query params to the requests. I.e mfmode=2. Doing this in
request headers and altering edge config is unnecessary and a bad design
pattern. On the analytics side, if parsing query params seems challenging
vs.
squids or image caches.
On Sunday, February 3, 2013, Asher Feldman wrote:
If you want to differentiate categories of API requests in logs, add
descriptive noop query params to the requests. I.e mfmode=2. Doing this in
request headers and altering edge config is unnecessary and a bad design
:
MobileFrontend: 1/2 a/b/s
work just as fine?
*--*
*Tyler Romeo*
Stevens Institute of Technology, Class of 2015
Major in Computer Science
www.whizkidztech.com | tylerro...@gmail.com javascript:;
On Sun, Feb 3, 2013 at 4:35 AM, Asher Feldman
afeld...@wikimedia.orgjavascript:;
wrote
javascript:;
On Sun, Feb 3, 2013 at 5:12 AM, Asher Feldman
afeld...@wikimedia.orgjavascript:;
wrote:
That's not at all true in the real world. Look at the actual requests for
google analytics on a high percentage of sites, etc.
Setting new request headers for mobile that map to new
On Wed, Dec 12, 2012 at 6:45 AM, Antoine Musso hashar+...@free.fr wrote:
Le 12/12/12 01:10, Asher Feldman a écrit :
This afternoon, I migrated one of the main production English Wikipedia
slaves, db59, to MariaDB 5.5.28.
Congratulations :-)
Out of curiosity, have you looked at Drizzle
On Wednesday, December 12, 2012, David Gerard wrote:
On 12 December 2012 15:32, Thomas Fellows
thomas.fell...@gmail.comjavascript:;
wrote:
This is awesome! Is there any write-up of the migration process floating
around?
+1
In fact, this would be a nice thing to put on the WMF blog.
Hi,
This afternoon, I migrated one of the main production English Wikipedia
slaves, db59, to MariaDB 5.5.28. We've previously been testing 5.5.27 on
the primary research slave, and I've been testing the current build for the
last few days on a slave in eqiad. All has looked good, and I spent
On Tue, Dec 11, 2012 at 5:49 PM, Terry Chay tc...@wikimedia.org wrote:
Nice!
The main goal of migrating to MariaDB is not performance driven. More
so, I think it's in WMF's and the open source communities interest to
coalesce around the MariaDB Foundation as the best route to ensuring a
If anyone is interested, or knows of a worthy entrant, the application
deadline for the 2013 Antonia Pizzigati prize, honoring open source
software development in the public interest, has been extended to Friday,
14 December 2012.
http://www.tides.org/impact/awards-prizes/pizzigati-prize/
The
Hi all,
I'm excited to see that Max has made a lot of great progress in adding Solr
support to the GeoData extension so that we don't have to use mysql for
spatial search - https://gerrit.wikimedia.org/r/#/c/27610/
GeoData makes use of the Solarium php client, which is currently included
as a
I think the latter case is a likely candidate, as we see a couple hundred
apache worker segfaults daily, in both php5 and libxml2 space. The first
case is likely a bug in php core and it's worth checking whether we'd see
the same behavior running a current php release. Core analysis may help us
On Wednesday, September 26, 2012, Daniel Kinzler wrote:
I see your point. But if we have the choice between lock contention and
silent
data loss, which is better?
This isn't really a choice - by default, when a statement in mysql hits a
lock timeout, it is rolled back but the transaction
On Wed, Sep 26, 2012 at 4:07 AM, Daniel Kinzler dan...@brightbyte.dewrote:
On 26.09.2012 12:06, Asher Feldman wrote:
On Wednesday, September 26, 2012, Daniel Kinzler wrote:
I see your point. But if we have the choice between lock contention and
silent
data loss, which is better
As we've increased our use of sha1 hashes to identify unique content over
the past year, I occasionally see changesets or discussions about indexing
sha1's in mysql. When indexing a text field, it's generally beneficial to
define the smallest index that still uniquely matches a high percentage of
Base36 certainly isn't the most efficient way to store a sha1, but it's
what is in use all over mediawiki. I think there was some discussion on
this list of the tradeoffs of different methods when revision.rev_sha1 was
added, and base36 was picked as a compromise. I don't know why base36 was
Here's the slide deck for the mediawiki profiling presentation I gave at
WMF Tech Days 2012 yesterday:
https://commons.wikimedia.org/wiki/File:MediaWikiPerformanceProfiling.pdf
-Asher
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
On Tue, Sep 4, 2012 at 3:11 PM, Platonides platoni...@gmail.com wrote:
On 03/09/12 02:59, Tim Starling wrote:
I'll go for option 4. You can't delete the images from the backend
while they are still in Squid, because then they would not be purged
when the image is updated or action=purge is
While I don't agree with the negative sentiment around experimentation, I
think there's value both in MZMcBride's op-ed, and in the comment thread
that follows. He correctly calls out some of our long term organizational
failings around product planning, resource allocation, execution, and
Mission accomplished!
On Tue, Jul 17, 2012 at 5:26 PM, Asher Feldman afeld...@wikimedia.orgwrote:
Hi All,
Ryan Lane and I are migrating gerrit's db to a server in eqiad (where the
gerrit app server is located) on Friday, and have a downtime window of
18:00-19:00 UTC (11am-12pm PDT). Actual
Hi All,
Ryan Lane and I are migrating gerrit's db to a server in eqiad (where the
gerrit app server is located) on Friday, and have a downtime window of
18:00-19:00 UTC (11am-12pm PDT). Actual downtime should be shorter.
Gerrit makes many mysql queries for some page requests; this will improve
Hi all,
I'd like to remind everyone involved in development that requires db schema
migrations - please keep in mind the three related guidelines in our
official deployment policies -
http://www.mediawiki.org/wiki/Development_policy#Database_patches -
especially the third, which is to make schema
commons was read-only for about 60 seconds while I was switching the master
this afternoon due to issues necessitating a kernel upgrade.
On Mon, May 14, 2012 at 6:00 PM, Bináris wikipo...@gmail.com wrote:
I deleted a test article from huwiki, and this was the result instead of
the usual
I am generally in favor of all of this and in the meeting that proceeded
Rob's email, proposed that we develop a new schema migration tool for
mediawiki along similar lines. Such a beast would have to work in all
deployment cases without modifications (stock single wiki installs and at
wmf with
Thanks, hashar!
On Wed, Apr 25, 2012 at 12:12 AM, Antoine Musso hashar+...@free.fr wrote:
Le 25/04/12 02:52, Rob Lanphier a écrit :
3. For anything that involves a schema change to the production dbs,
make sure Asher Feldman (afeld...@wikimedia.org) is on the reviewer
list. He's already
On Fri, Apr 20, 2012 at 8:01 AM, Mark A. Hershberger m...@wikimedia.orgwrote:
Sumana Harihareswara suma...@wikimedia.org writes:
Please leave your comments at bug 1450 so we can decide how to write
the rewrite rule.
Since Gerrit makes review possible and the relevant Apache config
MW needs full etag support, with hooks for extensions. Without it, we can't
widely support caching in the case you've outlined.
Different browsers handle the Vary header differently. Some treat Varies
as don't cache. Chrome (possibly other webkit browsers) treats it as a
marker to revalidate
On Thu, Apr 5, 2012 at 5:25 PM, Ryan Lane rlan...@gmail.com wrote:
How many languages can we reasonably support? We're currently using
PHP, Python, Java, OCaml and Javascript (and probably more). Should we
also throw Ruby in here as well? What level of support are the
Selenium tests really
On Tuesday, March 20, 2012, Roan Kattouw roan.katt...@gmail.com wrote:
So yeah /normally/
you hit DB servers at random and different servers might respond
differently (or be lagged to different degrees), but in this
particular case it was always the same DB server returning the same
lag
On Tuesday, March 20, 2012, Roan Kattouw roan.katt...@gmail.com wrote:
On Tue, Mar 20, 2012 at 11:35 AM, Asher Feldman afeld...@wikimedia.org
wrote:
How do you feel about a switch to change that behavior (maxlag - 1)? It
would be nice to be continue guiding developers towards throttling API
Just a heads up that the last of the 1.19 migrations, to add the sha1
column to enwiki.revision is going to be running throughout this week.
Don't be alarmed by replication lag messages for s1 dbs in irc. I'm going
to juggle which db watchlist queries go to during the migration, so nothing
should
I've temporarily commented out db36 from db.php on the cluster.
This is a flaw in the how the client-side use of maxlag interacts with our
schema migration process - we run migrations on slaves one by one in an
automated fashion, only moving to the next after replication lag catches
up.
On Wed, Mar 14, 2012 at 5:08 PM, Arthur Richards aricha...@wikimedia.orgwrote:
To follow up on this, I actually made some additional changes to how
useformat works to simplify manually switching between mobile and desktop
views which had been suggested by Brion Vibber. Take a look at:
On Thu, Mar 15, 2012 at 12:10 PM, Brion Vibber br...@pobox.com wrote:
Let's please kill the m. domain.
IMO desktop and mobile users should use the same URLs; there should be sane
device detection; and an easy override in both directions available at all
times.
This is blocked on migrating
The larger time range includes the removal of the mysql based parsercache
which is the cause of the primary decline and not related to 1.19. The time
range of the originally mentioned graph just shows a bit of context before
the enwiki deployment up until the time I posted it to irc last night.
A
+1 to adding to a modified version of change_tag, or something like it.
While unfamiliar with the current tagging interface(s), the content of
ct_tag seems arbitrary (possible movie studio tagger appears 4 times in
enwiki.change_tag.ct_tag out of 2mil rows) and it probably makes sense to
keep
This is great!! Welcome Andrew!
On Fri, Jan 6, 2012 at 10:08 AM, Rob Lanphier ro...@wikimedia.org wrote:
Hi everyone,
I'm pleased to announce Andrew Otto will be coming to Wikimedia
Foundation as a software developer in Platform Engineering, focused on
analytics. We've been hiring for this
will be
interrupted for a similar length of time.
-Asher
On Mon, Dec 19, 2011 at 5:17 PM, Asher Feldman afeld...@wikimedia.orgwrote:
Hi,
We need around 20 minutes of downtime to all services that write to db9
for replication maintenance. Outside of services that support the ops
team, this primarily means
Hi,
We need around 20 minutes of downtime to all services that write to db9 for
replication maintenance. Outside of services that support the ops team,
this primarily means bugzilla, etherpad, and civicrm. It will remain
available for read queries, however, so read usage of services such as the
On Mon, Nov 28, 2011 at 12:06 PM, Roan Kattouw roan.katt...@gmail.comwrote:
On Mon, Nov 28, 2011 at 8:59 PM, Neil Harris n...@tonal.clara.co.uk
wrote:
I hadn't thought properly about cache stampedes: since the parser cache
is
only part of page rendering, this might also explain some of the
It appears that we were actually taken down by the reddit community, after
a link to the fundraising stats page was posted under Brandon's IAMA there.
sq71.wikimedia.org 943326197 2011-11-27T22:51:09.075 62032 109.125.42.71
TCP_MISS/200 1035 GET
Yay! Welcome!
On Mon, Oct 10, 2011 at 9:52 AM, CT Woo ct...@wikimedia.org wrote:
All,
Technical Operations department is pleased to announce another new fabulous
staff member to its team. Please join us to welcome Leslie Carr , our
Network Operations Engineer, starting today, 10/10/11.
On Thursday, October 6, 2011, IAlex ialex.w...@gmail.com wrote:
Le 7 oct. 2011 à 06:21, Chad a écrit :
Well we do serve the logged out cookie. What real purpose
that serves, I don't know :)
It's to bypass the browser cache, and to not let the user see a page with
it's user name at the top
From the wmf office in San Francisco,
webcache.googleusercontent.comresolves to something geographically
close and network RT time is around
25ms versus 86ms for en.wikipedia.org, 61ms in googles favor.
From chrome in incognito mode to avoid sending wikipedia cookies, it takes
me 391ms to fetch
Since the primary use case here seems to be offline analysis and it may not
be of much interest to mediawiki users outside of wmf, can we store the
checksums in new tables (i.e. revision_sha1) instead of running large
alters, and implement the code to generate checksums on new edits via an
Would it be possible to generate offline hashes for the bulk of our revision
corpus via dumps and load that into prod to minimize the time and impact of
the backfill?
When using for analysis, will we wish the new columns had partial indexes
(first 6 characters?)
Is code written to populate
From the orig post Recent Intel CPU has a fature called
AES-NIhttp://en.wikipedia.org/wiki/AES_instruction_set that
accelerates AES processing. A CPU with AES-NI can perform 5 to 10 times
faster than a CPU without it. We observe that a single core can perform 5
Gbps and 15 Gbps for encryption and
On Thu, Aug 4, 2011 at 10:31 AM, Aryeh Gregor a...@aryeh.name wrote:
I was under the impression that the biggest cost in TLS isn't the
symmetric encryption for an ongoing connection, it's the asymmetric
encryption for the connection setup. If so, AES acceleration isn't
going to help with the
Woo! Looking forward to working with you, Jeff!
On Thu, Jun 30, 2011 at 2:31 PM, CT Woo ct...@wikimedia.org wrote:
All,
Please join me to welcome Jeff Green to Wikimedia Foundation.
Jeff is taking up the Special Ops position in the Tech Ops department where
one of his responsibilities is
81 matches
Mail list logo
