Re: [Wikitech-l] [Analytics] [WikimediaMobile] Mobile stats

, Andrew Otto o...@wikimedia.org wrote: These zero.tsv.log* files to which I refer seem to be, basically Varnish log lines that correspond to Wikipedia Zero-targeted traffic. Yup! Correct. zero.tsv.log* files are captured unsampled and based on the presence of a zero= tag in the X

Re: [Wikitech-l] Need help from Wikimedia github admins to transfer bingle repo

Hm, someone correct me if I’m wrong, but if you want to have Bingle hosted on the wikimedia Github, you should create a repository for it in Gerrit, and then add a special replication rule in puppet in role/gerrit.pp to get the repository in the Github URL that you want. On Feb 21, 2014, at

Re: [Wikitech-l] CirrusSearch outage Feb 28 ~19:30 UTC

* We're going to figure out why we only got half the settings. This is complicated because we can't let puppet restart Elasticsearch because Elasticsearch restarts must be done one node at a time. Ah, I think I see it in elasticsearch/init.pp. If you don’t want to subscribe the service to

Re: [Wikitech-l] [Analytics] RFC: Tab as field delimiter in logging format of cache servers

the format. We will most likely change the file names back to their original names in a month or so. Thanks all! -Andrew Otto On Jan 28, 2013, at 11:33 AM, Matthew Flaschen mflasc...@wikimedia.org wrote: On 01/27/2013 08:07 AM, Erik Zachte wrote: The code to change existing tabs into some less

Re: [Wikitech-l] [Analytics] RFC: Tab as field delimiter in logging format of cache servers

Ah, no I mean change the future ones back to their original names. We'd leave the ones that are being generated as '.tab.' now as they are. We could see these filenames in the archives. But! If everybody loves '.tab.', forever, that's fine with me too! On Jan 31, 2013, at 6:51 PM,

Re: [Wikitech-l] [Analytics] RFC: Tab as field delimiter in logging format of cache servers

wrote: On Thu, Jan 31, 2013 at 7:12 PM, Andrew Otto o...@wikimedia.org wrote: Ah, no I mean change the future ones back to their original names. We'd leave the ones that are being generated as '.tab.' now as they are. We could see these filenames in the archives. But! If everybody loves

Re: [Wikitech-l] Github/Gerrit mirroring

I've been hosting my puppet-cdh4 (Hadoop) repository on Github for a while now. I am planning on moving this into Gerrit. I've been getting pretty high quality pull requests for the last month or so from a couple of different users. (Including CentOS support, supporting MapReduce v1 as well

Re: [Wikitech-l] Github/Gerrit mirroring

14, 2013, at 4:11 PM, Juliusz Gonera jgon...@wikimedia.org wrote: On 03/08/2013 08:55 AM, Andrew Otto wrote: I've been hosting my puppet-cdh4 (Hadoop) repository on Github for a while now. I am planning on moving this into Gerrit. I've been getting pretty high quality pull requests

Re: [Wikitech-l] [Analytics] Page view stats failure

Yesterday morning I received an alert that the gadolinium udp2log process was experiencing packet loss. In addition to being the webstats-collector host (which generates the pagecounts files), gadolinium is a socat relay. It is responsible for feeding about 5 total udp2log instances all of

Re: [Wikitech-l] [WikimediaMobile] [Analytics] Mobile stats

These zero.tsv.log* files to which I refer seem to be, basically Varnish log lines that correspond to Wikipedia Zero-targeted traffic. Yup! Correct. zero.tsv.log* files are captured unsampled and based on the presence of a zero= tag in the X-Analytics header:

Re: [Wikitech-l] Guidelines for db schema changes

I once wrote a pretty decent schema migration tool that fits most if not all of these requirements. It was built for the Kohana PHP framework, but a lot of it is pretty independent of that. If someone ends up working on this I'd love to help and maybe share some code and ideas. -Andrew

Re: [Wikitech-l] Guidelines for db schema changes

Here's the migrations library I wrote. :) https://github.com/ottomata/cs_migrations -Andrew Otto On Apr 26, 2012, at 11:30 AM, Andrew Otto wrote: I once wrote a pretty decent schema migration tool that fits most if not all of these requirements. It was built for the Kohana PHP framework

Re: [Wikitech-l] Extensions queue for Git conversion

migrated to git? -Andrew Otto On May 21, 2012, at 2:01 PM, Chad wrote: On Mon, May 21, 2012 at 10:50 AM, Stephan Gambke s7ep...@gmail.com wrote: Hi, the next batch of extensions to be transferred to Git is overdue for more than a week now. Any idea, when it will actually happen? (http

Re: [Wikitech-l] Extensions queue for Git conversion

Full history, but apparently I didn't do some fancy things that Chad usually does. Chad is going to redo this conversion this week. On May 21, 2012, at 4:10 PM, K. Peachey wrote: On Tue, May 22, 2012 at 4:17 AM, Andrew Otto o...@wikimedia.org wrote: Chad, I'm not sure if this matters

Re: [Wikitech-l] Personal sandbox space in Gerrit

Ah that is useful! Thanks Chad! On Jul 2, 2012, at 1:35 PM, Chad wrote: Hi everyone, I've just come across (and enabled) a feature in Gerrit that I think many will find useful. I'm calling them personal sandboxes. The basic premise is that each user can have a personal branch space that

Re: [Wikitech-l] Appreciation thread

Many many many thanks Rob H, Peter Y, Leslie C, Ben H, Ryan L, Faidon, Daniel Z, Mark B, Chris J, and everyone else on the ops team that has put up with my IRC poking and prodding thus far. You guys are a huge help to the analytics team. Thanks for guiding me through and teaching me the

Re: [Wikitech-l] Appreciation thread

Oh and thanks to Jeremy B too! He's been super helpful at directing my questions to the proper know-it-all. On Aug 24, 2012, at 9:58 AM, Andrew Otto o...@wikimedia.org wrote: Many many many thanks Rob H, Peter Y, Leslie C, Ben H, Ryan L, Faidon, Daniel Z, Mark B, Chris J, and everyone else

Re: [Wikitech-l] Github replication

Awesome! I have a repo I'd love to try this with right now. I'll find you on IRC… On Oct 3, 2012, at 12:27 PM, Chad innocentkil...@gmail.com wrote: Hi everyone, Just letting everyone know: mediawiki/core is now replicating from gerrit to github. https://github.com/mediawiki/core

Re: [Wikitech-l] Mediawiki + Vagrant

If you don't want to use Puppet or Chef, you can just configure an instance by hand (by SSHing into it, usually) and regenerate a Vagrant box from the result. Actually, even if you do want to use Puppet, a VM can be useful. I have a local VM (not Vagrant) set up that I use to test new

[Wikitech-l] Git + Gerrit is a toughy

Hi all! And here's another hi: Hi! This is my first post to this list, so here is a quick intro in case you missed the other ones. I'm Andrew Otto, an engineering on the new Analytics team. I'm working with David Schoonover (new hire as well), Fabian Kaelin, and Diederik van Liere. Right

Re: [Wikitech-l] Git + Gerrit is a toughy

get this ironed out fo sho. -otto On Feb 18, 2012, at 2:31 AM, Roan Kattouw wrote: On Sat, Feb 18, 2012 at 2:47 AM, Andrew Otto o...@wikimedia.org wrote: 2. Do I need to rebase every time I push for review? I don't quite understand what is going on here. I've installed git-review and am

Re: [Wikitech-l] Git + Gerrit is a toughy

On Feb 18, 2012, at 2:31 AM, Roan Kattouw wrote: On Sat, Feb 18, 2012 at 2:47 AM, Andrew Otto o...@wikimedia.org wrote: 2. Do I need to rebase every time I push for review? I don't quite understand what is going on here. I've installed git-review and am using this to push to git. It does

Re: [Wikitech-l] Git + Gerrit is a toughy

, Antoine Musso wrote: Le 21/02/12 18:44, Andrew Otto a écrit : [~/Projects/wm/analytics/reportcard] (master)[29c6b47]$ git-review You have more than one commit that you are about to submit. The outstanding commits are: 29c6b47 (HEAD, master) observation.py - comments 14a771a test commit

[Wikitech-l] Analytics Kafka Upgrade next week

Hiya, We’ll be upgrading the Analytics Kafka cluster from 0.8.2 to 0.9.0.1 next week. This is scheduled to start at Wednesday May 11th at 13:00 UTC (9:00 EST, 6:00 PST). If all goes well*, this should be a rolling upgrade with no downtime. Just a heads up, thanks! -Andrew & Luca *everybody

Re: [Wikitech-l] Unique Devices data available on API

Ha, perhaps Nuria’s quote should read: > it does not include any cookie by which your brows*ING* history can be tracked [3]. s/browser/browsing/ On Tue, Apr 19, 2016 at 4:38 PM, bawolff wrote: > > it does not include any > > cookie by which your browser history can be

Re: [Wikitech-l] [Ops] Migrating Node.JS services' deployment to Scap3

> I think that’s the only python based service deployed via scap3 EventLogging is python and deployed with scap3. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Ops] Migrating Node.JS services' deployment to Scap3

Oo also! Mukunda​ and Tyler and I did some work over the last couple of weeks to make bootstrapping new repos on deployment servers easier and decoupled from trebuchet. You don’t need to do this now if your repo is already cloned on tin, but you might want to anyway. snap deployed repos should

[Wikitech-l] Fwd: EventStreams launch and RCStream deprecation

kpm967t> - Would you like to consume streams other than RecentChanges? <https://www.mediawiki.org/wiki/Topic:Tkjk4ezxb4u01a61> (Currently available events are described the event-schemas repository <https://github.com/wikimedia/mediawiki-event-schemas/tree/master/

Re: [Wikitech-l] Fwd: EventStreams launch and RCStream deprecation

25, 2017 at 2:31 PM, MZMcBride <z...@mzmcbride.com> wrote: > Congratulations on the launch of EventStreams. > > Andrew Otto wrote: > >I did say deprecated! Okay okay, we may never be able to fully deprecate > >irc.wikimedia.org. It’s used by too many (probably senti

[Wikitech-l] Public Event Streams (AKA RCStream replacement) question

directly with HTTP? There seem to be good clients for socket.io and for SSE/EventSource in many languages. https://phabricator.wikimedia.org/T130651 has more context, but don’t worry about reading it; it is getting a little long. Feel free to chime in there or on this thread. Thanks! -Andrew Otto

Re: [Wikitech-l] Public Event Streams (AKA RCStream replacement) question

​So, since most of the dev work for a socket.io implementation is already done, you can see what the protocol would look like here: https://github.com/wikimedia/kasocki#socketio-client-set-up Kasocki is just a library, the actual WMF deployment and documentation would be more specific about

Re: [Wikitech-l] Recent changes, notifications & pageprops

Could we emit a page/properties-change event to EventBus when page props are updated? Similar to how we emit an event for revision visibility changes: https://github.com/wikimedia/mediawiki-event-schemas/blob/master/jsonschema/mediawiki/revision/visibility-change/1.yaml These events would be

Re: [Wikitech-l] Recent changes, notifications & pageprops

You can seek back on EventBus events, but not permanently (by default, only up to 1 week). If you want to respond to changes in an event stream, you should consume the full event stream realtime and react to the events as they come in. A proper Stream Processing system (like Flink or Spark

Re: [Wikitech-l] Public Event Streams (AKA RCStream replacement) question

ntire conversation, so the answers might already be > there - > my appologies if they are, just point me there. > > Anyway, if anyone has a good solution for sending wiki-events to a large > number > of subscribers, yes, please let us (WMDE/Wikidata) know about it! > > Am 26

Re: [Wikitech-l] Public Event Streams (AKA RCStream replacement) question

ayers that > provide other features like socket.io be implemented in other proxy > servers. > > > > On Sun, Sep 25, 2016 at 4:02 PM, Merlijn van Deen (valhallasw) < > valhall...@arctus.nl> wrote: > > > Hi Andrew, > > > > On 23 September 2016 at 23:15, A

Re: [Wikitech-l] Public Event Streams (AKA RCStream replacement) question

s/Fabricator/Phabricator/ (gmail auto correct GRR) On Thu, Oct 20, 2016 at 4:00 PM, Andrew Otto <o...@wikimedia.org> wrote: > Thanks for the feedback everyone! > > Due to the simplicity of the HTTP stream model, we are moving forward with > that, instead of websockets/socket.io

Re: [Wikitech-l] Public Event Streams (AKA RCStream replacement) question

received from the service. > > I hope this helps. Would be happy to talk more about this specific topic > some more. > > Cheers, > Marko > > > > > > I have not read the entire conversation, so the answers might already be > > there - > > my appolog

Re: [Wikitech-l] Can we drop revision hashes (rev_sha1)?

We should hear from Joseph, Dan, Marcel, and Aaron H on this I think, but from the little I know: Most analytical computations (for things like reverts, as you say) don’t have easy access to content, so computing SHAs on the fly is pretty hard. MediaWiki history reconstruction relies on the SHA

Re: [Wikitech-l] Can we drop revision hashes (rev_sha1)?

> can it be a dataset generated from each revision and then published separately? Perhaps it be generated asynchronously via a job? Either stored in revision or a separate table. On Fri, Sep 15, 2017 at 4:06 PM, Andrew Otto <o...@wikimedia.org> wrote: > > As a rand

Re: [Wikitech-l] Can we drop revision hashes (rev_sha1)?

> As a random idea - would it be possible to calculate the hashes when data is transitioned from SQL to Hadoop storage? We take monthly snapshots of the entire history, so every month we’d have to pull the content of every revision ever made :o On Fri, Sep 15, 2017 at 4:01 PM, Stas Malyshev

Re: [Wikitech-l] Question Pertaining to the "stats.grok.se" Page Containing Pre-2015 Page-View Data

That thread links to https://meta.wikimedia.org/wiki/Community_Tech/Pageview_stats_tool, which has some good info about the history and status. On Mon, Sep 25, 2017 at 5:22 PM, Toby Negrin wrote: > Hi Karl, Daniel -- > > Erik doesn't support stats.grok.se. > > There's

Re: [Wikitech-l] Sunsetting Trending Edits Service before the holiday

> This is a little inferior to the production version as it is unable to use production kafka and if it has any outages it will lose data. ​EventStreams isn’t as good as using Kafka, but an outage shouldn’t be a reason to lose data. Store the Last-Event-ID

Re: [Wikitech-l] RevisionInsertComplete vs. RevisionRecordInserted

This is the first I’ve heard of it! So, we don’t have a plan to change it, but I suppose we should if RevisionInsertComplete is deprecated. I haven’t looked at RevisionRecordInserted yet so I can’t answer questions about schema changes, but I doubt it would change anything. Just created

Re: [Wikitech-l] ganglia.wikimedia.org has been retired

NICE! On Fri, Dec 22, 2017 at 6:52 PM, Daniel Zahn wrote: > This is a notice that as part of "T177195 Reduce technical debt in metrics > monitoring " the service > > ganglia.wikimedia.org has been retired and removed from DNS. > >

Re: [Wikitech-l] Announcing MediaWiki code search

Super cool! On Thu, Dec 21, 2017 at 9:31 AM, zppix e wrote: > Thank you very much Kunal! > > -- > Zppix > Volunteer Wikimedia Developer > Volunteer Wikimedia GCI2017 Mentor > enwp.org/User:Zppix > **Note: I do not work for Wikimedia Foundation, or any of its

Re: [Wikitech-l] TechCom Radar 2018-08-08

Is there a reason there is no TechCom IRC meeting next week? I’d love to have one! https://phabricator.wikimedia.org/T201643 :) On Fri, Aug 10, 2018 at 4:01 PM Kate Chapman wrote: > Hi All, > > Here are the minutes from this week's TechCom meeting: > > * IRC meeting scheduled for 22 August at

[Wikitech-l] stats.wikimedia.org maintenance downtime

. Thanks! -Andrew Otto Systems Engineer Wikimedia Foundation ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] stats.wikimedia.org maintenance downtime

This has been done, thanks all! On Tue, Aug 28, 2018 at 12:53 PM Andrew Otto wrote: > Hi all, > > On Wednesday September 5th at around 13:30 UTC we will be taking > stats.wikimedia.org and analytics.wikimedia.org offline for a server > upgrade. We expect this downtime to tak

Re: [Wikitech-l] Change to Wikitech logins: Username now case-sensitive

Great! Is this just for Wikitech itself or all ldap/wikitech authentication? On Mon, Apr 15, 2019 at 7:56 PM Bryan Davis wrote: > A change was deployed to the Wikitech config 2019-04-15T23:16 UTC > which prevents users from logging into the wiki with a username that > differs in case from the

Re: [Wikitech-l] New CI Promote Step for Service Developers

Hey wow this is really great! Excited to use this. THANK YOU! On Thu, Sep 24, 2020 at 1:59 PM Jeena Huneidi wrote: > Hi Everyone, > > I'd like to announce a new CI feature for developers of services deployed > to kubernetes! > > It's now possible for your helmfile values files to be

Re: [Wikitech-l] Replacement for Helm chart repository

GREAT STUFF! Thank you! On Mon, Aug 3, 2020 at 11:02 AM Janis Meybohm wrote: > Hello, > > We are replacing our current repository for Helm charts > (https://releases.wikimedia.org/charts/) with an instance of > ChartMuseum. The new repository index can be found at: >

[Wikitech-l] Re: Stream of recent changes diffs

This isn't helpful now, but your use case is relevant to something I hope to pursue in the future: comprehensive mediawiki change events, including content. I don't have a great place yet for collecting these use cases, so I added it to Modern Event Platform parent ticket

[Wikitech-l] Re: Enabling translating RCFeed log entry messages(IRC log entry messages)

I can't help you with your problem, but: > The main cause of this is probably that the messages are only for irc.wikimedia.org and irc.wikimedia.org will be replaced with EventStreams? I don't think this is the cause. We'd love to deprecate irc.wikimedia.org, but doing so is probably

[Wikitech-l] Re: [Wiki-research-l] Wikimedia Enterprise HTML dumps available for public download

Wow very cool! On Tue, Oct 19, 2021 at 10:57 AM Ariel Glenn WMF wrote: > I am pleased to announce that Wikimedia Enterprise's HTML dumps [1] for > October 17-18th are available for public download; see > https://dumps.wikimedia.org/other/enterprise_html/ for more information. > We > expect to

[Wikitech-l] Re: major upgrade of PageProperties extension (a proof of concept for the use of SLOTS)

> This also reminds me, that like for namespaces and content handlers, we should probably keep a list of known rvslot names, to avoid potential conflicts. Would be nice if extensions could register with mediawiki what rvslot names they provide/manage. :) On Thu, Jan 12, 2023 at 10:18 AM Derk-Jan

[Wikitech-l] Re: Python requests broken by urllib3 version 2.x

> Tangent: is it worthwhile to establish a consensus for best practices with package pinning and package management for Python projects in the Wikimedia ecosystem? Yes! That would be awesome. I have spent a lot of time floundering in this area trying to make decisions; it'd be nice if we had a

[Wikitech-l] Re: Python requests broken by urllib3 version 2.x

> For Java, we run an instance of Archiva: https://archiva.wikimedia.org/ > It's not a perfect approach but I think we can and should move in that direction with all our other ecosystems Gitlab package registries may help us here! On Mon, May 8, 2023 at 8:59 AM Andrew Otto wrote: >

[Wikitech-l] Re: Python requests broken by urllib3 version 2.x

ty of the current) > Python packaging ecosystem, I can recommend this blog post by > PyPA member and PSF fellow Pradyum Gedam: > https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/ > <https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/> > > &g

[Wikitech-l] Re: Enabling canary events for all MediaWiki event streams

Hi everyone, I just enabled canary events for the mentioned streams. Please comment here or on the task if you encounter any issues. Thank you! -Andrew Otto & the WMF Data Engineering team <https://wikitech.wikimedia.org/wiki/Data_Engineering> On Tue, Nov 7, 2023 at 1:57 PM Andrew O

[Wikitech-l] Fwd: Enabling canary events for all MediaWiki event streams

at stream.wikimedia.org <https://stream.wikimedia.org/?doc#/streams>.) Thank you, -Andrew Otto & the WMF Data Engineering team <https://wikitech.wikimedia.org/wiki/Data_Engineering> References - T266798 - Enable canary events for all MediaWiki streams <https://phabricator.wi

[Wikitech-l] Re: Fwd: Enabling canary events for all MediaWiki event streams

diawiki.page-undelete >> >> - mediawiki.revision-create >> >> - mediawiki.revision-visibility-change >> >> - mediawiki.user-blocks-change >> >> - mediawiki.centralnotice.campaign-change >> >> - mediawiki.centralnotice.campai