Ottomata added a subscriber: Ottomata.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak, JanZerebecki,
bd808, MZMcBride
Ottomata added a comment.
BTW, https://phabricator.wikimedia.org/T102082 is mainly about analytics
eventlogging, but the confluent stuff would be good for an event bus used for
application stuff too.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https
Ottomata added a comment.
> topics named something like mw-edit and mw-edit-private perhaps (where the
> latter contains this extra info).
I'd prefer if we did this the other way around. The 'private' topic will have
more data and be the main source of truth. The public one will c
Ottomata added a comment.
I'd like an actual timestamp to be part of the framing for all events too. I'm
all for a reqid, (although I'd bikeshed about the name a bit), but Having a
standardized canonical timestamp in all events is very useful. Can we add:
- **ts**: iso 8601 timestamp
Ottomata added a comment.
etherpad from today's meeting:
https://etherpad.wikimedia.org/p/eventbus-events
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Smalyshev, yuvipanda
Ottomata added a blocking task: T114191: Setup a 3 server Kafka instance in
both eqiad and codfw for reliable purge streams.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Smalyshev
Ottomata added a comment.
COOL. As part of this discussion, I'd like us to think about not only fields
that are relevant to edit events, but also those fields that might be useful
for most, if not all, standardized WMF events to have. These might be
required for all events to share. Things
Ottomata added a comment.
> If we have a use case for emitting two secondary events *to the same topic*
> that were both triggered by the same primary event (user click / request id),
> then we can generate a new ID for at least one of those events, and record
> the par
Ottomata added a comment.
What do y'all think about keeping these 'framing' fields in a nested object?
I'm not sure if this is a good or bad idea. If later we decide we do want to
use $ref to share common schema fields between different schemas, it'll be
easier to do so
Ottomata added a comment.
> @Ottomata, I think understanding the semantics of an event primarily requires
> knowledge of the topic.
Hm, I don't think this is true. You will need some understanding of what a
historical dataset is, but that's all. The historical datasets are going to be
Ottomata added a comment.
Producer A has schema version 1.
Producer B has schema version 2, which has added field "name" with default
"nonya".
All of these events are being imported into Hadoop. An analyst looks at the
latest schema and wants to do some analysis on "
Ottomata added a comment.
Have we decided that defaults will be filled in for missing fields?
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: mobrovac, Ottomata
Cc: intracer, EBernhardson
Ottomata added a comment.
Or produce time.
But really, even if we fill in defaults during production or consumption, this
will still be a problem for historical data. Data is only consumed into Hadoop
once, and schema changes can happen after consumption time. If you have no
way
Ottomata added a comment.
Events will be consumed into Hadoop close to production time (within an hour
usually). Schema changes made years after the fact cannot be reflected in
years old historical data unless it is reprocessed and rewritten.
TASK DETAIL
https://phabricator.wikimedia.org
Ottomata added a comment.
> > If we adopt a convention of always storing schema name and/or revision in
> > the schemas themselves, then we can do like EventLogging does and infer and
> > validate the schema based on this value. This would especially be helpful
> >
Ottomata added a comment.
To avoid possible conflicts, I'd suggest we call this not just `id`. How about
`uuid`? That's what EventLogging capsule does:
https://meta.wikimedia.org/wiki/Schema:EventCapsule
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https
Ottomata added a comment.
I'm still a little confused about how this reqid/id will work? You are
suggesting that it comes from the x-request-id that we want varnish to set,
right? Won't this mean that multiple events (those produced during the same
http request at varnish level) will have
Ottomata added a comment.
Also, this is just a personal preference, but I'd prefer if we had a convention
differentiating integer/second based 'timestamps' and string/date based
'datetimes'. For webrequest data, the ISO8601 is called `dt`.
https://wikitech.wikimedia.org/wiki/Analytics/Data
Ottomata added a comment.
> Hm, I think duplicates should be detected based on the content of the message
> itself and the time stamp.
EventLogging explicitly uses the uuid in MySQL as a unique key for all tables.
Having it standardized on a single field means that the unique index cr
Ottomata added a comment.
> Manual schema versions. We could increase the schema version every time we
> change something in the schema. Easy to achieve but it's also easy to forget
> to bump the version when something has been changed.
FWIW, this is how EventLogging does it,
Ottomata added a comment.
> I don't see a conflicting problem with id (even though id is a JSONSchema
> keyword, but it relates to the schema, not its properties, so we're good
> there). uuid is not a good choice, IMHO, it's like naming a field string
> because its value is a stri
Ottomata added a comment.
> So the producer would store the same time stamp twice? UUID v1 already
> contains it.
Could you provide an example of what this UUID would look like?
A reason for having a timestamp only field is so that applications can use it
for time based logic without
Ottomata added a comment.
Right, but how would you do this in say, Hive? Or in bash?
Timestamp logic should be easy and immediate.
> Regarding a separate timestamp in the framing information: Which time would
> this correspond to?
This is up to the producer, I think. If there ar
Ottomata added a comment.
Ok, cool, I'm cool with that, so:
`request_id` - UUID1 from Varnish, not necessarily unique for an individual
event
`event_id` (or maybe just `uuid`? since that is what EL uses?) - Actual UUID
for an event.
`dt` - IS08601 timestamp, usually derived from the timestamp
Ottomata added a comment.
I'm not very opinionated here, but I think an analytics graphite instance will
be very useful for other things than just this ticket, especially if these
metrics are backed up by data in HDFS, and graphite is used as a visualization
and rollup tool.
TASK DETAIL
Ottomata added a comment.
Sounds good. Shall I just find a time and set one up?
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Milimetric, RobLa-WMF, brion, intracer, Smalyshev
Ottomata added a comment.
Just made a calendar event for Tuesday at 10:30 PST. Happy to move it if some
other time is better.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc
Ottomata added a comment.
Cassandra is a complicated clustered solution, graphite will just require a
simple single instance running and intaking events somewhere.
TASK DETAIL
https://phabricator.wikimedia.org/T117732
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
Ottomata closed blocking task T118578: Package EventLogging and dependencies
for Jessie as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Milimetric, RobLa-
Ottomata added a comment.
@daniel
This schema repo will be used by many codebases. EventLogging, Mediawiki,
analytics refinery, etc. etc. Anyone creating events will need this code.
There are various ways to share these schemas, but one idea is to use git
submodules.
TASK DETAIL
https
Ottomata added a comment.
Is it time to consider creating a standalone repo for these schemas? If so,
then that means it is time for repo name bikeshed, woohoo!
No idea what to call this or where to put it. `mediawiki/schemas`?
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL
Ottomata closed blocking task T118761: Move EventLogging/server to its own repo
and set up CI as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Milimetric,
Ottomata created this task.
Ottomata claimed this task.
Ottomata added subscribers: Milimetric, Nuria, madhuvishy.
Ottomata added projects: Analytics-Kanban, operations, MediaWiki-RfCs,
Wikidata-Query-Service, Service-Architecture, Services,
MediaWiki-General-or-Unknown, Wikidata, Analytics
Ottomata added a comment.
> In my recollection of the discussion & the log you linked to, the question of
> which REST producer proxy to use was left open.
I think you may be referring to the first link of the meetbot notes, which was
ended before we stopped discussing. Starting at
Ottomata added a comment.
Today's EventBus RFC
<https://tools.wmflabs.org/meetbot/wikimedia-office/2015/wikimedia-office.2015-10-30-18.17.html>
discussion
<http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20151030.txt> ended
with the general consensus that we wi
Ottomata added a comment.
Cool, added some comments.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: mobrovac, Ottomata
Cc: intracer, EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron
Ottomata added a subscriber: Milimetric.
Ottomata added a comment.
Ok, still various TODOs around the code, but this is ready for review.
https://gerrit.wikimedia.org/r/#/c/235671
There are concepts that it'll be good to do close review with folks familiar
with EventLogging (probably @nuria
Ottomata moved this task to In Progress on the Analytics-Kanban workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
WORKBOARD
https://phabricator.wikimedia.org/project/board/1030/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
Ottomata added a project: Analytics-Kanban.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: RobLa-WMF, brion, intracer, Smalyshev, mark, MZMcBride, Krinkle,
EBernhardson, bd808, Joe
Ottomata added a comment.
> Until very recently (last days), there wasn't actually an EventBus-like REST
> proxy with schema validation in the EventLogging repository.
Not quite true, this was started Sept 3.
https://phabricator.wikimedia.org/T88459#1601022
TASK DETAIL
Ottomata added a comment.
So, we need to be really careful here. This MVP as of yet has zero buy in from
anyone in ops. In addition, both @ori and @eevans point out that EventLogging
already does everything that this MVP encompasses, minus the HTTP service part.
Now it is time for me
Ottomata added a comment.
Hey yalls,
I've had requests that we postpone the RFC for this one more week, until Oct
28th. I'd like for one opsen and @ori to be able to attend, and the relevant
opsens are all traveling, and Ori can't make this one either.
TASK DETAIL
https
Ottomata added a comment.
@Joe, there are two parts to this MVP:
- Centralized (and CI controlled) schema sharing
- An easy way to get valid data into Kafka.
With eventlogging right now, we are spending a lot of resources just processing
incoming data and making sure it is valid. When data
Ottomata added a comment.
@joe, yes, please see this ticket:
https://phabricator.wikimedia.org/T88459
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: EBernhardson, bd808, Joe
Ottomata added a comment.
@ori I just edited the EventBus project description
<https://phabricator.wikimedia.org/project/sprint/profile/1474/> to include a
version of the problem statement I sent out to engineering@ back in July.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
Ottomata added a project: EventBus.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion,
Krenair, Halfak
Ottomata closed blocking task T110748: Event Bus as "Invalid".
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevan
Ottomata added a comment.
Does the description look ok? Feel free to edit and discuss here.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: madhuvishy, Nuria, ori, faidon, aaron
Ottomata added a comment.
Nuria pointed out to me that (as engineers love to do) we are focusing a lot
here on technical architecture, but haven't thought a lot about what use case
this MVP will actual serve. I agree, and it would be good for us to pick a
simple use case for this system
Ottomata added a subscriber: dr0ptp4kt.
Ottomata set Security to None.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke
Ottomata created this task.
Ottomata added subscribers: Aklapper, Ottomata, Eevans, Halfak, mobrovac,
GWicke, aaron, faidon, ori, Nuria, madhuvishy.
Ottomata added projects: EventBus, Discovery, Epic, Analytics, Wikidata,
operations, MediaWiki-General-or-Unknown, Services, Service-Architecture
Ottomata added a comment.
Edit events would be awesome and totally doable with this MVP, but I'm a little
worried about the amount of bike shedding that will go into designing that
schema!
What about api.php logging? See https://phabricator.wikimedia.org/T108618.
Those logs are still being
Ottomata added a comment.
@bd808, what's your timeline on this? Producing directly into Kafka is fine,
but we are trying to do two things with this MVP:
- centralize and standardize schemas
- provide a REST API for producing to Kafka
We aren't going to block you on your work, but if you could
Ottomata added a comment.
Oh, so the service is planned to be kafka agnostic? (I will come hang on your
side after lunch :) )
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc
Ottomata added a comment.
Hm, @gwicke, if Mediawiki has access to schemas, and is relatively sure that it
can produce valid messages for a given topic, why should a Mediawiki client use
EventBus at all? They can produce directly to Kafka, bypassing the HTTP and
proxying overhead of EventBus
Ottomata added a blocking task: T110748: Event Bus.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion,
Krenair
Ottomata added a comment.
Hm, not sure I follow. We are proposing that a schema be ID-able via a URI,
and also remotely locatable if that URI happens to be a full URL with schema
and domain information. Is the opaqueness issue the fact that it is
`/` that IDs the schema, instead of just
Ottomata added a comment.
Hm, I think I see. We are coupling the URI to the ID, which according to the
W3C should not be relied upon. Ok, noted.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
Ottomata added a comment.
No, consumption is not part of the MVP.
There may be future work to make consumption from Kafka via websockets easy to
set up, but we will not make any Events public by default. We will have to set
up special endpoints for approved event streams.
TASK DETAIL
Ottomata closed blocking task T118869: Send HTTP stats about
eventlogging-service to statsd [3 pts] as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc
Ottomata closed blocking task T118780: Puppetize eventlogging-service as
"Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Glaisher, Milimetric, RobLa-WMF, brion
Ottomata added a subscriber: Nuria.
Ottomata added a comment.
@gwicke and I discussed the schema/revision in meta issue in IRC today. He had
an idea that I quite like!
@gwicke suggested that instead of using (schema, revision) to uniquely ID a
schema, that we just use a URI. EventLogging does
Ottomata added a comment.
FYI, the repo is here, waiting for some schemas! :)
https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/event-schemas
For Avro, @ebernhardson, you can go ahead and submit a patch there in an avro/
directory. We should should probably maintain the usual Java
Ottomata added a comment.
I believe we can close this task, ja? Got a few defined here:
https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings
Ottomata added a subscriber: Ottomata.
Ottomata added a comment.
Hey sorry, I don't think I've seen this ticket before, hence the silence! I
just commented on ticket about pull vs. push.
TASK DETAIL
https://phabricator.wikimedia.org/T118739
EMAIL PREFERENCES
https
Ottomata closed blocking task T88459: Implementing the reliable event bus using
Kafka as "Resolved".
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke, Ottomata
Cc: Aklappe
Ottomata added a comment.
We will resolve this after https://phabricator.wikimedia.org/T120212 is
closed, and after we have the first consumer (change propagation) in production.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org
Ottomata moved this task to In Progress on the EventBus workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
WORKBOARD
https://phabricator.wikimedia.org/project/board/1474/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc
Ottomata added a comment.
YEEHAW
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: yuvipanda, JanZerebecki, Glaisher, Milimetric, RobLa-WMF, brion, intracer,
Smalyshev, mark
Ottomata added a comment.
Done!
TASK DETAIL
https://phabricator.wikimedia.org/T119070
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Ottomata
Cc: Ottomata, Aklapper, Addshore, StudiesWorld, D3r1ck01, Izno, Wikidata-bugs,
aude, Mbch331
Ottomata added a comment.
@Smalyshev let's jump in a hangout sometime to discuss this more.
Just a few quick points:
Does not have data back more than 7 days
We could probably bump this up to 14 days for specific topics like recentchange.
Scalable - there's no hard limit on the number
Ottomata added a comment.
If so, you may want to consider consuming from Kafka rather than EventStreams.
I am considering this too, but I assume it's more code for me to write (maybe wrongly, I didn't look at it closely).
It will be more, a lot more. What language are you working in?
But what
Ottomata added a comment.
I'd rather have some intermediary that cleans up, deduplicates, etc. the changes.
FYI, neither base Kafka Consumer clients nor EventStreams does this.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Ottomata triaged this task as "Low" priority.Ottomata removed a project: Operations.Herald added a project: Operations.
TASK DETAILhttps://phabricator.wikimedia.org/T154017EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Aklapper, Lydia
Ottomata removed a project: Operations.Ottomata added a comment.Herald added a project: Operations.
This has been placed on the Traffic board, removing operations tag.TASK DETAILhttps://phabricator.wikimedia.org/T154015EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Ottomata triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T154015EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Ricordisamoa, Boshomi, Lydia_Pintscher, Aklapper, Esc3300, Th3d3v1ls, EBju
Ottomata added a comment.
I haven't fully grokked this ticket, but in general I am for events! :) 2 qs:
How many events per second?
What would be emitting this event? Mediawiki?
TASK DETAILhttps://phabricator.wikimedia.org/T151717EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
Ottomata edited projects, added Analytics-Kanban; removed Analytics.
TASK DETAILhttps://phabricator.wikimedia.org/T173850EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Liuxinyu970226, WMDE-leszek, Anomie, Aklapper, Jdforrester-WMF, Addshore
Ottomata added a comment.
I fixed the two warnings from the EventLogging extension, but I have not been able to test them. If anyone knows how, I would be much obliged.
https://gerrit.wikimedia.org/r/#/c/374816/TASK DETAILhttps://phabricator.wikimedia.org/T173850EMAIL PREFERENCEShttps
Ottomata added a comment.
Sure, I suppose! You can connect to it with a Kafka client now. The Kafka brokers are kafka-jumbo100[1-6].eqiad.wmnet:9092
I think you are most interested in the eqiad.mediawiki.revision-create topic. I haven't tried yet at all, but these topics should have a broker
Ottomata added a comment.
Seems like the mirroring is done by 0.9 MirrorMaker and timestamp handling was added only in 0.10 MirrorMaker.
Hm, ya but I had thought that if a timestamp was not set by the producer, it would be set to server receive time. Maybe I was wrong!TASK DETAILhttps
Ottomata added a comment.
Woot, that did it ^. We need topics to default to LogAppendTime.
[@stat1005:/home/otto] $ ./kafkacat -Q -b kafka-jumbo1001.eqiad.wmnet:9092 -t eqiad.mediawiki.revision-create:0:151275919
eqiad.mediawiki.revision-create [0] offset 3658631TASK DETAILhttps
Ottomata added a comment.
So, FYI, the timestamps as they are now are the timestamp that the kafka jumbo-eqiad cluster received the messages. These are replicated from the main-eqiad cluster, and might have a short (seconds usually, minutes max) delay.
Eventually (work not planned yet) we
Ottomata added a subtask: T175461: Port Clients to new jumbo cluster.
TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ladsgroup, Nuria, Anomie, Aklapper, Smalyshev, Lahi, Gq86, Vacio
Ottomata added a comment.
Unfortunately not yet! We are very close...the cluster is up and running, but porting clients has been blocked on getting proper keys and certificates for SSL support for a long time now. SSL is finally moving now, so we should be able to start porting clients over soon
Ottomata added a comment.
You can easily 'quickbuild' kafkacat with a statically linked librdkafka. I've just done this on a stretch labs host, and copied the kafkacat binary to stat1005 at /home/otto/kafkacat. Try it out!TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps
Ottomata added a comment.
mediawiki eventbus topics should now be retained for 31 days in main Kafka clusters. If we add a new mediawiki topic, we need to remember to run this command for it.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org
Ottomata added a project: Analytics-Kanban.
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smalyshev, Lahi, Gq86
Ottomata added a comment.
Doing the following for all main-eqiad and main-codfw:
for t in \
mediawiki.page-create\
mediawiki.page-delete\
mediawiki.page-edit \
mediawiki.page-move
Ottomata renamed this task from "Increase kafka event retention to 14 or 21 days" to "Increase kafka event retention to 31".Ottomata claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/pa
Ottomata set the point value for this task to "2".
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup, Pchelolo, JAllemandou, Smaly
Ottomata merged a task: T196409: Consider increasing retention for mediawiki event topics.
TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata, Aklapper, Nuria, Ladsgroup
Ottomata added a comment.
I'll make this 31 days just to bump it up to a month. We have plenty of space for this.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: mforns, elukey, Ottomata
Ottomata added a comment.
OO yes @Smalyshev and in case you didn't see, we also increased retention of mediawiki topics to 31 days in the main kafka clusters.TASK DETAILhttps://phabricator.wikimedia.org/T161731EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
Ottomata added a comment.
I think we can do this just for the mediawiki eventbus topics on the jumbo cluster.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Ottomata, Aklapper, Nuria, Ladsgroup
Ottomata added a comment.
Is there a reason we want to do this on main instead of jumbo? Stas will be consuming from jumbo, since it has timestamp offset support.TASK DETAILhttps://phabricator.wikimedia.org/T187296EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
Ottomata triaged this task as "Low" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T177099EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Lydia_Pintscher, herron, Aklapper, Qtn1293, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn
Ottomata triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T181988EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Aklapper, Smalyshev, Gehel, Qtn1293, Lahi, Gq86, Darkminds3113, Lucas_Werkme
Ottomata triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T178445EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: OttomataCc: Joe, Volans, mobrovac, Smalyshev, Gehel, Stashbot, Aklapper, Dzahn, Qtn1293,
Ottomata added subscribers: Pchelolo, Ottomata.Ottomata added a comment.
Ping @Pchelolo.
This is happening because of https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/468482/ as part of a fix for T207329: Clear watchlist on enwiki only removes 50 items at a time.
Interesting
1 - 100 of 185 matches
Mail list logo
