GWicke added a comment.
Is 48k transaction records all transactions committed up to that point?
TASK DETAIL
https://phabricator.wikimedia.org/T76509
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
To: Smalyshev, GWicke
Cc
GWicke added a comment.
! In T76373#799513, @Smalyshev wrote:
Technical issues:
# On import, titan sometimes slows down and gets into GC loops.
# On querying, for vertices with a lot of edges (such as
`wd(Q5).in(P31)`, i.e. humans, titan produces a backend exception:
```
Caused
GWicke added a comment.
! In T76373#802449, @Smalyshev wrote:
Note, that running Titan with Cassandra embedded requires GC tuning. While
embedded Cassandra can provide lower latency query answering, its GC behavior
under load is less predictable.
Yeah, agreed. GC scaling limits are kind
GWicke added a subscriber: wikidata-query-service.
TASK DETAIL
https://phabricator.wikimedia.org/T77897
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
To: GWicke
Cc: Aklapper, GWicke, mark, RobH, Smalyshev, Eloquence, aude
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T77897
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
To: GWicke
Cc: Aklapper, GWicke, mark, RobH, Smalyshev, Eloquence, aude, Tobi_WMDE_SW
GWicke added a subscriber: GWicke.
GWicke added a comment.
Please make sure that the language(s) used are static for anything in the
content of a page. In particular, it should not depend on the user interface
language, as that would impose a heavy performance penalty on the user.
TASK DETAIL
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T77897
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a comment.
Might be worth starting with Tinkerpop3
https://github.com/tinkerpop/tinkerpop3/blob/master/CHANGELOG.asciidoc.
TASK DETAIL
https://phabricator.wikimedia.org/T76372
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T76372
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a comment.
@smalyshev started testing yesterday. Things are working fine so far. Thanks!
TASK DETAIL
https://phabricator.wikimedia.org/T77897
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
GWicke created this task.
GWicke added a subscriber: GWicke.
GWicke added a project: wikidata-query-service.
GWicke changed Security from none to none.
TASK DESCRIPTION
We need a reliable way to distribute a variety of update events emitted from
MediaWiki core (and other services) to various
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke triaged this task as Normal priority.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
GWicke created this task.
GWicke added a subscriber: GWicke.
GWicke added a project: wikidata-query-service.
TASK DESCRIPTION
We need to investigate design a public query API. Goals for this API are:
- Easy to build queries with a variety of clients (probably JSON) without
dealing
GWicke removed a subscriber: Aklapper.
GWicke added a project: Services.
GWicke set Security to none.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https
GWicke added a comment.
Re performance and indexing, from a mail thread:
Earlier today Stas I were looking a bit into what is happening behind the
scenes in some of the slower queries like
https://www.mediawiki.org/wiki/Wikibase/Indexing/Benchmarks
GWicke added a comment.
In https://phabricator.wikimedia.org/T76373#941917, @Manybubbles wrote:
I'd spin up a new one - probably just on a single node. I think in the long
run we probably can run this on the production search cluster but for now
lets keep it off just in case it does
GWicke added a comment.
In https://phabricator.wikimedia.org/T84923#940155, @JanZerebecki wrote:
The nature of these event type candidates is such that they are changes with
a log existing at the provider.
Wikidata might be the exeception here. Most other events are not available
GWicke added a comment.
In https://phabricator.wikimedia.org/T85181#940893, @Smalyshev wrote:
Are we considering supporting WDQ API mini-language as the option for the
queries or it's not a viable option?
The problem I see with the WDQ language is the need to perform error-prone
custom
GWicke added a comment.
Another fun article for dates:
http://en.wikipedia.org/wiki/Timeline_of_the_far_future
TASK DETAIL
https://phabricator.wikimedia.org/T76373
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a comment.
In https://phabricator.wikimedia.org/T76373#943426, @Smalyshev wrote:
Proposed storage format for dates:
1. Dates are stored as long signed integers, representing number of seconds
since 1970-01-01 00:00:00 UTC.
2. This gives us range of 292 bln years
http
GWicke added a comment.
In https://phabricator.wikimedia.org/T76373#943454, @Smalyshev wrote:
seconds(292M-12-31T23:59:59) seconds(292M+1)
It's pretty likely that there were more than 356 leap years between -292M and
1970, so it's very possible for the years to be non-monotonic if we don't
GWicke added a comment.
In https://phabricator.wikimedia.org/T76373#943459, @Smalyshev wrote:
For dates beyond real Gregorian calendar, the values more precise than years
have little meaning anyway, so I don't think it matters too much as long as
comparisons and lookups (i.e. which Greek
GWicke added a comment.
@jeroendedauw: I think it's possible to implement a simple query interface
(such as the current wikibase API, as far as I understand it) on top of a more
powerful one such as the one we are discussing here. The reverse is not
necessarily true.
What is your view
GWicke added a comment.
In https://phabricator.wikimedia.org/T84923#968636, @JanZerebecki wrote:
http://www.fedmsg.com might fit this need. It is used/developed by Fedora and
Debian people and is a federated, reliable message bus with history of
cryptographically authenticated json messages
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a comment.
In https://phabricator.wikimedia.org/T85181#971763, @JanZerebecki wrote:
No, just a list of things that were brought up, when people talked about
querying Wikidata. I don't like Ask. Only ever used Ask and Cypher. I think
the way Cypher goes about the problem is neat
GWicke added a comment.
In https://phabricator.wikimedia.org/T84923#943464, @JanZerebecki wrote:
https://www.wikidata.org/w/api.php?action=helpmodules=wbgetentities and
recent changes, see also https://phabricator.wikimedia.org/T85103 and
https://phabricator.wikimedia.org/T85100
GWicke added a blocking task: T85181: Investigate design public API, possibly
using MQL.
TASK DETAIL
https://phabricator.wikimedia.org/T85159
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https
GWicke added a comment.
In https://phabricator.wikimedia.org/T86561#971491, @JanZerebecki wrote:
Do we want to have the Cassandra and Titan nodes be in the same rack as I
assume that query performance is very latency sensitive?
I don't think that there is a huge enough difference in latency
GWicke edited the task description.
GWicke removed a subscriber: Manybubbles.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https
GWicke added a subscriber: GWicke.
GWicke added a comment.
I would actually argue for starting with a simple subset of whatever we plan to
use in the longer term (MQL?), and then expand from there. Lets not expose n
different interfaces for the same thing just because we can. Doing so would
GWicke added a comment.
@janzerebecki: Are there any in that list that you would prefer over MQL? If
so, for which reasons?
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username
GWicke added a subscriber: Manybubbles.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a comment.
@halfak has written up very similar ideas at
https://meta.wikimedia.org/wiki/Research:MediaWiki_events:_a_generalized_public_event_datasource
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a project: Analytics.
GWicke added a comment.
In https://phabricator.wikimedia.org/T84923#961622, @bd808 wrote:
can support large delays (order of days) for individual consumers
Do you have a strong use case to support this need?
Yes. Hosts can go down for multiple days
GWicke added a comment.
In https://phabricator.wikimedia.org/T85181#958820, @JeroenDeDauw wrote:
As far as I can tell, decisions are currently made based on the use case the
WMF has, rather than also holding the Wkidata plans into account. I'm getting
this impression because I'm not seeing
GWicke placed this task up for grabs.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke moved this task to Future on the Services workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
WORKBOARD
https://phabricator.wikimedia.org/project/board/69/
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a subscriber: Eevans.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a subscriber: GWicke.
TASK DETAIL
https://phabricator.wikimedia.org/T90109
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T85181
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a project: RESTBase-Usecase.
TASK DETAIL
https://phabricator.wikimedia.org/T93913
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added subscribers: mobrovac, Eevans.
TASK DETAIL
https://phabricator.wikimedia.org/T93913
REPLY HANDLER ACTIONS
Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign
username.
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: mmodell, Eevans, chasemp, brion, Krenair, Halfak, JanZerebecki, bd808,
MZMcBride, mobrovac, GWicke
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: mmodell, Eevans, chasemp, brion, Krenair, Halfak, JanZerebecki, bd808,
MZMcBride, mobrovac, GWicke
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak,
JanZerebecki, bd808
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion,
Krenair, Halfak
GWicke added a comment.
@jeroendedauw: https://en.wikipedia.org/api/rest_v1/?doc
TASK DETAIL
https://phabricator.wikimedia.org/T107602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Joe, GWicke
Cc: JohnLewis, hoo, GWicke, greg, Lydia_Pintscher
GWicke added a subscriber: GWicke.
GWicke added a comment.
Will the query service return raw HTML or SVG content? If it's only returning
other content types like JSON, then CORS might not end up mattering too much.
An alternative to a separate domain could be to use
`https://wikidata.org/api
GWicke edited the task description.
Herald added subscribers: Matanya, Aklapper.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell
GWicke added a comment.
@ottomata, UUIDs are described in
https://en.wikipedia.org/wiki/Universally_unique_identifier. An example for a
v1 UUID is `b54adc00-67f9-11d9-9669-0800200c9a66`. There are libraries to
extract the high-resolution timestamp for most environments.
Regarding a separate
GWicke added a comment.
@JanZerebecki: Suppression information would indeed be needed for public access
to older events. One option would be to key this on the event's UUID. We could
also consider superseding the message using Kafka's deduplication (compaction)
based on the same UUID.
TASK
GWicke added a subscriber: EBernhardson.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke,
mobrovac, MZMcBride, bd808
GWicke added a comment.
Some notes from the meeting:
1. Framing, for all events
- **uri**: string; path or url. Example:
/en.wikipedia.org/v1/page/title/San_Francisco
- **reqid**: v1 UUID
<https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_1_.28MAC_address_.26_date-time
GWicke added a comment.
In https://phabricator.wikimedia.org/T116247#1754698, @Ottomata wrote:
> > If we have a use case for emitting two secondary events *to the same topic*
> > that were both triggered by the same primary event (user click / request
> > id), then we can
GWicke added a comment.
@ottomata, they will be filled in somewhere, but I think we haven't necessarily
decided on filling them in at production time. To me it seems that filling in
either at production or consumption time will work, as long as defaults don't
change. It sounds like you have
GWicke added a comment.
@ottomata: Based on our backwards-compatibility rules, the latest schema will
be a superset of previous schemas. This means that you will be able to
understand both old and new data in a given topic using the //latest// schema.
TASK DETAIL
https
GWicke added a comment.
@ottomata, you are basically making the case for filling in the defaults at
consumption time.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: mobrovac, GWicke
Cc
GWicke added a comment.
@ottomata: If you fill in the defaults at consumption time, then you have a
choice of how you want to treat old events. You can either fill in the defaults
from the latest schema (probably what you want in most cases), or choose to
explicitly distinguish fields
GWicke added a comment.
> I've been thinking about it too. Ideally, we could leave these fields out of
> schema defs, simply reference them. But, that seems not to be in correlation
> with storing them in a git repo. What I see as a possible solution is to put
> these c
GWicke added a comment.
@ottomata, I think understanding the semantics of an event primarily requires
knowledge of the topic. The topic in turn provides access to the schema, which
describes the structure of the events. It is likely that we'll have multiple
topics record similarly-structured
GWicke added a comment.
> If we adopt a convention of always storing schema name and/or revision in the
> schemas themselves, then we can do like EventLogging does and infer and
> validate the schema based on this value. This would especially be helpful in
> associating a message
GWicke added a comment.
> I'm not so sure actually that these will always be redundant. I think the
> request ID should be persisted to track the same event throughout the system.
> Imagine a user clicks on something which produces an event in the queue and
> that event triggers
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke,
mobrovac, MZMcBride, bd808
GWicke added a comment.
> Right, but how would you do this in say, Hive? Or in bash? Timestamp logic
> should be easy and immediate.
Yeah, Hive really seems to be lacking built-in support for UUIDs. There seems
to be UDF code to deal with them, but it's definitely not as conv
GWicke added a comment.
I went ahead and updated the task description with the current framing /
per-event schema. I renamed the `reqid` to just `id`, and added a `ts` field
containing the same timestamp in ISO 8601 format.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL
GWicke added a comment.
We are having a hangout meeting tomorrow (Thursday, 22nd) between 11&12am SF
time. Please let us know if you'd like to join.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailprefere
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion,
Krenair, Halfak
GWicke added a blocked task: T102476: RFC: Requirements for change propagation.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac
GWicke added a blocked task: T102476: RFC: Requirements for change propagation.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans
GWicke created this task.
GWicke added subscribers: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell,
Eevans, chasemp, brion, Krenair, Halfak, JanZerebecki, bd808, MZMcBride,
mobrovac, GWicke, aaron, daniel, Hardikj, yuvipanda.
GWicke added projects: operations, EventBus, Discovery, Epic
GWicke raised the priority of this task from "Normal" to "High".
GWicke set Security to None.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: yuvipanda, Hardikj, d
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808,
JanZerebecki, Halfak, Krenair
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T116247
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808,
JanZerebecki, Halfak, Krenair
GWicke added a comment.
@faidon: Until very recently (last days), there wasn't actually any REST proxy
with schema validation in the EventLogging repository. @ottomata now has a
patch implementing such a service
<https://gerrit.wikimedia.org/r/#/c/235671/24/server/bin/eventlogging-serv
GWicke added a subscriber: GWicke.
GWicke added a comment.
There is some related high-level discussion about recent changes and page
history as event streams in https://phabricator.wikimedia.org/T107595. One idea
is to layer event streams, which would potentially let us integrate related
GWicke added a subscriber: mobrovac.
TASK DETAIL
https://phabricator.wikimedia.org/T105638
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: JanZerebecki, GWicke
Cc: mobrovac, GWicke, Addshore, Qgil, Spage, greg, tstarling, aude, hoo,
daniel
GWicke added a subscriber: GWicke.
GWicke added a comment.
Here is an idea for a workflow-based solution that would work for nodejs as
well:
1. Each code project has a corresponding deploy repository. For nodejs, current
practice is to have the code as a submodule of the deploy repository
GWicke added a comment.
@ottomata: In my recollection of the discussion & the log you linked to, the
question of which REST producer proxy to use was left open. Our priority is to
get basic events into Kafka before the end of this month, so that we can start
building on top of this for ch
GWicke added a comment.
In https://phabricator.wikimedia.org/T114443#1731399, @ori wrote:
> In https://phabricator.wikimedia.org/T114443#1731284, @GWicke wrote:
>
> > See https://phabricator.wikimedia.org/T88459#1604768. tl;dr: It's not
> > necessarily clear that saving ver
GWicke added a comment.
In https://phabricator.wikimedia.org/T114443#1730753, @Eevans wrote:
> 1. Already leverages a (really slick) JSON schema registry
> <https://meta.wikimedia.org/wiki/Category:Schemas_%28active%29?status=active>
Optionally fetching schemas from a URL isn
GWicke added a comment.
> For starters, it means that we have alternatives for environments where Kafka
> is overkill (small third-party installations, dev environments, mw-vagrant,
> etc). Using, for example, sqlite instead of Kafka is already something
> supported.
As far
GWicke added a comment.
A PR adding remote schema support to the nodejs frontend is now available at
https://github.com/wikimedia/restevent/pull/1. This means that we can now
choose to use local or remote schemas per-topic in the configuration.
TASK DETAIL
https://phabricator.wikimedia.org
GWicke added a project: MediaWiki-RfCs.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron,
GWicke, mobrovac
GWicke moved this task to Ready for RFC meeting on the MediaWiki-RfCs workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
WORKBOARD
https://phabricator.wikimedia.org/project/board/52/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
GWicke added a comment.
Etherpad notes: https://etherpad.wikimedia.org/p/scalable_events_system
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Aklapper, Matanya, Mattflaschen
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T84923
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion,
Krenair, Halfak
GWicke added a comment.
I have now integrated some of those changes into the description.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: dr0ptp4kt, madhuvishy, Nuria, ori, faidon
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak,
Eevans, Ottomata
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron,
GWicke, mobrovac, Halfak
GWicke added a comment.
@ori, I changed the text to clarify which of those are potential, and which are
concrete plans for this quarter. Please follow the provided links if things are
still unclear.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https
GWicke edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron,
GWicke, mobrovac, Halfak
GWicke added a comment.
@Nuria, see the task description, heading "Initial use cases".
TASK DETAIL
https://phabricator.wikimedia.org/T114443
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: GWicke
Cc: EBernhardson, bd808, Joe, dr0ptp4kt,
GWicke added a comment.
@ottomata, main reason would be the ability to work with $simple_queue,
$binary_kafka, $amazon_queue and so on without changes in MW code. This isn't
so theoretical. We'll want a lighter-weight queue for testing, developers and
third party users rather soon.
TASK
1 - 100 of 155 matches
Mail list logo
