[Wikidata-bugs] [Maniphest] [Unassigned] T105845: Page components / content widgets

2017-10-11 Thread GWicke
GWicke removed GWicke as the assignee of this task. TASK DETAILhttps://phabricator.wikimedia.org/T105845EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: leila, Reasno, SBisson, MZMcBride, Mholloway, RandomDSdevel, jmadler, Bianjiang, LikeLifer

[Wikidata-bugs] [Maniphest] [Unassigned] T99088: [RFC] Evolving our content platform: Content adaptability, structured data and caching

2017-10-11 Thread GWicke
GWicke removed GWicke as the assignee of this task. TASK DETAILhttps://phabricator.wikimedia.org/T99088EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: jmadler, Bianjiang, gpaumier, LikeLifer, Mholloway, MZMcBride, RobLa-WMF, StudiesWorld, Qgil, Tgr

[Wikidata-bugs] [Maniphest] [Commented On] T175316: Very large jobs posted by Wikidata

2017-09-14 Thread GWicke
GWicke added a comment. Looks like adding the JSON_UNESCAPED_UNICODE flag should do it: http://php.net/manual/en/function.json-encode.phpTASK DETAILhttps://phabricator.wikimedia.org/T175316EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pchelolo, GWickeCc

[Wikidata-bugs] [Maniphest] [Updated] T175316: Very large jobs posted by Wikidata

2017-09-13 Thread GWicke
GWicke added a comment. Raised priority, as this is a) blocking the migration to the Kafka job queue backend (T157088), and b) is likely already causing performance and possibly reliability issues in the current job queue.TASK DETAILhttps://phabricator.wikimedia.org/T175316EMAIL PREFERENCEShttps

[Wikidata-bugs] [Maniphest] [Raised Priority] T175316: Very large jobs posted by Wikidata

2017-09-13 Thread GWicke
GWicke raised the priority of this task from "Normal" to "High". TASK DETAILhttps://phabricator.wikimedia.org/T175316EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pchelolo, GWickeCc: Ladsgroup, daniel, GWicke, Aklapper, Pchelolo, GoranS

[Wikidata-bugs] [Maniphest] [Edited] T175316: Very large jobs posted by Wikidata

2017-09-08 Thread GWicke
GWicke updated the task description. (Show Details) CHANGES TO TASK DESCRIPTION..."2653965": [6, "Meyers_b9_s0043.jpg"], (..a few million further page entries...) },...TASK DETAILhttps://phabricator.wikimedia.org/T175316EM

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-08-31 Thread GWicke
GWicke added a comment. I updated https://gerrit.wikimedia.org/r/#/c/295027/ to apply on current master. This removes CDN purges from HTMLCacheUpdate, and only performs them after RefreshLinks, and only if nothing else caused a re-render since. With this patch applied, we should be able to reduce

[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop

2017-08-30 Thread GWicke
GWicke added a comment. HTMLCacheUpdate root job timestamp distribution, jobs executed within the last 15 hours: 1233 20170407 8237 20170408 18 20170423 18 20170426 20 20170429 50 20170430 18 20170502 18 20170504 20 20170509 10 20170512 18

[Wikidata-bugs] [Maniphest] [Updated] T173710: Job queue is increasing non-stop

2017-08-30 Thread GWicke
GWicke added a comment. A possible contribution to the backlog building could be the infinite retry / immortal job problem described in T73853. Looking for old htmlCacheUpdate root jobs from April still executing over four months later (!) via grep htmlCacheUpdate runJobs.log | grep -c

[Wikidata-bugs] [Maniphest] [Updated] T173710: Job queue is increasing non-stop

2017-08-30 Thread GWicke
GWicke added a project: Services (watching). TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen

[Wikidata-bugs] [Maniphest] [Commented On] T172832: Investigate use-cases for delayed job executions

2017-08-28 Thread GWicke
GWicke added a comment. In T172832#3540031, @Mattflaschen-WMF wrote: There are three considerations relevant to Echo: Delayed notifications (T156808: Back-end infrastructure for timed notifications in Echo) 1a. Article reminder notifications (T2582: Remind me of this article in X days) 1b. User

[Wikidata-bugs] [Maniphest] [Commented On] T172832: Investigate use-cases for delayed job executions

2017-08-16 Thread GWicke
GWicke added a comment. Some use cases from today's eventbus sync discussion: Batching & rate limiting for Echo notifications CirrusSearch for rate limiting / batching Wikidata for rate limiting, partly work-around for lack of dependency tracking Delayed HTCP purging (by ~10s) to acc

[Wikidata-bugs] [Maniphest] [Updated] T170860: Need ability to block specific query sources for WDQS

2017-07-17 Thread GWicke
GWicke added a comment. T167906: Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting might help with this problem, especially if you can reduce the allowed concurrency to a fairly small value.TASK DETAILhttps

[Wikidata-bugs] [Maniphest] [Updated] T167787: [Spike 2hr] Investigate ability for page previews in wikidata to appear in user's preferred language

2017-06-20 Thread GWicke
GWicke added a comment. In T167787#3364625, @phuedx wrote: In T167787#3364314, @GWicke wrote: We discussed this in the Reading / Services sync meeting. One question that came up in the discussion is whether including all languages in the response would be feasible from a performance perspective

[Wikidata-bugs] [Maniphest] [Updated] T167787: [Spike 2hr] Investigate ability for page previews in wikidata to appear in user's preferred language

2017-06-20 Thread GWicke
GWicke added a comment. We discussed this in the Reading / Services sync meeting. One question that came up in the discussion is whether including all languages in the response would be feasible from a performance perspective. The advantage of this direction would be no cache fragmentation

[Wikidata-bugs] [Maniphest] [Commented On] T157014: CONSULTATION/PLAN: Managing Complex State and GUI on MediaWiki (e.g. for Wikidata/Wikibase UI)

2017-04-10 Thread GWicke
GWicke added a comment. In T157014#3168034, @daniel wrote: AFAIK we do have infrastructure for running node servers with proper caching, as part of the rest services layer. If you create endpoints that return structured data you could consume those from the PHP backend or the JS client. We have

[Wikidata-bugs] [Maniphest] [Commented On] T161527: Canonical data URIs and URLs for machine readable page content

2017-03-29 Thread GWicke
GWicke added a comment. I think the URL most users would consider canonical is /wiki/{title}. Wouldn't this already provide a reasonable URL for the concept of the page?TASK DETAILhttps://phabricator.wikimedia.org/T161527EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] [Commented On] T161527: Canonical data URIs and URLs for machine readable page content

2017-03-29 Thread GWicke
GWicke added a comment. In T161527#3135095, @Smalyshev wrote: I think we have several concepts there that needs to be refined. Canonical object URI - this is the URI that uniquely identifies an object in Wikimedia world, and, by extension, in the whole world of linked data. I think

[Wikidata-bugs] [Maniphest] [Updated] T161527: Canonical data URIs and URLs for machine readable page content

2017-03-27 Thread GWicke
GWicke added a project: Services (watching). TASK DETAILhttps://phabricator.wikimedia.org/T161527EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: Dzahn, GWicke, tstarling, Aklapper, Jonas, Smalyshev, mkroetzsch, Lydia_Pintscher, daniel, QZanden

[Wikidata-bugs] [Maniphest] [Commented On] T161527: Canonical data URIs and URLs for machine readable page content

2017-03-27 Thread GWicke
GWicke added a comment. However, URIs by nature should not include interface version information, because they identify the resource independently of representation. The REST API versioning policy explicitly describes how representation concerns are hadled through content negotiation

[Wikidata-bugs] [Maniphest] [Commented On] T161527: Canonical data URIs and URLs for machine readable page content

2017-03-27 Thread GWicke
GWicke added a comment. The description of the requirements seems to fit the REST API: API versioning & content negotiation. REST URL structure. Integration with CDN layer. Machine and user readable API specs & documentation. The REST URL hierarchy makes it quite easy to route spec

[Wikidata-bugs] [Maniphest] [Created] T149114: Reconsider wikidata support in the REST API

2016-10-25 Thread GWicke
GWicke created this task.GWicke added projects: Services (next), Wikidata, RESTBase-API, Mobile-Content-Service.Herald added a subscriber: Aklapper. TASK DESCRIPTIONBasically none of the current REST API end points in https://www.wikidata.org/api/rest_v1/ are actually useful currently. HTML

[Wikidata-bugs] [Maniphest] [Triaged] T149114: Reconsider wikidata support in the REST API

2016-10-25 Thread GWicke
GWicke triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T149114EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: mobrovac, bearND, Mholloway, Pchelolo, Eevans, daniel, Aklapper, GWicke, D3r1ck01, Izn

[Wikidata-bugs] [Maniphest] [Lowered Priority] T102476: RFC: Requirements for change propagation

2016-10-12 Thread GWicke
GWicke lowered the priority of this task from "High" to "Low".GWicke edited projects, added Services (watching); removed Services. TASK DETAILhttps://phabricator.wikimedia.org/T102476EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Updated] T88633: [EPIC] Image-positioning service for storing and retrieving image focal points

2016-10-12 Thread GWicke
GWicke edited projects, added Services (watching); removed Services (later). TASK DETAILhttps://phabricator.wikimedia.org/T88633EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: MZMcBride, Jdlrobson, Ricordisamoa, Spage, bearND, kaldari, KHammerstein

[Wikidata-bugs] [Maniphest] [Updated] T88633: [EPIC] Image-positioning service for storing and retrieving image focal points

2016-10-12 Thread GWicke
GWicke edited projects, added Services (backlog); removed Services. TASK DETAILhttps://phabricator.wikimedia.org/T88633EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: MZMcBride, Jdlrobson, Ricordisamoa, Spage, bearND, kaldari, KHammerstein, Mhurd

[Wikidata-bugs] [Maniphest] [Edited] T102476: RFC: Requirements for change propagation

2016-10-05 Thread GWicke
GWicke edited the task description. (Show Details) EDIT DETAILS...- **Reliable RCStream**: @Ottomata has been looking into leveraging Kafka events in RCStream. This can potentially let clients catch up after being disconnected. This can potentially let clients catch up after being disconnectedSee

[Wikidata-bugs] [Maniphest] [Updated] T38881: Wiktionary needs usable API

2016-07-22 Thread GWicke
GWicke added a comment. There is now an experimental API end point for wiktionary definitions at https://en.wiktionary.org/api/rest_v1/?doc#!/Page_content/get_page_definition_term This API is used by the Android app to provide definitions for words using wiktionary data, but it is currently only

[Wikidata-bugs] [Maniphest] [Updated] T38881: Wiktionary needs usable API

2016-07-22 Thread GWicke
GWicke added a subtask: T138709: Use microformats on Wiktionary to improve term parsing. TASK DETAILhttps://phabricator.wikimedia.org/T38881EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GWickeCc: Alkamid, TheDaveRoss, dg711, intracer, Aklapper, Hippietrail

[Wikidata-bugs] [Maniphest] [Commented On] T102476: RFC: Requirements for change propagation

2016-05-18 Thread GWicke
GWicke added a comment. I moved the current status / next steps section back here, so that only the general background section is now on MediaWiki.org. Status & next steps is more volatile & links directly to ongoing work, so benefits from being in this task. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Edited] T102476: RFC: Requirements for change propagation

2016-05-18 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobLa-WMF, GWicke Cc: Scott_WUaS, Ottomata, Smalyshev, ArielGlenn, hoo, Addshore, RobLa-WMF, StudiesWorld

[Wikidata-bugs] [Maniphest] [Edited] T102476: RFC: Requirements for change propagation

2016-05-13 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobLa-WMF, GWicke Cc: Ottomata, Smalyshev, ArielGlenn, hoo, Addshore, RobLa-WMF, StudiesWorld, intracer

[Wikidata-bugs] [Maniphest] [Commented On] T102476: RFC: Requirements for change propagation

2016-05-13 Thread GWicke
GWicke added a comment. @RobLa-WMF, I updated the task summary with a status summary & a sketch of next steps & open questions. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: R

[Wikidata-bugs] [Maniphest] [Edited] T102476: RFC: Requirements for change propagation

2016-05-13 Thread GWicke
GWicke added subscribers: Smalyshev, Ottomata. GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RobLa-WMF, GWicke Cc: Ottomata, Smalyshev, ArielGlenn, hoo

[Wikidata-bugs] [Maniphest] [Commented On] T107595: [RFC] Multi-Content Revisions

2016-05-13 Thread GWicke
GWicke added a comment. > As I understand it, restbase is a front-end caching proxy store, exposed to the public internet. For most use cases (including HTML), it is actually *storing*, and not just caching. It is the equivalent of ExternalStore and most of the text table, includ

[Wikidata-bugs] [Maniphest] [Commented On] T107595: [RFC] Multi-Content Revisions

2016-05-04 Thread GWicke
GWicke added a comment. The use case for providing metadata is so that we can use stores like RESTBase, which already provide an API keyed on title, revision & render ID. It also already deals with the complexities you mention. Basically, if we don't have a way to provide this

[Wikidata-bugs] [Maniphest] [Commented On] T107595: [RFC] Multi-Content Revisions

2016-05-04 Thread GWicke
GWicke added a comment. > In any case, the PageUpdater / WikiPage code needs to trigger notifications (produce events). I don't care what mechanism it used for that. Or rather: I'm very happy if we get a generalized mechanism. We'll have to agree on some kind of schema for revisions, sl

[Wikidata-bugs] [Maniphest] [Commented On] T107595: [RFC] Multi-Content Revisions

2016-05-04 Thread GWicke
GWicke added a comment. > Blobs would typically be shared by different revisions of the same page. This happens every time one primary slot is edited, but another is not changed. E.g. the free wikitext description of a file is edited, but the structured data isn't (or vice ve

[Wikidata-bugs] [Maniphest] [Updated] T107595: [RFC] Multi-Content Revisions

2016-05-04 Thread GWicke
GWicke added a comment. > Where do I propose another mechanism for change propagation? The PageUpdater would do exactly what Revision does now: schedule DataUpdates. EventBus & the change propagation service are moving away from scheduling "jobs", and towards an

[Wikidata-bugs] [Maniphest] [Commented On] T107595: [RFC] Multi-Content Revisions

2016-04-28 Thread GWicke
GWicke added a comment. Some notes: - PageUpdater aims to provide similar functionality as the change propagation service (using EventBus) & the job queue. Could you clarify why we need another mechanism for change propagation? - The blob store does not provide any loca

[Wikidata-bugs] [Maniphest] [Up For Grabs] T102476: RFC: Requirements for change propagation

2016-03-23 Thread GWicke
GWicke placed this task up for grabs. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: ArielGlenn, hoo, Addshore, RobLa-WMF, StudiesWorld, intracer, JanZerebecki, brion, Ltrlg, Anomie

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidata Sparql queries

2016-02-17 Thread GWicke
GWicke added a comment. > some of my concerns about the affect of including a graph (with a slowish > query) on page save timing, e.g. when just fixing a typo on a wikipedia page. To get sensible hit rates for relatively rare events like edits, we would need to cache results for a lon

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidata Sparql queries

2016-02-17 Thread GWicke
GWicke added a comment. > Since running the query each time graph is displayed is too expensive, we > want some intermediate caching store that would store the results, possibly > for the time defined in the query. Is the graph extension actually re-requesting the data on each view,

[Wikidata-bugs] [Maniphest] [Commented On] T126730: [RFC] Caching for results of wikidatasparql queries for Graphs

2016-02-15 Thread GWicke
GWicke added a comment. @smalyshev, I would need more context to usefully comment on this. In particular, I wonder if there are a small number of queries that get a lot of hits, and if those queries can be cached for long enough to result in worthwhile hit rates. When discussing a use case

[Wikidata-bugs] [Maniphest] [Edited] T102476: RFC: Requirements for change propagation

2016-02-11 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: ArielGlenn, hoo, Addshore, RobLa-WMF, StudiesWorld, intracer, Qgil, JanZerebecki, brion, Ltrlg

[Wikidata-bugs] [Maniphest] [Updated] T102476: RFC: Requirements for change propagation

2016-02-11 Thread GWicke
GWicke removed a project: Wikimedia-Developer-Summit-2016. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: ArielGlenn, hoo, Addshore, RobLa-WMF, StudiesWorld, intracer, Qgil

[Wikidata-bugs] [Maniphest] [Closed] T84923: Reliable publish / subscribe event bus

2016-01-22 Thread GWicke
GWicke closed this task as "Resolved". GWicke claimed this task. GWicke added a comment. A basic event bus is now available in production, and is being populated with edit events from MediaWiki. Consumption is directly from Kafka at this point. This means that the core proposal of

[Wikidata-bugs] [Maniphest] [Unblock] T102476: RFC: Requirements for change propagation

2016-01-22 Thread GWicke
GWicke closed blocking task T84923: Reliable publish / subscribe event bus as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: Addshore, RobLa-WMF, StudiesWorld

[Wikidata-bugs] [Maniphest] [Commented On] T114474: More flexible and modernized Recent Changes code

2015-12-28 Thread GWicke
GWicke added a comment. @aude: Which questions would you like to resolve at the summit? Do you think this topic could also be reasonably discussed in a regular RFC meeting? TASK DETAIL https://phabricator.wikimedia.org/T114474 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings

[Wikidata-bugs] [Maniphest] [Edited] T102476: RFC: Requirements for change propagation

2015-12-28 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T102476 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: Addshore, RobLa-WMF, StudiesWorld, intracer, Qgil, JanZerebecki, brion, Ltrlg, Anomie, Milimetric

[Wikidata-bugs] [Maniphest] [Commented On] T102476: RFC: Requirements for change propagation

2015-12-28 Thread GWicke
GWicke added a comment. @janzerebecki: The current intention is to keep change propagation relatively simple and efficient. Many services can be implemented with very small relative per-request overheads, and services with high per-request overheads can consider applying opportunistic batching

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

2015-12-22 Thread GWicke
GWicke added a comment. @ArielGlenn: To me it seems that the discussion so far lacks a shared agreement on what the most pressing problems with dumps are. This makes it difficult to evaluate candidate solutions and their trade-offs relative to the top priorities. With the right preparation

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T105638: RFC: Streamlining Composer usage

2015-11-06 Thread GWicke
GWicke added a subscriber: mobrovac. TASK DETAIL https://phabricator.wikimedia.org/T105638 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JanZerebecki, GWicke Cc: mobrovac, GWicke, Addshore, Qgil, Spage, greg, tstarling, aude, hoo, daniel

[Wikidata-bugs] [Maniphest] [Commented On] T105638: RFC: Streamlining Composer usage

2015-11-06 Thread GWicke
GWicke added a subscriber: GWicke. GWicke added a comment. Here is an idea for a workflow-based solution that would work for nodejs as well: 1. Each code project has a corresponding deploy repository. For nodejs, current practice is to have the code as a submodule of the deploy repository

[Wikidata-bugs] [Maniphest] [Updated] T114474: More flexible and modernized Recent Changes code

2015-11-05 Thread GWicke
GWicke added a subscriber: GWicke. GWicke added a comment. There is some related high-level discussion about recent changes and page history as event streams in https://phabricator.wikimedia.org/T107595. One idea is to layer event streams, which would potentially let us integrate related

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-04 Thread GWicke
GWicke added a comment. @faidon: Until very recently (last days), there wasn't actually any REST proxy with schema validation in the EventLogging repository. @ottomata now has a patch implementing such a service <https://gerrit.wikimedia.org/r/#/c/235671/24/server/bin/eventlogging-serv

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-11-02 Thread GWicke
GWicke added a comment. @ottomata: In my recollection of the discussion & the log you linked to, the question of which REST producer proxy to use was left open. Our priority is to get basic events into Kafka before the end of this month, so that we can start building on top of this for ch

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-29 Thread GWicke
GWicke added a comment. @ottomata, they will be filled in somewhere, but I think we haven't necessarily decided on filling them in at production time. To me it seems that filling in either at production or consumption time will work, as long as defaults don't change. It sounds like you have

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-29 Thread GWicke
GWicke added a comment. @ottomata: Based on our backwards-compatibility rules, the latest schema will be a superset of previous schemas. This means that you will be able to understand both old and new data in a given topic using the //latest// schema. TASK DETAIL https

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-29 Thread GWicke
GWicke added a comment. @ottomata, you are basically making the case for filling in the defaults at consumption time. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: mobrovac, GWicke Cc

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-29 Thread GWicke
GWicke added a comment. @ottomata: If you fill in the defaults at consumption time, then you have a choice of how you want to treat old events. You can either fill in the defaults from the latest schema (probably what you want in most cases), or choose to explicitly distinguish fields

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-28 Thread GWicke
GWicke added a comment. @ottomata, I think understanding the semantics of an event primarily requires knowledge of the topic. The topic in turn provides access to the schema, which describes the structure of the events. It is likely that we'll have multiple topics record similarly-structured

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-27 Thread GWicke
GWicke added a comment. > I've been thinking about it too. Ideally, we could leave these fields out of > schema defs, simply reference them. But, that seems not to be in correlation > with storing them in a git repo. What I see as a possible solution is to put > these c

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-26 Thread GWicke
GWicke added a comment. In https://phabricator.wikimedia.org/T116247#1754698, @Ottomata wrote: > > If we have a use case for emitting two secondary events *to the same topic* > > that were both triggered by the same primary event (user click / request > > id), then we can

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-26 Thread GWicke
GWicke added a comment. > If we adopt a convention of always storing schema name and/or revision in the > schemas themselves, then we can do like EventLogging does and infer and > validate the schema based on this value. This would especially be helpful in > associating a message

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-26 Thread GWicke
GWicke added a comment. > I'm not so sure actually that these will always be redundant. I think the > request ID should be persisted to track the same event throughout the system. > Imagine a user clicks on something which produces an event in the queue and > that event triggers

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-23 Thread GWicke
GWicke added a comment. @ottomata, UUIDs are described in https://en.wikipedia.org/wiki/Universally_unique_identifier. An example for a v1 UUID is `b54adc00-67f9-11d9-9669-0800200c9a66`. There are libraries to extract the high-resolution timestamp for most environments. Regarding a separate

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-23 Thread GWicke
GWicke added a comment. @JanZerebecki: Suppression information would indeed be needed for public access to older events. One option would be to key this on the event's UUID. We could also consider superseding the message using Kafka's deduplication (compaction) based on the same UUID. TASK

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T116247: Define edit related events for change propagation

2015-10-23 Thread GWicke
GWicke added a subscriber: EBernhardson. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808

[Wikidata-bugs] [Maniphest] [Edited] T116247: Define edit related events for change propagation

2015-10-23 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson, Smalyshev, yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-23 Thread GWicke
GWicke added a comment. > Right, but how would you do this in say, Hive? Or in bash? Timestamp logic > should be easy and immediate. Yeah, Hive really seems to be lacking built-in support for UUIDs. There seems to be UDF code to deal with them, but it's definitely not as conv

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-23 Thread GWicke
GWicke added a comment. I went ahead and updated the task description with the current framing / per-event schema. I renamed the `reqid` to just `id`, and added a `ts` field containing the same timestamp in ISO 8601 format. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T116247: Define edit related events for change propagation

2015-10-22 Thread GWicke
GWicke added a comment. Some notes from the meeting: 1. Framing, for all events - **uri**: string; path or url. Example: /en.wikipedia.org/v1/page/title/San_Francisco - **reqid**: v1 UUID <https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_1_.28MAC_address_.26_date-time

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-21 Thread GWicke
GWicke added a comment. We are having a hangout meeting tomorrow (Thursday, 22nd) between 11&12am SF time. Please let us know if you'd like to join. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailprefere

[Wikidata-bugs] [Maniphest] [Edited] T84923: Reliable publish / subscribe event bus

2015-10-21 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T84923 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak

[Wikidata-bugs] [Maniphest] [Updated] T116247: Define edit related events for change propagation

2015-10-21 Thread GWicke
GWicke added a blocked task: T102476: RFC: Requirements for change propagation. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac

[Wikidata-bugs] [Maniphest] [Updated] T84923: Reliable publish / subscribe event bus

2015-10-21 Thread GWicke
GWicke added a blocked task: T102476: RFC: Requirements for change propagation. TASK DETAIL https://phabricator.wikimedia.org/T84923 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans

[Wikidata-bugs] [Maniphest] [Created] T116247: Define edit related events for change propagation

2015-10-21 Thread GWicke
GWicke created this task. GWicke added subscribers: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak, JanZerebecki, bd808, MZMcBride, mobrovac, GWicke, aaron, daniel, Hardikj, yuvipanda. GWicke added projects: operations, EventBus, Discovery, Epic

[Wikidata-bugs] [Maniphest] [Raised Priority] T116247: Define edit related events for change propagation

2015-10-21 Thread GWicke
GWicke raised the priority of this task from "Normal" to "High". GWicke set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: yuvipanda, Hardikj, d

[Wikidata-bugs] [Maniphest] [Edited] T116247: Define edit related events for change propagation

2015-10-21 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair

[Wikidata-bugs] [Maniphest] [Edited] T116247: Define edit related events for change propagation

2015-10-21 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T116247 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: yuvipanda, Hardikj, daniel, aaron, GWicke, mobrovac, MZMcBride, bd808, JanZerebecki, Halfak, Krenair

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-19 Thread GWicke
GWicke added a comment. A PR adding remote schema support to the nodejs frontend is now available at https://github.com/wikimedia/restevent/pull/1. This means that we can now choose to use local or remote schemas per-topic in the configuration. TASK DETAIL https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-16 Thread GWicke
GWicke added a comment. In https://phabricator.wikimedia.org/T114443#1731399, @ori wrote: > In https://phabricator.wikimedia.org/T114443#1731284, @GWicke wrote: > > > See https://phabricator.wikimedia.org/T88459#1604768. tl;dr: It's not > > necessarily clear that saving ver

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-16 Thread GWicke
GWicke added a comment. In https://phabricator.wikimedia.org/T114443#1730753, @Eevans wrote: > 1. Already leverages a (really slick) JSON schema registry > <https://meta.wikimedia.org/wiki/Category:Schemas_%28active%29?status=active> Optionally fetching schemas from a URL isn

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-16 Thread GWicke
GWicke added a comment. > For starters, it means that we have alternatives for environments where Kafka > is overkill (small third-party installations, dev environments, mw-vagrant, > etc). Using, for example, sqlite instead of Kafka is already something > supported. As far

[Wikidata-bugs] [Maniphest] [Updated] T114443: EventBus MVP

2015-10-07 Thread GWicke
GWicke added a project: MediaWiki-RfCs. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac

[Wikidata-bugs] [Maniphest] [Changed Project Column] T114443: EventBus MVP

2015-10-07 Thread GWicke
GWicke moved this task to Ready for RFC meeting on the MediaWiki-RfCs workboard. TASK DETAIL https://phabricator.wikimedia.org/T114443 WORKBOARD https://phabricator.wikimedia.org/project/board/52/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-05 Thread GWicke
GWicke added a comment. I guess we have slightly different ideas about what a message bus should be: 1. a way to get blobs from a to b, and 2. a way to expose a stream of events in a defined format that can be consumed easily by a range of clients. The use cases I care about require 2

[Wikidata-bugs] [Maniphest] [Edited] T114443: EventBus MVP

2015-10-04 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-04 Thread GWicke
GWicke added a comment. @ori, I changed the text to clarify which of those are potential, and which are concrete plans for this quarter. Please follow the provided links if things are still unclear. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] [Edited] T114443: EventBus MVP

2015-10-04 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson, bd808, Joe, dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-04 Thread GWicke
GWicke added a comment. @Nuria, see the task description, heading "Initial use cases". TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson, bd808, Joe, dr0ptp4kt,

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-02 Thread GWicke
GWicke added a comment. @ottomata, main reason would be the ability to work with $simple_queue, $binary_kafka, $amazon_queue and so on without changes in MW code. This isn't so theoretical. We'll want a lighter-weight queue for testing, developers and third party users rather soon. TASK

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-02 Thread GWicke
GWicke added a comment. @ottomata, yes. One of the motivations for having a REST interface is having,... an interface. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: EBernhardson

[Wikidata-bugs] [Maniphest] [Commented On] T114443: EventBus MVP

2015-10-01 Thread GWicke
GWicke added a comment. I have now integrated some of those changes into the description. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: dr0ptp4kt, madhuvishy, Nuria, ori, faidon

[Wikidata-bugs] [Maniphest] [Edited] T114443: EventBus MVP

2015-10-01 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T114443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: dr0ptp4kt, madhuvishy, Nuria, ori, faidon, aaron, GWicke, mobrovac, Halfak, Eevans, Ottomata

[Wikidata-bugs] [Maniphest] [Commented On] T107595: RFC: Multi-Content Revisions

2015-09-25 Thread GWicke
GWicke added a comment. @daniel, your revised version seems to focus even more on implementing storage systems, change propagation etc, rather than defining a data access interface for MediaWiki, which can be backed by services. Could you clarify how you see this relate to ongoing efforts

[Wikidata-bugs] [Maniphest] [Commented On] T84923: Reliable publish / subscribe event bus

2015-08-31 Thread GWicke
GWicke added a comment. Etherpad notes: https://etherpad.wikimedia.org/p/scalable_events_system TASK DETAIL https://phabricator.wikimedia.org/T84923 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: Aklapper, Matanya, Mattflaschen

[Wikidata-bugs] [Maniphest] [Edited] T84923: Reliable publish / subscribe event bus

2015-08-31 Thread GWicke
GWicke edited the task description. TASK DETAIL https://phabricator.wikimedia.org/T84923 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: GWicke Cc: Aklapper, Matanya, Mattflaschen, Ottomata, mmodell, Eevans, chasemp, brion, Krenair, Halfak

[Wikidata-bugs] [Maniphest] [Commented On] T107602: Set up a public interface to the wikidata query service

2015-08-04 Thread GWicke
GWicke added a comment. @jeroendedauw: https://en.wikipedia.org/api/rest_v1/?doc TASK DETAIL https://phabricator.wikimedia.org/T107602 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, GWicke Cc: JohnLewis, hoo, GWicke, greg, Lydia_Pintscher

  1   2   >