Re: [Wikidata] WDQS updater switched to Kafka

[Guillaume Lederrey]

On Wed, Mar 7, 2018 at 6:47 PM, Yuri Astrakhan  wrote:
> Thanks Stas. How does this affect non-WMF clones of Wikidata QS?

It does not affect non-WMF clones. The previous recent change poller
is still part of the code. And we still use it on our test servers. We
have no plans at this point to remove it. So you should be all good!

> On Tue, Mar 6, 2018 at 2:51 PM, Stas Malyshev 
> wrote:
>>
>> Hi!
>>
>> This morning we have switched the polling mechanism for Wikidata Query
>> Service from using Recent Changes API to using Kafka events
>> (https://wikitech.wikimedia.org/wiki/EventBus). This should improve
>> reliability of the updates and reduce load on Wikidata servers. Given
>> that we are putting more data into Wikidata and more servers for SPARQL
>> services, having a scalable update mechanism is important, and that's
>> what we are doing with Kafka updater.
>>
>> If everything works well, you should not notice any change at all.
>>
>> So far, we have inspected it in action for a few days and it seems to be
>> working fine. However, since the functionality is new, it is possible
>> that some bugs exist. The bugs might manifest themselves in certain new
>> edits not being reflected in Wikidata Query Service data. If you notice
>> this happening, please tell me (preferably via Phabricator ticket
>> specifying the details of the problem, but if it doesn't work for you
>> you can email me or ping me on IRC or on Wiki: Smalyshev_(WMF)).
>>
>> Thanks,
>> --
>> Stas Malyshev
>> smalys...@wikimedia.org
>>
>> ___
>> Wikidata mailing list
>> Wikidata@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikidata
>
>
>
> ___
> Wikidata mailing list
> Wikidata@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikidata
>



-- 
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation
UTC+2 / CEST

___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

Re: [Wikidata] WDQS updater switched to Kafka

[Yuri Astrakhan]

Thanks Stas. How does this affect non-WMF clones of Wikidata QS?

On Tue, Mar 6, 2018 at 2:51 PM, Stas Malyshev 
wrote:

> Hi!
>
> This morning we have switched the polling mechanism for Wikidata Query
> Service from using Recent Changes API to using Kafka events
> (https://wikitech.wikimedia.org/wiki/EventBus). This should improve
> reliability of the updates and reduce load on Wikidata servers. Given
> that we are putting more data into Wikidata and more servers for SPARQL
> services, having a scalable update mechanism is important, and that's
> what we are doing with Kafka updater.
>
> If everything works well, you should not notice any change at all.
>
> So far, we have inspected it in action for a few days and it seems to be
> working fine. However, since the functionality is new, it is possible
> that some bugs exist. The bugs might manifest themselves in certain new
> edits not being reflected in Wikidata Query Service data. If you notice
> this happening, please tell me (preferably via Phabricator ticket
> specifying the details of the problem, but if it doesn't work for you
> you can email me or ping me on IRC or on Wiki: Smalyshev_(WMF)).
>
> Thanks,
> --
> Stas Malyshev
> smalys...@wikimedia.org
>
> ___
> Wikidata mailing list
> Wikidata@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikidata
>
___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata