[Wikidata-bugs] [Maniphest] [Created] T176927: WDQS updater crashed

[Yurik]

Yurik created this task.Yurik added a project: Wikidata-Query-Service.Herald added a subscriber: Aklapper.Herald added projects: Wikidata, Discovery.
TASK DESCRIPTIONWDQS updater crashed, possibly due to a DNS outage? (everything else seemed fine, professional datacenter)

05:12:21.954 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Backoff overflow, advancing next time to 2017-09-28T03:10:23Z
05:12:21.954 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got no real changes
05:12:21.954 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got only old changes, next is: {"rccontinue":"20170928031020|602559438","continue":"-||"}
05:12:22.126 [main] INFO  o.w.q.r.t.change.RecentChangesPoller - Got 10 changes, from Q38130793@567600187@20170928031020|602559438 to Q24791441@567600192@20170928031021|6025
59447
05:12:23.083 [main] INFO  org.wikidata.query.rdf.tool.Updater - Polled up to 2017-09-28T03:10:23Z (next: 20170928031021|602559448) at (11.7, 10.9, 10.3) updates per second a
nd (823.3, 863.1, 704.6) milliseconds per second
Exception in thread "main" java.lang.RuntimeException: java.net.UnknownHostException: www.wikidata.org: Temporary failure in name resolution
at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:226)
at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:296)
at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:310)
at org.wikidata.query.rdf.tool.change.RecentChangesPoller.nextBatch(RecentChangesPoller.java:140)
at org.wikidata.query.rdf.tool.change.RecentChangesPoller.nextBatch(RecentChangesPoller.java:30)
at org.wikidata.query.rdf.tool.Updater.nextBatch(Updater.java:263)
at org.wikidata.query.rdf.tool.Updater.run(Updater.java:155)
at org.wikidata.query.rdf.tool.Update.main(Update.java:73)
Caused by: java.net.UnknownHostException: www.wikidata.org: Temporary failure in name resolution
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:385)
at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:223)
... 7 moreTASK DETAILhttps://phabricator.wikimedia.org/T176927EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: YurikCc: Smalyshev, Aklapper, Yurik, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Gehel, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T12342: Use user language for namespace names

[Quiddity]

Quiddity added a comment.
Hi! Can we consider re-opening this?  
Even just having the English fallback available would be especially useful for most developers and for many cross-wiki maintainers/admins/stewards/etc, who might not be able to read the local script. 
E.g. when trying to filter RecentChanges or Contributions, or trying to refine a search result.
E.g. specifically I was trying to search for a string in a specific namespace at hewiki, but I ended up having to open a search at another wiki  to find the correct namespace numbers in order to do a URL-hack. 
Like so: https://he.wikipedia.org/w/index.php?search=asdfghjk=מיוחד:חיפוש="">

I'm hoping it's feasible, at least for the standard namespaces, as the mainspace name is already given in the requested language. 
E.g. https://he.wikipedia.org/w/index.php?search=asdfghjk=מיוחד:חיפוש="">

ty!TASK DETAILhttps://phabricator.wikimedia.org/T12342EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: QuiddityCc: Quiddity, matmarex, Ricordisamoa, bzimport, Wikidata-bugs, Yair_rand, Denny, DanielFriesen, daniel, demon, Catrope, Dereckson, Logicwiki, NehalDaveND, Arrbee, Aklapper, TTO, siebrand, Nemo_bis, He7d3r, aude, JackPotte, wikibugs-l-list, Nikerabbit, MuhammadShuaib, LNDDYL, Psychoslave, KartikMistry, Jay8g___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T175109: beta: Wikidata dispatchChanges.php causes a lot of "Wikimedia\Rdbms\DatabaseMysqlBase::unlock failed to release lock"

[greg]

greg added a project: Wikimedia-log-errors.
TASK DETAILhttps://phabricator.wikimedia.org/T175109EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gregCc: Krinkle, thiemowmde, Ladsgroup, aude, Aklapper, hashar, GoranSMilovanovic, QZanden, TerraCodes, Izno, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-09-27T23:17:05Z]  Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)TASK DETAILhttps://phabricator.wikimedia.org/T175741EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot, gerritbot, Sjoerddebruin, Aklapper, EBernhardson, dcausse, Lydia_Pintscher, daniel, Smalyshev, Lea_Lacroix_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Lewizho99, Maathavan, Jdrewniak, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata

[gerritbot]

gerritbot added a comment.
Change 381025 merged by jenkins-bot:
[operations/mediawiki-config@master] Make using CirrusSearch engine default for wbsearchentities on test.wikidata.org

https://gerrit.wikimedia.org/r/381025TASK DETAILhttps://phabricator.wikimedia.org/T175741EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Sjoerddebruin, Aklapper, EBernhardson, dcausse, Lydia_Pintscher, daniel, Smalyshev, Lea_Lacroix_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Lewizho99, Maathavan, Jdrewniak, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175109: beta: Wikidata dispatchChanges.php causes a lot of "Wikimedia\Rdbms\DatabaseMysqlBase::unlock failed to release lock"

[Krinkle]

Krinkle added a comment.
Still happening. 750,000+ log entries in the last 30 days on Beta Cluster. Most are like:

mediawiki	WARNING	wikidatawiki	deployment-tin	Wikimedia\Rdbms\DatabaseMysqlBase::unlock failed to release lock ''

or

mediawiki	WARNING	wikidatawiki	deployment-tin	Wikimedia\Rdbms\DatabaseMysqlBase::unlock failed to release lock ''Wikibase.wikidatawiki.dispatchChanges.''TASK DETAILhttps://phabricator.wikimedia.org/T175109EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: KrinkleCc: Krinkle, thiemowmde, Ladsgroup, aude, Aklapper, hashar, GoranSMilovanovic, QZanden, TerraCodes, Izno, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair, greg___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T106287: [Task] Track descriptions usages separately (Create a new description usage aspect "D")

[gerritbot]

gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T106287EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, eranroz, PokestarFan, jmadler, hoo, daniel, zhuyifei1999, Eloquence, Lydia_Pintscher, Sannita, Ainali, Liuxinyu970226, MZMcBride, Ricordisamoa, Micru, jayvdb, Daniel_Mietchen, Tobi_WMDE_SW, Legoktm, Abraham, greg, Wikidata-bugs, liangent, jeremyb, aude, Bianjiang, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T106287: [Task] Track descriptions usages separately (Create a new description usage aspect "D")

[gerritbot]

gerritbot added a comment.
Change 381140 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/Wikibase@master] Add EntityUsage::DESCRIPTION_USAGE for descriptions

https://gerrit.wikimedia.org/r/381140TASK DETAILhttps://phabricator.wikimedia.org/T106287EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, eranroz, PokestarFan, jmadler, hoo, daniel, zhuyifei1999, Eloquence, Lydia_Pintscher, Sannita, Ainali, Liuxinyu970226, MZMcBride, Ricordisamoa, Micru, jayvdb, Daniel_Mietchen, Tobi_WMDE_SW, Legoktm, Abraham, greg, Wikidata-bugs, liangent, jeremyb, aude, Bianjiang, Aklapper, GoranSMilovanovic, QZanden, Izno, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T173144: Tracking implicit (extensions) usages of Wikidata

[hoo]

hoo added a subtask: T106287: [Task] Track descriptions usages separately (Create a new description usage aspect "D").
TASK DETAILhttps://phabricator.wikimedia.org/T173144EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: hoo, PokestarFan, Halfak, Aklapper, RP88, eranroz, Doc_James, GoranSMilovanovic, QZanden, Winter, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T106287: [Task] Track descriptions usages separately (Create a new description usage aspect "D")

[hoo]

hoo added a parent task: T173144: Tracking implicit (extensions) usages of Wikidata.
TASK DETAILhttps://phabricator.wikimedia.org/T106287EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: eranroz, PokestarFan, jmadler, hoo, daniel, zhuyifei1999, Eloquence, Lydia_Pintscher, Sannita, Ainali, Liuxinyu970226, MZMcBride, Ricordisamoa, Micru, jayvdb, Daniel_Mietchen, Tobi_WMDE_SW, Legoktm, Abraham, greg, Wikidata-bugs, liangent, jeremyb, aude, Bianjiang, Aklapper, GoranSMilovanovic, QZanden, Izno, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Claimed] T106287: [Task] Track descriptions usages separately (Create a new description usage aspect "D")

[hoo]

hoo claimed this task.hoo added a project: Wikidata-Sprint.
TASK DETAILhttps://phabricator.wikimedia.org/T106287EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: eranroz, PokestarFan, jmadler, hoo, daniel, zhuyifei1999, Eloquence, Lydia_Pintscher, Sannita, Ainali, Liuxinyu970226, MZMcBride, Ricordisamoa, Micru, jayvdb, Daniel_Mietchen, Tobi_WMDE_SW, Legoktm, Abraham, greg, Wikidata-bugs, liangent, jeremyb, aude, Bianjiang, Aklapper, GoranSMilovanovic, QZanden, Izno, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T113468: [Task] Use compact representation of diffs in EntityChange.

[hoo]

hoo added a comment.
I suggest replacing the Diff in DiffChange with a new EntityDiffChangedAspects object. Initially both should be present, but only one should be in the serialized form (EntityChange::getSerializedInfo).

I would introduce a new boolean feature flag which changes which of the two are included in the serialization. Once all Client code works with the EntityDiffChangedAspects object, we can change the flag and remove it + the legacy client code N days after.

The new class would probably look something like this:

class EntityDiffChangedAspects {
	private int $formatVersion = 123;
	private array[] $siteLinkChanges = [ 'dewiki' => [ /* means sitelink changed or removed or added */ ], 'enwiki' => [ 'badges' ] ];
	private string[] $labelChanges = [ 'de', 'en' ];
	private string[] $descriptionChanges = [ 'de', 'es' ];
	private string[] $changedStatements = [ 'P31', 'P1337' ];
}TASK DETAILhttps://phabricator.wikimedia.org/T113468EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: PokestarFan, hoo, gerritbot, aude, Aklapper, daniel, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176124: Lua: Make it possible to load all statements for only one property of an entity using mw.wikibase

[Andrei_Stroe]

Andrei_Stroe added a comment.
Yes, it's perfectly OK, because the code I'm going to replace with a call to this function is brutely iterating through the claims table.
Further filtering should be left to the calling module.TASK DETAILhttps://phabricator.wikimedia.org/T176124EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, Andrei_StroeCc: gerritbot, Lydia_Pintscher, eranroz, Aklapper, hoo, Andrei_Stroe, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

Re: [Wikidata] Do you use the Wikidata entity dump dcatap.rdf?

[Stas Malyshev]

Hi!

> is anyone using the Wikidata entity dump dcatap.rdf at
> https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf?
> 
> It is very rarely used and is thus causing us a (probably) undue
> maintenance burden, because of which we plan to remove it.

What's the issue with it? I don't use it but it seems to be part of
standard for dataset descriptions, so I wonder if the issues can be
fixed. I don't know too much about it but from the description is seems
to be very automatable.

-- 
Stas Malyshev
smalys...@wikimedia.org

___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

[Wikidata-bugs] [Maniphest] [Commented On] T173749: QuickStatements to RDF converter

[Pintoch]

Pintoch added a comment.
@Lydia_Pintscher that makes sense. Okay, thank you to you both, we are on the same page! Given all these tickets on the topic I was worried that I had missed something obvious about this issue…TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: PintochCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

Re: [Wikidata] Do you use the Wikidata entity dump dcatap.rdf?

[Jan Ainali]

If it is used, (although rarely) and you are not sure if it is causing you
any undue burden, why remove a metadata description for linked data
recommended by the EU?

Med vänliga hälsningar
Jan Ainali
http://ainali.com

2017-09-27 12:04 GMT+02:00 Marius Hoch :

> Hi folks,
>
> is anyone using the Wikidata entity dump dcatap.rdf at
> https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf?
>
> It is very rarely used and is thus causing us a (probably) undue
> maintenance burden, because of which we plan to remove it.
>
> If anyone is making use of it, please speak up so that we can keep it or
> find a viable alternative.
>
> Cheers,
> Marius
>
> ___
> Wikidata mailing list
> Wikidata@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikidata
>
___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

[Wikidata-bugs] [Maniphest] [Commented On] T173749: QuickStatements to RDF converter

[Lydia_Pintscher]

Lydia_Pintscher added a comment.
Right and I don't think there is desire to change that. But for tools like the primary sources tool that could ingest rdf and then feed data into Wikidata through the usual API I think that'd be ok.TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T173749: QuickStatements to RDF converter

[Smalyshev]

Smalyshev added a comment.
Right now RDF is only a secondary database format, so all imports are supposed to go through Wikibase instance, using one of Wikibase import formats. Using RDF for import AFAIK is not a use case we currently implement.TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T173749: QuickStatements to RDF converter

[Lydia_Pintscher]

Lydia_Pintscher added a comment.
I am not aware of any "standard" but this is the time to make one then I guess ;-)TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch

[Smalyshev]

Smalyshev updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION...[] Figure out how to properly index descriptions (may require some work with analyzers due to different languages, probably need to discuss with @dcausseT176903)...TASK DETAILhttps://phabricator.wikimedia.org/T125500EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: K4-713, Nikki, Lydia_Pintscher, Lea_Lacroix_WMDE, Stashbot, PokestarFan, Ladsgroup, Sjoerddebruin, dcausse, Jonas, gerritbot, JanZerebecki, daniel, thiemowmde, Tobi_WMDE_SW, Smalyshev, Aklapper, StudiesWorld, Deskana, aude, Ricordisamoa, hoo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Avner, Lewizho99, Maathavan, debt, Gehel, Jdrewniak, FloNight, Izno, Wikidata-bugs, jayvdb, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch

[Smalyshev]

Smalyshev added a subtask: T176903: Index wikidata descriptions.
TASK DETAILhttps://phabricator.wikimedia.org/T125500EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: K4-713, Nikki, Lydia_Pintscher, Lea_Lacroix_WMDE, Stashbot, PokestarFan, Ladsgroup, Sjoerddebruin, dcausse, Jonas, gerritbot, JanZerebecki, daniel, thiemowmde, Tobi_WMDE_SW, Smalyshev, Aklapper, StudiesWorld, Deskana, aude, Ricordisamoa, hoo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Avner, Lewizho99, Maathavan, debt, Gehel, Jdrewniak, FloNight, Izno, Wikidata-bugs, jayvdb, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T173749: QuickStatements to RDF converter

[Pintoch]

Pintoch added a comment.
@Smalyshev thanks for your quick reply! Just for clarity, I am not personally working on the PST, I was just trying to find out if there was any established way to use RDF to represent a data import. If that is the case, then other tools could use that format too (for instance, OpenRefine could export datasets to this format). I'd be happy to work on that but I can only do it if a RDF model is agreed on.

It looks like the Munger from wikidata-query-rdf can be used to fill a Wikibase instance, but that is a different use case.TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: PintochCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T173749: QuickStatements to RDF converter

[Smalyshev]

Smalyshev added a comment.
If I understand the use case right, you may not need the full data set, and you won't be actually importing the data into WDQS database. So you can change the RDF model according to your case.

For identifiers, right now they are non-portable and implementation-dependant (see T167759) - except for statement ones, which are just GUIDs. But again I think based on use case that you have, you can choose any consistent scheme (preferably which would return same ID for the same data) as you won't be mixing actual WDQS data and imported data directly.TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T167759: Reference hash is not stable

[Smalyshev]

Smalyshev added a comment.
It would be nice if we could make hashes at least reproducible by other tools... Though it might be non-trivial I admit. But now if some tool wants to generate RDF representation of Wikidata statement, there's basically no way besides using the same PHP code.TASK DETAILhttps://phabricator.wikimedia.org/T167759EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: thiemowmde, daniel, Aklapper, Smalyshev, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Gehel, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T173749: QuickStatements to RDF converter

[Pintoch]

Pintoch added a comment.
@Lydia_Pintscher , @Smalyshev and @Tpt : is there any info about how RDF is expected to behave as an import format for Wikidata? As far as I can tell, the RDF that gets fed into the Query Service is not designed for import at all:


first, there is a lot of redundancy: values are represented by simple values and value nodes, truthy statements are redundant with statement nodes, and other things like that. (this is absolutely not a criticism of the RDF serialization strategy: it totally makes sense as an export format!) So is there any designated subset of the exported triples that data producers would need to emit? I assume that subset would need to be as expressive as possible (so, for instance, the truthy triples would be dropped in favor of the full statement nodes). That is going to be very verbose, right?
second, the identifiers on the nodes are generated by Wikibase: so, how does a data producer picks identifiers? Is it just going to impose its own hashes that Wikibase will have to respect?


It would be great to have something else than QuickStatements to represent a data import, but I still have doubts about why RDF is suitable for that in the first place. The good thing about RDF is that it is a standard, so many tools can deal with it. But given the issues mentioned above, I expect it is going to be quite painful to reuse these tools to produce data in the right schema, as everything is deeply reified. Anyway, if that is the path you have chosen, we need specs please!

Also, it seems that this project uses Java, so may I suggest that the reusable parts go to the Wikidata-Toolkit rather than the Primary Sources Tool? Wikidata-Toolkit has already got RDF export, so it would make sense to have RDF import (from RDF statements to the datamodel representation, say).TASK DETAILhttps://phabricator.wikimedia.org/T173749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: PintochCc: Pintoch, Aklapper, Hjfocs, Tpt, Lydia_Pintscher, Smalyshev, GoranSMilovanovic, Kiailandi, QZanden, dachary, Izno, Wikidata-bugs, aude, Ricordisamoa, Sjoerddebruin, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

[Gehel]

Gehel added a comment.
Doing some quick math:


wdqs eqiad cluster has between 20 and 40 requests per seconds according to our varnish stats
3 servers in that cluster
let's assume requests are routed equally to all servers, that's 10 req/second/server
1.3 GB/sec allocation rate
130MB allocated per request
the calculation above do not take writes into account, to the allocation per request is probably quite a bit lower
130MB / request is high, but not entirely crazy


Note that I also find anecdotal evidence through Google that an allocation rate of 1GB/core is sustainable (this does not match my experience, where I've seen issues much below that).

I need to dig more into G1 documentation to see how to tune generations sizes to optimize for short lived objects (which seems to be our case).TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Triaged] T175919: investigate GC times on wikidata query service

[Smalyshev]

Smalyshev triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

[Smalyshev]

Smalyshev added a comment.
I think we should be careful about cutting down heap size. The peaks is what I am worried about, and Java has a nasty habit of getting stuck once the heap is exhausted. I'd rather it not happen in the middle of the night because somebody run some heavy query...TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176124: Lua: Make it possible to load all statements for only one property of an entity using mw.wikibase

[hoo]

hoo added a comment.
My proposed implementation includes "deprecated" statements as well, I guess this ok here?TASK DETAILhttps://phabricator.wikimedia.org/T176124EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: gerritbot, Lydia_Pintscher, eranroz, Aklapper, hoo, Andrei_Stroe, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T168298: UI for editing a Lexeme's language

[gerritbot]

gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T168298EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Jakob_WMDE, gerritbotCc: gerritbot, WMDE-leszek, Jan_Dittrich, Aklapper, daniel, Lordiis, Cinemantique, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, JGirault, Izno, Wikidata-bugs, aude, Darkdadaah, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T176124: Lua: Make it possible to load all statements for only one property of an entity using mw.wikibase

[gerritbot]

gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T176124EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Lydia_Pintscher, eranroz, Aklapper, hoo, Andrei_Stroe, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176124: Lua: Make it possible to load all statements for only one property of an entity using mw.wikibase

[gerritbot]

gerritbot added a comment.
Change 381028 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[mediawiki/extensions/Wikibase@master] Lua: Add mw.wikibase.getStatements

https://gerrit.wikimedia.org/r/381028TASK DETAILhttps://phabricator.wikimedia.org/T176124EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Lydia_Pintscher, eranroz, Aklapper, hoo, Andrei_Stroe, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata

[gerritbot]

gerritbot added a comment.
Change 381025 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/mediawiki-config@master] Make using CirrusSearch engine default for wbsearchentities on test.wikidata.org

https://gerrit.wikimedia.org/r/381025TASK DETAILhttps://phabricator.wikimedia.org/T175741EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Sjoerddebruin, Aklapper, EBernhardson, dcausse, Lydia_Pintscher, daniel, Smalyshev, Lea_Lacroix_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, Lewizho99, Maathavan, Jdrewniak, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.
If the TTL isn’t too long (I saw a cap of 1 day in the puppet config, is that correct?), then normal expiry is probably enough.TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[Ladsgroup]

Ladsgroup added a comment.
Nope :D We need this for all clients not just Commons.TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.
Okay, in that case we can close this issue now, right?TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T102749: [Task] Implement "Value only constraint"

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE moved this task from ready to go to in current sprint on the Wikidata board.Lucas_Werkmeister_WMDE added a project: Wikidata-Sprint.
TASK DETAILhttps://phabricator.wikimedia.org/T102749WORKBOARDhttps://phabricator.wikimedia.org/project/board/71/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: gerritbot, Lucas_Werkmeister_WMDE, PokestarFan, Mbch331, Yair_rand, Aklapper, Jonaskeutel, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T102749: [Task] Implement "Value only constraint"

[gerritbot]

gerritbot added a comment.
Change 381005 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/WikibaseQualityConstraints@master] Add “used as value only” constraint type

https://gerrit.wikimedia.org/r/381005TASK DETAILhttps://phabricator.wikimedia.org/T102749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Lucas_Werkmeister_WMDE, PokestarFan, Mbch331, Yair_rand, Aklapper, Jonaskeutel, GoranSMilovanovic, QZanden, Agabi10, Izno, Wikidata-bugs, aude___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T102749: [Task] Implement "Value only constraint"

[gerritbot]

gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T102749EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Lucas_Werkmeister_WMDE, PokestarFan, Mbch331, Yair_rand, Aklapper, Jonaskeutel, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176841: Create wikibase/wikiba.se-deploy repo

[Dzahn]

Dzahn added a comment.
New Gerrit repos (projects) might have to be requested on wiki instead, afaict.

https://www.mediawiki.org/wiki/Gerrit/New_repositories/RequestsTASK DETAILhttps://phabricator.wikimedia.org/T176841EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: DzahnCc: hashar, Aklapper, JanZerebecki, hoo, thiemowmde, JeroenDeDauw, Jonas, Addshore, Ivanhercaz, faidon, PokestarFan, greg, mark, Lydia_Pintscher, Dzahn, Ladsgroup, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Zppix, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[akosiaris]

akosiaris added a comment.

In T163922#3639831, @Lucas_Werkmeister_WMDE wrote:
If the TTL isn’t too long (I saw a cap of 1 day in the puppet config, is that correct?), then normal expiry is probably enough.


It doesn't work like that. The time that request can be cached is determined by the headers the response sets. http://book.varnish-software.com/3.0/HTTP.html is a pretty interesting read if you 've never read it before. It's also a rabbithole (ableit not a very big one ;-). In absence of these (like in this case where only the Age header was set) it's not easy to deduce when the page is going to be removed from all existing caches (some of which we don't really control, like the browser cache)

Anyway, I 've purged the caches in order to resolve this faster instead of waiting it out. For the interested the commands were (in that sequence)

varnishadm  ban "obj.status == 301 && req.http.host ~ commons.wikimedia.org"
varnishadm -n frontend  ban "obj.status == 301 && req.http.host ~ commons.wikimedia.org"

Do remember to force refresh to test it as your browser probably has the result cached as well.TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: akosiarisCc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T168298: UI for editing a Lexeme's language

[gerritbot]

gerritbot added a comment.
Change 381000 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/WikibaseLexeme@master] [WIP] Make lexeme language and lexical category editable.

https://gerrit.wikimedia.org/r/381000TASK DETAILhttps://phabricator.wikimedia.org/T168298EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Jakob_WMDE, gerritbotCc: gerritbot, WMDE-leszek, Jan_Dittrich, Aklapper, daniel, Cinemantique, GoranSMilovanovic, QZanden, JGirault, Izno, Wikidata-bugs, aude, Darkdadaah, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Claimed] T176124: Lua: Make it possible to load all statements for only one property of an entity using mw.wikibase

[hoo]

hoo claimed this task.hoo added projects: MediaWiki-extensions-WikibaseClient, Wikidata-Sprint.
TASK DETAILhttps://phabricator.wikimedia.org/T176124EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Lydia_Pintscher, eranroz, Aklapper, hoo, Andrei_Stroe, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[elukey]

elukey added a comment.

In T163922#3639504, @Lucas_Werkmeister_WMDE wrote:
Is there any way to find out which URLs are garbled? Can we look for RW_PROTO in all the cached redirects, or something like that?


There is a way (https://wikitech.wikimedia.org/wiki/Varnish#One-off_purges_.28bans.29) but it is going to be risky if we don't get the purge pattern right, since we might risk to cut too many objects from the cache. I would avoid it if possible (and wait the normal caching expiry workflow), but we can discuss with the Traffic team another approach if you want.TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: elukeyCc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[daniel]

daniel added a comment.
@elukey, @Lucas_Werkmeister_WMDE, @Ladsgroup Thanks for the quick fix!TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

[Gehel]

Gehel added a comment.
For comparison, wdqs2001 (which is seeing much les user traffic, but the same amount of writes) is showing a GC overhead of ~3.2%, allocation rate of 200mb/sec, max GC time of 1.7 seconds.

Obvious solution to improve throughput: stop sending user traffic to our servers :)

More seriously, this indicates that the write traffic does not appear to be the main issue (good news!).

I see some advice in the blazgraph docs for performance tuning, but it seems mostly related to IO and even more to write load. It might make sense to increase the "write retention queue", since reducing disk writes might indirectly reduce memory throughput as well (just guessing).

I was expecting to see some read caches that might be increased, but I don't find them in the docs.TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888

[Addshore]

Addshore updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION...Would it be possible to also access wdqs.svc.eqiad.wmnet on port 

Allowing this would allow me to remove the hard coding of an individual machine added in https://gerrit.wikimedia.org/r/#/c/380974/TASK DETAILhttps://phabricator.wikimedia.org/T176875EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Addshore, Aklapper, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Gehel, Jonas, FloNight, Xmlizer, Izno, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T176875: Allow access to wdqs.svc.eqiad.wmnet on port 8888

[Addshore]

Addshore created this task.Addshore added projects: Wikidata, User-Addshore, WMDE-Analytics-Engineering, Operations, Wikidata-Query-Service.Herald added a subscriber: Aklapper.Herald added a project: Discovery.
TASK DESCRIPTIONAs far as I can tell wdqs.svc.eqiad.wmnet will direct me to an active wdqs server 'always'.
https://github.com/wikimedia/puppet/blob/6698ee49e2f04292ba6f8041aed0f524bcf48753/hieradata/role/common/cache/misc.yaml#L135

Port  was opened on the wdqs servers to allow for internal queries to run with a longer timeout T119941
https://github.com/wikimedia/puppet/blob/6698ee49e2f04292ba6f8041aed0f524bcf48753/hieradata/role/common/cache/misc.yaml#L135
https://github.com/wikimedia/puppet/blob/a0b0f48ca009934342e3710e42c6732994c6fbbd/modules/wdqs/manifests/gui.pp#L15
https://github.com/wikimedia/puppet/blob/a0b0f48ca009934342e3710e42c6732994c6fbbd/modules/wdqs/templates/nginx.erb#L80

Would it be possible to also access wdqs.svc.eqiad.wmnet on port TASK DETAILhttps://phabricator.wikimedia.org/T176875EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Addshore, Aklapper, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, Avner, Zppix, debt, Gehel, Jonas, FloNight, Xmlizer, Izno, jkroll, Smalyshev, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176577: Some values of "Instance of counts" are not updating anymore

[gerritbot]

gerritbot added a comment.
Change 380974 merged by jenkins-bot:
[analytics/wmde/scripts@master] instanceof.php talk to wdqs1003: directly for now

https://gerrit.wikimedia.org/r/380974TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176577: Some values of "Instance of counts" are not updating anymore

[gerritbot]

gerritbot added a comment.
Change 380975 merged by jenkins-bot:
[analytics/wmde/scripts@production] instanceof.php talk to wdqs1003: directly for now

https://gerrit.wikimedia.org/r/380975TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176577: Some values of "Instance of counts" are not updating anymore

[gerritbot]

gerritbot added a comment.
Change 380975 had a related patch set uploaded (by Addshore; owner: Addshore):
[analytics/wmde/scripts@production] instanceof.php talk to wdqs1003: directly for now

https://gerrit.wikimedia.org/r/380975TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, gerritbotCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T176577: Some values of "Instance of counts" are not updating anymore

[gerritbot]

gerritbot added a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Claimed] T176577: Some values of "Instance of counts" are not updating anymore

[Addshore]

Addshore claimed this task.Addshore triaged this task as "High" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176577: Some values of "Instance of counts" are not updating anymore

[gerritbot]

gerritbot added a comment.
Change 380974 had a related patch set uploaded (by Addshore; owner: Addshore):
[analytics/wmde/scripts@master] instanceof.php talk to wdqs1003: directly for now

https://gerrit.wikimedia.org/r/380974TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176577: Some values of "Instance of counts" are not updating anymore

[Lydia_Pintscher]

Lydia_Pintscher added a comment.
I'd say pretty important because it lets us see huge imports happening or changes to the ontology that have huge impacts.
I'd say let's try to go with the internal endpoint if that is ok. If/when that fails we can switch to using the dumps?TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174794: [Task] investigate emails about Wikidata changes from Wikipedia

[IKhitron]

IKhitron added a comment.

In T174794#3639561, @Lydia_Pintscher wrote:
I believe that would be helpful.


enwiki:
Talk page message V, Thanks X, Translations V, Mention V, Failed mention -, Successful mention -, Page link V, Connection with Wikidata X, Failed login attempts V, Login from an unfamiliar device V, Page review V, User rights change V, Edit revert V, Email from other user -, Education Program V

ruwiki:
Talk page message V, Thanks V, Structured Discussion V, Translations V, Mention V, Failed mention -, Successful mention -, Page link V, Connection with Wikidata X, Failed login attempts X, Login from an unfamiliar device X, User rights change V, Edit revert V, Email from other user -, Education Program V

common turned on parts:
Talk page message, Translations, Mention, Page link, User rights change, Edit revert, Education ProgramTASK DETAILhttps://phabricator.wikimedia.org/T174794EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aude, IKhitronCc: jeremyb, Noella94, IKhitron, matej_suchanek, Aklapper, Lydia_Pintscher, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

[gerritbot]

gerritbot added a comment.
Change 380972 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] wdqs: reduce blazegraph heap size to 10GB

https://gerrit.wikimedia.org/r/380972TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T176577: Some values of "Instance of counts" are not updating anymore

[Addshore]

Addshore added a subscriber: Lydia_Pintscher.Addshore added a comment.
I have unassigned myself as the investigation of the cause is now done!

@Lydia_Pintscher what priority is this?TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Lydia_Pintscher, gerritbot, Addshore, Sjoerddebruin, Aklapper, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176577: Some values of "Instance of counts" are not updating anymore

[Sjoerddebruin]

Sjoerddebruin added a comment.
I'm okay with weekly data for this specific case.TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SjoerddebruinCc: gerritbot, Addshore, Sjoerddebruin, Aklapper, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Unassigned] T176577: Some values of "Instance of counts" are not updating anymore

[Addshore]

Addshore removed Addshore as the assignee of this task.Addshore removed a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: gerritbot, Addshore, Sjoerddebruin, Aklapper, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175919: investigate GC times on wikidata query service

[Gehel]

Gehel added a comment.
Looking at 21 hours of GC logs on wdqs1004, I still see 1.3GB/sec allocation rate (almost 100TB allocated over 21 hours!). GC overhead is still high at ~8%. It still looks like the heap is oversized, so I'll try to reduce more to see if that increases throughput (but I doubt it will improve enough). We had 5 stop the world events longer than 10 seconds (all ~30 seconds) over the last 21h.TASK DETAILhttps://phabricator.wikimedia.org/T175919EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Stashbot, gerritbot, Smalyshev, Gehel, Aklapper, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, debt, Jonas, FloNight, Xmlizer, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T176577: Some values of "Instance of counts" are not updating anymore

[Addshore]

Addshore added a comment.
addshore@stat1005:~$ cat /srv/analytics-wmde/graphite/log/daily.03.log |grep wikidata-sparql-instanceof |grep Q11173
2017-09-26 03:01:59 wikidata-sparql-instanceof Result for: Q11173 null
2017-09-27 03:02:07 wikidata-sparql-instanceof WikidataInstanceOf::doSparqlQuery: PREFIX wd: PREFIX wdt: SELECT (count(distinct(?s)) AS ?scount) WHERE {?s wdt:P31/wdt:P279* wd:Q11173} [{"http_code":"HTTP\/1.1 500 Server Error","Date":"Wed, 27 Sep 2017 03:02:07 GMT","Content-Type":"text\/plain","Content-Length":"3093","Connection":"keep-alive","Server":"nginx\/1.11.13","X-Served-By":"wdqs1004","Access-Control-Allow-Origin":"*","Vary":"Accept-Encoding","X-Varnish":"21878624, 304358865","Via":"1.1 varnish-v4, 1.1 varnish-v4","Age":"0","X-Cache":"cp1051 miss, cp1058 miss","X-Cache-Status":"miss","Strict-Transport-Security":"max-age=106384710; includeSubDomains; preload","Set-Cookie":"WMF-Last-Access-Global=27-Sep-2017;Path=\/;Domain=.wikidata.org;HttpOnly;secure;Expires=Sun, 29 Oct 2017 00:00:00 GMT","X-Analytics":"https=1;nocookies=1","X-Client-IP":"2620:0:861:108:1a66:daff:fefa:bc4"},"SPARQL-QUERY: queryStr=PREFIX wd: PREFIX wdt: SELECT (count(distinct(?s)) AS ?scount) WHERE {?s wdt:P31\/wdt:P279* wd:Q11173}\njava.util.concurrent.TimeoutException\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:205)\n\tat com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:293)\n\tat com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:654)\n\tat com.bigdata.rdf.sail.webapp.QueryServlet.doGet(QueryServlet.java:288)\n\tat com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240)\n\tat com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:687)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)\n\tat org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:244)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)\n\tat ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)\n\tat org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:42)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:497)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)\n\tat java.lang.Thread.run(Thread.java:748)\n"]

So it does indeed look like these are timing out.

We have 2 options:


Rewrite this so that the numbers come from the dumps instead of from the SPARQL endpoint. This will mean only having weekly data
Use the open port (for the analytics cluster only) that allows for longer running SPARQL queries. This will mean we keep the daily data. I guess one day in the future we might run into the same problem though. (Relevant ticket T119941)
TASK DETAILhttps://phabricator.wikimedia.org/T176577EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: gerritbot, Addshore, Sjoerddebruin, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, 

[Wikidata-bugs] [Maniphest] [Edited] T166892: Implement “Reference” constraint

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION~~Extremely trivial (always violated, because we only check the main value) and~~ completely analogous to “Qualifier”TASK DETAILhttps://phabricator.wikimedia.org/T166892EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: Aklapper, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T168532: Check constraints on qualifiers and references

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a subtask: T166892: Implement “Reference” constraint.
TASK DETAILhttps://phabricator.wikimedia.org/T168532EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: abian, Ivan_A_Krestinin, gerritbot, PokestarFan, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Updated] T166892: Implement “Reference” constraint

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a parent task: T168532: Check constraints on qualifiers and references.
TASK DETAILhttps://phabricator.wikimedia.org/T166892EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: Aklapper, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174794: [Task] investigate emails about Wikidata changes from Wikipedia

[Lydia_Pintscher]

Lydia_Pintscher added a comment.
I believe that would be helpful.TASK DETAILhttps://phabricator.wikimedia.org/T174794EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aude, Lydia_PintscherCc: jeremyb, Noella94, IKhitron, matej_suchanek, Aklapper, Lydia_Pintscher, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.
Is there any way to find out which URLs are garbled? Can we look for RW_PROTO in all the cached redirects, or something like that?TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T174794: [Task] investigate emails about Wikidata changes from Wikipedia

[IKhitron]

IKhitron added a comment.

In T174794#3636534, @Noella94 wrote:
Hi, What is common about those receiving those mails? The mails are sent from other wikis due to wikidata changes but the mails are not sent to all wikimedians (users).


Maybe we should just put here the list of our email preferences. Those that will be turned off for all affected peopls will give the answer. I can bring two lists, from enwiki and ruwiki.TASK DETAILhttps://phabricator.wikimedia.org/T174794EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aude, IKhitronCc: jeremyb, Noella94, IKhitron, matej_suchanek, Aklapper, Lydia_Pintscher, GoranSMilovanovic, QZanden, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T175561: Check distinct values (unique value) constraint on qualifiers and references

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.
Two semantics questions:


Is it a constraint violation if the same value is used in a statement and elsewhere in a qualifier? Or should values only be distinct between all main snaks / all qualifiers / all references? (My current WIP implementation only checks uniqueness within one “kind” of snak – main snak, qualifier, or reference.)
Is it a constraint violation if the exact same reference (same hash, same RDF node) is used several times (on the same entity or on different entities) and includes a “distinct values” property?
TASK DETAILhttps://phabricator.wikimedia.org/T175561EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: gerritbot, Lucas_Werkmeister_WMDE, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Edited] T168532: Check constraints on qualifiers and references

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTION...- [ ] #patch-for-review: [Add config for checking constraints on qualifiers and references](https://gerrit.wikimedia.org/r/#/c/376290/) ~~(might be the same as the one for switching to the new API format)~~
- [ ] #patch-for-review: [Add support for checking constraints on qualifiers and references to extension](https://gerrit.wikimedia.org/r/#/c/379746/)
- [ ] #patch-for-review: [Add support for checking constraints on qualifiers and references to extensiongadget](https://gerrit.wikimedia.org/r/#/c/379746/7/)
- [ ] [Add support for checkingEnable constraintsfig on qualifiers and references to gadget](https://gerrit.wikimedia.org/r/#/c/379747/)Wikidata: {T176863}
- [ ] Remove support for old API format (and config setting)
- [ ] Default config for checking constraints on qualifiers and references to `true`
- …TASK DETAILhttps://phabricator.wikimedia.org/T168532EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: abian, Ivan_A_Krestinin, gerritbot, PokestarFan, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Block] T168532: Check constraints on qualifiers and references

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE created subtask T176863: Enable constraint checks on qualifiers and references on Wikidata.
TASK DETAILhttps://phabricator.wikimedia.org/T168532EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: abian, Ivan_A_Krestinin, gerritbot, PokestarFan, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Aklapper, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T176863: Enable constraint checks on qualifiers and references on Wikidata

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE created this task.Lucas_Werkmeister_WMDE added projects: Wikidata, Wikibase-Quality, Wikibase-Quality-Constraints, Wikidata-Sprint, Patch-For-Review.Herald added a subscriber: Aklapper.Herald removed a project: Patch-For-Review.
TASK DESCRIPTIONOnce all the code for T168532: Check constraints on qualifiers and references is ready and deployed, and once T175590: Switch to new wbcheckconstraints API format on Wikidata is active on Wikidata, we also need to enable constraint checks on qualifiers and references with the following config snippet:

$wgWBQualityConstraintsCheckQualifiers = true;
$wgWBQualityConstraintsCheckReferences = true;

(I don’t think it makes sense to enable them separately, though we might want to disable one of them if it turns out the results are unexpectedly bad.)

We probably don’t want to do this on the same day as T175590: Switch to new wbcheckconstraints API format on Wikidata (Tuesday, 2017-10-10), but on the other hand I don’t think we need to wait a lot longer with it either. Perhaps do it on Wednesday or Thursday? @Ladsgroup would you be available to deploy this? (Assuming, of course, that the required changes are reviewed and deployed by then.)TASK DETAILhttps://phabricator.wikimedia.org/T176863EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDECc: Aklapper, Lucas_Werkmeister_WMDE, Ladsgroup, GoranSMilovanovic, QZanden, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

Re: [Wikidata] Encoders/feature extractors for neural nets

[John Erling Blad]

The most important thing for my problem would be to encode quantity and
geopos. The test case is lake sizes to encode proper localized descriptions.

Unless someone already have a working solution I would encode this as
sparse logarithmic vectors, probably also with log of pairwise differences.

Encoding of qualifiers is interesting, but would require encoding of a
topic map, and that adds an additional layer of complexity.

How to encode the values are not so much the problem, but avoiding
reimplementing this yet another time… ;)

On Wed, Sep 27, 2017 at 1:23 PM, Thomas Pellissier Tanon <
tho...@pellissier-tanon.fr> wrote:

> Just an idea of a very sparse but hopefully not so bad encoding (I have
> not actually tested it).
>
> NB: I am going to use a lot the terms defined in the glossary [1].
>
> A value could be encoded by a vector:
> - for entity ids it is a vector V that have the dimension of the number of
> existing entities such that V[q] = 1 if, and only if, it is the entity q
> and V[q] = 0 if not.
> - for time : a vector with year, month, day, hours, minutes, seconds,
> is_precision_year, is_precision_month, ..., is_gregorian, is_julian (or
> something similar)
> - for geo coordinates latitude, longitude, is_earth, is_moon...
> - string/language strings: an encoding depending on your use case
> ...
> Example : To encode "Q2" you would have the vector {0,1,0}
> To encode the year 2000 you would have {2000,0..., is_precision_decade =
> 0,is_precision_year=1,is_precision_month=0,...,is_gregorian=true,...}
>
> To encode a snak you build a big vector by concatenating the vector of the
> value if it is P1, if it is P2... (you use the property datatype to pick a
> good vector shape) + you add two cells per property to encode is_novalue,
> is_somevalue. To encode "P31: Q5" you would have a vector V =
> {0,,0,0,0,0,1,0,} with 1 only for  V[P31_offset + Q5_offset]
>
> To encode a claim you could concatenate the main snak vector + the
> qualifiers vectors that is the merge of the snak vector for all qualifiers
> (i.e. you build the vector for all snak and you sum them) such that the
> qualifier vectors encode all qualifiers at the same time. it allows to
> check that a qualifiers is set just by picking the right cell in the
> vector. But it will do bad things if there are two qualifiers with the same
> property and having a datatype like time or geocoordinates. But I don't
> think it really a problem.
> Example: to encode the claim with "P31: Q5" main snak and qualifiers "P42:
> Q42, P42: Q44" we would have a vector V such that V[P31_offset + Q5_offset]
> = 1, V[qualifiers_offset + P42_offset + Q42_offset] = 1 and
> V[qualifiers_offset + P42_offset + Q44_offset] = 1 and 0 elsewhere.
>
> I am not sure how to encode statements references (merge all of them and
> encode it just like the qualifiers vector is maybe a first step but is bad
> if we have multiple references).  For the rank you just need 3 booleans
> is_preferred, is_normal and is_deprecated.
>
> Cheers,
>
> Thomas
>
> [1] https://www.wikidata.org/wiki/Wikidata:Glossary
>
>
> > Le 27 sept. 2017 à 12:41, John Erling Blad  a écrit :
> >
> > Is there anyone that has done any work on how to encode statements as
> features for neural nets? I'm mostly interested in sparse encoders for
> online training of live networks.
> >
> >
> > ___
> > Wikidata mailing list
> > Wikidata@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikidata
>
>
> ___
> Wikidata mailing list
> Wikidata@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikidata
>
>
___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

[Wikidata-bugs] [Maniphest] [Closed] T176771: Enable ArticlePlaceholder on bnwiki

[hoo]

hoo closed this task as "Resolved".hoo added a comment.
Deployed now: https://bn.wikipedia.org/wiki/%E0%A6%AC%E0%A6%BF%E0%A6%B6%E0%A7%87%E0%A6%B7:AboutTopic/Q53TASK DETAILhttps://phabricator.wikimedia.org/T176771EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Stashbot, gerritbot, Jayprakash12345, Lucie, Lea_Lacroix_WMDE, Lydia_Pintscher, hoo, Mahir256, Aklapper, Aftabuzzaman, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, cmadeo, Lewizho99, Maathavan, Bodhisattwa, Izno, Wikidata-bugs, aude, Pmlineditor, jayvdb, Ricordisamoa, Praveenp, Mbch331, jayantanth___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Triaged] T176858: Represent Structured Data on Commons at WikidataCon 2017 in Berlin

[SandraF_WMF]

SandraF_WMF triaged this task as "Normal" priority.
TASK DETAILhttps://phabricator.wikimedia.org/T176858EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Aklapper, SandraF_WMF, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T176771: Enable ArticlePlaceholder on bnwiki

[hoo]

hoo moved this task from Doing to Done on the ArticlePlaceholder board.hoo removed a project: Patch-For-Review.
TASK DETAILhttps://phabricator.wikimedia.org/T176771WORKBOARDhttps://phabricator.wikimedia.org/project/board/1416/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Stashbot, gerritbot, Jayprakash12345, Lucie, Lea_Lacroix_WMDE, Lydia_Pintscher, hoo, Mahir256, Aklapper, Aftabuzzaman, GoranSMilovanovic, QZanden, cmadeo, Bodhisattwa, Izno, Wikidata-bugs, aude, Pmlineditor, jayvdb, Ricordisamoa, Praveenp, Mbch331, jayantanth___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata] Do you use the Wikidata entity dump dcatap.rdf?

[Marius Hoch]

Hi folks,

is anyone using the Wikidata entity dump dcatap.rdf at 
https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf?


It is very rarely used and is thus causing us a (probably) undue 
maintenance burden, because of which we plan to remove it.


If anyone is making use of it, please speak up so that we can keep it or 
find a viable alternative.


Cheers,
Marius

___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

[Wikidata-bugs] [Maniphest] [Created] T176858: Represent Structured Data on Commons at WikidataCon 2017 in Berlin

[SandraF_WMF]

SandraF_WMF created this task.SandraF_WMF added projects: Community-Liaisons (Oct-Dec 2017), Structured-Data-Commons.Herald added a subscriber: Aklapper.Herald added a project: Wikidata.
TASK DESCRIPTIONGive a presentation about Structured Data on Commons at WikidataCon, and interact/network with many interested participants there!TASK DETAILhttps://phabricator.wikimedia.org/T176858WORKBOARDhttps://phabricator.wikimedia.org/project/board/2706/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Aklapper, SandraF_WMF, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176771: Enable ArticlePlaceholder on bnwiki

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-09-27T12:06:48Z]  Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T176771EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, StashbotCc: Stashbot, gerritbot, Jayprakash12345, Lucie, Lea_Lacroix_WMDE, Lydia_Pintscher, hoo, Mahir256, Aklapper, Aftabuzzaman, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, cmadeo, Lewizho99, Maathavan, Bodhisattwa, Izno, Wikidata-bugs, aude, Pmlineditor, jayvdb, Ricordisamoa, Praveenp, Mbch331, jayantanth___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176771: Enable ArticlePlaceholder on bnwiki

[Stashbot]

Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-09-27T12:05:34Z]  Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)TASK DETAILhttps://phabricator.wikimedia.org/T176771EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, StashbotCc: Stashbot, gerritbot, Jayprakash12345, Lucie, Lea_Lacroix_WMDE, Lydia_Pintscher, hoo, Mahir256, Aklapper, Aftabuzzaman, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, cmadeo, Lewizho99, Maathavan, Bodhisattwa, Izno, Wikidata-bugs, aude, Pmlineditor, jayvdb, Ricordisamoa, Praveenp, Mbch331, jayantanth___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T176771: Enable ArticlePlaceholder on bnwiki

[gerritbot]

gerritbot added a comment.
Change 380793 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ArticlePlaceholder on bnwiki

https://gerrit.wikimedia.org/r/380793TASK DETAILhttps://phabricator.wikimedia.org/T176771EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Jayprakash12345, Lucie, Lea_Lacroix_WMDE, Lydia_Pintscher, hoo, Mahir256, Aklapper, Aftabuzzaman, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, cmadeo, Lewizho99, Maathavan, Bodhisattwa, Izno, Wikidata-bugs, aude, Pmlineditor, jayvdb, Ricordisamoa, Praveenp, Mbch331, jayantanth___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T173840: Send out Structured Data on Commons Newsletter - The October 2017 Edition

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T173840WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Aklapper, Sadads, PokestarFan, Elitre, SandraF_WMF, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Retitled] T173840: Send out Structured Data on Commons Newsletter - The October 2017 Edition

[SandraF_WMF]

SandraF_WMF renamed this task from "Send out Structured Data on Commons Newsletter - The Post-Wikimania 2017 Edition" to "Send out Structured Data on Commons Newsletter - The October 2017 Edition".
TASK DETAILhttps://phabricator.wikimedia.org/T173840EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Aklapper, Sadads, PokestarFan, Elitre, SandraF_WMF, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T173924: Revisit old Structured Data on Commons documentation (2014) + summarize and archive

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T173924WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, SandraF_WMF, Aklapper, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Jay8g, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Closed] T173945: Create basic communication and partner outreach plan for Structured Data on Commons

[SandraF_WMF]

SandraF_WMF closed this task as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T173945EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, BVershbow_WMF, SandraF_WMF, Aklapper, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T173945: Create basic communication and partner outreach plan for Structured Data on Commons

[SandraF_WMF]

SandraF_WMF added a comment.
The first version of the plan can be considered READY. It will need lots of updating as our experience with the project grows, but the basis is there!TASK DETAILhttps://phabricator.wikimedia.org/T173945EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, BVershbow_WMF, SandraF_WMF, Aklapper, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T174142: Build up and facilitate a focus group of Commons and Wikidata contributors for (long-term) feedback and input on Structured Data on Common

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T174142WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Raymond, SandraF_WMF, Aklapper, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T174928: Initiate v1 of a glossary about Structured Data on Commons.

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T174928WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, PokestarFan, Elitre, Aklapper, Ramsey-WMF, SandraF_WMF, E1presidente, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Jay8g, Keegan, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T174168: Update the Frequently Asked Questions (FAQ) pages about Structured Data on Commons

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T174168WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, PokestarFan, Elitre, Aklapper, Ramsey-WMF, SandraF_WMF, E1presidente, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Jay8g, Keegan, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T173971: Get community input on tools, bots and workflows on Wikimedia Commons that will likely be affected by Structured Data on Commons.

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T173971WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Raymond, SandraF_WMF, Aklapper, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T174134: Build up and facilitate a focus group of GLAM representatives for (long-term) feedback on Structured Data on Commons

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T174134WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Aklapper, BVershbow_WMF, Sadads, SandraF_WMF, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T173979: Get community input on a wide variety of interesting Commons files that can be used as examples to prepare for metadata modelling

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T173979WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: PKM, Deskana, EBernhardson, Abit, El_Grafo, Yann, Keegan, PierreSelim, SandraF_WMF, Aklapper, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Wikidata-bugs, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Changed Project Column] T173978: Collect and get community input on user stories related to Structured Data on Commons

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).
TASK DETAILhttps://phabricator.wikimedia.org/T173978WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Ramsey-WMF, Aklapper, SandraF_WMF, E1presidente, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, JGirault, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Lowered Priority] T173820: Cleanup and update drive of information pages / portal about Structured Data on Commons

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).SandraF_WMF lowered the priority of this task from "High" to "Normal".
TASK DETAILhttps://phabricator.wikimedia.org/T173820WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Ramsey-WMF, Aklapper, Elitre, PokestarFan, SandraF_WMF, E1presidente, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Jay8g, Keegan, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Lowered Priority] T174565: Gather first community input and create focus groups for Structured Data on Wikimedia Commons

[SandraF_WMF]

SandraF_WMF moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Community-Liaisons board.SandraF_WMF edited projects, added Community-Liaisons (Oct-Dec 2017); removed Community-Liaisons (Jul-Sep 2017).SandraF_WMF lowered the priority of this task from "High" to "Normal".
TASK DETAILhttps://phabricator.wikimedia.org/T174565WORKBOARDhttps://phabricator.wikimedia.org/project/board/27/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SandraF_WMFCc: Abit, Aklapper, SandraF_WMF, E1presidente, Ramsey-WMF, Jmmuguerza, GoranSMilovanovic, Ivana_Isadora, QZanden, Acer, Jseddon, FloNight, Trizek-WMF, Susannaanas, Izno, Yann, Wikidata-bugs, PKM, Base, matthiasmullie, aude, Ricordisamoa, Fabrice_Florin, Raymond, Mbch331, Keegan, Elitre, Qgil___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

Re: [Wikidata] Encoders/feature extractors for neural nets

[Thomas Pellissier Tanon]

Just an idea of a very sparse but hopefully not so bad encoding (I have not 
actually tested it).

NB: I am going to use a lot the terms defined in the glossary [1].

A value could be encoded by a vector:
- for entity ids it is a vector V that have the dimension of the number of 
existing entities such that V[q] = 1 if, and only if, it is the entity q and 
V[q] = 0 if not.
- for time : a vector with year, month, day, hours, minutes, seconds, 
is_precision_year, is_precision_month, ..., is_gregorian, is_julian (or 
something similar)
- for geo coordinates latitude, longitude, is_earth, is_moon...
- string/language strings: an encoding depending on your use case
...
Example : To encode "Q2" you would have the vector {0,1,0}
To encode the year 2000 you would have {2000,0..., is_precision_decade = 
0,is_precision_year=1,is_precision_month=0,...,is_gregorian=true,...}

To encode a snak you build a big vector by concatenating the vector of the 
value if it is P1, if it is P2... (you use the property datatype to pick a good 
vector shape) + you add two cells per property to encode is_novalue, 
is_somevalue. To encode "P31: Q5" you would have a vector V = 
{0,,0,0,0,0,1,0,} with 1 only for  V[P31_offset + Q5_offset]

To encode a claim you could concatenate the main snak vector + the qualifiers 
vectors that is the merge of the snak vector for all qualifiers (i.e. you build 
the vector for all snak and you sum them) such that the qualifier vectors 
encode all qualifiers at the same time. it allows to check that a qualifiers is 
set just by picking the right cell in the vector. But it will do bad things if 
there are two qualifiers with the same property and having a datatype like time 
or geocoordinates. But I don't think it really a problem.
Example: to encode the claim with "P31: Q5" main snak and qualifiers "P42: Q42, 
P42: Q44" we would have a vector V such that V[P31_offset + Q5_offset] = 1, 
V[qualifiers_offset + P42_offset + Q42_offset] = 1 and V[qualifiers_offset + 
P42_offset + Q44_offset] = 1 and 0 elsewhere.

I am not sure how to encode statements references (merge all of them and encode 
it just like the qualifiers vector is maybe a first step but is bad if we have 
multiple references).  For the rank you just need 3 booleans is_preferred, 
is_normal and is_deprecated.

Cheers,

Thomas

[1] https://www.wikidata.org/wiki/Wikidata:Glossary


> Le 27 sept. 2017 à 12:41, John Erling Blad  a écrit :
> 
> Is there anyone that has done any work on how to encode statements as 
> features for neural nets? I'm mostly interested in sparse encoders for online 
> training of live networks.
> 
> 
> ___
> Wikidata mailing list
> Wikidata@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikidata



signature.asc
Description: Message signed with OpenPGP
___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata

[Wikidata-bugs] [Maniphest] [Changed Subscribers] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[elukey]

elukey added a subscriber: ema.elukey added a comment.
I believe this is only a matter of cleaning up urls that show up garbled, @ema just did it for https://commons.wikimedia.org/data/main/Data:Bundestagswahl2017/wahlkreis46.map via https://wikitech.wikimedia.org/wiki/Multicast_HTCP_purging#One-off_purge)TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: elukeyCc: ema, elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Lowered Priority] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content

[elukey]

elukey lowered the priority of this task from "Unbreak Now!" to "Normal".
TASK DETAILhttps://phabricator.wikimedia.org/T163922EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: elukeyCc: elukey, TerraCodes, Jay8g, Liuxinyu970226, Lucas_Werkmeister_WMDE, Dereckson, akosiaris, fgiunchedi, Stashbot, gerritbot, Ladsgroup, Aklapper, daniel, Lordiis, Adik2382, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, Zoranzoki21, Lewizho99, Maathavan, DatGuy, Devwaker, Urbanecm, JEumerus, Tulsi_Bhagat, suriyaa, Luke081515, biplabanand, Wikidata-bugs, Snowolf, Dcljr, Southparkfan, Jdforrester-WMF, Matanya, Rxy, Glaisher, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Created] T176852: Redirects should not be in wb_terms table

[Ladsgroup]

Ladsgroup created this task.Ladsgroup added projects: Wikidata, MediaWiki-extensions-WikibaseRepository.Herald added a subscriber: Aklapper.
TASK DESCRIPTIONFound them while checking the table:

mysql:wikiadmin@db1092 [wikidatawiki]> select * from wb_terms where term_full_entity_id is null order by term_entity_id asc limit 100;
+-++-+--+---+-+++-+
| term_row_id | term_entity_id | term_full_entity_id | term_entity_type | term_language | term_type   | term_text  | term_search_key| term_weight |
+-++-+--+---+-+++-+
|   512962682 |3091008 | NULL| item | fr| alias   | Caméléon verruqueux| caméléon verruqueux|   0 |
|   512962687 |3091008 | NULL| item | ceb   | alias   | Chamaeleon monilifer   | chamaeleon monilifer   |   0 |
|   512962686 |3091008 | NULL| item | fr| alias   | Furcifer verrucosus verrucosus | furcifer verrucosus verrucosus |   0 |
|   512962685 |3091008 | NULL| item | fr| alias   | Furcifer verrucosus semicristatus  | furcifer verrucosus semicristatus  |   0 |
|   512962684 |3091008 | NULL| item | fr| alias   | Chamaeleo verrucosus   | chamaeleo verrucosus   |   0 |
|   512962683 |3091008 | NULL| item | fr| alias   | Chamaeleon semicristatus   | chamaeleon semicristatus   |   0 |
|   512962688 |3091008 | NULL| item | ceb   | alias   | Chamaeleo verrucosus   | chamaeleo verrucosus   |   0 |
|   648180164 |3411758 | NULL| item | zh-hant   | description | 維基媒體消歧義頁   | 維基媒體消歧義頁   |   0 |
|   648180157 |3411758 | NULL| item | gl| description | páxina de homónimos de Wikimedia   | páxina de homónimos de wikimedia   |   0 |
|   648180160 |3411758 | NULL| item | nds   | description | Sied för en mehrdüdig Begreep op Wikimedia | sied för en mehrdüdig begreep op wikimedia |   0 |
|   648180156 |3411758 | NULL| item | cs| description | rozcestník na projektech Wikimedia | rozcestník na projektech wikimedia |   0 |
|   648180158 |3411758 | NULL| item | ja| description | ウィキメディアの曖昧さ回避ページ   | ウィキメディアの曖昧さ回避ページ   |   0 |
|   648180155 |3411758 | NULL| item | ru| description | страница значений в проекте Викимедиа  | страница значений в проекте викимедиа  |   0 |
|   648180161 |3411758 | NULL| item | nl| description | Wikimedia-doorverwijspagina| wikimedia-doorverwijspagina|   0 |
|   648180154 |3411758 | NULL| item | en| description | Wikimedia disambiguation page

All are redirects.TASK DETAILhttps://phabricator.wikimedia.org/T176852EMAIL 

[Wikidata] WikiPedia to WikiData URIs

[Timothy Holborn]

any chance of converting
https://en.wikipedia.org/wiki/Linked_data#Principles to say -
https://en.wikidata.org/wiki/Linked_data#Principles via automated tools?

or perhaps some sort  of agent method that adds literals from wikipedia for
an RDF agent?

Or some other similar method / idea?

Tim.H.
___
Wikidata mailing list
Wikidata@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata