[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
Hadyelsahar added a comment. @hoo Your help is much appreciated :) thanks a lotTASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, HadyelsaharCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
hoo added a comment. The first truthy nt dump should appear next Tuesday (probably late UTC). I'll keep this open until we actually have it.TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 348096 merged by ArielGlenn: [operations/puppet@production] Create truthy nt Wikidata entity dump each Monday https://gerrit.wikimedia.org/r/348096TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 348095 merged by ArielGlenn: [operations/puppet@production] Wikidata entity dumps: Allow nt RDF dumps https://gerrit.wikimedia.org/r/348095TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 347838 merged by ArielGlenn: [operations/puppet@production] Allow running two dumpwikidatattl dumps side by side https://gerrit.wikimedia.org/r/347838TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 347234 merged by ArielGlenn: [operations/puppet@production] Change dumpwikidatattl to allow producing other flavors https://gerrit.wikimedia.org/r/347234TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 348096 had a related patch set uploaded (by Hoo man): [operations/puppet@production] Create truthy nt Wikidata entity dump each Monday https://gerrit.wikimedia.org/r/348096TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 348095 had a related patch set uploaded (by Hoo man): [operations/puppet@production] Wikidata entity dumps: Allow nt RDF dumps https://gerrit.wikimedia.org/r/348095TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
hoo added a comment. In T155103#3177252, @Smalyshev wrote: Should truthy dump also include full property definitions? Because if you use only /prop/direct/ there's not much use to include other predicates, though technically it doesn't hurt anything. I guess we don't strictly need if for now. Adding things later on is trivial, so we could just go with this and expand it if there's an actual need for these?TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
Smalyshev added a comment. Should truthy dump also include full property definitions? Because if you use only /prop/direct/ there's not much use to include other predicates, though technically it doesn't hurt anything. @Hadyelsahar Not sure what you mean by "skipped". It's normal for labels to be encoded with \u sequences, since not all tools can handle all Unicode properly, unfortunately. Any tool that reads TTL should be able to handle encoded sequences though.TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, SmalyshevCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
Hadyelsahar added a comment. thanks a lot for the help it looks ok, only a small question . is it normal to have UTF labels being skipped inside the ASCII like that ?, cant we just output everything in UTF-8 or 16 "\u30DD\u30BA\u30CA\u30F3"@ja .TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, HadyelsaharCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
hoo added a comment. I did a (test) dump of test.wikidata.org just now: https://people.wikimedia.org/~hoo/tmp/testwikidata-20170412-truthy-BETA.nt.gz Please have a look at it!TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 347840 merged by jenkins-bot: [mediawiki/extensions/Wikibase@wmf/1.29.0-wmf.19] dumpRdf: Allow creating truthy dumps https://gerrit.wikimedia.org/r/347840TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 347840 had a related patch set uploaded (by Hoo man): [mediawiki/extensions/Wikibase@wmf/1.29.0-wmf.19] dumpRdf: Allow creating truthy dumps https://gerrit.wikimedia.org/r/347840TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 347838 had a related patch set uploaded (by Hoo man): [operations/puppet@production] Allow running two dumpwikidatattl dumps side by side https://gerrit.wikimedia.org/r/347838TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 347234 had a related patch set uploaded (by Hoo man): [operations/puppet@production] Change dumpwikidatattl to allow producing other flavors https://gerrit.wikimedia.org/r/347234TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 346636 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] dumpRdf: Allow creating truthy dumps https://gerrit.wikimedia.org/r/346636TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
hoo added a comment. I would suggest to put the dumps like https://dumps.wikimedia.org/wikidatawiki/entities/20170403/wikidata-20170403-truthy-BETA.nt.gz (compared to https://dumps.wikimedia.org/wikidatawiki/entities/20170403/wikidata-20170403-all-BETA.ttl.gz for the current full ttl dump).TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Adik2382, Lunewa, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Salgo60, Lewizho99, Maathavan, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
gerritbot added a comment. Change 346636 had a related patch set uploaded (by Hoo man): [mediawiki/extensions/Wikibase@master] dumpRdf: Allow creating truthy dumps https://gerrit.wikimedia.org/r/346636TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Lunewa, QZanden, Salgo60, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
Lucie added a comment. To answer all of those to my best knowledge at once: bzip2 should be fine Including property definitions will most likely be very handy, though I am not sure what that means exactly- it just includes all statements for properties? Than yes! Version info absolutely Normalized values make a lot of sense Not sure what resolved entities should therefore I can't say whether it makes sense Thanks, looking forward to the truthy dump :DTASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, LucieCc: Hadyelsahar, Smalyshev, Lydia_Pintscher, hoo, Lucie, Lunewa, QZanden, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
Smalyshev added a comment. It should include all the statements ttl dump includes, i.e. flavor=dump. So, RdfProducer::PRODUCE_TRUTHY_STATEMENTS should be in. Property/entity resolution is not necessary for the dump, since all entities/properties are included anyway, by virtue of it being full dump.TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, SmalyshevCc: Smalyshev, Lydia_Pintscher, hoo, Lucie, Lunewa, QZanden, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T155103: Create a truthy nt dump
hoo added a comment.
Shall this just include RdfProducer::PRODUCE_TRUTHY_STATEMENTS?
We potentially at least also want RdfProducer::PRODUCE_PROPERTIES ("Add entity definitions for properties used in the dump"), RdfProducer::PRODUCE_VERSION_INFO ("Produce metadata header containing software version info and copyright.") and RdfProducer::PRODUCE_NORMALIZED_VALUES ("Produce normalized values for values with units."). Possibly we also want RdfProducer::PRODUCE_RESOLVED_ENTITIES ("Produce definitions for all entities used in the dump"), although I'm not sure what the indications of that exactly are.
Maybe interesting: EntityDataSerializationService::getFlavor (although that doesn't have a "truthy" flavor).TASK DETAILhttps://phabricator.wikimedia.org/T155103EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Lydia_Pintscher, hoo, Lucie, Lunewa, QZanden, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, Hydriz, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
