[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T94019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Pppery Cc: dcausse, Addshore, toan, Tonina_Zhelyazkova_WMDE,

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

Addshore added a comment. When T120242: Consistent MediaWiki state change events | MediaWiki events as source of truth is ready we could probably change some of the architecture and process around dumping for Wikidata.org We would likely keep

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

Addshore removed a project: Wikidata-Campsite. TASK DETAIL https://phabricator.wikimedia.org/T94019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: dcausse, Addshore, toan, Tonina_Zhelyazkova_WMDE, JAllemandou, Pintoch, Smalyshev, hoo,

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

dcausse added a comment. Indeed, the RDF data is available in the hive table `discovery.wikibase_rdf` but it is generated reading the TTL dumps so it might not help for this particular task. Using hadoop will indeed allow to process the json efficiently but has drawbacks as already

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

JAllemandou added a subscriber: dcausse. JAllemandou added a comment. Info: There already is in the cluster a job doing `TTL -> RDF` conversion. The TTL dumps are imported weekly, and converted to blazegraph RDF once available. The job is maintained by the Search Platform team (ping

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

Addshore added a comment. In T94019#5131531 , @JAllemandou wrote: > The analytics hadoop cluster could also be of use here: the task can easily take advantage of parallelization. Indeed, and it already gets the JSON dumps loaded

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

Addshore added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T94019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: toan, Tonina_Zhelyazkova_WMDE, JAllemandou, Pintoch, Smalyshev, hoo, Liuxinyu970226, mkroetzsch,

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

toan added a project: Wikidata-Campsite. TASK DETAIL https://phabricator.wikimedia.org/T94019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: toan Cc: toan, Tonina_Zhelyazkova_WMDE, JAllemandou, Pintoch, Smalyshev, hoo, Liuxinyu970226, mkroetzsch,

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T94019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: Tonina_Zhelyazkova_WMDE, JAllemandou, Pintoch, Smalyshev, hoo, Liuxinyu970226,

[Wikidata-bugs] [Maniphest] T94019: Generate RDF from JSON

gerritbot added a comment. Change 617153 had a related patch set uploaded (by Hoo man; owner: Hoo man): [mediawiki/extensions/Wikibase@master] Experimental support for creating dumps from JSON dumps https://gerrit.wikimedia.org/r/617153 TASK DETAIL