[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
gerritbot added a comment. Change 438021 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] SqlEntityIdPager: filter entity types on DB level https://gerrit.wikimedia.org/r/438021TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: Pablo-WMDE, gerritbot, Addshore, hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
gerritbot added a comment. Change 437501 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] DumpRDF: Omit entity types disabled for RDF https://gerrit.wikimedia.org/r/437501TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pablo-WMDE, gerritbotCc: gerritbot, Addshore, hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
WMDE-leszek added a comment. I think it should respect it. This script generates dumps that are then given out to the public, so it should not allow to override wiki settings, e.g. in the case of accidentally passing wrong parameters. In theory it seems reasonable. If my IDE is correct, SqlEntityIdPager is only used in two places: in the rdf export script, and in ItemsPerSiteBuilder which only cares about items. So this extension of Pager might not be worth an effort either. I sadly don't know the history of this class. Ability to specify a "list" of entity types to RDF dump script is also really new (like since two weeks), before it was either all types, or a single type filtering only, AFAIR. I think it might simply not be needed, as long as there were only two entity types in play. TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pablo-WMDE, WMDE-leszekCc: Addshore, hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
WMDE-leszek added a comment. @Pablo-WMDE: correct.TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pablo-WMDE, WMDE-leszekCc: hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
Pablo-WMDE added a comment. @WMDE-leszek Could you please confirm that https://gerrit.wikimedia.org/r/#/c/437501/ conceptually (never mind the how for now) does what this ticket tries to achieve? This assumes that LocalSettings.php contains a line $wgWBRepoSettings['disabledRdfExportEntityTypes'] = ['lexeme'];TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pablo-WMDECc: hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
Pablo-WMDE added a comment. Observations: SqlEntityIdPager can be used to select entities (their ids) by page_namespace (which can be derived from the entity type) from the database currently only supports this for one entityType at a time when querying for more than one entity type, filtering is performed in application code (DumpGenerator) as DumpEntities supports two id stream providers, SqlEntityIdPager and a file based EntityIdReader, the second of which does not support filtering by entity type on its own, the DumpGenerator will have to preserve its ability to filter by entity type regardless of potential optimization in the SQL department TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Pablo-WMDECc: hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194627: dumpRdf script should take the disabledRdfExportEntityTypes flag into account
WMDE-leszek added a comment. @hoo: that would be a workaround, yes. Not a fix really, though.TASK DETAILhttps://phabricator.wikimedia.org/T194627EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: WMDE-leszekCc: hoo, WMDE-leszek, Aklapper, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
