| thiemowmde added a comment. |
As you said, Special:GoToLinkedPage redirects and does not output HTML (except for the form, which is a single page, and the reason why Disallow: /wiki/Special:GoToLinkedPage should not be used). The target pages of these redirects are Wikipedia articles. They should be indexed, and they already have canonical tags.
I'm not sure what an X-Robots-Tag will do when used with a redirect.
- I believe there is no point in crawling Special:GoToLinkedPage URLs, because they are guaranteed to do nothing but redirect to Wikipedia articles. We know each redirect represents a sitelink, and sitelinks are already accessible on the ordinary item page. We know all this. Google does not.
- Similar for Special:ItemByTitle, which redirects to a Wikidata item. The exact same links already exist in the sidebars of the connected Wikipedia articles.
@Mbch331, please add the following lines to https://www.wikidata.org/wiki/MediaWiki:Robots.txt:
Disallow: /wiki/Special:GoToLinkedPage? Disallow: /wiki/Special:ItemByTitle? Disallow: /wiki/Special:SetSiteLink?
Do not remove the slashes, because this would exclude the special page forms itself. We want these to appear in a Google search.
TASK DETAIL
EMAIL PREFERENCES
To: Mbch331, thiemowmde
Cc: Sjoerddebruin, TheDJ, Mbch331, Jonas, hoo, aude, Lydia_Pintscher, Tobi_WMDE_SW, Aklapper, thiemowmde, TerraCodes, D3r1ck01, MuhammadShuaib, Izno, Wikidata-bugs
Cc: Sjoerddebruin, TheDJ, Mbch331, Jonas, hoo, aude, Lydia_Pintscher, Tobi_WMDE_SW, Aklapper, thiemowmde, TerraCodes, D3r1ck01, MuhammadShuaib, Izno, Wikidata-bugs
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
