Sure we could do robots. But I suspect that we put ourselves in this situation through https://issues.apache.org/jira/browse/SOLR-10595 ourselves Check out the attachment solr_redirects.conf on that JIRA (also here https://gist.github.com/janhoy/a3149e1ed27df020194a2de1a7fa2c16)
Here, we explicitly map all pages that used to be in Confluence (which there are a ton of links to on the net), to version 6.6 of the guide. Of course, if we change those to "latest", some of those links will break, but perhaps it would still be better? Or can we be more intelligent in the rewrite rules on Solr site - that if you try a "/guide/foo.html" link and it is not found, that you display a custom error page or go to front page of latest guide? Jan > 4. mar. 2021 kl. 10:21 skrev Ishan Chattopadhyaya <[email protected]>: > > We can add robots.txt to stop Google from indexing/showing in results. > > On Thu, 4 Mar, 2021, 2:34 pm Jan Høydahl, <[email protected] > <mailto:[email protected]>> wrote: > Hi, sending to this list since dev@solr list is not yet announced properly. > > We have a few days of traffic to the new site and can see the most visited > pages at https://uls.apache.org/exports/solr.apache.org.yaml > <https://uls.apache.org/exports/solr.apache.org.yaml> (see copy below). > When I search google for "solr query parser", I get the 6.6 guide on top, > which is probably why /guide/6_6/the-standard-query-parser.html shows up, and > the same for the other /guide/6_6/ links. > Some questions: > > How can we make Google forget about version 6.6? I know we had a bunch of > redirects from Confluence to the 6.6 guide, are they still in place? > Why is /docs/6_6_0/solr-core/index.html the 2nd most visited page? Anywhere > that links to it? > Why is /docs/4_8_1/solr-solrj/index.html so high? Ahywhere that links to it? > The /mirrors-solr-latest-redir.html redirect was not working. I just pushed a > fix > > > Sheet3: > Name: Most visited pages, past month > Values: > /index.html: 443 > /docs/6_6_0/solr-core/index.html: 281 > /guide/8_8/solr-tutorial.html: 104 > /news.html: 92 > /guide/solr-tutorial.html: 91 > /resources.html: 75 > /features.html: 69 > /docs/8_7_0/solr-core/index.html: 68 > /downloads.html: 68 > /guide/6_6/the-standard-query-parser.html: 65 > /docs/4_8_1/solr-solrj/index.html: 62 > /guide/6_6/common-query-parameters.html: 50 > /guide/index.html: 46 > /docs/8_8_1/solr-solrj/index.html: 44 > /docs/8_7_0/solr-solrj/index.html: 38 > /docs/8_8_1/solr-core/index.html: 37 > /community.html: 24 > /guide/8_8/: 23 > /guide/6_6/uploading-data-with-index-handlers.html: 22 > /docs/8_6_3/solr-core/index.html: 21 > /guide/6_6/filter-descriptions.html: 21 > /guide/6_6/collections-api.html: 18 > /docs/8_6_2/solr-solrj/overview-summary.html: 16 > /guide/6_6/faceting.html: 16 > /mirrors-solr-latest-redir.html: 15 > /whoweare.html: 15 > /guide/6_6/solrcloud.html: 14 > /guide/6_6/tokenizers.html: 13 > /guide/7_0/solr-configuration-files.html: 13 > /guide/8_8/query-syntax-and-parsing.html: 13 > /security.html: 13 > /guide/6_6/introduction-to-solr-indexing.html: 12 > /guide/8_8/solr-upgrade-notes.html: 12 > /docs/8_0_0/solr-solrj/allclasses-frame.html: 11 > /docs/8_6_3/solr-solrj/index.html: 11 > /guide/6_6/the-dismax-query-parser.html: 11 > /guide/8_8/getting-started.html: 11 > /guide/solr-upgrade-notes.html: 11 > /docs/8_0_0/solr-solrj/overview-summary.html: 10 > /docs/8_6_2/solr-solrj/index.html: 10 > /guide/6_6/running-solr.html: 10 > /docs/7_2_1/solr-solrj/overview-summary.html: 9 > /docs/8_0_0/solr-solrj/overview-frame.html: 9 > /guide/6_6/format-of-solr-xml.html: 9 > /guide/6_6/index.html: 9 > /guide/6_6/learning-to-rank.html: 9 > /guide/6_6/making-and-restoring-backups.html: 9 > /guide/6_6/working-with-dates.html: 9 > /guide/8_0/reindexing.html: 9 > /docs/7_2_1/solr-solrj/allclasses-frame.html: 8 > >
