Re: no mailing list hits in google
On Wed, Aug 28, 2019 at 10:31 PM Alvaro Herrera wrote: > On 2019-Aug-28, Thomas Kellerer wrote: > > > Merlin Moncure schrieb am 28.08.2019 um 18:22: > > > My test case here is the query: pgsql-hackers > > > > That search term is the first hit on DuckDuckGo: > > https://duckduckgo.com/?q=pgsql-hackers+ExecHashJoinNewBatch=h_=web > > Yes, but that's an old post, not the one from this year. > > It does show another interesting point though -- it *also* includes hits from third party list archiving sites, which are *also* gone from Google at this point. And those are definitely not gone from Google because we have a robots.txt blocking /list/ -- it must be something else. //Magnus
Re: no mailing list hits in google
On 2019-Aug-28, Thomas Kellerer wrote: > Merlin Moncure schrieb am 28.08.2019 um 18:22: > > My test case here is the query: pgsql-hackers > > That search term is the first hit on DuckDuckGo: > https://duckduckgo.com/?q=pgsql-hackers+ExecHashJoinNewBatch=h_=web Yes, but that's an old post, not the one from this year. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: no mailing list hits in google
Merlin Moncure schrieb am 28.08.2019 um 18:22: My test case here is the query: pgsql-hackers That search term is the first hit on DuckDuckGo: https://duckduckgo.com/?q=pgsql-hackers+ExecHashJoinNewBatch=h_=web Searching for "postgres ExecHashJoinNewBatch" returns that ot position 4 https://duckduckgo.com/?q=postgres+ExecHashJoinNewBatch=h_=web
Re: no mailing list hits in google
Hi, On 2019-08-28 10:26:35 -0700, Andres Freund wrote: > On August 28, 2019 9:22:44 AM PDT, Merlin Moncure wrote: > >Hackers, > >[apologies if this is the incorrect list or is already discussed > >material] > > Probably should be on the -www list. Redirecting. Please trim in future > replies. > > >I've noticed that mailing list discussions in -hackers and other > >mailing lists appear to not be indexed by google -- at all. > > I noticed that there's fewer and fewer hits too. Pretty annoying. I have an > online archive I can search, but that's not something everyone should have to > do. > > I think it's because robots.txt tells search engines to ignore the lists. > Quite hard to understand how that's a good idea. > > https://www.postgresql.org/robots.txt > > User-agent: * > Disallow: /admin/ > Disallow: /account/ > Disallow: /docs/devel/ > Disallow: /list/ > Disallow: /search/ > Disallow: /message-id/raw/ > Disallow: /message-id/flat/ > > Sitemap: https://www.postgresql.org/sitemap.xml > > Without /list, there's no links to the individual messages. So there needs to > be another external reference for a search engine to arrive at individual > messages. > > Andres > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. For reasons that I do not understand, the previous mail had a broken html part, making the above message invisible for people viewing the html part. Greetings, Andres Freund
Re: no mailing list hits in google
Hi, On August 28, 2019 9:22:44 AM PDT, Merlin Moncure wrote: >Hackers, >[apologies if this is the incorrect list or is already discussed >material] Probably should be on the -www list. Redirecting. Please trim in future replies. >I've noticed that mailing list discussions in -hackers and other >mailing lists appear to not be indexed by google -- at all. I noticed that there's fewer and fewer hits too. Pretty annoying. I have an online archive I can search, but that's not something everyone should have to do. I think it's because robots.txt tells search engines to ignore the lists. Quite hard to understand how that's a good idea. https://www.postgresql.org/robots.txt User-agent: * Disallow: /admin/ Disallow: /account/ Disallow: /docs/devel/ Disallow: /list/ Disallow: /search/ Disallow: /message-id/raw/ Disallow: /message-id/flat/ Sitemap: https://www.postgresql.org/sitemap.xml Without /list, there's no links to the individual messages. So there needs to be another external reference for a search engine to arrive at individual messages. Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
