o-nikolas commented on issue #29283: URL: https://github.com/apache/airflow/issues/29283#issuecomment-1491019607
> I was looking into working on this issue but could not find where the `robots.txt` file of Airflow website was defined. Do you know by any chance? We let Hugo generate it: https://github.com/apache/airflow-site/blob/bc075d5a2049fed3ea71f27fac08712b043beb18/landing-pages/site/config.toml#L20 Right now it's on "/" so search engines basically search all pages. But we can customize it by providing a robots.txt in a few locations (see [the docs here](https://gohugo.io/templates/robots/#robotstxt-template-lookup-order)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
