vincbeck opened a new pull request, #761: URL: https://github.com/apache/airflow-site/pull/761
See issue [#29283](https://github.com/apache/airflow/issues/29283). The current robots.txt allows all pages to be indexed by search engines. As a a consequence, some old version of documentation ends up being indexed which can be confusing for users. They end up reading old version of documentation thinking it is the latest one. An example of user getting confused [here](https://apache-airflow.slack.com/archives/CSS36QQS1/p1680524708746749). Which this change, all pages under `/docs/apache-airflow/` but `/docs/apache-airflow/stable/` **will stop being crawled by search engines**. !! Please confirm this is fine since this change is quite impactful !! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
