A new Request for Comments is now available in online RFC libraries.
RFC 9309
Title: Robots Exclusion Protocol
Author: M. Koster,
G. Illyes,
H. Zeller,
L. Sassman
Status: Standards Track
Stream: IETF
Date: September 2022
Mailbox: [email protected],
[email protected],
[email protected],
[email protected]
Pages: 12
Updates/Obsoletes/SeeAlso: None
I-D Tag: draft-koster-rep-12.txt
URL: https://www.rfc-editor.org/info/rfc9309
DOI: 10.17487/RFC9309
This document specifies and extends the "Robots Exclusion Protocol"
method originally defined by Martijn Koster in 1994 for service
owners to control how content served by their services may be
accessed, if at all, by automatic clients known as crawlers.
Specifically, it adds definition language for the protocol,
instructions for handling errors, and instructions for caching.
This is now a Proposed Standard.
STANDARDS TRACK: This document specifies an Internet Standards Track
protocol for the Internet community, and requests discussion and suggestions
for improvements. Please refer to the current edition of the Official
Internet Protocol Standards (https://www.rfc-editor.org/standards) for the
standardization state and status of this protocol. Distribution of this
memo is unlimited.
This announcement is sent to the IETF-Announce and rfc-dist lists.
To subscribe or unsubscribe, see
https://www.ietf.org/mailman/listinfo/ietf-announce
https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist
For searching the RFC series, see https://www.rfc-editor.org/search
For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk
Requests for special distribution should be addressed to either the
author of the RFC in question, or to [email protected]. Unless
specifically noted otherwise on the RFC itself, all RFCs are for
unlimited distribution.
The RFC Editor Team
Association Management Solutions, LLC
_______________________________________________
IETF-Announce mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ietf-announce
