jnioche commented on code in PR #2109:
URL: https://github.com/apache/stormcrawler/pull/2109#discussion_r3905264253
##########
external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:
##########
@@ -89,6 +89,39 @@ public class WARCRecordFormat implements RecordFormat {
protected static final Pattern HTTP_STATUS_CODE_PATTERN =
Pattern.compile("^[0-9]{3}$");
protected static final String HTTP_VERSION_FALLBACK = "HTTP/1.1";
+ /*
+ * Named fields (WARC header fields and the fields of an
application/warc-fields payload) are
+ * terminated by CRLF: their names are limited to printable ASCII
characters excluding the
+ * colon, cf. RFC 5322 section 2.2, and their values must not contain CR
or LF, otherwise the
+ * remainder of the value would be read as additional field lines.
+ */
+ private static final Pattern WARC_FIELD_NAME_PATTERN =
Pattern.compile("[!-9;-~]+");
+
+ /**
+ * Check whether a string is a valid WARC field name, i.e. consists of
printable ASCII
+ * characters without a colon, cf. RFC 5322 section 2.2.
+ *
+ * @param name field name candidate
+ * @return true if the name can safely be written as the name of a WARC
field
+ */
+ static boolean isValidWarcFieldName(String name) {
+ return name != null && WARC_FIELD_NAME_PATTERN.matcher(name).matches();
Review Comment:
you are right, I had misread the code. Better to have is more explicitly
##########
external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:
##########
@@ -89,6 +89,39 @@ public class WARCRecordFormat implements RecordFormat {
protected static final Pattern HTTP_STATUS_CODE_PATTERN =
Pattern.compile("^[0-9]{3}$");
protected static final String HTTP_VERSION_FALLBACK = "HTTP/1.1";
+ /*
+ * Named fields (WARC header fields and the fields of an
application/warc-fields payload) are
+ * terminated by CRLF: their names are limited to printable ASCII
characters excluding the
+ * colon, cf. RFC 5322 section 2.2, and their values must not contain CR
or LF, otherwise the
+ * remainder of the value would be read as additional field lines.
+ */
+ private static final Pattern WARC_FIELD_NAME_PATTERN =
Pattern.compile("[!-9;-~]+");
+
+ /**
+ * Check whether a string is a valid WARC field name, i.e. consists of
printable ASCII
+ * characters without a colon, cf. RFC 5322 section 2.2.
+ *
+ * @param name field name candidate
+ * @return true if the name can safely be written as the name of a WARC
field
+ */
+ static boolean isValidWarcFieldName(String name) {
+ return name != null && WARC_FIELD_NAME_PATTERN.matcher(name).matches();
Review Comment:
you are right, I had misread the code. Better to have it more explicitly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
