abhinav-phi commented on code in PR #2112:
URL: https://github.com/apache/stormcrawler/pull/2112#discussion_r3915823071
##########
core/src/main/java/org/apache/stormcrawler/parse/ParseResult.java:
##########
@@ -66,19 +66,51 @@ public void setOutlinks(List<Outlink> outlinks) {
}
/**
- * @return An existent instance of Parse for the given URL or an empty one
if none can be found,
- * useful to avoid unnecessary checks in the parse plugins
+ * Convenience accessor which returns the ParseData for the given URL,
creating an empty entry
+ * in this ParseResult if none can be found, useful to avoid unnecessary
checks in the parse
+ * plugins. Note that looking up a URL which has not been parsed modifies
this ParseResult: the
+ * entry created for it is treated like any other document and gets
emitted by the parser bolts.
+ *
+ * @deprecated use {@link #getIfPresent(String)} for a read-only lookup or
{@link
+ * #getOrCreate(String)} to create an entry, so that the intent is
explicit at the call site
+ * @return An existent instance of Parse for the given URL or an empty one
if none can be found
*/
+ @Deprecated
Review Comment:
Agreed — since the next release is a major one, `get(url)` is now removed
outright in 8aea8e9c instead of being deprecated. All in-tree callers had
already been migrated to `getOrCreate(url)`, and filters that only want to
check for an existing entry can use the new `getIfPresent(url)`.
##########
core/src/main/java/org/apache/stormcrawler/bolt/JSoupParserBolt.java:
##########
@@ -493,10 +493,20 @@ public void execute(Tuple tuple) {
}
// emit each document/subdocument in the ParseResult object
- // there should be at least one ParseData item for the "parent" URL
-
+ // there should be at least one ParseData item for the "parent" URL.
Review Comment:
Kept the check and the log (as a backstop for external filters that still
create empty entries via `put`/`set`/`getOrCreate`), plus the
`skipped_empty_documents` counter. The lengthy block comment above the loop is
now two lines, and the new accessors' Javadoc has been cut down to the
essentials — 8aea8e9c.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
