dpol1 commented on code in PR #2012:
URL: https://github.com/apache/stormcrawler/pull/2012#discussion_r3637422577
##########
docs/src/main/asciidoc/configuration.adoc:
##########
@@ -318,6 +318,7 @@ The values below are used by sub-classes of
`AbstractIndexerBolt`.
| indexer.canonical.name | canonical | Metadata key for the canonical URL.
Used to replace the URL with its canonical form before indexing.
| indexer.ignore.empty.fields | false | If true, skip fields with empty values
when indexing.
+| indexer.md.docid | - | Metadata key whose value, if present, is used as the
document ID instead of the default SHA-256 digest of the URL. Useful e.g. for
content-based deduplication when a `ParseFilter` stores a content hash in the
metadata.
Review Comment:
Two precisions for this row: the value is digested, not used raw (SHA-256
here, SHA-512 via `CloudSearchUtils.getID` for CloudSearch); and the SOLR/SQL
indexers never call `getDocumentID()`, so the setting is silently ignored
there. Worth stating the supported backends.
##########
core/src/main/java/org/apache/stormcrawler/indexing/AbstractIndexerBolt.java:
##########
@@ -75,8 +75,20 @@ public abstract class AbstractIndexerBolt extends
BaseRichBolt {
/** Indicates that empty field values should not be emitted at all. */
public static final String ignoreEmptyFieldValueParamName =
"indexer.ignore.empty.fields";
+ /**
+ * Metadata key whose value should be used to derive the document ID, if
present. The value is
+ * passed through the same SHA-256 digest used for the URL-based ID, so
this can be used e.g. by
+ * a {@link org.apache.stormcrawler.parse.ParseFilter} generating a hash
of the content so that
+ * duplicate content ends up under the same document ID regardless of the
URL(s) it was found
+ * at. Falls back to the default (a SHA-256 digest of the URL) if not set
or if the metadata
+ * does not contain a value for the configured key.
+ */
+ public static final String DOC_ID_METADATA_PARAM_NAME = "indexer.md.docid";
Review Comment:
Worth documenting: on the deletion path (gone pages) there's no parsing, so
the DeletionBolt only sees this key if it's in `metadata.persist` and
round-trips through the status index. Otherwise it falls back to `sha256(url)`
and the document indexed under the metadata ID is silently orphaned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
