Hi users@, I am new to Nutch (v.1.17) and my current project requires the indexing of the html of crawled pages. It also requires fields that can be derived from the raw html such as image count, and charset.
I have looked on StackOverflow for how to achieve this and most people from my understanding seem to be recommending processing the segments to extract the html and modify the documents post-crawl. This doesn't fit my use case as I need to calculate these fields at crawl time before they are indexed into Elasticsearch. The other recommendations I have seen mention creating a plugin to override the parse-html plugin. However, I have found rather limited documentation on how to do this correctly and am not sure on how to return from the plugin in a way that the field propagates into the NutchDocument which will be processed in the Indexers' write method. Do any of you have any advice or links to documentation that explains how to modify what gets set in the NutchDocument? Thank you in advance
