[
https://issues.apache.org/jira/browse/NUTCH-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471260
]
Dogacan Güney commented on NUTCH-443:
-------------------------------------
Ok, this is the second attempt(sorry that I am sending patches in a frenzy, I
will slow down now).
In the first patch, I just put Map<String, Parse> to FetcherOutput but that
doesn't work,
since the keys are not necessarily ordered.
I mean;
Assume, we have two <key, Map<String, Parse>> pairs:
<"a.com", <"z.com", some_parse>> and <"b.com", <"b.com", some_other_parse>>
With the first patch, we would first get a.com (and thus write z.com) then get
b.com (and try to write b.com)
but this would fail since "b.com" < "z.com".
I completely removed FetcherOutput class. What it does can be done with
wrapping the objects
in ObjectWritable. I know this is heavier but I couldn't think of another way
around the issue of ordering
of keys.
I tested this a bit with a small set of urls. Both parsing seperately and
parsing during fetching seems to work.
> allow parsers to return multiple Parse object, this will speed up the rss
> parser
> --------------------------------------------------------------------------------
>
> Key: NUTCH-443
> URL: https://issues.apache.org/jira/browse/NUTCH-443
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.9.0
> Reporter: Renaud Richardet
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: parse-map-core-untested.patch
>
>
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser
> can return multiple parse objects, that will all be indexed separately.
> Advantage: no need to fetch all feed-items separately.
> see the discussion at
> http://www.nabble.com/RSS-fecter-and-index-individul-how-can-i-realize-this-function-tf3146271.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
