[
https://issues.apache.org/jira/browse/NUTCH-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473147
]
Doğacan Güney commented on NUTCH-443:
-------------------------------------
> Hmm, actually this is an important question. I don't think FetcherOutput is
> persisted anywhere, it's just an aggregate class to
> keep things together before they hit the disk. I propose to leave a comment
> in MapWritable like this "// code -123 was
> reserved for FetcherOutput - no longer in use". As for the class itself -
> again, since it's not persisted we don't have to keep it
> around, just remove it.
I implemented this approach in one of the earlier patches. The problem is that,
the code in MapWritable does this:
addIdEntry((byte) (-128 + CLASS_ID_MAP.size() + ++fIdCount), // ...
Now, I don't claim to understand the code perfectly but because of the "-128 +
CLASS_ID_MAP.size()" part I think CLASS_ID_MAP must have consecutive values
always, so not having -123 breaks it. IIRC, removing that line and running
TestMapWritable fails.
> Sections in Fetcher.FetcherThread.output() and similar in Fetcher2 that
> output the data need to be synchronized now -
> output.collect() is no longer a single atomic operation. Perhaps it's better
> to leave FetcherOutput after all?
This causes key ordering problems. See my admittedly-could-have-been-clearer
2nd comment.
Anyway, I am assumming that you are OK with removing
ParseUtil.getFirstParseEntry and just using Map.get?
> allow parsers to return multiple Parse object, this will speed up the rss
> parser
> --------------------------------------------------------------------------------
>
> Key: NUTCH-443
> URL: https://issues.apache.org/jira/browse/NUTCH-443
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.9.0
> Reporter: Renaud Richardet
> Assigned To: Chris A. Mattmann
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch,
> NUTCH-443-draft-v3.patch, NUTCH-443-draft-v4.patch, NUTCH-443-draft-v5.patch,
> NUTCH-443-draft-v6.patch, parse-map-core-draft-v1.patch,
> parse-map-core-untested.patch, parsers.diff
>
>
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser
> can return multiple parse objects, that will all be indexed separately.
> Advantage: no need to fetch all feed-items separately.
> see the discussion at
> http://www.nabble.com/RSS-fecter-and-index-individul-how-can-i-realize-this-function-tf3146271.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
