[ 
https://issues.apache.org/jira/browse/NUTCH-443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495696
 ] 

Doğacan Güney commented on NUTCH-443:
-------------------------------------

I am not sure I follow you Andrzej. My patch already does a very similar thing 
in Fetchers . Actually, the only difference between our patches - w.r.t Fetcher 
code - is in your patch the parsing condition also includes (content != null) 
check. Beyond that our code is pretty much the same. (I will send an updated 
patch that does that, btw). Besides the code change in Fetchers, we also need 
to remove the redir != null condition for indexer to work correctly. See my 
comment above for a hopefully more understandable description.

Indexer has to read crawl_parse, because that is where ParseSegment pushes 
sub-urls fetch datums. So, it is not related to the redirection issue. It is 
related to the "Oh man, I forgot to include that line in my patch" issue:).

If reading crawl_parse turns out to be a big burden to Indexer, perhaps we can 
make ParseSegment push these datums to another file.  (crawl_late_fetch? Yeah, 
I know that name sucks:) It would be awesome if hadoop allowed us to reopen 
SequenceFiles to append data(so we could have just pushed them to crawl_fetch). 
AFAIK, hadoop doesn't have that yet.




> allow parsers to return multiple Parse object, this will speed up the rss 
> parser
> --------------------------------------------------------------------------------
>
>                 Key: NUTCH-443
>                 URL: https://issues.apache.org/jira/browse/NUTCH-443
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 0.9.0
>            Reporter: Renaud Richardet
>         Assigned To: Chris A. Mattmann
>            Priority: Minor
>             Fix For: 1.0.0
>
>         Attachments: NUTCH-443-draft-v1.patch, NUTCH-443-draft-v2.patch, 
> NUTCH-443-draft-v3.patch, NUTCH-443-draft-v4.patch, NUTCH-443-draft-v5.patch, 
> NUTCH-443-draft-v6.patch, NUTCH-443-draft-v7.patch, 
> NUTCH-443.022507.patch.txt, NUTCH-443.02282007-v2.patch, 
> NUTCH-443.02282007.patch, NUTCH-443.08052007.patch, 
> parse-map-core-draft-v1.patch, parse-map-core-untested.patch, parsers.diff, 
> patch.txt, redirect_and_index.patch
>
>
> allow Parser#parse to return a Map<String,Parse>. This way, the RSS parser 
> can return multiple parse objects, that will all be indexed separately. 
> Advantage: no need to fetch all feed-items separately.
> see the discussion at 
> http://www.nabble.com/RSS-fecter-and-index-individul-how-can-i-realize-this-function-tf3146271.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers

Reply via email to