[jira] Created: (NUTCH-776) Configurable queue depth
Configurable queue depth Key: NUTCH-776 URL: https://issues.apache.org/jira/browse/NUTCH-776 Project: Nutch Issue Type: Improvement Components: fetcher Affects Versions: 1.1 Reporter: MilleBii Priority: Minor Fix For: 1.1 I propose that we create a configurable item for the queuedepth in Fetcher.java instead of the hard-coded value of 50. key name : fetcher.queues.depth Default value : remains 50 (of course) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-770) Timebomb for Fetcher
[ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12786443#action_12786443 ] MilleBii commented on NUTCH-770: Tried it succesfully on a windows platform. It does not work on a Ubuntu, pseudo-distributed hadoop configuration with mappers running in parallel Timebomb for Fetcher Key: NUTCH-770 URL: https://issues.apache.org/jira/browse/NUTCH-770 Project: Nutch Issue Type: Improvement Reporter: Julien Nioche Assignee: Andrzej Bialecki Fix For: 1.1 Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (NUTCH-770) Timebomb for Fetcher
[ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12786443#action_12786443 ] MilleBii edited comment on NUTCH-770 at 12/5/09 4:50 PM: - Tried it succesfully on a windows platform. It does not work on a Ubuntu, pseudo-distributed hadoop configuration with two mappers running in parallel was (Author: millebii): Tried it succesfully on a windows platform. It does not work on a Ubuntu, pseudo-distributed hadoop configuration with mappers running in parallel Timebomb for Fetcher Key: NUTCH-770 URL: https://issues.apache.org/jira/browse/NUTCH-770 Project: Nutch Issue Type: Improvement Reporter: Julien Nioche Assignee: Andrzej Bialecki Fix For: 1.1 Attachments: log-770, NUTCH-770-v2.patch, NUTCH-770-v3.patch, NUTCH-770.patch This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (NUTCH-770) Timebomb for Fetcher
[ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783252#action_12783252 ] MilleBii edited comment on NUTCH-770 at 11/29/09 8:47 PM: -- That's what I did and just retried ... so I'm a bit suprised too. Other patches worked fine so far. Changed my method and used patching by Eclipse and I get the following compiling error : 992: cannot find symbol [javac] symbol : method checkTimeBomb() [javac] location: class org.apache.nutch.fetcher.Fetcher.FetchItemQueues [javac] int timeBombed =fetchQueues.checkTimeBomb(); [javac] ^ [javac] 1 error was (Author: millebii): That's what I did and just retried ... so I'm a bit suprised too. Other patches worked fine so far. ??? Timebomb for Fetcher Key: NUTCH-770 URL: https://issues.apache.org/jira/browse/NUTCH-770 Project: Nutch Issue Type: Improvement Reporter: Julien Nioche Attachments: log-770, NUTCH-770.patch This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (NUTCH-770) Timebomb for Fetcher
[ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MilleBii updated NUTCH-770: --- Attachment: log-770 Please find the logs of the patch... I did effectively try it but I could not compile after it. Timebomb for Fetcher Key: NUTCH-770 URL: https://issues.apache.org/jira/browse/NUTCH-770 Project: Nutch Issue Type: Improvement Reporter: Julien Nioche Attachments: log-770, NUTCH-770.patch This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (NUTCH-770) Timebomb for Fetcher
[ https://issues.apache.org/jira/browse/NUTCH-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12783252#action_12783252 ] MilleBii commented on NUTCH-770: That's what I did and just retried ... so I'm a bit suprised too. Other patches worked fine so far. ??? Timebomb for Fetcher Key: NUTCH-770 URL: https://issues.apache.org/jira/browse/NUTCH-770 Project: Nutch Issue Type: Improvement Reporter: Julien Nioche Attachments: log-770, NUTCH-770.patch This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
