Re: Running multiple fetch map tasks on a Hadoop Cluster.
Julien, How would you achieve parallelism then on a Hadoop cluster , am I missing something here? My understanding was that we could scale the crawl by allowing fetch to happen in multiple map tasks in multiple nodes in a Hadoop cluster , otherwise I am stuck in sequentially crawling a large set of urls spread across mutiple domains. If that is indeed the way to scale the crawl , then we would need to generate multiple segments at the generate time so that these could be fetched in paralle. So I guess I really need help in . 1. Making the generate phase generate multiple segments 2. Being able to fetch these segments in parallel. Can you please let me know if my approach to scale the crawl sounds right to you ? Thanks and much appreciated, all the help I have gotten so far On Fri, Sep 19, 2014 at 10:40 AM, Julien Nioche lists.digitalpeb...@gmail.com wrote: The fetching operates segment by segment and won't fetch more than one at the same time. You can get the generation step to build multiple segments in one go but you'd need to modify the script so that the fetching step is called as many times as you have segments + you'd probably need to add some logic for detecting that they've all finished before you move on to the update step. Out of curiosity : why do you want to fetch multiple segments at the same time? On 19 September 2014 06:00, Meraj A. Khan mera...@gmail.com wrote: Hello Folks, I am unable to run multiple fetch Map taks for Nutch 1.7 on Hadoop YARN. Based on Julien's suggestion I am using the bin/crawl script and did the following tweaks to trigger a fetch with multiple map tasks , however I am unable to do so. 1. Added maxNumSegments and numFetchers parameters to the generate phase. $bin/nutch generate $commonOptions $CRAWL_PATH/crawldb $CRAWL_PATH/segments -maxNumSegments $numFetchers -numFetchers $numFetchers -noFilter 2. Removed the topN paramter and removed the noParsing parameter because I want the parsing to happen at the time of fetch. $bin/nutch fetch $commonOptions -D fetcher.timelimit.mins=$timeLimitFetch $CRAWL_PATH/segments/$SEGMENT -threads $numThreads #-noParsing# The generate phase is not generating more than one segment. And as a result the fetch phase is not creating multiple map tasks, also I belive the way the script is written it does not allow the fecth to fecth multiple segements in parallel even if the generate were to generate multiple segments. Can someone please let me know , how they go the script to run in a distributed Hadoop cluster ? Or if there is a different version of script that should be used? Thanks. -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble
Re: Running multiple fetch map tasks on a Hadoop Cluster.
Hi Meraj, Nutch and Hadoop abstract all of that for you, so you don’t need to worry about it. When you execute the fetch command for a segment, it will be parallelized across the nodes in your cluster. Cheers Jake On Sep 19, 2014, at 1:52 PM, Meraj A. Khan mera...@gmail.com wrote: Julien, How would you achieve parallelism then on a Hadoop cluster , am I missing something here? My understanding was that we could scale the crawl by allowing fetch to happen in multiple map tasks in multiple nodes in a Hadoop cluster , otherwise I am stuck in sequentially crawling a large set of urls spread across mutiple domains. If that is indeed the way to scale the crawl , then we would need to generate multiple segments at the generate time so that these could be fetched in paralle. So I guess I really need help in . 1. Making the generate phase generate multiple segments 2. Being able to fetch these segments in parallel. Can you please let me know if my approach to scale the crawl sounds right to you ? Thanks and much appreciated, all the help I have gotten so far On Fri, Sep 19, 2014 at 10:40 AM, Julien Nioche lists.digitalpeb...@gmail.com wrote: The fetching operates segment by segment and won't fetch more than one at the same time. You can get the generation step to build multiple segments in one go but you'd need to modify the script so that the fetching step is called as many times as you have segments + you'd probably need to add some logic for detecting that they've all finished before you move on to the update step. Out of curiosity : why do you want to fetch multiple segments at the same time? On 19 September 2014 06:00, Meraj A. Khan mera...@gmail.com wrote: Hello Folks, I am unable to run multiple fetch Map taks for Nutch 1.7 on Hadoop YARN. Based on Julien's suggestion I am using the bin/crawl script and did the following tweaks to trigger a fetch with multiple map tasks , however I am unable to do so. 1. Added maxNumSegments and numFetchers parameters to the generate phase. $bin/nutch generate $commonOptions $CRAWL_PATH/crawldb $CRAWL_PATH/segments -maxNumSegments $numFetchers -numFetchers $numFetchers -noFilter 2. Removed the topN paramter and removed the noParsing parameter because I want the parsing to happen at the time of fetch. $bin/nutch fetch $commonOptions -D fetcher.timelimit.mins=$timeLimitFetch $CRAWL_PATH/segments/$SEGMENT -threads $numThreads #-noParsing# The generate phase is not generating more than one segment. And as a result the fetch phase is not creating multiple map tasks, also I belive the way the script is written it does not allow the fecth to fecth multiple segements in parallel even if the generate were to generate multiple segments. Can someone please let me know , how they go the script to run in a distributed Hadoop cluster ? Or if there is a different version of script that should be used? Thanks. -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble
Re: Running multiple fetch map tasks on a Hadoop Cluster.
Jake, I am not sure how to make that happen, every time I run the nutch 1.7 job on YARN , I see a single segment being generated a nd a single map task bein launched,underutilizing the capacity of the cluster and slowing the crawl. Are you suggesting I should be seeing multiple fetch map tasks for a single segment, if so I am not. Thanks. On Sep 19, 2014 5:13 PM, Jake Dodd j...@ontopic.io wrote: Hi Meraj, Nutch and Hadoop abstract all of that for you, so you don’t need to worry about it. When you execute the fetch command for a segment, it will be parallelized across the nodes in your cluster. Cheers Jake On Sep 19, 2014, at 1:52 PM, Meraj A. Khan mera...@gmail.com wrote: Julien, How would you achieve parallelism then on a Hadoop cluster , am I missing something here? My understanding was that we could scale the crawl by allowing fetch to happen in multiple map tasks in multiple nodes in a Hadoop cluster , otherwise I am stuck in sequentially crawling a large set of urls spread across mutiple domains. If that is indeed the way to scale the crawl , then we would need to generate multiple segments at the generate time so that these could be fetched in paralle. So I guess I really need help in . 1. Making the generate phase generate multiple segments 2. Being able to fetch these segments in parallel. Can you please let me know if my approach to scale the crawl sounds right to you ? Thanks and much appreciated, all the help I have gotten so far On Fri, Sep 19, 2014 at 10:40 AM, Julien Nioche lists.digitalpeb...@gmail.com wrote: The fetching operates segment by segment and won't fetch more than one at the same time. You can get the generation step to build multiple segments in one go but you'd need to modify the script so that the fetching step is called as many times as you have segments + you'd probably need to add some logic for detecting that they've all finished before you move on to the update step. Out of curiosity : why do you want to fetch multiple segments at the same time? On 19 September 2014 06:00, Meraj A. Khan mera...@gmail.com wrote: Hello Folks, I am unable to run multiple fetch Map taks for Nutch 1.7 on Hadoop YARN. Based on Julien's suggestion I am using the bin/crawl script and did the following tweaks to trigger a fetch with multiple map tasks , however I am unable to do so. 1. Added maxNumSegments and numFetchers parameters to the generate phase. $bin/nutch generate $commonOptions $CRAWL_PATH/crawldb $CRAWL_PATH/segments -maxNumSegments $numFetchers -numFetchers $numFetchers -noFilter 2. Removed the topN paramter and removed the noParsing parameter because I want the parsing to happen at the time of fetch. $bin/nutch fetch $commonOptions -D fetcher.timelimit.mins=$timeLimitFetch $CRAWL_PATH/segments/$SEGMENT -threads $numThreads #-noParsing# The generate phase is not generating more than one segment. And as a result the fetch phase is not creating multiple map tasks, also I belive the way the script is written it does not allow the fecth to fecth multiple segements in parallel even if the generate were to generate multiple segments. Can someone please let me know , how they go the script to run in a distributed Hadoop cluster ? Or if there is a different version of script that should be used? Thanks. -- Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com http://twitter.com/digitalpebble
