Further to this, I have found that I can only submit a maximum of 256 steps to EMR. Some of our crawls take over 100 rounds, so defining an arbitrary number of (generate,fetch,parse,updatedb,index,solrdedup) rounds each with 6 steps isn't going to work either :-(
Has nobody automated this? Thanks, Jim Sent: Thursday, November 17, 2016 at 11:30 AM From: "Jim Lamb" <[email protected]> To: [email protected] Subject: Re: Automating Nutch 2.3.1 on Amazon EMR Hi Sebastian, Thanks for coming back to me. > Adding > set -x > to bin/nutch and then running bin/crawl with a sample crawl which includes > all steps > should log all commands with a full list of arguments. Yes, that's a great idea. Thanks. > But on EMR it should be possible to directly reference the Nutch job file > by a s3:// URL. (but haven't tried it this way) Yes, that is possible. You add an S3 URL to the Jar= argument in your step definition of the create-cluster command. > aws emr terminate-cluster ... Ah, yes. I did wonder if the master instance had appropriate instance role privilege to do this. I'll try. Unfortunately, it still doesn't solve the iteration issue. Short of defining many many repeated sets of steps, I don't see how I would get multiple rounds. What am I missing? Thanks, Jim
