On Fri, 2005-11-04 at 22:57 -0500, Rod Taylor wrote:
> On Fri, 2005-11-04 at 19:43 -0800, Doug Cutting wrote:
> > Rod Taylor wrote:
> > > I tried running one datanode per machine connecting back to the same SAN
> > > but it seemed pretty clunky. A crash of any datanode would take down
> > > the entire system (no data replication since it's a common data-store in
> > > the end). Reducing it to a single datanode did not have this impact.
> >
> > Why use NDFS at all? Why not just mount the SAN on all hosts? You're
> > not using NDFS as a distributed file system, but rather as a centralized
> > file system.
>
> I was unable to make the mapred branch work by using 'local' as the
> filesystem and having more than one tasktracker. Tasktrackers were
> unable to complete any work, although it was quite a while ago when I
> last tried (September).
Here you go. local filesystem and a single job tracker on another
machine. When the tasktracker and jobtracker are on the same box there
isn't a problem. When they are on different machines it runs into
issues.
This is using mapred.local.dir on the local machine (not sharedd between
sbider4 and sbider5):
051104 230802 parsing
file:/opt/nutch-0.8_7/conf/nutch-default.xml
051104 230802 parsing
file:/opt/nutch-0.8_7/conf/mapred-default.xml
051104 230802
parsing /home/sitesell/localt/taskTracker/task_m_o59djj/job.xml
[Fatal Error] :-1:-1: Premature end of file.
051104 230802 SEVERE error parsing conf file:
org.xml.sax.SAXParseException: Premature end of file.
java.lang.RuntimeException: org.xml.sax.SAXParseException:
Premature end of file.
at
org.apache.nutch.util.NutchConf.loadResource(NutchConf.java:358)
at
org.apache.nutch.util.NutchConf.getProps(NutchConf.java:293)
at
org.apache.nutch.util.NutchConf.get(NutchConf.java:94)
at
org.apache.nutch.mapred.JobConf.getJar(JobConf.java:81)
at org.apache.nutch.mapred.TaskTracker
$TaskInProgress.localizeTask(TaskTracker.java:332)
at org.apache.nutch.mapred.TaskTracker
$TaskInProgress.<init>(TaskTracker.java:314)
at
org.apache.nutch.mapred.TaskTracker.offerService(TaskTracker.java:214)
at
org.apache.nutch.mapred.TaskTracker.run(TaskTracker.java:268)
at
org.apache.nutch.mapred.TaskTracker.main(TaskTracker.java:633)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown
Source)
at
org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at
javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:172)
at
org.apache.nutch.util.NutchConf.loadResource(NutchConf.java:318)
... 8 more
051104 230802 Lost connection to JobTracker
[sbider5.sitebuildit.com/192.168.100.14:5464]. Retrying...
This is using a shared mapred.local.dir on the SAN:
051104 232115 parsing
file:/opt/nutch-0.8_7/conf/nutch-default.xml
051104 232115 parsing
file:/opt/nutch-0.8_7/conf/mapred-default.xml
051104 232115
parsing
/opt/sitesell/sbider_data/test/local/taskTracker/task_m_l86ntl/job.xml
[Fatal Error] :-1:-1: Premature end of file.
051104 232116 SEVERE error parsing conf file:
org.xml.sax.SAXParseException: Premature end of file.
java.lang.RuntimeException: org.xml.sax.SAXParseException:
Premature end of file.
at
org.apache.nutch.util.NutchConf.loadResource(NutchConf.java:358)
at
org.apache.nutch.util.NutchConf.getProps(NutchConf.java:293)
at
org.apache.nutch.util.NutchConf.get(NutchConf.java:94)
at
org.apache.nutch.mapred.JobConf.getJar(JobConf.java:81)
at org.apache.nutch.mapred.TaskTracker
$TaskInProgress.localizeTask(TaskTracker.java:332)
at org.apache.nutch.mapred.TaskTracker
$TaskInProgress.<init>(TaskTracker.java:314)
at
org.apache.nutch.mapred.TaskTracker.offerService(TaskTracker.java:214)
at
org.apache.nutch.mapred.TaskTracker.run(TaskTracker.java:268)
at
org.apache.nutch.mapred.TaskTracker.main(TaskTracker.java:633)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown
Source)
at
org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at
javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:172)
at
org.apache.nutch.util.NutchConf.loadResource(NutchConf.java:318)
... 8 more
051104 232116 Lost connection to JobTracker
[sbider5.sitebuildit.com/192.168.100.14:5464]. Retrying...
--
Rod Taylor <[EMAIL PROTECTED]>
