Hi
Nutch uses Log4j and with it you can write log output from different
classes or different log levels to different output files. I'm sure this
will work with Nutch in local mode so i believe you can make it happen
with Hadoop but may be tricky, or not possible.
Cheers
On Fri, 11 May 2012 14:59:13 +0200, Ferdy Galema
<[email protected]> wrote:
There is, every task gets run a temporary working directory. But in
general
the output is cleaned after the task completes. If you want to save
"side
data" you have to figure a workaround. This page should give you a
few
pointers:
http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Task+Side-Effect+Files
On Fri, May 11, 2012 at 2:36 PM, Vijith <[email protected]>
wrote:
Thanks Ferdy.
So does this mean that there is no way nutch can connect to a flat
file /
database etc. while in deploy mode.
On Fri, May 11, 2012 at 5:44 PM, Ferdy Galema
<[email protected]
>wrote:
> When running hadoop in deploy mode the actual tasks are ran by the
> mapreduce framework so you have to check the mapreduce "user"
logs.
Either
> use the jobtracker interface or check them directly on the nodes
in
> HADOOP_HOME/logs/userlogs or something like that.
>
> On Fri, May 11, 2012 at 1:11 PM, Vijith <[email protected]>
wrote:
>
> > I have tried with a seperate logger and a printWriter objects to
do
this.
> > It works in local mode but not in deploy mode.
> > I am running the nutch job file. Its running and generating the
hadoop
> log
> > without any errors. But the files are not created in any of the
nodes.
> >
> > On Fri, May 11, 2012 at 3:07 PM, Vijith <[email protected]>
wrote:
> >
> > > Hi,
> > >
> > > How can I create a separate project specific log in addition
to the
> > > existing log.
> > > I am running nutch in eploy mode.
> > > Also I want some urls filtered by my urlfilter to be stored in
an
> > external
> > > flat file. How can I achieve this.
> > >
> > > --
> > > *Thanks & Regards*
> > > *
> > > *
> > > *Vijith V*
> > >
> > >
> > >
> >
> >
> > --
> > *Thanks & Regards*
> > *
> > *
> > *Vijith V*
> >
>
--
*Thanks & Regards*
*
*
*Vijith V*
--
Markus Jelsma - CTO - Openindex
