Hi,
I took a look around on the Internet, but I didn't find any docs about
MiniDFS
and MiniMRCluster. Is there docs about them?
It remember me this phrase I got from the Herriot [1] page.
"As always your best source of information and knowledge about any software
system is its source code" :)
Do you think is possible to have just one tool to cover all kinds of tests?
Another question, do you know if is possible to evaluate a MR program, eg
sort,
with Herriot considering several test data?
Thanks in Advance
--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/
On Mon, Feb 7, 2011 at 10:29 PM, Konstantin Boudnik <c...@apache.org> wrote:
> On Mon, Feb 7, 2011 at 04:20, Edson Ramiro <erlfi...@gmail.com> wrote:
> > Well, I'm studying the Hadoop test tools to evaluate some (if there are)
> > deficiences, also trying to compare these tools to see what one cover
> that
> > other doesn't and what is possible to do with each one.
>
> There's also a simulated test cluster infrastructure called MiniDFS
> and MiniMRCluster to allow you to develop functional tests without
> actual cluster deployment.
>
> > As far as I know we have just Herriot and MRUnit for test, and them do
> > different things as you said me :)
> >
> > I'm very interested in your initial version, is there a link?
>
> Not at the moment, but I will send it here as soon as a initial
> version is pushed out.
>
> >
> > Thanks in advance
> >
> > --
> > Edson Ramiro Lucas Filho
> > {skype, twitter, gtalk}: erlfilho
> > http://www.inf.ufpr.br/erlf07/
> >
> >
> > On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <c...@apache.org>
> wrote:
> >
> >> Yes, Herriot can be used for integration tests of MR. Unit test is a
> very
> >> different thing and normally is done against a 'unit of compilation'
> e.g. a
> >> class, etc. Typically you won't expect to do unit tests against a
> deployed
> >> cluster.
> >>
> >> There is fault injection framework wich works at the level of functional
> >> tests
> >> (with mini-clusters). Shortly we'll be opening an initial version of
> smoke
> >> and
> >> integration test framework (maven and JUnit based).
> >>
> >> It'd be easier to provide you with a hint if you care to explain what
> >> you're
> >> trying to solve.
> >>
> >> Cos
> >>
> >> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
> >> > Thank you a lot Konstantin, you cleared my mind.
> >> >
> >> > So, Herriot is a framework designed to test Hadoop as a whole, and
> (IMHO)
> >> is
> >> > a tool for help Hadoop developers and not for who is developing MR
> >> programs,
> >> > but can we use Herriot to do unit, integration or other tests on our
> MR
> >> > jobs?
> >> >
> >> > Do you know another test tool or test framework for Hadoop?
> >> >
> >> > Thanks in Advance
> >> >
> >> > --
> >> > Edson Ramiro Lucas Filho
> >> > {skype, twitter, gtalk}: erlfilho
> >> > http://www.inf.ufpr.br/erlf07/
> >> >
> >> >
> >> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <c...@apache.org>
> >> wrote:
> >> >
> >> > > (Moving to common-user where this belongs)
> >> > >
> >> > > Herriot is system test framework which runs against a real physical
> >> > > cluster deployed with a specially crafted build of Hadoop. That
> >> > > instrumented build of provides an extra APIs not available in Hadoop
> >> > > otherwise. These APIs are created to facilitate cluster software
> >> > > testability. Herriot isn't limited by MR but also covered (although
> in
> >> > > a somewhat lesser extend) HDFS side of Hadoop.
> >> > >
> >> > > MRunit is for MR job "unit" testing as in making sure that your MR
> job
> >> > > is ok and/or to allow you to debug it locally before scale
> deployment.
> >> > >
> >> > > So, long story short - they are very different ;) Herriot can do
> >> > > intricate fault injection and can work closely with a deployed
> cluster
> >> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
> >> > > testing.
> >> > >
> >> > > Hope it helps.
> >> > > --
> >> > > Take care,
> >> > > Konstantin (Cos) Boudnik
> >> > >
> >> > >
> >> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <erlfi...@gmail.com>
> wrote:
> >> > > > Hi all,
> >> > > >
> >> > > > Plz, could you explain me the difference between MRUnit and
> Herriot?
> >> > > >
> >> > > > I've read the documentation of both and they seem very similar to
> me.
> >> > > >
> >> > > > Is Herriot an evolution of MRUnit?
> >> > > >
> >> > > > What can Herriot do that MRUnit can't?
> >> > > >
> >> > > > Thanks in Advance
> >> > > >
> >> > > > --
> >> > > > Edson Ramiro Lucas Filho
> >> > > > {skype, twitter, gtalk}: erlfilho
> >> > > > http://www.inf.ufpr.br/erlf07/
> >> > > >
> >> > >
> >>
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v1.4.10 (GNU/Linux)
> >>
> >> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe
> >> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD
> >> =h/66
> >> -----END PGP SIGNATURE-----
> >>
> >>
> >
>
