Thanks, Suneel!
I just followed your suggestion and added the annotation (and fixed the
project to add the relevant jars in the classpath), but it still generated
similar error messages.
It seems carrort search still tried to interrupt the threads (ignoring the
annotation). I am not sure why is this. Thanks!
Wei
My code:
@ThreadLeakAction({ThreadLeakAction.Action.WARN})
@Test
public void simpleTest() throws IOException{
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path filenamePath = new Path("/tmp/tmp.txt");
FSDataOutputStream out = fs.create(filenamePath);
out.close();
fs.close();
}
Error message:
Jun 17, 2014 7:23:25 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
WARNING: Will linger awaiting termination of 1 leaked thread(s).
Jun 17, 2014 7:23:45 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
SEVERE: 1 thread leaked from SUITE scope at
org.apache.mahout.wei.inc.util.HDFSUtilTest:
1) Thread[id=15, name=org.apache.hadoop.hdfs.LeaseRenewer$1@7b968017,
state=TIMED_WAITING, group=TGRP-HDFSUtilTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:397)
at org.apache.hadoop.hdfs.LeaseRenewer.access$600(
LeaseRenewer.java:69)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273)
at java.lang.Thread.run(Unknown Source)
Jun 17, 2014 7:23:45 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
INFO: Starting to interrupt leaked threads:
1) Thread[id=15, name=org.apache.hadoop.hdfs.LeaseRenewer$1@7b968017,
state=TIMED_WAITING, group=TGRP-HDFSUtilTest]
Jun 17, 2014 7:23:45 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
INFO: All leaked threads terminated.
From: Suneel Marthi <[email protected]>
To: "[email protected]" <[email protected]>,
Date: 06/17/2014 01:57 AM
Subject: Re: a seemingly benign test that fails MahoutTestCase
Annotate ur test case with the following:
@ThreadLeakAction({ThreadLeakAction.Action.WARN)
(This is from Carrot Randomized Test framework, ensure that u have the
relevant jars in ur classpath for this to compile)
that should throw a Warning as opposed to interrupting the thread. As
Ted's said Hadoop code's indeed cavalier about thread leaks, so
warnings/errors like this can be expected when running in Carrot Randomized
testcontainer.
On Tue, Jun 17, 2014 at 1:35 AM, Ted Dunning <[email protected]> wrote:
> Well, the problem is that there is a thread leak. It is not clear
whether
> the file creation succeeded.
>
> It isn't that surprising that there is a thread leak since Hadoop code is
> pretty cavalier about this and the Hadoop test cases don't check for
this.
>
> I think that there might be a way to signal that this is ok, possibly
with
> annotations. Somebody who knows the test randomizer framework would have
> to comment on that.
>
>
> On Mon, Jun 16, 2014 at 6:38 PM, Wei Zhang <[email protected]> wrote:
>
> > Thanks, Ted. I did try to do the fs.close(); The code looks like this:
> > @Test
> > *public* *void* simpleTest() *throws* IOException{
> > Configuration conf = *new* Configuration();
> > FileSystem fs = FileSystem.*get*(conf);
> > Path filenamePath = *new* Path("/tmp/tmp.txt");
> > FSDataOutputStream out = fs.create(filenamePath);
> > out.close();
> > fs.close();
> > }
> >
> > It still generates the same error message.
> >
> > Thanks!
> >
> > Wei
> >
> > [image: Inactive hide details for Ted Dunning ---06/16/2014 09:29:28
> > PM---Close the FileSystem object (fs). On Mon, Jun 16, 2014 at 6:2]Ted
> > Dunning ---06/16/2014 09:29:28 PM---Close the FileSystem object (fs).
On
> > Mon, Jun 16, 2014 at 6:22 PM, Wei Zhang <[email protected]> wrote
> >
> > From: Ted Dunning <[email protected]>
> > To: "[email protected]" <[email protected]>,
> > Date: 06/16/2014 09:29 PM
> > Subject: Re: a seemingly benign test that fails MahoutTestCase
> > ------------------------------
> >
> >
> >
> > Close the FileSystem object (fs).
> >
> >
> >
> >
> > On Mon, Jun 16, 2014 at 6:22 PM, Wei Zhang <[email protected]> wrote:
> >
> > > Hello Ted,
> > >
> > > Thank you very much for the reply! I have created a github gist that
> > > records the error message at
> > > https://gist.github.com/anonymous/21a2f36c995c1d39717b
> > >
> > > I am sorry that I forgot to attach the error log, it is something
that
> > > looks like this (just in case that github gist is displaying
> > incorrectly):
> > > Jun 16, 2014 6:07:10 PM
> > > com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> > > WARNING: Will linger awaiting termination of 2 leaked thread(s).
> > > Jun 16, 2014 6:07:30 PM
> > > com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> > > SEVERE: 1 thread leaked from SUITE scope at
> > > org.apache.mahout.wei.inc.util.HDFSUtilTest:
> > > 1) Thread[id=15, name=org.apache.hadoop.hdfs.LeaseRenewer
$1@7b968017
> ,
> > > state=TIMED_WAITING, group=TGRP-HDFSUtilTest]
> > > at java.lang.Thread.sleep(*Native Method*)
> > > at
> > org.apache.hadoop.hdfs.LeaseRenewer.run(*LeaseRenewer.java:397*
> > > )
> > > at org.apache.hadoop.hdfs.LeaseRenewer.access$600(
> > > *LeaseRenewer.java:69*)
> > > at org.apache.hadoop.hdfs.LeaseRenewer$1.run(
> > > *LeaseRenewer.java:273*)
> > > at java.lang.Thread.run(Unknown Source)
> > > Jun 16, 2014 6:07:30 PM
> > > com.carrotsearch.randomizedtesting.ThreadLeakControl
tryToInterruptAll
> > > INFO: Starting to interrupt leaked threads:
> > > 1) Thread[id=15, name=org.apache.hadoop.hdfs.LeaseRenewer
$1@7b968017
> ,
> > > state=TIMED_WAITING, group=TGRP-HDFSUtilTest]
> > > Jun 16, 2014 6:07:30 PM
> > > com.carrotsearch.randomizedtesting.ThreadLeakControl
tryToInterruptAll
> > > INFO: All leaked threads terminated.
> > >
> > > Thanks!
> > >
> > > Wei
> > >
> > >
> > > [image: Inactive hide details for Ted Dunning ---06/16/2014 07:50:30
> > > PM---Attachments are stripped by the mailing list. Can you throw]Ted
> > > Dunning ---06/16/2014 07:50:30 PM---Attachments are stripped by the
> > mailing
> > > list. Can you throw the stack trace onto a github gist or s
> > >
> > > From: Ted Dunning <[email protected]>
> > > To: "[email protected]" <[email protected]>,
> > > Date: 06/16/2014 07:50 PM
> > > Subject: Re: a seemingly benign test that fails MahoutTestCase
> > > ------------------------------
> > >
> > >
> > >
> > > Attachments are stripped by the mailing list. Can you throw the
stack
> > > trace onto a github gist or something?
> > >
> > >
> > > On Mon, Jun 16, 2014 at 3:27 PM, Wei Zhang <[email protected]> wrote:
> > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > I am wrting a simple Unit test, which extends MahoutTestCase. I am
> > using
> > > > Mahout 0.9 and Hadoop 1.0.3.
> > > >
> > > > The test is quite simple, it just creates an empty file on HDFS.
The
> > > entire
> > > > coding is the following:
> > > >
> > > > @Test
> > > > public void simpleTest() throws IOException{
> > > > Configuration conf = new Configuration();
> > > > FileSystem fs = FileSystem.get(conf);
> > > > Path filenamePath = new Path("/tmp/tmp.txt");
> > > > FSDataOutputStream out = fs.create(filenamePath);
> > > > out.close();
> > > > }
> > > >
> > > > However, this will fail the MahoutTestCase. In particular, the
> moment I
> > > > added the line 4 (FSDataOutputStream out = fs.create
(filenamePath);),
> > > this
> > > > test would fail.
> > > > I am wondering can anyone give me some pointers regarding why this
> > naive
> > > > program would fail ?
> > > > The error log is attached below.
> > > >
> > > > Thank you very much !
> > > >
> > > > Wei
> > >
> > >
> >
> >
>
