Hello, with this commit I added a cleanup of the data dir before the DfsMiniCluster is started. I also use absolute file names to make debugging a bit easier and I moved initialisation code to the setUp() method
http://svn.apache.org/r1650847 & http://svn.apache.org/r1650852 This way the test do not error out anymore. But I have no idea why this was happening on one machine and not on others (maybe a race, the failing machine had SSD?). So this means, now I can concentrate on merging the new version. Gruss Bernd Am Sun, 11 Jan 2015 01:25:48 +0100 schrieb Bernd Eckenfels <e...@zusammenkunft.net>: > Hello, > > Am Sat, 10 Jan 2015 03:12:19 +0000 (UTC) > schrieb dlmar...@comcast.net: > > > Bernd, > > > > Regarding the Hadoop version for VFS 2.1, why not use the latest on > > the first release of the HDFS provider? The Hadoop 1.1.2 release was > > released in Feb 2013. > > Yes, you are right. We dont need to care about 2.0 as this is a new > provider. I will make the changes, just want to fix the current test > failures I see first. > > > > I just built 2.1-SNAPSHOT over the holidays with JDK 6, 7, and 8 on > > Ubuntu. What type of test errors are you getting? Testing is > > disabled on Windows unless you decide to pull in windows artifacts > > attached to VFS-530. However, those artifacts are associated with > > patch 3 and are for Hadoop 2.4.0. Updating to 2.4.0 would also be > > sufficient in my opinion. > > Yes, what I mean is: I typically build under Windows so I would not > notice if the test starts to fail. However it seems to pass on the > integration build: > > https://continuum-ci.apache.org/continuum/projectView.action?projectId=129&projectGroupId=16 > > Running > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest > Starting DataNode 0 with dfs.data.dir: > target/build/test/data/dfs/data/data1,target/build/test/data/dfs/data/data2 > Cluster is active Cluster is active Tests run: 13, Failures: 0, > Errors: 0, Skipped: 0, Time elapsed: 11.821 sec - in > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest > Running > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase > Starting DataNode 0 with dfs.data.dir: > target/build/test2/data/dfs/data/data1,target/build/test2/data/dfs/data/data2 > Cluster is active Cluster is active Tests run: 76, Failures: 0, > Errors: 0, Skipped: 0, Time elapsed: 18.853 sec - in > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase > > Anyway, on a Ubuntu, I get this exception currently: > > Running > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase > Starting DataNode 0 with dfs.data.dir: > target/build/test/data/dfs/data/data1,tar > get/build/test/data/dfs/data/data2 Cluster is active Cluster is > active Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time > elapsed: 1.486 sec <<< FA > ILURE! - in > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase > junit.framework.TestSuite@56c77035(org.apache.commons.vfs2.provider.hdfs.test.Hd > fsFileProviderTestCase$HdfsProviderTestSuite) Time elapsed: 1.479 > sec <<< ERRO R! > java.lang.RuntimeException: Error setting up mini cluster at > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase$H > dfsProviderTestSuite.setUp(HdfsFileProviderTestCase.java:112) at > org.apache.commons.vfs2.test.AbstractTestSuite$1.protect(AbstractTest > Suite.java:148) at > junit.framework.TestResult.runProtected(TestResult.java:142) at > org.apache.commons.vfs2.test.AbstractTestSuite.run(AbstractTestSuite. > java:154) at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner. > java:86) at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provide > r.java:283) at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUni > t4Provider.java:173) at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4 > Provider.java:153) at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider > .java:128) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameCla > ssLoader(ForkedBooter.java:203) at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Fork > edBooter.java:155) at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java: > 103) Caused by: java.io.IOException: Cannot lock storage > target/build/test/data/dfs/n > ame1. The directory is already locked. at > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(St > orage.java:599) at > org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13 > 27) at > org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:13 > 45) at > org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java: > 1207) at > org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java: > 187) at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:268) > at > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTestCase$H > dfsProviderTestSuite.setUp(HdfsFileProviderTestCase.java:107) ... 11 > more > > Running > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest Tests > run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.445 sec > - in > > When I delete the core/target/build/test/data/dfs/ directory and then > run the ProviderTest I can do that multiple times and it works: > > mvn surefire:test > -Dtest=org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest > > But when I run all tests or the HdfsFileProviderTestCase then it > fails and afterwards not even the ProviderTest suceeds until I delete > that dir. > > (I suspect the "locking" is a missleading error, looks more like the > data pool has some kind of instance ID which it does not have at the > next run) > > Looks like TestCase has a problem and ProviderTest does no proper > pre-cleaning. Will check the source. More generally speaking it > should not use a fixed working directory anyway. > > > > I started up Hadoop 2.6.0 on my laptop, created a directory and > > file, then used the VFS shell to list and view the contents > > (remember, HDFS provider is read-only currently). Here is the what > > I did: > > Looks good. I will shorten it a bit and add it to the wiki. BTW: the > warning, is this something we can change? > > Gruss > Bernd --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
