[
https://issues.apache.org/jira/browse/BIGTOP-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766712#comment-13766712
]
Martin Bukatovic commented on BIGTOP-1032:
------------------------------------------
I went through the code of hdfs testaces and it seem to me that there are more
changes needed. I propose the following:
1) Change TestCLI.java (and FSCmdExecutor.java which is used by TestCLI) to use
hadoop executable (/usr/lib/hadoop/bin/hadoop) instead of using
org.apache.hadoop.fs.FsShell directly. The choice of which hadoop filesystem
will be used can be done via setting few system properties. This way we
solve several problems:
* fs specific configuration doesn't need to be included in the TestCLI
code, we assume that these tests are run on properly configured hadoop
cluster.
* there is zero additional work (besides specification of the properties)
to run this test with any hadoop compatible fs. Eg. with current state, to
make it run with glusterfs, one would have to rewrite the test code and
edit pom.xml for classpath to include jar with glusterfs implementation
(which is reduntant as you already specified this in core-site.xml).
* and last but not least using command line is more appropriate for this
kind of testing anyway
My first attempt how this may look like is attached - it's not a patch, I
changed FSCmdExecutor.java and TestCLI.java - and it's not fully working
code yet.
2) Most of the other tests are HDFS specific and as such should not be run on
hadoop pools using alternative hadoop compatible filesystems:
* TestDFSAdmin.groovy
* TestFsck.groovy
* TestFuseDFS.groovy
* TestHDFSBalancer.groovy
* TestHDFSQuota.groovy
On the other hand TestDistCpIntra.groovy, TestFileAppend.groovy and
TestTextSnappy.groovy may be executed even on non HDFS pool, but may
require minor updates - I didn't try this yet though).
To make it work with any HCFS, I would create 2 JUnit groups, one for HDFS
specific tests, the other for HCFS tests - but I'm not sure how this would
fit into current workflow. Any better ideas here?
Does this whole idea make sense for you? What would you change?
> Refactor hdfs/TestCLI and hdfs/TestAppend to be generic filesystem tests
> -------------------------------------------------------------------------
>
> Key: BIGTOP-1032
> URL: https://issues.apache.org/jira/browse/BIGTOP-1032
> Project: Bigtop
> Issue Type: Improvement
> Reporter: jay vyas
>
> The TestCLI and TestAppend classes both have hard references to an HDFS user
> shell, although most of their functionality is actually geared towards
> testing the abstract FileSystem interface.
> Removing a couple of lines from TestCLI:
> String[] createTestcliDirCmds = {"hadoop fs -mkdir -p " +
> TEST_DIR_ABSOLUTE,
> "hadoop fs -chmod 777 " + TEST_DIR_ABSOLUTE};
> shHDFS.exec(createTestcliDirCmds);
> And a this line from TestAppend:
> shHDFS.exec("hadoop fsck
> /user/$USERNAME/$testAppendInput/test2.file$date");
> This will essentially allow bigtop suites to be used to validate file system
> functionality in clusters with heterogenous
> https://wiki.apache.org/hadoop/HCFS/ implementations.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
