[
https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-10892:
---------------------------------
Description:
I did not find unit test in {{trunk}} code for following cases:
- HDFS command {{dfs -tail}}
- HDFS command {{dfs -stat}}
I think it still merits to have one though the commands have served us for
years.
was:
I did not find unit test in {{trunk}} code for following cases:
- HDFS command {{dfs -tail}}
- HDFS command {{dfs -stat}}
- file name or content with UTF-8 characters
I think it still merits to have one though the commands have served us for
years.
The {{hadoop.hdfs.TestDFSShell#testUtf8Encoding}} was not able to run in
Jenkins. I was able to pass the test locally without any problem. Perhaps it
has something to with the {{LANG}} environment variable according to
discussions on the Internet. I can confirm that my local test machine is using
following settings. I also think Yetus is setting the local correctly. [~aw] Do
you have any idea about this? Thanks.
{code}
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
{code}
As this test case is not tightly related to the commands "-tail" or "-stat", I
remove this test case in the latest patch. Just for what it's worth, the test
code is:
{code}
/**
* Test that the file name and content can have UTF-8 chars.
*/
@Test (timeout = 30000)
public void testUtf8Encoding() throws Exception {
final int blockSize = 1024;
final Configuration conf = new HdfsConfiguration();
conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize);
try (MiniDFSCluster cluster =
new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) {
cluster.waitActive();
final DistributedFileSystem dfs = cluster.getFileSystem();
final Path workDir= new Path("/testUtf8Encoding");
dfs.mkdirs(workDir);
System.setProperty("sun.jnu.encoding", "UTF-8");
System.setProperty("file.encoding", "UTF-8");
final String chineseStr = "哈杜普.txt";
final File testFile = new File(TEST_ROOT_DIR, chineseStr);
// create a local file; its content contains the Chinese file name
createLocalFile(testFile);
dfs.copyFromLocalFile(new Path(testFile.getPath()), workDir);
assertTrue(dfs.exists(new Path(workDir, testFile.getName())));
final ByteArrayOutputStream out = new ByteArrayOutputStream();
System.setOut(new PrintStream(out));
final String argv[] = new String[]{
"-cat", workDir + "/" + testFile.getName()};
final int ret = ToolRunner.run(new FsShell(conf), argv);
assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret,
0, ret);
assertTrue("Unexpected -cat output: " + out,
out.toString().contains(chineseStr));
}
}
{code}
If anyone suggests a working approach, I'd like to submit separate JIRA for
tracking this test. Otherwise, we test it elsewhere (say, nightly system tests).
> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> -----------------------------------------------------------
>
> Key: HDFS-10892
> URL: https://issues.apache.org/jira/browse/HDFS-10892
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, shell, test
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Attachments: HDFS-10892.000.patch, HDFS-10892.001.patch,
> HDFS-10892.002.patch, HDFS-10892.003.patch
>
>
> I did not find unit test in {{trunk}} code for following cases:
> - HDFS command {{dfs -tail}}
> - HDFS command {{dfs -stat}}
> I think it still merits to have one though the commands have served us for
> years.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
