Martin Bukatovic created BIGTOP-1307:
----------------------------------------
Summary: Some TestCLI cases fail with 'No such file or directory'
Key: BIGTOP-1307
URL: https://issues.apache.org/jira/browse/BIGTOP-1307
Project: Bigtop
Issue Type: Bug
Components: Tests
Affects Versions: 0.8.0
Environment: HDP 2.0.6
Reporter: Martin Bukatovic
Priority: Critical
I observe weird results of xml-defined test cases of TestCLI bigtop test.
{{{
14/05/15 16:50:41 INFO cli.CLITestHelper: Testing mode: test
14/05/15 16:50:41 INFO cli.CLITestHelper:
14/05/15 16:50:41 INFO cli.CLITestHelper: Overall result: --- FAIL
---
14/05/15 16:50:41 INFO cli.CLITestHelper: # Tests pass: 322 (65%)
14/05/15 16:50:41 INFO cli.CLITestHelper: # Tests fail: 169 (34%)
14/05/15 16:50:41 INFO cli.CLITestHelper: # Validations done: 1251
(each test may do multiple validations)
}}}
Where 137 testcases failed because of 'No such file or directory' error.
To show what the problem is, see testcase #1:
{{{
14/05/15 16:50:40 INFO cli.CLITestHelper:
-------------------------------------------
14/05/15 16:50:40 INFO cli.CLITestHelper: Test ID: [1]
14/05/15 16:50:40 INFO cli.CLITestHelper: Test Description: [ls:
file using absolute path]
14/05/15 16:50:40 INFO cli.CLITestHelper:
14/05/15 16:50:40 INFO cli.CLITestHelper: Test Commands: [-fs
hdfs://dhcp-lab-203.local:8020 -touchz /tmp/testcli_1400165386646/file1]
14/05/15 16:50:40 INFO cli.CLITestHelper: Test Commands: [-fs
hdfs://dhcp-lab-203.local:8020 -ls /tmp/testcli_1400165386646/file1]
14/05/15 16:50:40 INFO cli.CLITestHelper:
14/05/15 16:50:40 INFO cli.CLITestHelper: Cleanup Commands: [-fs
hdfs://dhcp-lab-203.local:8020 -rm /tmp/testcli_1400165386646/file1]
14/05/15 16:50:40 INFO cli.CLITestHelper:
14/05/15 16:50:40 INFO cli.CLITestHelper: Comparator:
[TokenComparator]
14/05/15 16:50:40 INFO cli.CLITestHelper: Comparision result: [fail]
14/05/15 16:50:40 INFO cli.CLITestHelper: Expected output: [Found
1 items]
14/05/15 16:50:40 INFO cli.CLITestHelper: Actual output: [ls:
`/tmp/testcli_1400165386646/file1': No such file or directory
]
14/05/15 16:50:40 INFO cli.CLITestHelper: Comparator:
[RegexpComparator]
14/05/15 16:50:40 INFO cli.CLITestHelper: Comparision result: [fail]
14/05/15 16:50:40 INFO cli.CLITestHelper: Expected output:
[^-rw-r--r--( )*1( )*[a-z]*( )*hdfs( )*0( )*[0-9]{4,}-[0-9]{2,}-[0-9]{2,}
[0-9]{2,}:[0-9]{2,}( )*/tmp/testcli_1400165386646/file1]
14/05/15 16:50:40 INFO cli.CLITestHelper: Actual output: [ls:
`/tmp/testcli_1400165386646/file1': No such file or directory
]
14/05/15 16:50:40 INFO cli.CLITestHelper:
14/05/15 16:50:40 INFO cli.CLITestHelper:
-------------------------------------------
}}}
The results looks as if there were someting wrong with hadoop/hdfs.
Nevertheless when I checked this particular case manually, it worked just fine:
{{{
[bigtop@dhcp-lab-203 testcli]$ hadoop fs -mkdir /tmp/testcli_1400165386646
[bigtop@dhcp-lab-203 testcli]$ hadoop fs -fs hdfs://dhcp-lab-203.local:8020
-touchz /tmp/testcli_1400165386646/file1
[bigtop@dhcp-lab-203 testcli]$ hadoop fs -fs hdfs://dhcp-lab-203.local:8020 -ls
/tmp/testcli_1400165386646/file1
Found 1 items
-rw-r--r-- 3 bigtop hdfs 0 2014-05-15 17:08
/tmp/testcli_1400165386646/file1
[bigtop@dhcp-lab-203 testcli]$ hadoop fs -fs hdfs://dhcp-lab-203.local:8020 -rm
/tmp/testcli_1400165386646/file1
14/05/15 17:08:27 INFO fs.TrashPolicyDefault: Namenode trash configuration:
Deletion interval = 21600000 minutes, Emptier interval = 0 minutes.
Moved: 'hdfs://dhcp-lab-203.local:8020/tmp/testcli_1400165386646/file1' to
trash at: hdfs://dhcp-lab-203.local:8020/user/bigtop/.Trash/Current
[bigtop@dhcp-lab-203 testcli]$
}}}
I manually checked 5 other cases with the same result: when the testcase is done
manually, it works without any problems.
Moreover I rerunt all TestCLI cases 5 times, and the set of failed cases
was always the same.
Note: I used the following script to parse the logs for me to filter which
cases fails on 'No such file' error.
{{{
#!/bin/bash
sed 's/.*CLITestHelper://' ${1} \
| awk '
/Test ID:/, /^-{43}$/ {
# save the testcase id
if ($1 == "Test" && $2 == "ID:") TEST_ID=$3;
# increment a counter of IO errors of current testcase
if ($0 ~ "No such file or directory") TESTCASES[TEST_ID]+=1;
}
END {
for (i in TESTCASES) { printf("%5s %5d\n", i, TESTCASES[i]) }
}
' | sort
}}}
Have anybody seen similar behaviour? I have executed TestCLI cases via wrapper
which sets system classpath instead of maven defined enviroment. Can this
caseud the issue, or is it likely that the problem is the bigtop tests? Also
feel free to propose a way to debug this further.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
