[
https://issues.apache.org/jira/browse/FLUME-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324471#comment-14324471
]
Tony Reix commented on FLUME-2625:
----------------------------------
Here are more details about the unstable tests.
T: Total of tests
F: Failure
E: Error
S: Skipped
I've launched the tests 8 times (files mvn.Test.OpenJDK.res ,
mvn.Test.OpenJDK.res4 .... mvn.Test.OpenJDK.res10), within the same environment.
Details of results of each test is given for the 8 files (numbered from 0 to 7).
As an example, about .source.avroLegacy.TestLegacyAvroSource test, there are 1
or 2 errors:
T F E S
.source.avroLegacy.TestLegacyAvroSource 0: 2 0 2 0 |
.source.avroLegacy.TestLegacyAvroSource 1: 2 0 2 0 |
.source.avroLegacy.TestLegacyAvroSource 2: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 3: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 4: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 5: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 6: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 7: 2 0 1 0 |
$ ./BuildUnstable.sh -A mvn.Test.OpenJDK.res mvn.Test.OpenJDK.res4
mvn.Test.OpenJDK.res5 mvn.Test.OpenJDK.res6 mvn.Test.OpenJDK.res7
mvn.Test.OpenJDK.res8 mvn.Test.OpenJDK.res9 mvn.Test.OpenJDK.res10
Files:
0: mvn.Test.OpenJDK.res
1: mvn.Test.OpenJDK.res4
2: mvn.Test.OpenJDK.res5
3: mvn.Test.OpenJDK.res6
4: mvn.Test.OpenJDK.res7
5: mvn.Test.OpenJDK.res8
6: mvn.Test.OpenJDK.res9
7: mvn.Test.OpenJDK.res10
Unstable Tests:
------------------------------------------- T F E S |
.sink.TestThriftSink 0: 3 0 0 0 |
.sink.TestThriftSink 1: 3 0 3 0 |
.sink.TestThriftSink 2: 3 0 3 0 |
.sink.TestThriftSink 3: 3 0 3 0 |
.sink.TestThriftSink 4: 3 0 3 0 |
.sink.TestThriftSink 5: 3 0 3 0 |
.sink.TestThriftSink 6: 3 0 3 0 |
.sink.TestThriftSink 7: 3 0 3 0 |
------------------------------------------- T F E S |
.source.TestSyslogUdpSource 0: 3 0 0 0 |
.source.TestSyslogUdpSource 1: 3 3 0 0 |
.source.TestSyslogUdpSource 2: 3 3 0 0 |
.source.TestSyslogUdpSource 3: 3 3 0 0 |
.source.TestSyslogUdpSource 4: 3 3 0 0 |
.source.TestSyslogUdpSource 5: 3 3 0 0 |
.source.TestSyslogUdpSource 6: 3 3 0 0 |
.source.TestSyslogUdpSource 7: 3 3 0 0 |
------------------------------------------- T F E S |
.instrumentation.http.TestHTTPMetricsServer 0: 3 0 0 0 |
.instrumentation.http.TestHTTPMetricsServer 1: 3 0 3 0 |
.instrumentation.http.TestHTTPMetricsServer 2: 3 0 3 0 |
.instrumentation.http.TestHTTPMetricsServer 3: 3 0 3 0 |
.instrumentation.http.TestHTTPMetricsServer 4: 3 0 3 0 |
.instrumentation.http.TestHTTPMetricsServer 5: 3 0 3 0 |
.instrumentation.http.TestHTTPMetricsServer 6: 3 0 3 0 |
.instrumentation.http.TestHTTPMetricsServer 7: 3 0 3 0 |
------------------------------------------- T F E S |
.source.TestThriftSource 0: 4 0 0 0 |
.source.TestThriftSource 1: 4 1 3 0 |
.source.TestThriftSource 2: 4 1 3 0 |
.source.TestThriftSource 3: 4 1 3 0 |
.source.TestThriftSource 4: 4 1 3 0 |
.source.TestThriftSource 5: 4 1 3 0 |
.source.TestThriftSource 6: 4 1 3 0 |
.source.TestThriftSource 7: 4 1 3 0 |
------------------------------------------- T F E S |
.channel.TestSpillableMemoryChannel 0: 14 0 0 0 |
.channel.TestSpillableMemoryChannel 1: 0 0 0 0 |
.channel.TestSpillableMemoryChannel 2: 14 0 0 0 |
.channel.TestSpillableMemoryChannel 3: 0 0 0 0 |
.channel.TestSpillableMemoryChannel 4: 0 0 0 0 |
.channel.TestSpillableMemoryChannel 5: 0 0 0 0 |
.channel.TestSpillableMemoryChannel 6: 14 0 0 0 |
.channel.TestSpillableMemoryChannel 7: 0 0 0 0 |
------------------------------------------- T F E S |
.api.TestThriftRpcClient 0: 6 0 0 0 |
.api.TestThriftRpcClient 1: 6 0 3 0 |
.api.TestThriftRpcClient 2: 6 0 3 0 |
.api.TestThriftRpcClient 3: 6 0 3 0 |
.api.TestThriftRpcClient 4: 6 0 3 0 |
.api.TestThriftRpcClient 5: 6 0 3 0 |
.api.TestThriftRpcClient 6: 6 0 3 0 |
.api.TestThriftRpcClient 7: 6 0 3 0 |
------------------------------------------- T F E S |
.source.thriftLegacy.TestThriftLegacySource 0: 3 0 0 0 |
.source.thriftLegacy.TestThriftLegacySource 1: 3 2 0 0 |
.source.thriftLegacy.TestThriftLegacySource 2: 3 2 0 0 |
.source.thriftLegacy.TestThriftLegacySource 3: 3 2 0 0 |
.source.thriftLegacy.TestThriftLegacySource 4: 3 2 0 0 |
.source.thriftLegacy.TestThriftLegacySource 5: 3 2 0 0 |
.source.thriftLegacy.TestThriftLegacySource 6: 3 2 0 0 |
.source.thriftLegacy.TestThriftLegacySource 7: 3 2 0 0 |
------------------------------------------- T F E S |
.source.TestAvroSource 0: |
.source.TestAvroSource 1: 12 0 8 0 |
.source.TestAvroSource 2: 12 0 8 0 |
.source.TestAvroSource 3: 12 0 8 0 |
.source.TestAvroSource 4: 12 0 8 0 |
.source.TestAvroSource 5: 12 0 8 0 |
.source.TestAvroSource 6: 12 0 8 0 |
.source.TestAvroSource 7: 12 0 8 0 |
------------------------------------------- T F E S |
.source.avroLegacy.TestLegacyAvroSource 0: 2 0 2 0 |
.source.avroLegacy.TestLegacyAvroSource 1: 2 0 2 0 |
.source.avroLegacy.TestLegacyAvroSource 2: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 3: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 4: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 5: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 6: 2 0 1 0 |
.source.avroLegacy.TestLegacyAvroSource 7: 2 0 1 0 |
------------------------------------------- T F E S |
.source.TestMultiportSyslogTCPSource 0: 4 0 0 0 |
.source.TestMultiportSyslogTCPSource 1: 4 0 1 0 |
.source.TestMultiportSyslogTCPSource 2: 4 0 1 0 |
.source.TestMultiportSyslogTCPSource 3: 4 0 1 0 |
.source.TestMultiportSyslogTCPSource 4: 4 0 1 0 |
.source.TestMultiportSyslogTCPSource 5: 4 0 1 0 |
.source.TestMultiportSyslogTCPSource 6: 4 0 1 0 |
.source.TestMultiportSyslogTCPSource 7: 4 0 1 0 |
------------------------------------------- T F E S |
.api.TestLoadBalancingRpcClient 0: 15 0 12 0 |
.api.TestLoadBalancingRpcClient 1: 15 0 0 0 |
.api.TestLoadBalancingRpcClient 2: 15 0 0 0 |
.api.TestLoadBalancingRpcClient 3: 15 0 0 0 |
.api.TestLoadBalancingRpcClient 4: 15 0 0 0 |
.api.TestLoadBalancingRpcClient 5: 15 0 0 0 |
.api.TestLoadBalancingRpcClient 6: 15 0 0 0 |
.api.TestLoadBalancingRpcClient 7: 15 0 0 0 |
------------------------------------------- T F E S |
.source.TestSyslogTcpSource 0: 2 0 0 0 |
.source.TestSyslogTcpSource 1: 2 0 2 0 |
.source.TestSyslogTcpSource 2: 2 0 2 0 |
.source.TestSyslogTcpSource 3: 2 0 2 0 |
.source.TestSyslogTcpSource 4: 2 0 2 0 |
.source.TestSyslogTcpSource 5: 2 0 2 0 |
.source.TestSyslogTcpSource 6: 2 0 2 0 |
.source.TestSyslogTcpSource 7: 2 0 2 0 |
------------------------------------------- T F E S |
.source.TestExecSource 0: 10 0 0 0 |
.source.TestExecSource 1: 10 0 0 0 |
.source.TestExecSource 2: 10 0 0 0 |
.source.TestExecSource 3: 10 0 0 0 |
.source.TestExecSource 4: 10 1 0 0 |
.source.TestExecSource 5: 10 0 0 0 |
.source.TestExecSource 6: 10 0 0 0 |
.source.TestExecSource 7: 10 1 0 0 |
------------------------------------------- T F E S |
.channel.file.TestFileChannelRestart 0: 40 0 0 0 |
.channel.file.TestFileChannelRestart 1: 40 0 0 0 |
.channel.file.TestFileChannelRestart 2: 40 0 0 0 |
.channel.file.TestFileChannelRestart 3: 40 0 0 0 |
.channel.file.TestFileChannelRestart 4: 40 1 0 0 |
.channel.file.TestFileChannelRestart 5: 40 0 0 0 |
.channel.file.TestFileChannelRestart 6: 40 0 0 0 |
.channel.file.TestFileChannelRestart 7: 40 0 0 0 |
------------------------------------------- T F E S |
.sink.TestAvroSink 0: 15 0 0 0 |
.sink.TestAvroSink 1: 15 0 15 0 |
.sink.TestAvroSink 2: 15 0 0 0 |
.sink.TestAvroSink 3: 15 0 0 0 |
.sink.TestAvroSink 4: 15 0 0 0 |
.sink.TestAvroSink 5: 15 0 0 0 |
.sink.TestAvroSink 6: 15 0 0 0 |
.sink.TestAvroSink 7: 15 0 0 0 |
------------------------------------------- T F E S |
.source.http.TestHTTPSource 0: 14 0 0 0 |
.source.http.TestHTTPSource 1: 14 1 12 0 |
.source.http.TestHTTPSource 2: 14 1 12 0 |
.source.http.TestHTTPSource 3: 14 1 12 0 |
.source.http.TestHTTPSource 4: 14 1 12 0 |
.source.http.TestHTTPSource 5: 14 1 12 0 |
.source.http.TestHTTPSource 6: 14 1 12 0 |
.source.http.TestHTTPSource 7: 14 1 12 0 |
My first test file (mvn.Test.OpenJDK.res) shows much more success than the 7
other files. Maybe I did something special...
Anyway, it is clear that there is some random failures for tests:
.channel.TestSpillableMemoryChannel
.source.avroLegacy.TestLegacyAvroSource
.api.TestLoadBalancingRpcClient
.source.TestExecSource
.channel.file.TestFileChannelRestart
.sink.TestAvroSink
> There are several unstable tests within FLUME
> ---------------------------------------------
>
> Key: FLUME-2625
> URL: https://issues.apache.org/jira/browse/FLUME-2625
> Project: Flume
> Issue Type: Bug
> Components: Test
> Affects Versions: v1.5.0.1
> Environment: RHEL 7.1 / x86_64 / Open JDK 1.7
> Reporter: Tony Reix
>
> Hi,
> I'm working on porting FLUME in a RHEL 7.1 / PPC64LE / IBM JVM 1.7
> environment.
> As an example, I've found that the test .source.TestSyslogUdpSource fails,
> but not always, only 7 times out of 10 tries. Testing on RHEL 7.1 / x86_64 /
> IBM JVM, I've also had random failures.
> Running the same .source.TestSyslogUdpSource test in RHEL 7.1 / x86_64 / Open
> JDK 1.7 environment, I've found that this test fails only once out of 30
> tries: it is an "unstable" test.
> In order to find which test issues are specific to PPC64 or IBMJVM
> environment, I've run 10 times all the FLUME tests in the RHEL 7.1 / x86_64 /
> Open JDK 1.7 environment, which I call my "reference" environment.
> Then, using a tool that compares all the results, I've found that there are
> 16 tests that are "unstable" in my "reference" (x86_64/OpenJDK) .
> By "unstable", I mean to say that the results vary, though the environment is
> exactly the same.
> These tests are:
> .api.TestLoadBalancingRpcClient
> .api.TestThriftRpcClient
> .channel.file.TestFileChannelRestart
> .channel.TestSpillableMemoryChannel
> .instrumentation.http.TestHTTPMetricsServer
> .sink.TestAvroSink
> .sink.TestThriftSink
> .source.avroLegacy.TestLegacyAvroSource
> .source.http.TestHTTPSource
> .source.TestAvroSource
> .source.TestExecSource
> .source.TestMultiportSyslogTCPSource
> .source.TestSyslogTcpSource
> .source.TestSyslogUdpSource
> .source.TestThriftSource
> .source.thriftLegacy.TestThriftLegacySource
> About ".source.TestSyslogUdpSource" test, my analysis is that the test code
> is not reliable since the test checks that some data is correct without
> checking that all the "messages" have arrived (sometimes, a message has not
> arrived in time, and a reference is NULL).
> Adding "sleep(1000) to the test with IBM JVM, the test then failed only 3
> times out of 10.
> So, I think that several FLUME tests are coded in a way that is not 100%
> reliable. Or it could also be that some core code of FLUME is not 100%
> reliable.
> I mean to say that some code may have been written based on the specific
> behaviour of the OpenJDK Java Virtual Machine, which was used for testing.
> Some change about how the order of threads are launched, or about the time
> needed to send messages in the JVM/OS, may lead to issues that are not
> correctly handled by the code (mainly test code, but maybe core code too).
> And it seems that, though being perfectly correct, the IBM JVM does not work
> the same way compared to OpenJDK.
> So, this is a pain. Mainly in my PPC64LE/IBMJVM environment.
> I think that these 16 tests must be analysed and improved.
> Also, running tests with OpenJDK AND IBM JVM in your development and
> test/Jenkins environments would help to see these random issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
