[ 
https://issues.apache.org/jira/browse/BAHIR-83?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kadner updated BAHIR-83:
----------------------------------
    Description: 
The test case _"Recovering offset from the last processed offset."_ in 
{{BasicMQTTSourceSuite}} fails sporadically.

I did repeated runs of that particular test and it failed 6 of 100 runs.
{code}
for i in `seq 100` ; do
  mvn scalatest:test -pl sql-streaming-mqtt -q \
    -Dsuites='*.BasicMQTTSourceSuite @ Recovering offset from the last 
processed offset.' | \
    grep -q "TEST FAILED" && echo "$i: failed"
done
1: failed
6: failed
22: failed
46: failed
77: failed
83: failed
{code}

The complete test suite BasicMQTTSourceSuite failed 56/100 times.


*_EDIT 2017/01/13:_*

 - the two test cases {{Send and receive 100 messages}} and {{Recovering offset 
from the last processed offset}} in {{BasicMQTTSourceSuite}} appear to fail 
when system resources are limited or under stress

- I was able to reproduce similar test failures by putting additional stress on 
CPU, memory and disk I/O using the 
[stress|http://people.seas.harvard.edu/~apw/stress/] tool

- when CPU-bound, the test cases can be modified to succeed by increasing the 
stream write timeout from 10 seconds to 15 or 20 seconds. However that 
increased timeout did not help when memory or I/O is limited

- it may be worth identifying and document minimum resource requirements for 
the SQL-MQTT connector. If we want to get sophisticated, that information could 
possibly be used to dynamically enable/disable test cases based on available 
system resources


  was:
The test case _"Recovering offset from the last processed offset."_ in 
{{BasicMQTTSourceSuite}} fails sporadically.

I did repeated runs of that particular test and it failed 6 of 100 runs.
{code}
for i in `seq 100` ; do
  mvn scalatest:test -pl sql-streaming-mqtt -q \
    -Dsuites='*.BasicMQTTSourceSuite @ Recovering offset from the last 
processed offset.' | \
    grep -q "TEST FAILED" && echo "$i: failed"
done
1: failed
6: failed
22: failed
46: failed
77: failed
83: failed
{code}

The complete test suite BasicMQTTSourceSuite failed 56/100 times.

EDIT



> Flaky test in BasicMQTTSourceSuite
> ----------------------------------
>
>                 Key: BAHIR-83
>                 URL: https://issues.apache.org/jira/browse/BAHIR-83
>             Project: Bahir
>          Issue Type: Test
>          Components: Spark Structured Streaming Connectors
>    Affects Versions: Spark-2.0.1
>         Environment: Maven 3.3.9, JDK 1.8, 2015 MacBook Pro, 16 GB RAM, 50 GB 
> free storage
>            Reporter: Christian Kadner
>            Assignee: Prashant Sharma
>
> The test case _"Recovering offset from the last processed offset."_ in 
> {{BasicMQTTSourceSuite}} fails sporadically.
> I did repeated runs of that particular test and it failed 6 of 100 runs.
> {code}
> for i in `seq 100` ; do
>   mvn scalatest:test -pl sql-streaming-mqtt -q \
>     -Dsuites='*.BasicMQTTSourceSuite @ Recovering offset from the last 
> processed offset.' | \
>     grep -q "TEST FAILED" && echo "$i: failed"
> done
> 1: failed
> 6: failed
> 22: failed
> 46: failed
> 77: failed
> 83: failed
> {code}
> The complete test suite BasicMQTTSourceSuite failed 56/100 times.
> *_EDIT 2017/01/13:_*
>  - the two test cases {{Send and receive 100 messages}} and {{Recovering 
> offset from the last processed offset}} in {{BasicMQTTSourceSuite}} appear to 
> fail when system resources are limited or under stress
> - I was able to reproduce similar test failures by putting additional stress 
> on CPU, memory and disk I/O using the 
> [stress|http://people.seas.harvard.edu/~apw/stress/] tool
> - when CPU-bound, the test cases can be modified to succeed by increasing the 
> stream write timeout from 10 seconds to 15 or 20 seconds. However that 
> increased timeout did not help when memory or I/O is limited
> - it may be worth identifying and document minimum resource requirements for 
> the SQL-MQTT connector. If we want to get sophisticated, that information 
> could possibly be used to dynamically enable/disable test cases based on 
> available system resources



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to