[
https://issues.apache.org/jira/browse/EXEC-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17722296#comment-17722296
]
Gary D. Gregory commented on EXEC-120:
--------------------------------------
Sigh
> Some flaky tests
> ----------------
>
> Key: EXEC-120
> URL: https://issues.apache.org/jira/browse/EXEC-120
> Project: Commons Exec
> Issue Type: Bug
> Reporter: Agor
> Priority: Major
>
> Hello,
> We tried running this project and discovered that it contains some flaky
> tests (i.e., tests that nondeterministically pass and fail). We found these
> tests to fail more frequently when running them on certain machines of ours.
> To prevent others from running this project and its tests in machines that
> may result in flaky tests, we suggest adding information to the README.md
> file indicating the minimum resource configuration for running the tests of
> this project as to prevent observation of test flakiness.
> If we run this project in a machine with 1cpu and 1gb ram, we observe flaky
> tests. We found that the tests in this project did not have any flaky tests
> when we ran it on machines with 2cpu and 2gb ram.
> Here is a list of the tests we have identified and their likelihood of
> failure on a system with less than the recommended 2 CPUs and 2 GB RAM.
> # org.apache.commons.exec.issues.Exec34Test#testExec34_1 (2 out 10)
> #
> org.apache.commons.exec.issues.Exec65Test#testExec65WithSudoUsingShellScript
> (2 out 10)
> Please let me know if you would like us to create a pull request on this
> matter (possibly to the readme of this project).
> Thank you for your attention to this matter. We hope that our recommendations
> will be helpful in improving the quality and performance of your project,
> especially for others to use.
> Reproducing
> {code:java}
> FROM maven:3.5.4-jdk-11
> WORKDIR /home/
> RUN git clone https://github.com/apache/commons-exec && \
> cd commons-exec && \
> git checkout 7204a355ba5ffba21d828ce993b27c81b5565f43
> WORKDIR /home/commons-exec
> RUN mvn install -DskipTests
> ENTRYPOINT ["mvn", "test", "-fn"]
> {code}
> Build the image:
> {code:java}
> $> mkdir tmp
> $> cp Dockerfile tmp
> $> cd tmp
> $> docker build -t commons-exec . # estimated time of build 3m
> {code}
>
> Running:
> this configuration likely prevents flakiness (no flakiness in 10 runs)
> {code:java}
> $> docker run --rm --memory=2g --cpus=2 --memory-swap=-1 commons-exec | tee
> output.txt
> $> grep "Failures:" output.txt # checking results{code}
> this other configuration –similar to the previous– can’t prevent flaky tests
> (observation in 10 runs)
> {code:java}
> $> docker run --rm --memory=1g --cpus=1 --memory-swap=-1 commons-exec | tee
> output2.txt
> $> grep "Failures:" output2.txt # checking results{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
