[
https://issues.apache.org/jira/browse/HTTPCORE-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732740#comment-17732740
]
Agor commented on HTTPCORE-747:
-------------------------------
[~olegk] , sorry for the delay, and thank you for or your feedback.
I understand that the test cases are passing consistently for you locally and
on the CI. However, I'd like to share the significance of including a minimum
specification in the project README. This edition aims to prevent potential
flaky test scenarios that may occur for other users or contributors in their
machines.
The main goal of our contribution is to educate developers and users about the
importance of having a minimum requirement for stable test execution. By
specifying the minimum required RAM and CPU in the README, we can avoid
unexpected test failures caused by insufficient resources on developers'
machines.
Including this information requires minimal effort, yet it can save developers
from the frustration of debugging flaky test failures if their machines don't
meet the recommended specifications.
Thank you for your consideration.
> Some flaky tests
> ----------------
>
> Key: HTTPCORE-747
> URL: https://issues.apache.org/jira/browse/HTTPCORE-747
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: Documentation, HttpCore
> Affects Versions: 5.2.1
> Reporter: Agor
> Priority: Major
> Fix For: 5.2.2, 5.3-alpha1
>
>
> Hello,
> We tried running this project and discovered that it contains some flaky
> tests (i.e., tests that nondeterministically pass and fail). We found these
> tests to fail more frequently when running them on certain machines of ours.
> To prevent others from running this project and its tests in machines that
> may result in flaky tests, we suggest adding information to the README.md
> file indicating the minimum resource configuration for running the tests of
> this project as to prevent observation of test flakiness.
> If we run this project in a machine with 1cpu and 1gb ram, we observe flaky
> tests. We found that the tests in this project did not have any flaky tests
> when we ran it on machines with 2cpu and 2gb ram.
> Here is a list of the tests we have identified and their likelihood of
> failure on a system with less than the recommended 2 CPUs and 2 GB RAM.
> # org.apache.hc.core5.pool.TestPoolEntry#testExpiry
> # org.apache.hc.core5.pool.TestLaxConnPool#testCloseIdle
> #
> org.apache.hc.core5.testing.classic.ClassicIntegrationTest#testHttpPostsWithExpectContinue
> Please let me know if you would like us to create a pull request on this
> matter (possibly to the readme of this project).
> Thank you for your attention to this matter. We hope that our recommendations
> will be helpful in improving the quality and performance of your project,
> especially for others to use.
> h3. Reproducing
> {code:java}
> FROM maven:3.5.4-jdk-11
> WORKDIR /home/
> RUN git clone https://github.com/apache/httpcomponents-core && \
> cd httpcomponents-core && \
> git checkout 9f8cdb17c85defb267fc814d6094bfda3c8ff1a7
> WORKDIR /home/httpcomponents-core
> RUN mvn install -DskipTests
> ENTRYPOINT ["mvn", "test", "-fn"]
> {code}
> Build the image:
> {code:java}
> $> mkdir tmp
> $> cp Dockerfile tmp
> $> cd tmp
> $> docker build -t http-core . # estimated time of build 3m
> {code}
> Running:
> this configuration likely prevents flakiness (no flakiness in 10 runs)
> {code:java}
> $> docker run --rm --memory=2g --cpus=2 --memory-swap=-1 http-core | tee
> output.txt
> $> grep "Failures:" output.txt # checking results{code}
> this other configuration –similar to the previous– can’t prevent flaky tests
> (observation in 10 runs)
> {code:java}
> $> docker run --rm --memory=500m --cpus=1 --memory-swap=-1 http-core | tee
> output2.txt
> $> grep "Failures:" output2.txt # checking results{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org
For additional commands, e-mail: dev-h...@hc.apache.org
