[
https://issues.apache.org/jira/browse/FLINK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510448#comment-17510448
]
Aitozi edited comment on FLINK-25480 at 3/22/22, 12:15 PM:
-----------------------------------------------------------
FYI, I encounter the same problem with 1.14.4 when running test in container. I
test in 16C 32G container, and {{mvn -Dflink.forkCount=2
-Dflink.forkCountTestPackage=2 -Dmaven.test.failure.ignore=true verify}}
command exit 137 finally. At the meantime, I opened another screen to run
{{vsar --cpu --mem -l}} to monitor the memory usage. But I still not catch the
memory stroke. Hope to be helpful to your guys. I'm curious about it, because
it stop me from building our stable CI pipeline.
was (Author: aitozi):
FYI, I encounter the same problem with 1.14.4 when running test in container. I
test in 16C 32G container, and {{mvn verify}} command exit 137 finally. At the
meantime, I opened another screen to run {{vsar --cpu --mem -l}} to monitor
the memory usage. But I still not catch the memory stroke. Hope to be helpful
to your guys. I'm curious about it, because it stop me from building our stable
CI pipeline.
> Create dashboard/monitoring to see resource usage per E2E test
> --------------------------------------------------------------
>
> Key: FLINK-25480
> URL: https://issues.apache.org/jira/browse/FLINK-25480
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
> Affects Versions: 1.15.0, 1.13.6, 1.14.3
> Reporter: Martijn Visser
> Priority: Critical
> Labels: test-stability
>
> Over the past couple of weeks, we've encountered multiple problems with tests
> failing due to out-of-memory errors and/or exit code 137 happening. These are
> happening both on Alibaba CI machines, as well as Azure hosted agents. For
> the Alibaba CI machines, we've mitigated the problem by reducing the number
> of workers per CI machine from 7 to 5. These workers can spin up multiple
> Docker containers, especially with Testcontainers getting used more and more.
> If we can get insights in the resource usage per end-to-end test, it will
> also help in debugging test infrastructure problems more quickly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
