[
https://issues.apache.org/jira/browse/CASSANDRA-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318726#comment-17318726
]
Michael Semb Wever edited comment on CASSANDRA-16587 at 4/11/21, 10:37 AM:
---------------------------------------------------------------------------
I can see two different paths forward here.
1) Admit default that our different 'test' targets are not parallel JVM
friendly, and remove everything related to 'testparallel' and 'test.runners'
from the build.xml. This would solve a number of problems: distributed in-jvm
tests, conf files, and byteman agent.
2) Fix the issues one-by-one, the best we can…
For (1), the performance issue is of concern: running tests one at a time on
ci-cassandra agents that have 4 cores and 16gb ram available to each executor
is a waste of resources and time. Going from one test.runner to two on these
agents, doubled the performance. If we were to remove test parallelism from the
build.xml (like [this
patch|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587-2/trunk])
it could be replaced with parallelism at the docker level, example patch
[here|https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16587-2/trunk].
For (2) the following patch, which is 100% completely reliable but shows a lot
of promise (see CI), makes byteman tests wait until the byteman port becomes
available before running the test…
-
https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587/trunk
CI
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/616/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/616/pipeline]
was (Author: michaelsembwever):
I can see two different paths forward here.
1) Admit default that our different 'test' targets are not parallel JVM
friendly, and remove everything related to 'testparallel' and 'test.runners'
from the build.xml. This would solve a number of problems: distributed in-jvm
tests, conf files, and byteman agent.
2) Fix the issues on-by-one, the best we can…
For (1), the performance issue is of concern: running tests one at a time on
ci-cassandra agents that have 4 cores and 16gb ram available to each executor
is a waste of resources and time. Going from one test.runner to two on these
agents, doubled the performance. If we were to remove test parallelism from the
build.xml (like [this
patch|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587-2/trunk])
it could be replaced with parallelism at the docker level, example patch
[here|https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16587-2/trunk].
For (2) the following patch, which is 100% completely reliable but shows a lot
of promise (see CI), makes byteman tests wait until the byteman port becomes
available before running the test…
-
https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587/trunk
CI
[!https://ci-cassandra.apache.org/job/Cassandra-devbranch/616/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/616/pipeline]
> flaky AntiCompactionBytemanTest.testRedundantTransitions byteman Connection
> refused
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-16587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16587
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
> Reporter: Michael Semb Wever
> Priority: Normal
> Fix For: 4.0.x
>
>
> Failing occasionally with
> {noformat}
> java.net.ConnectException: Connection refused (Connection refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:607)
> at java.net.Socket.connect(Socket.java:556)
> at java.net.Socket.<init>(Socket.java:452)
> at java.net.Socket.<init>(Socket.java:229)
> at org.jboss.byteman.agent.submit.Submit$Comm.<init>(Submit.java:881)
> at org.jboss.byteman.agent.submit.Submit.submitRequest(Submit.java:787)
> at org.jboss.byteman.agent.submit.Submit.addScripts(Submit.java:603)
> at
> org.jboss.byteman.contrib.bmunit.BMUnit.loadScriptText(BMUnit.java:291)
> at
> org.jboss.byteman.contrib.bmunit.BMUnitRunner$9.evaluate(BMUnitRunner.java:362)
> at
> org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:263)
> at
> org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:97)
> {noformat}
> Looking through the [logs
> |https://nightlies.apache.org/cassandra/trunk/Cassandra-trunk-test-cdc/650/Cassandra-trunk-test-cdc/jdk=jdk_11_latest,label=cassandra,split=6/build/test/logs/cdc/TEST-org.apache.cassandra.db.compaction.AntiCompactionBytemanTest.log.xz]
> this correlates to
> {noformat}
> Setting org.jboss.byteman.allow.config.update=true
> TransformListener() : unexpected exception opening server socket
> java.net.BindException: Address already in use (Bind failed)
> java.net.BindException: Address already in use (Bind failed)
> at java.net.PlainSocketImpl.socketBind(Native Method)
> at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
> at java.net.ServerSocket.bind(ServerSocket.java:390)
> at java.net.ServerSocket.bind(ServerSocket.java:344)
> at
> org.jboss.byteman.agent.TransformListener.initialize(TransformListener.java:81)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.byteman.agent.Main.premain(Main.java:286)
> at org.jboss.byteman.agent.Main.agentmain(Main.java:309)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
> at
> sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
