[ 
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429469#comment-17429469
 ] 

Andres de la Peña edited comment on CASSANDRA-17043 at 10/15/21, 7:38 PM:
--------------------------------------------------------------------------

I understand that the parallelisms for the regular test jobs are based on the 
number of tests in that job, while the parallelism for multiplexer jobs should 
be based on the number of repetitions, which defaults to 100.

I think that we can do is simply use dedicated executors for the multiplexer 
jobs, so they can use the same resource class as the original job, but with a 
different parallelism. I gave it a go in [the branch for 
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
 and I'll update the other branches if the approach looks good.

The default parallelisms for the multiplexer jobs are 4 for lowres, 25 for 
midres and 100 for highres. This is more or less in line with the previous 
values, but I wonder if we should go for higher number in lowres and midres.


was (Author: adelapena):
I understand that the parallelisms for the regular test jobs are based on the 
number of tests in that job, while the parallelism for multiplexer jobs should 
be based on the number of repetitions, which defaults to 100.

I think that we can do is simply use dedicated executors for the multiplexer 
jobs, so they can use the same resource class as the original job, but with a 
different parallelism. I gave it a go in [the branch for 
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
 and I'll update the other branches if the approach looks good.

The defaults parallelisms for the multiplexer jobs are 4 for lowres, 25 for 
midres and 100 for highres.

> CircleCI dtest multiplexer with MIDRES needs more resources
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-17043
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
>             Project: Cassandra
>          Issue Type: Task
>          Components: CI
>            Reporter: Andres de la Peña
>            Assignee: Andres de la Peña
>            Priority: Low
>             Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES, 
> which is necessary for some dtests to reliably success. However, the dtest 
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as 
> the regular dtests. Using too small resources in the multiplexer can lead to 
> failures that don't reproduce in the regular dtest jobs, like the one we 
> found in 
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
>  when trying to repeatedly run a resource-hungry dtest, or like [this other 
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
>  while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the 
> multiplexer. This doesn't affect HIGHRES because in that case the patch 
> changes the configuration of the test executor, while in MIDRES a new 
> executor is defined.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to