[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430058#comment-17430058
]
Ekaterina Dimitrova edited comment on CASSANDRA-17043 at 10/18/21, 3:04 PM:
I agree with everything said, the point for not exhausting all resources of
your organization is a good one. I don't think there is perfect answer but what
you suggest sounds reasonable. What I can suggest and add to the wiki page is
to advise people around parallelism so they can make an informed decision
whether to use the default one or change it. What do you think?
I also looked at the current patches you posted and they look reasonable to me.
I can test also the patches later today. Thank you!
was (Author: e.dimitrova):
I agree with everything said, the point for not exhausting all resources of
your organization is a good one. I don't think there is perfect answer but what
you suggest sounds reasonable. What I can suggest and add to the wiki page is
to advise to people around parallelism so they can make an informed decision
whether to use the default one or change it. What do you think?
I also looked at the current patches you posted and they look reasonable to me.
I can test also the patches later today. Thank you!
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430051#comment-17430051
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/18/21, 3:04 PM:
--
{quote}In that sense it makes sense to me to raise the parallelism for the
MIDRES multiplexer for the upgrade in-jvm tests for example. Why ? Because the
whole suite is 10-20 test classes, but the default run in a loop which I would
assume most people will run is 100. So if we generalize this means to me 10
times more time for execution if we think that most tests need similar amount
of time to run(do they?).
{quote}
I think that the amount of time to run tests varies wildly even for the same
type of tests. Not only we have very different tests, but we can also multiplex
either a method or its containing suite. If the suite has ten methods, that run
will take ten times more. So I think it's difficult to adjust parallelism for
an expected run time if we don't know what test is going to be repeated,
differently to the relative fixed sets of tests that we have for the standard
jobs. I guess that the choice of parallelism for the multiplexer is going to be
based more on how much resources we want to invest than on the running time
that we aim to achieve.
The current parallelisms of 4/25/100 for low/mid/high resources save some of
the overhead of starting new runners in low and mid configs, although I don't
know how noticeable it would be in practice. Another reason not to use a very
high parallelism is that I understand that the maximum number of concurrent
runners is limited per organization, so a job with a very high parallelism can
produce starvation in other users. I understand that when users with access to
high resources choose to run with low/mid config they are not in a big hurry
and they can wait a bit longer in order not to exhaust the resources for other
users. Having a very high parallelism in these configs could make this type of
not-invasive runs more difficult to do.
That said, I have no clue about what parallelism would be better for the
average case, although the current values have been good to me by now. I'd be
happy to increase the parallelism of {{repeated_dtest}} if we find it too low,
what value for what config would you suggest?
Here are the updated patches with some multiplexer runs:
||Branch||CI low||CI mid||CI high||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1057/workflows/cd90268d-8af7-4d8c-9522-a277398acdc4]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1051/workflows/2550a673-5fc2-4748-b1c6-1ce56b5b2002]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1048/workflows/9d8a08dd-0db3-4a9e-911f-eb64a87741ff]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:17043-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1055/workflows/2412eeca-2a58-424c-9986-d6fb71cd13d4]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1054/workflows/fa78a6e8-be03-4ea8-9169-f92b03cc2a54]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1050/workflows/a7f29c05-ea57-40dd-84dd-e695514ca392]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:17043-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1056/workflows/08393fb6-17db-4d91-b3ed-5d7800c56dc1]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1056/workflows/23eeb750-e163-406c-9430-aa37e67b0717]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1053/workflows/15d171cd-cc45-4718-ac83-90710c725097]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1053/workflows/d5c4ef81-04d0-4081-8a9c-f5f1753debd8]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1049/workflows/3047dfae-3b8f-4716-8c18-db4a2fcc8fd5]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1049/workflows/7de5327d-360c-407d-8ab6-0b80fb58ddaf]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:17043-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1060/workflows/34a75acf-e786-4b6f-a4c6-9d878e7f1351]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1060/workflows/51beffb0-2ccc-4ce0-b7e4-0cabac0ac223]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1061/workflows/4842863d-b324-4e0f-a95f-a14ec1fb66d4]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1061/workflows/09bef172-f992-4ab5-b945-97966246a927]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1062/workflows/eff4b3c0-1a90-440a-a432-4ebabe716ff4]
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430058#comment-17430058
]
Ekaterina Dimitrova edited comment on CASSANDRA-17043 at 10/18/21, 3:03 PM:
I agree with everything said, the point for not exhausting all resources of
your organization is a good one. I don't think there is perfect answer but what
you suggest sounds reasonable. What I can suggest and add to the wiki page is
to advise to people around parallelism so they can make an informed decision
whether to use the default one or change it. What do you think?
I also looked at the current patches you posted and they look reasonable to me.
I can test also the patches later today. Thank you!
was (Author: e.dimitrova):
I agree with everything said, the point for not exhausting all resources of
your organization is a good one. I don't think there is perfect answer but what
you suggest sounds reasonable. What I can suggest and add to the wiki page is
advise to people around parallelism so they can make an informed decision
whether to use the default one or change it. What do you think?
I also looked at the current patches you posted and they look reasonable to me.
I can test also the patches later today. Thank you!
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430051#comment-17430051
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/18/21, 3:00 PM:
--
{quote}In that sense it makes sense to me to raise the parallelism for the
MIDRES multiplexer for the upgrade in-jvm tests for example. Why ? Because the
whole suite is 10-20 test classes, but the default run in a loop which I would
assume most people will run is 100. So if we generalize this means to me 10
times more time for execution if we think that most tests need similar amount
of time to run(do they?).
{quote}
I think that the amount of time to run tests varies wildly even for the same
type of tests. Not only we have very different tests, but we can also multiplex
either a method or its containing suite. If the suite has ten methods, that run
will take ten times more time. So I think it's difficult to adjust parallelism
for an expected run time if we don't know what test is going to be repeated,
differently to the relative fixed sets of tests that we have for the standard
jobs. I guess that the choice of parallelism for the multiplexer is going to be
based more in how much resources we want to invest than in the running time
that we aim to achieve.
The current parallelisms of 4/25/100 for low/mid/high resources save some of
the overhead of starting new runners in low and mid configs, although I don't
know how noticeable would it be in practice. Another reason to not use a very
high parallelism is that I understand that the maximum number of concurrent
runners is limited per organization, so a job with a very high parallelism can
produce starvation in other users. I understand that when users with access to
high resources choose to run with low/mid config they are not in a big hurry
and they can wait a bit longer in order to don't exhaust the resources for
other users. Having a very high parallelism in these configs could make this
type of not-invasive runs more difficult to do.
That said, I have no clue about what parallelism would be better for the
average case, although the current values have been good to me by now. I'd be
happy to increase the parallelism of {{repeated_dtest}} if we find it too low,
what value for what config would you suggest?
Here are the updated patches with some multiplexer runs:
||Branch||CI low||CI mid||CI high||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1057/workflows/cd90268d-8af7-4d8c-9522-a277398acdc4]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1051/workflows/2550a673-5fc2-4748-b1c6-1ce56b5b2002]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1048/workflows/9d8a08dd-0db3-4a9e-911f-eb64a87741ff]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:17043-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1055/workflows/2412eeca-2a58-424c-9986-d6fb71cd13d4]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1054/workflows/fa78a6e8-be03-4ea8-9169-f92b03cc2a54]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1050/workflows/a7f29c05-ea57-40dd-84dd-e695514ca392]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:17043-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1056/workflows/08393fb6-17db-4d91-b3ed-5d7800c56dc1]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1056/workflows/23eeb750-e163-406c-9430-aa37e67b0717]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1053/workflows/15d171cd-cc45-4718-ac83-90710c725097]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1053/workflows/d5c4ef81-04d0-4081-8a9c-f5f1753debd8]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1049/workflows/3047dfae-3b8f-4716-8c18-db4a2fcc8fd5]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1049/workflows/7de5327d-360c-407d-8ab6-0b80fb58ddaf]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:17043-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1060/workflows/34a75acf-e786-4b6f-a4c6-9d878e7f1351]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1060/workflows/51beffb0-2ccc-4ce0-b7e4-0cabac0ac223]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1061/workflows/4842863d-b324-4e0f-a95f-a14ec1fb66d4]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1061/workflows/09bef172-f992-4ab5-b945-97966246a927]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1062/workflows/eff4b3c0-1a90-440a-a432-4ebabe716ff4]
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429469#comment-17429469
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/15/21, 7:43 PM:
--
I understand that the parallelisms for the regular test jobs are based on the
number of tests in that job, while the parallelism for the multiplexer jobs
should be based on the number of repetitions, which defaults to 100.
I think that what we can do is simply use dedicated executors for the
multiplexer jobs, so they can use the same resource class as the original job,
but with a different parallelism. I gave it a go in [the branch for
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
and I'll update the other branches if the approach looks good.
The default parallelisms for the multiplexer jobs are 4 for lowres, 25 for
midres and 100 for highres. This is more or less in line with the previous
values, but I wonder if we should go for higher numbers in lowres and midres.
was (Author: adelapena):
I understand that the parallelisms for the regular test jobs are based on the
number of tests in that job, while the parallelism for multiplexer jobs should
be based on the number of repetitions, which defaults to 100.
I think that we can do is simply use dedicated executors for the multiplexer
jobs, so they can use the same resource class as the original job, but with a
different parallelism. I gave it a go in [the branch for
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
and I'll update the other branches if the approach looks good.
The default parallelisms for the multiplexer jobs are 4 for lowres, 25 for
midres and 100 for highres. This is more or less in line with the previous
values, but I wonder if we should go for higher numbers in lowres and midres.
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429469#comment-17429469
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/15/21, 7:39 PM:
--
I understand that the parallelisms for the regular test jobs are based on the
number of tests in that job, while the parallelism for multiplexer jobs should
be based on the number of repetitions, which defaults to 100.
I think that we can do is simply use dedicated executors for the multiplexer
jobs, so they can use the same resource class as the original job, but with a
different parallelism. I gave it a go in [the branch for
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
and I'll update the other branches if the approach looks good.
The default parallelisms for the multiplexer jobs are 4 for lowres, 25 for
midres and 100 for highres. This is more or less in line with the previous
values, but I wonder if we should go for higher numbers in lowres and midres.
was (Author: adelapena):
I understand that the parallelisms for the regular test jobs are based on the
number of tests in that job, while the parallelism for multiplexer jobs should
be based on the number of repetitions, which defaults to 100.
I think that we can do is simply use dedicated executors for the multiplexer
jobs, so they can use the same resource class as the original job, but with a
different parallelism. I gave it a go in [the branch for
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
and I'll update the other branches if the approach looks good.
The default parallelisms for the multiplexer jobs are 4 for lowres, 25 for
midres and 100 for highres. This is more or less in line with the previous
values, but I wonder if we should go for higher number in lowres and midres.
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429469#comment-17429469
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/15/21, 7:38 PM:
--
I understand that the parallelisms for the regular test jobs are based on the
number of tests in that job, while the parallelism for multiplexer jobs should
be based on the number of repetitions, which defaults to 100.
I think that we can do is simply use dedicated executors for the multiplexer
jobs, so they can use the same resource class as the original job, but with a
different parallelism. I gave it a go in [the branch for
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
and I'll update the other branches if the approach looks good.
The default parallelisms for the multiplexer jobs are 4 for lowres, 25 for
midres and 100 for highres. This is more or less in line with the previous
values, but I wonder if we should go for higher number in lowres and midres.
was (Author: adelapena):
I understand that the parallelisms for the regular test jobs are based on the
number of tests in that job, while the parallelism for multiplexer jobs should
be based on the number of repetitions, which defaults to 100.
I think that we can do is simply use dedicated executors for the multiplexer
jobs, so they can use the same resource class as the original job, but with a
different parallelism. I gave it a go in [the branch for
3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0],
and I'll update the other branches if the approach looks good.
The defaults parallelisms for the multiplexer jobs are 4 for lowres, 25 for
midres and 100 for highres.
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429299#comment-17429299
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/15/21, 3:04 PM:
--
Oh, I totally forgot about the multiplexer jobs for upgrade tests! I understand
that those jobs should use exactly the same resource class as the equivalent
job for regular upgrade tests. I have updated the patches and config files to
use the same executors as the corresponding jobs:
||Branch||CI||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1040/workflows/cd94b8ee-16ee-4e03-85d7-b51c0ede30ac]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:17043-3.11]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1041/workflows/60cfc2c6-1b87-4eca-bf0c-a76326741160]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:17043-4.0]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1037/workflows/3c412ef8-0e16-48f7-bd8c-5812ccbbcf12]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1037/workflows/f96639ad-e952-4727-90bf-a0228691bc50]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:17043-trunk]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1036/workflows/3e2b79da-609c-44e9-afd5-6337e429a0bc]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1036/workflows/eb1225b6-957d-446c-8b4f-8f6fea6b557b]|
However those executors have a quite low parallelism, which might be enough for
running the relatively small set of regular upgrade jobs but that can be too
low for the repeated runs of a specific test. Perhaps the multiplexer jobs
should use separate, dedicated executors with the same resource class and more
parallelism? Or could we just increase the parallelism of the current
executors? wdyt?
was (Author: adelapena):
Oh, I totally forgot about the multiplexer jobs for upgrade tests! I understand
that those jobs should use exactly the same resource class as the equivalent
job for regular upgrade tests. I have updated the patches and config files to
use the same executors as the corresponding jobs:
||Branch||
|[3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...adelapena:17043-3.0]|
|[3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...adelapena:17043-3.11]|
|[4.0|https://github.com/apache/cassandra/compare/cassandra-4.0...adelapena:17043-4.0]|
|[trunk|https://github.com/apache/cassandra/compare/trunk...adelapena:17043-trunk]|
However those executors have a quite low parallelism, which might be enough for
running the relatively small set of regular upgrade jobs but that can be too
low for the repeated runs of a specific test. Perhaps the multiplexer jobs
should use separate, dedicated executors with the same resource class and more
parallelism? Or could we just increase the parallelism of the current
executors? wdyt?
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands,
[jira] [Comment Edited] (CASSANDRA-17043) CircleCI dtest multiplexer with MIDRES needs more resources
[
https://issues.apache.org/jira/browse/CASSANDRA-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428902#comment-17428902
]
Andres de la Peña edited comment on CASSANDRA-17043 at 10/14/21, 4:40 PM:
--
These simple patches modify the patch file that we use to generate the MIDRES
CircleCI config file, so the dtest multiplexer jobs use the same resources as
the regular dtests jobs:
||patch||CI||
|[3.0|https://github.com/adelapena/cassandra/commit/3acd025e8f4f8396b95a3231275dc6b778cae9aa]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1030/workflows/deb0d4c5-3d2a-423a-93b7-5ad1bc692f78]|
|[3.11|https://github.com/adelapena/cassandra/commit/ca4a19d89380f46dd29d630ba7b5c6c25696c729]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1028/workflows/9309e88e-7386-4cac-871b-0a7641b64b96]|
|[4.0|https://github.com/adelapena/cassandra/commit/8d385d7462417a44d04cfc30c48fe5c6066a60ac]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1029/workflows/db1b57c8-4166-4d6e-a528-1949811e7565]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1029/workflows/bbdc1224-1bae-4b33-bf2e-c1df344e1146]|
|[trunk|https://github.com/adelapena/cassandra/commit/3973e410d34cb0d003fc74014b4b3bee8a165a21]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1027/workflows/ee7e4ffd-3eec-49fd-ad55-028731501811]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1027/workflows/eaa21c86-b148-45df-b01a-d100175ba8ca]|
[~edimitrova] could you please review?
was (Author: adelapena):
This simple patches modify the patch file that we use to generate the MIDRES
CircleCI config file, so the dtest multiplexer jobs use the same resources as
the regular dtests jobs:
||patch||CI||
|[3.0|https://github.com/adelapena/cassandra/commit/3acd025e8f4f8396b95a3231275dc6b778cae9aa]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1030/workflows/deb0d4c5-3d2a-423a-93b7-5ad1bc692f78]|
|[3.11|https://github.com/adelapena/cassandra/commit/ca4a19d89380f46dd29d630ba7b5c6c25696c729]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1028/workflows/9309e88e-7386-4cac-871b-0a7641b64b96]|
|[4.0|https://github.com/adelapena/cassandra/commit/8d385d7462417a44d04cfc30c48fe5c6066a60ac]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1029/workflows/db1b57c8-4166-4d6e-a528-1949811e7565]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1029/workflows/bbdc1224-1bae-4b33-bf2e-c1df344e1146]|
|[trunk|https://github.com/adelapena/cassandra/commit/3973e410d34cb0d003fc74014b4b3bee8a165a21]|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1027/workflows/ee7e4ffd-3eec-49fd-ad55-028731501811]
[j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1027/workflows/eaa21c86-b148-45df-b01a-d100175ba8ca]|
[~edimitrova] could you please review?
> CircleCI dtest multiplexer with MIDRES needs more resources
> ---
>
> Key: CASSANDRA-17043
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17043
> Project: Cassandra
> Issue Type: Task
> Components: CI
>Reporter: Andres de la Peña
>Assignee: Andres de la Peña
>Priority: Low
> Fix For: 3.0.x, 3.11.x, 4.0.x, 4.x
>
>
> The CircleCI jobs for regular dtests jobs have more resources in MIDRES,
> which is necessary for some dtests to reliably success. However, the dtest
> multiplexer uses the same resources for LOWRES and MIDRES.
> I think that the dtest multiplexer should always use the same resources as
> the regular dtests. Using too small resources in the multiplexer can lead to
> failures that don't reproduce in the regular dtest jobs, like the one we
> found in
> [CASSANDRA-16334|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> when trying to repeatedly run a resource-hungry dtest, or like [this other
> one|https://app.circleci.com/pipelines/github/adelapena/cassandra/1020/workflows/63908694-e4d7-40b1-9418-7a4b87826233/jobs/9422]
> while running {{test_network_topology}}.
> This happens because I forgot to update the diff patch when adding the
> multiplexer. This doesn't affect HIGHRES because in that case the patch
> changes the configuration of the test executor, while in MIDRES a new
> executor is defined.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
