Re: Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

On Thu, Feb 1, 2018 at 9:23 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On 1 Feb 2018 06:51, "kurt greaves"  wrote:
>
> Would you be able to create a JIRA ticket for this? Not sure if this is
> still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
> really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
> it's an issue there as well.​
>
>
> Our next planned step is to upgrade to 3.0.15, so we will see pretty soon
> if that's still an issue in the newer version.
>

Still the same issue with 3.0.  I've opened a ticket:
https://issues.apache.org/jira/browse/CASSANDRA-14210

--
Alex

Re: Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

On 1 Feb 2018 06:51, "kurt greaves"  wrote:

Would you be able to create a JIRA ticket for this? Not sure if this is
still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
it's an issue there as well.​


Our next planned step is to upgrade to 3.0.15, so we will see pretty soon
if that's still an issue in the newer version.

--
Alex

Re: Upgrading sstables not using all available compaction slots on version 2.2

[kurt greaves]

Would you be able to create a JIRA ticket for this? Not sure if this is
still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
it's an issue there as well.​

Re: Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

On Wed, Jan 24, 2018 at 10:40 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> Hello,
>
> In the process of upgrading our cluster from 2.1 to 2.2 we have triggered
> the SSTable rewriting process like this:
>
> $ nodetool upgradesstables -j 4  # concurrent_compactors=5
>
> Then if we would immediately check the compactionstats, we see that 4
> compactions of type 'Upgrade sstables' are running for the same
> keyspace/table.  Fine.
>
> What is surprising, though, is that when any of the tasks which are
> smaller in size complete, no new tasks are added and we are left waiting
> for a smaller number of long running tasks to complete, e.g:
>
> pending tasks: 4
>  idcompaction type   keyspace
>table   completed  totalunit   progress
>d535e962-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
>  event_history_unbounded32.87 GB   84.12 GB   bytes 39.08%
>d535e960-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
>  event_history_unbounded33.15 GB   84.79 GB   bytes 39.09%
>
> Periodically, an ordinary compaction would be listed, but no new Upgrade
> tasks appear.
> Only after all of the "current" tasks are finished, the new 4 tasks are
> scheduled and the process repeats.
>
> We would expect that new tasks would be scheduled as soon as a compaction
> slot is freed up, but this doesn't seem to happen.  Is this by design?  I
> do not see any open issue about this in Jira.
>

FYI: Because of this issue it has taken more than 7 days instead of the
estimated 1.5-2 days, based on the available throughput, to migrate ~50 TiB
of data files spread over 12 nodes.

--
Alex

Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

Hello,

In the process of upgrading our cluster from 2.1 to 2.2 we have triggered
the SSTable rewriting process like this:

$ nodetool upgradesstables -j 4  # concurrent_compactors=5

Then if we would immediately check the compactionstats, we see that 4
compactions of type 'Upgrade sstables' are running for the same
keyspace/table.  Fine.

What is surprising, though, is that when any of the tasks which are smaller
in size complete, no new tasks are added and we are left waiting for a
smaller number of long running tasks to complete, e.g:

pending tasks: 4
 idcompaction type   keyspace
   table   completed  totalunit   progress
   d535e962-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
 event_history_unbounded32.87 GB   84.12 GB   bytes 39.08%
   d535e960-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
 event_history_unbounded33.15 GB   84.79 GB   bytes 39.09%

Periodically, an ordinary compaction would be listed, but no new Upgrade
tasks appear.
Only after all of the "current" tasks are finished, the new 4 tasks are
scheduled and the process repeats.

We would expect that new tasks would be scheduled as soon as a compaction
slot is freed up, but this doesn't seem to happen.  Is this by design?  I
do not see any open issue about this in Jira.

Cheers,
-- 
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707