On Wed, Jan 24, 2018 at 10:40 AM, Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote:
> Hello, > > In the process of upgrading our cluster from 2.1 to 2.2 we have triggered > the SSTable rewriting process like this: > > $ nodetool upgradesstables -j 4 # concurrent_compactors=5 > > Then if we would immediately check the compactionstats, we see that 4 > compactions of type 'Upgrade sstables' are running for the same > keyspace/table. Fine. > > What is surprising, though, is that when any of the tasks which are > smaller in size complete, no new tasks are added and we are left waiting > for a smaller number of long running tasks to complete, e.g: > > pending tasks: 4 > id compaction type keyspace > table completed total unit progress > d535e962-00de-11e8-9c2c-ade7b0b922e3 Upgrade sstables cel_data > event_history_unbounded 32.87 GB 84.12 GB bytes 39.08% > d535e960-00de-11e8-9c2c-ade7b0b922e3 Upgrade sstables cel_data > event_history_unbounded 33.15 GB 84.79 GB bytes 39.09% > > Periodically, an ordinary compaction would be listed, but no new Upgrade > tasks appear. > Only after all of the "current" tasks are finished, the new 4 tasks are > scheduled and the process repeats. > > We would expect that new tasks would be scheduled as soon as a compaction > slot is freed up, but this doesn't seem to happen. Is this by design? I > do not see any open issue about this in Jira. > FYI: Because of this issue it has taken more than 7 days instead of the estimated 1.5-2 days, based on the available throughput, to migrate ~50 TiB of data files spread over 12 nodes. -- Alex
