Re: Connection Latency with Cassandra 4.0.x

[MyWorld]

In addition to this, the established connection count to db is increasing
on every new API hit. Ideally existing connection should be reused.

Moreover even if we restart Cassandra the connection from the API end do
not get killed and remains the same.

On Thu, 12 Jan, 2023, 02:16 Elliott Sims,  wrote:

> Consistently 200ms, during the back-and-forth negotiation rather than the
> handshake?  That sounds suspiciously like Nagle interacting with Delayed
> ACK.
>
> On Wed, Jan 11, 2023 at 8:41 AM MyWorld  wrote:
>
>> Hi all,
>> We are facing a connection latency of 200ms between API server and db
>> server during connection.
>> We are working with Apache cassandra 4.0.7 and open jdk ver 11.0.17. We
>> are using php on API side and connecting using php Cassandra driver (CPP
>> ver 2.7) with below string.
>> $cluster = Cassandra::cluster()
>>  ->withContactPoints('x.x.x.x')
>>  ->withPort(9042)
>>  ->withCredentials("user", "pswd")
>>  ->withPersistentSessions(true)
>>  ->withDefaultConsistency(Cassandra::CONSISTENCY_LOCAL_QUORUM)
>>   ->withDatacenterAwareRoundRobinLoadBalancingPolicy("dc1",0, false)
>>   ->build();
>>   $dbh = $cluster->connect('mykeyspace');
>>
>> We have earlier worked in 3.11.6 version and things were working fine.
>> This is the first time we have installed ver 4.0.x and started facing this
>> issue.No change have been done on driver side.
>> Just note both API and db server are on same location and ping time to db
>> server is less than 1ms.
>>
>> Unable to identify the root cause of this. Does anyone have any clue?
>>
>> Regards,
>> Ashish
>>
>>
> This email, including its contents and any attachment(s), may contain
> confidential and/or proprietary information and is solely for the review
> and use of the intended recipient(s). If you have received this email in
> error, please notify the sender and permanently delete this email, its
> content, and any attachment(s). Any disclosure, copying, or taking of any
> action in reliance on an email received in error is strictly prohibited.
>

Re: Connection Latency with Cassandra 4.0.x

[Elliott Sims]

Consistently 200ms, during the back-and-forth negotiation rather than the
handshake?  That sounds suspiciously like Nagle interacting with Delayed
ACK.

On Wed, Jan 11, 2023 at 8:41 AM MyWorld  wrote:

> Hi all,
> We are facing a connection latency of 200ms between API server and db
> server during connection.
> We are working with Apache cassandra 4.0.7 and open jdk ver 11.0.17. We
> are using php on API side and connecting using php Cassandra driver (CPP
> ver 2.7) with below string.
> $cluster = Cassandra::cluster()
>  ->withContactPoints('x.x.x.x')
>  ->withPort(9042)
>  ->withCredentials("user", "pswd")
>  ->withPersistentSessions(true)
>  ->withDefaultConsistency(Cassandra::CONSISTENCY_LOCAL_QUORUM)
>   ->withDatacenterAwareRoundRobinLoadBalancingPolicy("dc1",0, false)
>   ->build();
>   $dbh = $cluster->connect('mykeyspace');
>
> We have earlier worked in 3.11.6 version and things were working fine.
> This is the first time we have installed ver 4.0.x and started facing this
> issue.No change have been done on driver side.
> Just note both API and db server are on same location and ping time to db
> server is less than 1ms.
>
> Unable to identify the root cause of this. Does anyone have any clue?
>
> Regards,
> Ashish
>
>

-- 
This email, including its contents and any attachment(s), may contain 
confidential and/or proprietary information and is solely for the review 
and use of the intended recipient(s). If you have received this email in 
error, please notify the sender and permanently delete this email, its 
content, and any attachment(s).  Any disclosure, copying, or taking of any 
action in reliance on an email received in error is strictly prohibited.

Connection Latency with Cassandra 4.0.x

[MyWorld]

Hi all,
We are facing a connection latency of 200ms between API server and db
server during connection.
We are working with Apache cassandra 4.0.7 and open jdk ver 11.0.17. We are
using php on API side and connecting using php Cassandra driver (CPP ver
2.7) with below string.
$cluster = Cassandra::cluster()
 ->withContactPoints('x.x.x.x')
 ->withPort(9042)
 ->withCredentials("user", "pswd")
 ->withPersistentSessions(true)
 ->withDefaultConsistency(Cassandra::CONSISTENCY_LOCAL_QUORUM)
  ->withDatacenterAwareRoundRobinLoadBalancingPolicy("dc1",0, false)
  ->build();
  $dbh = $cluster->connect('mykeyspace');

We have earlier worked in 3.11.6 version and things were working fine. This
is the first time we have installed ver 4.0.x and started facing this
issue.No change have been done on driver side.
Just note both API and db server are on same location and ping time to db
server is less than 1ms.

Unable to identify the root cause of this. Does anyone have any clue?

Regards,
Ashish

Re: Cassandra 4.0 upgrade - Upgradesstables

[Jim Shaw]

Though it is not required to run upgradesstables, but upgradesstables -a
will re-write the file to kick out tombstones, in sizeTieredcompaction, the
largest files may stay a long time to wait for the next compaction to
kick out tombstones.
So it really depends,  to run it or not,  usually upgrades have a change
window, applications may be no load or less load, why don't take the chance
to run it.

Regards,

Jim

On Tue, Aug 16, 2022 at 3:17 PM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thank you
>
> On Tue, Aug 16, 2022 at 11:48 AM C. Scott Andreas 
> wrote:
>
>> No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1
>> SSTables incurred a performance hit).
>>
>> Many users of Cassandra don't run upgradesstables after 3.x -> 4.x
>> upgrades at all. It's not necessary to run until a hypothetical future time
>> if/when support for reading Cassandra 3.x SSTables is removed from
>> Cassandra. One of the most common reasons to avoid running upgradesstables
>> is because doing so causes 100% churn of the data files, meaning your
>> backup processes will need to upload a full copy of the data. Allowing
>> SSTables to organically churn into the new version via compaction avoids
>> this.
>>
>> If you're upgrading from 3.x to 4.x, don't feel like you have to - but it
>> does avoid the need to run upgradesstables in a hypothetical distant future.
>>
>> – Scott
>>
>> On Aug 16, 2022, at 6:32 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>
>> Thank you Erick,
>>
>> > it is going to be single-threaded by default so it will take a while to
>> get through all the sstables on dense nodes
>> Is there any downside if the upgradesstables take longer (example 1-2
>> days), other than I/O?
>>
>> Also when is the upgradesstable get triggered? after every node is
>> upgraded or it will kick in only when all the nodes in the cluster upgraded
>> to 4.0.x?
>>
>> On Tue, Aug 16, 2022 at 2:12 AM Erick Ramirez 
>> wrote:
>>
>>> As convenient as it is, there are a few caveats and it isn't a silver
>>> bullet. The automatic feature will only kick in if there are no other
>>> compactions scheduled. Also, it is going to be single-threaded by default
>>> so it will take a while to get through all the sstables on dense nodes.
>>>
>>> In contrast, you'll have a bit more control if you manually upgrade the
>>> sstables. For example, you can schedule the upgrade during low traffic
>>> periods so reads are not competing with compactions for IO. Cheers!
>>>


>>

Re: Cassandra 4.0 upgrade - Upgradesstables

[Jai Bheemsen Rao Dhanwada]

Thank you

On Tue, Aug 16, 2022 at 11:48 AM C. Scott Andreas 
wrote:

> No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1
> SSTables incurred a performance hit).
>
> Many users of Cassandra don't run upgradesstables after 3.x -> 4.x
> upgrades at all. It's not necessary to run until a hypothetical future time
> if/when support for reading Cassandra 3.x SSTables is removed from
> Cassandra. One of the most common reasons to avoid running upgradesstables
> is because doing so causes 100% churn of the data files, meaning your
> backup processes will need to upload a full copy of the data. Allowing
> SSTables to organically churn into the new version via compaction avoids
> this.
>
> If you're upgrading from 3.x to 4.x, don't feel like you have to - but it
> does avoid the need to run upgradesstables in a hypothetical distant future.
>
> – Scott
>
> On Aug 16, 2022, at 6:32 AM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>
> Thank you Erick,
>
> > it is going to be single-threaded by default so it will take a while to
> get through all the sstables on dense nodes
> Is there any downside if the upgradesstables take longer (example 1-2
> days), other than I/O?
>
> Also when is the upgradesstable get triggered? after every node is
> upgraded or it will kick in only when all the nodes in the cluster upgraded
> to 4.0.x?
>
> On Tue, Aug 16, 2022 at 2:12 AM Erick Ramirez 
> wrote:
>
>> As convenient as it is, there are a few caveats and it isn't a silver
>> bullet. The automatic feature will only kick in if there are no other
>> compactions scheduled. Also, it is going to be single-threaded by default
>> so it will take a while to get through all the sstables on dense nodes.
>>
>> In contrast, you'll have a bit more control if you manually upgrade the
>> sstables. For example, you can schedule the upgrade during low traffic
>> periods so reads are not competing with compactions for IO. Cheers!
>>
>>>
>>>
>

Re: Cassandra 4.0 upgrade - Upgradesstables

[C. Scott Andreas]

No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1 SSTables incurred a 
performance hit).Many users of Cassandra don't run upgradesstables after 3.x -> 4.x upgrades at 
all. It's not necessary to run until a hypothetical future time if/when support for reading 
Cassandra 3.x SSTables is removed from Cassandra. One of the most common reasons to avoid running 
upgradesstables is because doing so causes 100% churn of the data files, meaning your backup 
processes will need to upload a full copy of the data. Allowing SSTables to organically churn into 
the new version via compaction avoids this.If you're upgrading from 3.x to 4.x, don't feel like you 
have to - but it does avoid the need to run upgradesstables in a hypothetical distant future.– 
ScottOn Aug 16, 2022, at 6:32 AM, Jai Bheemsen Rao Dhanwada  
wrote:Thank you Erick,> it is going to be single-threaded by default so it will take a while to 
get through all the sstables on dense nodesIs there any downside if the upgradesstables take longer 
(example 1-2 days), other than I/O?Also when is the upgradesstable get triggered? after every node 
is upgraded or it will kick in only when all the nodes in the cluster upgraded to 4.0.x?On Tue, Aug 
16, 2022 at 2:12 AM Erick Ramirez  wrote:As convenient as it is, 
there are a few caveats and it isn't a silver bullet. The automatic feature will only kick in if 
there are no other compactions scheduled. Also, it is going to be single-threaded by default so it 
will take a while to get through all the sstables on dense nodes.In contrast, you'll have a bit more 
control if you manually upgrade the sstables. For example, you can schedule the upgrade during low 
traffic periods so reads are not competing with compactions for IO. Cheers!

Re: Cassandra 4.0 upgrade - Upgradesstables

[Jai Bheemsen Rao Dhanwada]

Thank you Erick,

> it is going to be single-threaded by default so it will take a while to
get through all the sstables on dense nodes
Is there any downside if the upgradesstables take longer (example 1-2
days), other than I/O?

Also when is the upgradesstable get triggered? after every node is
upgraded or it will kick in only when all the nodes in the cluster upgraded
to 4.0.x?

On Tue, Aug 16, 2022 at 2:12 AM Erick Ramirez 
wrote:

> As convenient as it is, there are a few caveats and it isn't a silver
> bullet. The automatic feature will only kick in if there are no other
> compactions scheduled. Also, it is going to be single-threaded by default
> so it will take a while to get through all the sstables on dense nodes.
>
> In contrast, you'll have a bit more control if you manually upgrade the
> sstables. For example, you can schedule the upgrade during low traffic
> periods so reads are not competing with compactions for IO. Cheers!
>
>>

Re: Cassandra 4.0 upgrade - Upgradesstables

[Erick Ramirez]

As convenient as it is, there are a few caveats and it isn't a silver
bullet. The automatic feature will only kick in if there are no other
compactions scheduled. Also, it is going to be single-threaded by default
so it will take a while to get through all the sstables on dense nodes.

In contrast, you'll have a bit more control if you manually upgrade the
sstables. For example, you can schedule the upgrade during low traffic
periods so reads are not competing with compactions for IO. Cheers!

>

Cassandra 4.0 upgrade - Upgradesstables

[Jai Bheemsen Rao Dhanwada]

Hello,

I am evaluating the upgrade from 3.11.x to 4.0.x and as per CASSANDRA-14197
 we don't need to
run upgradesstables any more. We have tested this in a test environment and
see that setting "-Dcassandra.automatic_sstable_upgrade=true" takes care of
the upgrading sstables to the new format. Since my actual cluster has a
very high data size and more datacenters, I would like to know the general
guidance on the upgradesstables option. Is this option guaranteed to take
care of upgradesstables or it's a best case scenario and expects the users
to run the upgradesstables (as I see some posts still run upgradesstables
for 4.x upgrade). If it's taken care of automatically, is there a way I can
see the progress/ time it takes or does the user need not worry about how
long it takes but can just proceed with the upgrade on all the data
centers sequentially one after another?

Thanks in advance.

Re: unlimited repair throughput in Cassandra 4.0

[Azamat Hackimov]

Hello.

I had the same issues on full repair. I've checked on various GC
settings, the most performant is ZGC on Java 11, but I had some
stability issues. I left G1GC settings from 3.11.x and got the same
issues as yours: CPU load over 90 %, and growing count of open file
descriptors (up to max allowed). It looks like the repair job doesn't
finish repairs on segments and waits when all segments get repaired.
so the job itself takes more and more resources until the node may
become irresponsible or all segments will be repaired.

Here my GC settings (Cassandra 4.0.2):

G1GC:

-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:MaxGCPauseMillis=500
-XX:InitiatingHeapOccupancyPercent=70
-XX:ParallelGCThreads=16
-XX:ConcGCThreads=16

ZGC:

-XX:+UnlockExperimentalVMOptions
-XX:+UseZGC
-XX:ConcGCThreads=8
-XX:ParallelGCThreads=8
-XX:+UseTransparentHugePages


ZGC works more effectively on repair tasks (it runs way faster, and
doesn't overuse much system resources), but I get random crushes on
various nodes, so I can't use it as production ready.


пн, 13 июн. 2022 г. в 15:26, onmstester onmstester :

>
> Hi
>
> I've been testing Cassandra 4.0.3 and when i run rull repair (on a single 
> table), all of bandwidth of my 1G link would be saturated (also CPU became > 
> 80% and disk util is 100%), stream_throughput been set to 200 Mb but not 
> affecting repair, all other configs are default and i could not find any 
> other configuration related to limiting the throughput of repair.
>
> IMHO, having a node with saturated resources would make the whole cluster's 
> response time be slow.
> Any workaround for this? Is this some sort of bug?
>
> Best Regards
>
>


--
>From Siberia with Love!

Re: Cassandra 4.0 randomly freezes on heavy load

[Azamat Hackimov]

OK, I think I figured out what was wrong in my setup. Appears that
during the upgrade course I totally missed Garbage Collector settings
in jvm11-server.options. After installation from RPM it defines the
default CMS garbage collector, which is totally wrong for my hi-load
environment. I returned back G1GC settings from my previous 3.11.x
configuration and it seems it works as expected.

Thank you all, you're very helpful!

Re: Cassandra 4.0 randomly freezes on heavy load

[Paulo Motta]

> I can reproduce this with a huge load using dsbulk, but still can't
determine the cause of the problem.

Can you get a thread dump (jstack ) when the system freezes? This
might be helpful to determine the cause of the freeze.

Also, can you reproduce this in a simpler environment (ccm + dsbulk)?

Em sex., 25 de fev. de 2022 às 07:03, Bowen Song  escreveu:

> Okay, that ruled it out. Anything interesting in the GC logs? Was
> Cassandra stuck at a GC safepoint? You may need to enable the detailed
> GC logs to see these.
>
> On 25/02/2022 10:02, Azamat Hackimov wrote:
> > Hello!
> >
> > No, I have a directly attached NVMe disk, and there are no IO or network
> issues.
> >
> > пт, 25 февр. 2022 г. в 12:50, Bowen Song :
> >> Do you have any network based mountpoint, such as NFS or samba? I have
> >> seen similar behaviour on other Java based applications at GC safepoint
> >> when the network based filesystem loses their connection and reconnects.
> >>
> >> On 25/02/2022 06:09, Azamat Hackimov wrote:
> >>> Hello!
> >>>
> >>> I recently migrated Cassandra from 3.11.x to 4.0 and got strange
> >>> freezes on heavy load. It looks like some nodes in DC stopped
> >>> responding and got DN status.
> >>> I cannot check status directly on node via nodetool status or even
> >>> restart Cassandra with `systemctl restart cassandra` command. Only
> >>> viable method is to `kill -9` hanging process and restart Cassandra
> >>> again. On 3.11.x there are no such problems.
> >>>
> >>> I have 2 DC with 8 nodes each deployed on good hardware servers, on
> >>> CentOS 7 and Java 11 environments with slightly changed default
> >>> settings inherited from 3.11.x installation.
> >>>
> >>> The problem shows randomly, I can't determine its source, in the
> >>> system.log and debug.log the last event that I could trace does not
> >>> have anything to do with the hang. The service just stops responding
> >>> and freezes. I can reproduce this with a huge load using dsbulk, but
> >>> still can't determine the cause of the problem.
> >>>
> >>> Has anyone encountered a similar problem, and is there any way other
> >>> than rolling back to the previous version?
> >>>
> >>> Here my config:
> >>>
> >>> cluster_name: 'mycluster'
> >>> num_tokens: '256'
> >>> allocate_tokens_for_local_replication_factor: 3
> >>> hinted_handoff_enabled: true
> >>> max_hint_window_in_ms: 1080
> >>> hinted_handoff_throttle_in_kb: 1024
> >>> max_hints_delivery_threads: 2
> >>> hints_directory: /data/cassandra/hints
> >>> hints_flush_period_in_ms: 1
> >>> max_hints_file_size_in_mb: 128
> >>> batchlog_replay_throttle_in_kb: 1024
> >>> authenticator: PasswordAuthenticator
> >>> authorizer: AllowAllAuthorizer
> >>> role_manager: CassandraRoleManager
> >>> network_authorizer: AllowAllNetworkAuthorizer
> >>> roles_validity_in_ms: 2000
> >>> permissions_validity_in_ms: 2000
> >>> credentials_validity_in_ms: 2000
> >>> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> >>> data_file_directories:
> >>>   - /data/cassandra/data
> >>> commitlog_directory: /data/cassandra/commitlog
> >>> cdc_enabled: false
> >>> disk_failure_policy: stop
> >>> commit_failure_policy: stop
> >>> prepared_statements_cache_size_mb:
> >>> key_cache_size_in_mb:
> >>> key_cache_save_period: 14400
> >>> row_cache_size_in_mb: 0
> >>> row_cache_save_period: 0
> >>> counter_cache_size_in_mb:
> >>> counter_cache_save_period: 7200
> >>> saved_caches_directory: /data/cassandra/saved_caches
> >>> commitlog_sync: periodic
> >>> commitlog_sync_period_in_ms: 1
> >>> commitlog_segment_size_in_mb: 32
> >>> seed_provider:
> >>>   - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> >>> parameters:
> >>> - seeds: 'node1-1,node1-4,node2-1,node2-4'
> >>> concurrent_reads: 32
> >>> concurrent_writes: 32
> >>> concurrent_counter_writes: 32
> >>> concurrent_materialized_view_writes: 32
> >>> file_cache_size_in_mb: '1024'
> >>> memtable_allocation_type: heap_buffers
> >>> index_summary_capacity_in_mb:
> >>> index_summary_resize_interval_in_minutes: 60
> >>> trickle_fsync: false
> >>> trickle_fsync_interval_in_kb: 10240
> >>> storage_port: 7000
> >>> ssl_storage_port: 7001
> >>> listen_address:
> >>> start_native_transport: true
> >>> native_transport_port: 9042
> >>> native_transport_allow_older_protocols: true
> >>> rpc_address:
> >>> rpc_keepalive: true
> >>> incremental_backups: false
> >>> snapshot_before_compaction: false
> >>> auto_snapshot: true
> >>> snapshot_links_per_second: 0
> >>> column_index_size_in_kb: 64
> >>> column_index_cache_size_in_kb: 2
> >>> concurrent_compactors: 5
> >>> concurrent_materialized_view_builders: 1
> >>> compaction_throughput_mb_per_sec: 200
> >>> sstable_preemptive_open_interval_in_mb: 50
> >>> read_request_timeout_in_ms: 5000
> >>> range_request_timeout_in_ms: 1
> >>> write_request_timeout_in_ms: 2000
> >>> counter_write_request_timeout_in_ms: 5000
> >>> cas_contention_timeout_in_ms: 1000
> >>> 

Re: Cassandra 4.0 randomly freezes on heavy load

[Bowen Song]

Okay, that ruled it out. Anything interesting in the GC logs? Was 
Cassandra stuck at a GC safepoint? You may need to enable the detailed 
GC logs to see these.


On 25/02/2022 10:02, Azamat Hackimov wrote:

Hello!

No, I have a directly attached NVMe disk, and there are no IO or network issues.

пт, 25 февр. 2022 г. в 12:50, Bowen Song :

Do you have any network based mountpoint, such as NFS or samba? I have
seen similar behaviour on other Java based applications at GC safepoint
when the network based filesystem loses their connection and reconnects.

On 25/02/2022 06:09, Azamat Hackimov wrote:

Hello!

I recently migrated Cassandra from 3.11.x to 4.0 and got strange
freezes on heavy load. It looks like some nodes in DC stopped
responding and got DN status.
I cannot check status directly on node via nodetool status or even
restart Cassandra with `systemctl restart cassandra` command. Only
viable method is to `kill -9` hanging process and restart Cassandra
again. On 3.11.x there are no such problems.

I have 2 DC with 8 nodes each deployed on good hardware servers, on
CentOS 7 and Java 11 environments with slightly changed default
settings inherited from 3.11.x installation.

The problem shows randomly, I can't determine its source, in the
system.log and debug.log the last event that I could trace does not
have anything to do with the hang. The service just stops responding
and freezes. I can reproduce this with a huge load using dsbulk, but
still can't determine the cause of the problem.

Has anyone encountered a similar problem, and is there any way other
than rolling back to the previous version?

Here my config:

cluster_name: 'mycluster'
num_tokens: '256'
allocate_tokens_for_local_replication_factor: 3
hinted_handoff_enabled: true
max_hint_window_in_ms: 1080
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /data/cassandra/hints
hints_flush_period_in_ms: 1
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: PasswordAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
network_authorizer: AllowAllNetworkAuthorizer
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
  - /data/cassandra/data
commitlog_directory: /data/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /data/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1
commitlog_segment_size_in_mb: 32
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: 'node1-1,node1-4,node2-1,node2-4'
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
file_cache_size_in_mb: '1024'
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address:
start_native_transport: true
native_transport_port: 9042
native_transport_allow_older_protocols: true
rpc_address:
rpc_keepalive: true
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
snapshot_links_per_second: 0
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
concurrent_compactors: 5
concurrent_materialized_view_builders: 1
compaction_throughput_mb_per_sec: 200
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 1
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 6
request_timeout_in_ms: 1
slow_query_log_timeout_in_ms: 500
endpoint_snitch: GossipingPropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 60
dynamic_snitch_badness_threshold: 1.0
server_encryption_options:
  internode_encryption: none
  enable_legacy_ssl_storage_port: false
  keystore: conf/.keystore
  keystore_password: cassandra
  require_client_auth: false
  truststore: conf/.truststore
  truststore_password: cassandra
  require_endpoint_verification: false
client_encryption_options:
  enabled: false
  keystore: conf/.keystore
  keystore_password: cassandra
  require_client_auth: false
internode_compression: dc
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
  enabled: false
  

Re: Cassandra 4.0 randomly freezes on heavy load

[Azamat Hackimov]

Hello!

No, I have a directly attached NVMe disk, and there are no IO or network issues.

пт, 25 февр. 2022 г. в 12:50, Bowen Song :
>
> Do you have any network based mountpoint, such as NFS or samba? I have
> seen similar behaviour on other Java based applications at GC safepoint
> when the network based filesystem loses their connection and reconnects.
>
> On 25/02/2022 06:09, Azamat Hackimov wrote:
> > Hello!
> >
> > I recently migrated Cassandra from 3.11.x to 4.0 and got strange
> > freezes on heavy load. It looks like some nodes in DC stopped
> > responding and got DN status.
> > I cannot check status directly on node via nodetool status or even
> > restart Cassandra with `systemctl restart cassandra` command. Only
> > viable method is to `kill -9` hanging process and restart Cassandra
> > again. On 3.11.x there are no such problems.
> >
> > I have 2 DC with 8 nodes each deployed on good hardware servers, on
> > CentOS 7 and Java 11 environments with slightly changed default
> > settings inherited from 3.11.x installation.
> >
> > The problem shows randomly, I can't determine its source, in the
> > system.log and debug.log the last event that I could trace does not
> > have anything to do with the hang. The service just stops responding
> > and freezes. I can reproduce this with a huge load using dsbulk, but
> > still can't determine the cause of the problem.
> >
> > Has anyone encountered a similar problem, and is there any way other
> > than rolling back to the previous version?
> >
> > Here my config:
> >
> > cluster_name: 'mycluster'
> > num_tokens: '256'
> > allocate_tokens_for_local_replication_factor: 3
> > hinted_handoff_enabled: true
> > max_hint_window_in_ms: 1080
> > hinted_handoff_throttle_in_kb: 1024
> > max_hints_delivery_threads: 2
> > hints_directory: /data/cassandra/hints
> > hints_flush_period_in_ms: 1
> > max_hints_file_size_in_mb: 128
> > batchlog_replay_throttle_in_kb: 1024
> > authenticator: PasswordAuthenticator
> > authorizer: AllowAllAuthorizer
> > role_manager: CassandraRoleManager
> > network_authorizer: AllowAllNetworkAuthorizer
> > roles_validity_in_ms: 2000
> > permissions_validity_in_ms: 2000
> > credentials_validity_in_ms: 2000
> > partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> > data_file_directories:
> >  - /data/cassandra/data
> > commitlog_directory: /data/cassandra/commitlog
> > cdc_enabled: false
> > disk_failure_policy: stop
> > commit_failure_policy: stop
> > prepared_statements_cache_size_mb:
> > key_cache_size_in_mb:
> > key_cache_save_period: 14400
> > row_cache_size_in_mb: 0
> > row_cache_save_period: 0
> > counter_cache_size_in_mb:
> > counter_cache_save_period: 7200
> > saved_caches_directory: /data/cassandra/saved_caches
> > commitlog_sync: periodic
> > commitlog_sync_period_in_ms: 1
> > commitlog_segment_size_in_mb: 32
> > seed_provider:
> >  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> >parameters:
> >- seeds: 'node1-1,node1-4,node2-1,node2-4'
> > concurrent_reads: 32
> > concurrent_writes: 32
> > concurrent_counter_writes: 32
> > concurrent_materialized_view_writes: 32
> > file_cache_size_in_mb: '1024'
> > memtable_allocation_type: heap_buffers
> > index_summary_capacity_in_mb:
> > index_summary_resize_interval_in_minutes: 60
> > trickle_fsync: false
> > trickle_fsync_interval_in_kb: 10240
> > storage_port: 7000
> > ssl_storage_port: 7001
> > listen_address:
> > start_native_transport: true
> > native_transport_port: 9042
> > native_transport_allow_older_protocols: true
> > rpc_address:
> > rpc_keepalive: true
> > incremental_backups: false
> > snapshot_before_compaction: false
> > auto_snapshot: true
> > snapshot_links_per_second: 0
> > column_index_size_in_kb: 64
> > column_index_cache_size_in_kb: 2
> > concurrent_compactors: 5
> > concurrent_materialized_view_builders: 1
> > compaction_throughput_mb_per_sec: 200
> > sstable_preemptive_open_interval_in_mb: 50
> > read_request_timeout_in_ms: 5000
> > range_request_timeout_in_ms: 1
> > write_request_timeout_in_ms: 2000
> > counter_write_request_timeout_in_ms: 5000
> > cas_contention_timeout_in_ms: 1000
> > truncate_request_timeout_in_ms: 6
> > request_timeout_in_ms: 1
> > slow_query_log_timeout_in_ms: 500
> > endpoint_snitch: GossipingPropertyFileSnitch
> > dynamic_snitch_update_interval_in_ms: 100
> > dynamic_snitch_reset_interval_in_ms: 60
> > dynamic_snitch_badness_threshold: 1.0
> > server_encryption_options:
> >  internode_encryption: none
> >  enable_legacy_ssl_storage_port: false
> >  keystore: conf/.keystore
> >  keystore_password: cassandra
> >  require_client_auth: false
> >  truststore: conf/.truststore
> >  truststore_password: cassandra
> >  require_endpoint_verification: false
> > client_encryption_options:
> >  enabled: false
> >  keystore: conf/.keystore
> >  keystore_password: cassandra
> >  require_client_auth: false
> > internode_compression: dc
> > 

Re: Cassandra 4.0 randomly freezes on heavy load

[Bowen Song]

Do you have any network based mountpoint, such as NFS or samba? I have 
seen similar behaviour on other Java based applications at GC safepoint 
when the network based filesystem loses their connection and reconnects.


On 25/02/2022 06:09, Azamat Hackimov wrote:

Hello!

I recently migrated Cassandra from 3.11.x to 4.0 and got strange
freezes on heavy load. It looks like some nodes in DC stopped
responding and got DN status.
I cannot check status directly on node via nodetool status or even
restart Cassandra with `systemctl restart cassandra` command. Only
viable method is to `kill -9` hanging process and restart Cassandra
again. On 3.11.x there are no such problems.

I have 2 DC with 8 nodes each deployed on good hardware servers, on
CentOS 7 and Java 11 environments with slightly changed default
settings inherited from 3.11.x installation.

The problem shows randomly, I can't determine its source, in the
system.log and debug.log the last event that I could trace does not
have anything to do with the hang. The service just stops responding
and freezes. I can reproduce this with a huge load using dsbulk, but
still can't determine the cause of the problem.

Has anyone encountered a similar problem, and is there any way other
than rolling back to the previous version?

Here my config:

cluster_name: 'mycluster'
num_tokens: '256'
allocate_tokens_for_local_replication_factor: 3
hinted_handoff_enabled: true
max_hint_window_in_ms: 1080
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /data/cassandra/hints
hints_flush_period_in_ms: 1
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: PasswordAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
network_authorizer: AllowAllNetworkAuthorizer
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
 - /data/cassandra/data
commitlog_directory: /data/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /data/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1
commitlog_segment_size_in_mb: 32
seed_provider:
 - class_name: org.apache.cassandra.locator.SimpleSeedProvider
   parameters:
   - seeds: 'node1-1,node1-4,node2-1,node2-4'
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
file_cache_size_in_mb: '1024'
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address:
start_native_transport: true
native_transport_port: 9042
native_transport_allow_older_protocols: true
rpc_address:
rpc_keepalive: true
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
snapshot_links_per_second: 0
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
concurrent_compactors: 5
concurrent_materialized_view_builders: 1
compaction_throughput_mb_per_sec: 200
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 1
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 6
request_timeout_in_ms: 1
slow_query_log_timeout_in_ms: 500
endpoint_snitch: GossipingPropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 60
dynamic_snitch_badness_threshold: 1.0
server_encryption_options:
 internode_encryption: none
 enable_legacy_ssl_storage_port: false
 keystore: conf/.keystore
 keystore_password: cassandra
 require_client_auth: false
 truststore: conf/.truststore
 truststore_password: cassandra
 require_endpoint_verification: false
client_encryption_options:
 enabled: false
 keystore: conf/.keystore
 keystore_password: cassandra
 require_client_auth: false
internode_compression: dc
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
 enabled: false
 chunk_length_kb: 64
 cipher: AES/CBC/PKCS5Padding
 key_alias: testing:1
 key_provider:
   - class_name: org.apache.cassandra.security.JKSKeyProvider
 parameters:
   - keystore: conf/.keystore
 keystore_password: cassandra
 store_type: JCEKS
 key_password: cassandra
tombstone_warn_threshold: 

Cassandra 4.0 randomly freezes on heavy load

[Azamat Hackimov]

Hello!

I recently migrated Cassandra from 3.11.x to 4.0 and got strange
freezes on heavy load. It looks like some nodes in DC stopped
responding and got DN status.
I cannot check status directly on node via nodetool status or even
restart Cassandra with `systemctl restart cassandra` command. Only
viable method is to `kill -9` hanging process and restart Cassandra
again. On 3.11.x there are no such problems.

I have 2 DC with 8 nodes each deployed on good hardware servers, on
CentOS 7 and Java 11 environments with slightly changed default
settings inherited from 3.11.x installation.

The problem shows randomly, I can't determine its source, in the
system.log and debug.log the last event that I could trace does not
have anything to do with the hang. The service just stops responding
and freezes. I can reproduce this with a huge load using dsbulk, but
still can't determine the cause of the problem.

Has anyone encountered a similar problem, and is there any way other
than rolling back to the previous version?

Here my config:

cluster_name: 'mycluster'
num_tokens: '256'
allocate_tokens_for_local_replication_factor: 3
hinted_handoff_enabled: true
max_hint_window_in_ms: 1080
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /data/cassandra/hints
hints_flush_period_in_ms: 1
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: PasswordAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
network_authorizer: AllowAllNetworkAuthorizer
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /data/cassandra/data
commitlog_directory: /data/cassandra/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /data/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 1
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
  parameters:
  - seeds: 'node1-1,node1-4,node2-1,node2-4'
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
file_cache_size_in_mb: '1024'
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address:
start_native_transport: true
native_transport_port: 9042
native_transport_allow_older_protocols: true
rpc_address:
rpc_keepalive: true
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
snapshot_links_per_second: 0
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
concurrent_compactors: 5
concurrent_materialized_view_builders: 1
compaction_throughput_mb_per_sec: 200
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 1
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 6
request_timeout_in_ms: 1
slow_query_log_timeout_in_ms: 500
endpoint_snitch: GossipingPropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 60
dynamic_snitch_badness_threshold: 1.0
server_encryption_options:
internode_encryption: none
enable_legacy_ssl_storage_port: false
keystore: conf/.keystore
keystore_password: cassandra
require_client_auth: false
truststore: conf/.truststore
truststore_password: cassandra
require_endpoint_verification: false
client_encryption_options:
enabled: false
keystore: conf/.keystore
keystore_password: cassandra
require_client_auth: false
internode_compression: dc
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
  - class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
  - keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 10
replica_filtering_protection:
cached_rows_warn_threshold: 2000
cached_rows_fail_threshold: 32000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10

Re: Cassandra 4.0 upgrade from Cassandra 3x

[manish khandelwal]

You can see upgrading instructions here
https://github.com/apache/cassandra/blob/cassandra-4.0.2/NEWS.txt.

On Fri, Feb 11, 2022 at 2:52 AM Abdul Patel  wrote:

> Hi
> apart from  standard upgrade process any thing specific needs ti be
> handled separately for this upgrade process?
>
> Any changes needed at client side w.r.t drivers?
>

Re: Cassandra 4.0 upgrade from Cassandra 3x

[Erick Ramirez]

Make sure you go through all the instructions in
https://github.com/apache/cassandra/blob/trunk/NEWS.txt. It's also highly
recommended that you upgrade to the latest 3.0.x or 3.11.x version before
upgrading to 4.0.

Generally there are no changes required on the client side apart from
setting the protocol version depending on the driver version. Cheers!

>

Cassandra 4.0 upgrade from Cassandra 3x

[Abdul Patel]

Hi
apart from  standard upgrade process any thing specific needs ti be handled
separately for this upgrade process?

Any changes needed at client side w.r.t drivers?

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

Hi Paul,


It's only problematic if you are trying to do *a lots of* subrange 
incremental repairs. The whole point of incremental repair is each 
repair is incremental and it will only touch the recently changed data, 
therefore you shouldn't need to split each node into too many subranges 
to reduce the required time and/or resources. For example, If you split 
each node into 16 subranges (e.g. 16 vnodes), on a cluster with a single 
DC and RF=3, only 48 rows will stay in the system.repairs table on each 
node. I don't see 48 rows will cause any noticeable performance impact.


Getting repair right is hard, which is why I recommended Cassandra 
Reaper. The cron job route will suffer from a lots of issues. For 
example, because you can't run repair with -st and -et cross the vnode 
boundary, each time you add or remove a node you will need to update the 
ranges not only on the newly added/removed node, but also on many other 
nodes. The amount of work to implement a script that does the token 
range auto-discovery is simply not worth the effort when you could spent 
the same time to secure the remote JMX and setup Cassandra Reaper instead.



Regards,

Bowen


On 27/01/2022 11:46, Paul Chandler wrote:

Thanks Erick and Bowen

I do find all the different parameters for repairs confusing, and even 
reading up on it now, I see Datastax warns against incremental repairs 
with -pr, but then the code here seems to negate the need for this 
warning.


Anyway running it like this, produces data in the system.repairs 
table, so I assume it is doing incremental repairs.


nodetool -h localhost -p 7199 repair -pr  -st +02596488670266845384 
-et +02613877898679419724


Then running it like this produces no data in the table, so again 
assuming that means it is full repairs


nodetool -h localhost -p 7199 repair -pr -full  -st 
+02596488670266845384 -et +02613877898679419724


Yesterday I recompiled the Cassandra 4.0.0 code with extra logging in 
the following method


https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L338

This showed me that the extra 10 minutes ( and more ) on some clusters 
is being taken up in the for loop reading the rows from the 
system.repairs table.


So this does seem to be issue if you are trying to do incremental 
range repairs in 4.0


Thanks

Paul


On 27 Jan 2022, at 10:27, Bowen Song  wrote:

Hi Erick,


From the source code: 
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/StorageService.java#L4042


The -pr option has no effect if -st and -et are specified. Therefore, 
the command results in an incremental repair.



Cheers,

Bowen

On 27/01/2022 01:32, Erick Ramirez wrote:
I just came across this thread and noted that you're running repairs 
with -pr which are not incremental repairs. Was that a typo? Cheers!

Re: Cassandra 4.0 hanging on restart

[Paul Chandler]

Thanks Erick and Bowen

I do find all the different parameters for repairs confusing, and even reading 
up on it now, I see Datastax warns against incremental repairs with -pr, but 
then the code here seems to negate the need for this warning.

Anyway running it like this, produces data in the system.repairs table, so I 
assume it is doing incremental repairs.

nodetool -h localhost -p 7199 repair -pr  -st +02596488670266845384 -et 
+02613877898679419724

Then running it like this produces no data in the table, so again assuming that 
means it is full repairs

nodetool -h localhost -p 7199 repair -pr -full-st +02596488670266845384 -et 
+02613877898679419724

Yesterday I recompiled the Cassandra 4.0.0 code with extra logging in the 
following method

https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L338
 


This showed me that the extra 10 minutes ( and more ) on some clusters is being 
taken up in the for loop reading the rows from the system.repairs table. 

So this does seem to be issue if you are trying to do incremental range repairs 
in 4.0

Thanks 

Paul 

> On 27 Jan 2022, at 10:27, Bowen Song  wrote:
> 
> Hi Erick,
> 
> 
> 
> From the source code: 
> https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/StorageService.java#L4042
>  
> 
> The -pr option has no effect if -st and -et are specified. Therefore, the 
> command results in an incremental repair.
> 
> 
> 
> Cheers,
> 
> Bowen
> 
> On 27/01/2022 01:32, Erick Ramirez wrote:
>> I just came across this thread and noted that you're running repairs with 
>> -pr which are not incremental repairs. Was that a typo? Cheers!

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

Hi Erick,


From the source code: 
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/StorageService.java#L4042


The -pr option has no effect if -st and -et are specified. Therefore, 
the command results in an incremental repair.



Cheers,

Bowen

On 27/01/2022 01:32, Erick Ramirez wrote:
I just came across this thread and noted that you're running repairs 
with -pr which are not incremental repairs. Was that a typo? Cheers!

Re: Cassandra 4.0 hanging on restart

[Erick Ramirez]

I just came across this thread and noted that you're running repairs with
-pr which are not incremental repairs. Was that a typo? Cheers!

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

Yes, I understand that exposing JMX securely outside localhost involves 
a fair amount of work. The sidecar option will use more RAM per 
Cassandra node, which may or may not be an issue depending on the 
individual circumstance. It is not a quick win if neither of those two 
options are optimal for you situation. However, based on my past 
experience with cron jobs and "nodetool repair", the wasted time on 
implementing, maintaining, monitoring and occasionally troubleshooting 
and fixing it totally justifies the amount of one-off work on exposing 
the JMX securely and setup Cassandra Reaper. Of course, mileage varies, 
you may find it easier and quicker to deal with nodetool cron jobs than 
other options.


On 26/01/2022 11:00, Paul Chandler wrote:
We don’t expose the JMX port outside localhost, so last time we looked 
it was not possible, I see now there is the sidecar option, but that 
sounds like there are number of caveats, particularly around 
resources, that may cause some issues with our setup. So at the moment 
reaper does not seem like a quick win for us.


Thanks

Paul




On 26 Jan 2022, at 10:33, Bowen Song  wrote:

I'm glad that it fixed the problem. Now, may I interest you with 
Cassandra Reaper ? In my experience it 
has managed the load fairly well on large clusters.


On 26/01/2022 10:19, Paul Chandler wrote:
I changed the the range repair to be full repair, reset the 
repairedAt for all SSTables and deleted the old data out of the 
system.repairs table.


This then did not create any new rows in the system.repairs table, 
and the node was able to restart without any problem, so this seems 
to be a solution


I am concerned about turning on full repair for some of our cluster 
and what that will do to the load, so I am now going to experiment 
with larger incremental range repairs to see if there is a sweet 
spot where this works ok.


Paul


On 25 Jan 2022, at 13:36, Bowen Song  wrote:

That would indicate the "isSuperseded(session)" call returned 
false. After looking at the source code, it seems the subrange 
incremental repair is likely causing this.


Would you mind to try either subrange full repair or full range 
incremental repair? You may need to reset the "repairedAt" value in 
all SSTables using the "sstablerepairedset" tool if you decide to 
move on to use subrange full repairs.



On 25/01/2022 12:39, Paul Chandler wrote:

Hi Bowen,

Yes there are a large number of "Skipping delete of FINALIZED 
LocalSession” messages.


We have a script that repairs ranges, stepping through the 
complete range in 5 days, this should create 1600 ranges over the 
5 days, this runs commands like this:


nodetool -h localhost -p 7199 repair -pr  -st 
+09152150533683801432 -et +09154639946886262655


I am also seeing lots of "Auto deleting repair session 
LocalSession” messages - these seem to be deleting the rows with a 
repairedAt value of more than 5 days, so it seems like that part 
is working correctly, but just taking 5 days to delete them.


Thanks

Paul





On 24 Jan 2022, at 22:12, Bowen Song  wrote:

From the source code I've read, by default Cassandra will run a 
clean up for the system.repairs table every 10 minutes, any row 
related to a repair that has completed over 1 day ago will be 
automatically removed. I highly doubt that you have ran 75,000 
repairs in the 24 hours prior to shutting down that node, because 
that's nearly one repair every second.


Do you see any logs like these?

  * Auto failing timed out repair session...
  * Skipping delete of FINALIZED LocalSession ... because it has
not been superseded by a more recent session
  * Skipping delete of LocalSession ... because it still contains
sstables

They are the logs from the cleanup() method in 
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416 
which indicates a record was not deleted during the cleaned up 
for a number of reasons.


On 24/01/2022 19:45, Paul Chandler wrote:

Hi Bowen,

Yes, there does seem to be a lot of rows, on one of the upgraded 
clusters there 75,000 rows.


I have been experimenting on a test cluster, this has about a 5 
minute pause, and around 15,000 rows.


If I clear the system.repairs table ( by deleting the sstables ) 
then this does not pause at all, so seems to fix the problem. 
However I don’t really understand what the implications are of 
just removing that data.


Thanks

Paul


On 24 Jan 2022, at 18:50, Bowen Song  wrote:

Hmm, interesting... Try "select * from system.repairs;" in 
cqlsh on a slow starting node, do you get a lots of rows? This 
is the most obvious loop run (indirectly) by the 
ActiveRepairService.start().


On 24/01/2022 13:30, Romain Anselin wrote:


Hi everyone,

We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that 
the time is spent in the ActiveRepairSession within the main 

Re: Cassandra 4.0 hanging on restart

[Paul Chandler]

We don’t expose the JMX port outside localhost, so last time we looked it was 
not possible, I see now there is the sidecar option, but that sounds like there 
are number of caveats, particularly around resources, that may cause some 
issues with our setup. So at the moment reaper does not seem like a quick win 
for us.

Thanks 

Paul

 

> On 26 Jan 2022, at 10:33, Bowen Song  wrote:
> 
> I'm glad that it fixed the problem. Now, may I interest you with Cassandra 
> Reaper ? In my experience it has managed the 
> load fairly well on large clusters.
> 
> On 26/01/2022 10:19, Paul Chandler wrote:
>> I changed the the range repair to be full repair, reset the repairedAt for 
>> all SSTables and deleted the old data out of the system.repairs table. 
>> 
>> This then did not create any new rows in the system.repairs table, and the 
>> node was able to restart without any problem, so this seems to be a solution
>> 
>> I am concerned about turning on full repair for some of our cluster and what 
>> that will do to the load, so I am now going to experiment with larger 
>> incremental range repairs to see if there is a sweet spot where this works 
>> ok.
>> 
>> Paul
>> 
>>> On 25 Jan 2022, at 13:36, Bowen Song mailto:bo...@bso.ng>> 
>>> wrote:
>>> 
>>> That would indicate the "isSuperseded(session)" call returned false. After 
>>> looking at the source code, it seems the subrange incremental repair is 
>>> likely causing this.
>>> 
>>> Would you mind to try either subrange full repair or full range incremental 
>>> repair? You may need to reset the "repairedAt" value in all SSTables using 
>>> the "sstablerepairedset" tool if you decide to move on to use subrange full 
>>> repairs.
>>> 
>>> 
>>> 
>>> On 25/01/2022 12:39, Paul Chandler wrote:
 Hi Bowen,
 
 Yes there are a large number of "Skipping delete of FINALIZED 
 LocalSession” messages.
 
 We have a script that repairs ranges, stepping through the complete range 
 in 5 days, this should create 1600 ranges over the 5 days, this runs 
 commands like this:
 
 nodetool -h localhost -p 7199 repair -pr  -st +09152150533683801432 -et 
 +09154639946886262655
 
 I am also seeing lots of "Auto deleting repair session LocalSession” 
 messages - these seem to be deleting the rows with a repairedAt value of 
 more than 5 days, so it seems like that part is working correctly, but 
 just taking 5 days to delete them. 
 
 Thanks 
 
 Paul
 
 
 
 
> On 24 Jan 2022, at 22:12, Bowen Song mailto:bo...@bso.ng>> 
> wrote:
> 
> From the source code I've read, by default Cassandra will run a clean up 
> for the system.repairs table every 10 minutes, any row related to a 
> repair that has completed over 1 day ago will be automatically removed. I 
> highly doubt that you have ran 75,000 repairs in the 24 hours prior to 
> shutting down that node, because that's nearly one repair every second.
> 
> Do you see any logs like these?
> 
> Auto failing timed out repair session...
> Skipping delete of FINALIZED LocalSession ... because it has not been 
> superseded by a more recent session
> Skipping delete of LocalSession ... because it still contains sstables
> They are the logs from the cleanup() method in 
> https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416
>  
> 
>  which indicates a record was not deleted during the cleaned up for a 
> number of reasons.
> 
> On 24/01/2022 19:45, Paul Chandler wrote:
>> Hi Bowen,
>> 
>> Yes, there does seem to be a lot of rows, on one of the upgraded 
>> clusters there 75,000 rows.
>> 
>> I have been experimenting on a test cluster, this has about a 5 minute 
>> pause, and around 15,000 rows. 
>> 
>> If I clear the system.repairs table ( by deleting the sstables ) then 
>> this does not pause at all, so seems to fix the problem. However I don’t 
>> really understand what the implications are of just removing that data.
>> 
>> Thanks 
>> 
>> Paul
>> 
>>> On 24 Jan 2022, at 18:50, Bowen Song >> > wrote:
>>> 
>>> Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a 
>>> slow starting node, do you get a lots of rows? This is the most obvious 
>>> loop run (indirectly) by the ActiveRepairService.start(). 
>>> 
>>> On 24/01/2022 13:30, Romain Anselin wrote:
 
 Hi everyone,
 
 We generated a JFR profile of the startup phase of Cassandra with 
 Paul, and it would appear that the time is spent in the 
 

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

I'm glad that it fixed the problem. Now, may I interest you with 
Cassandra Reaper ? In my experience it has 
managed the load fairly well on large clusters.


On 26/01/2022 10:19, Paul Chandler wrote:
I changed the the range repair to be full repair, reset the repairedAt 
for all SSTables and deleted the old data out of the system.repairs 
table.


This then did not create any new rows in the system.repairs table, and 
the node was able to restart without any problem, so this seems to be 
a solution


I am concerned about turning on full repair for some of our cluster 
and what that will do to the load, so I am now going to experiment 
with larger incremental range repairs to see if there is a sweet spot 
where this works ok.


Paul


On 25 Jan 2022, at 13:36, Bowen Song  wrote:

That would indicate the "isSuperseded(session)" call returned false. 
After looking at the source code, it seems the subrange incremental 
repair is likely causing this.


Would you mind to try either subrange full repair or full range 
incremental repair? You may need to reset the "repairedAt" value in 
all SSTables using the "sstablerepairedset" tool if you decide to 
move on to use subrange full repairs.



On 25/01/2022 12:39, Paul Chandler wrote:

Hi Bowen,

Yes there are a large number of "Skipping delete of FINALIZED 
LocalSession” messages.


We have a script that repairs ranges, stepping through the complete 
range in 5 days, this should create 1600 ranges over the 5 days, 
this runs commands like this:


nodetool -h localhost -p 7199 repair -pr  -st +09152150533683801432 
-et +09154639946886262655


I am also seeing lots of "Auto deleting repair session LocalSession” 
messages - these seem to be deleting the rows with a repairedAt 
value of more than 5 days, so it seems like that part is working 
correctly, but just taking 5 days to delete them.


Thanks

Paul





On 24 Jan 2022, at 22:12, Bowen Song  wrote:

From the source code I've read, by default Cassandra will run a 
clean up for the system.repairs table every 10 minutes, any row 
related to a repair that has completed over 1 day ago will be 
automatically removed. I highly doubt that you have ran 75,000 
repairs in the 24 hours prior to shutting down that node, because 
that's nearly one repair every second.


Do you see any logs like these?

  * Auto failing timed out repair session...
  * Skipping delete of FINALIZED LocalSession ... because it has
not been superseded by a more recent session
  * Skipping delete of LocalSession ... because it still contains
sstables

They are the logs from the cleanup() method in 
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416 
which indicates a record was not deleted during the cleaned up for 
a number of reasons.


On 24/01/2022 19:45, Paul Chandler wrote:

Hi Bowen,

Yes, there does seem to be a lot of rows, on one of the upgraded 
clusters there 75,000 rows.


I have been experimenting on a test cluster, this has about a 5 
minute pause, and around 15,000 rows.


If I clear the system.repairs table ( by deleting the sstables ) 
then this does not pause at all, so seems to fix the problem. 
However I don’t really understand what the implications are of 
just removing that data.


Thanks

Paul


On 24 Jan 2022, at 18:50, Bowen Song  wrote:

Hmm, interesting... Try "select * from system.repairs;" in cqlsh 
on a slow starting node, do you get a lots of rows? This is the 
most obvious loop run (indirectly) by the 
ActiveRepairService.start().


On 24/01/2022 13:30, Romain Anselin wrote:


Hi everyone,

We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that 
the time is spent in the ActiveRepairSession within the main thread (11mn of execution of the 
"main" thread in his environment, vs 15s in mine), which has been introduced in 
CASSANDRA-9143 based on a "view git blame" of the source code
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381

That seem to match with the gap we see in the logs, where the time is spent just before 
the "Preloaded" statement in the logs which comes just after in the 
CassandraDaemon code.

INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token metadata: 
Normal Tokens:
...  ...
WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  NoSpamLogger.java:95 - 
/IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] dropping message of type 
SCHEMA_VERSION_REQ whose timeout expired before reaching the network
INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 0 
prepared statements

Remains to determine what that is doing more in details and why it's taking 
longer and longer on startup.

We also exported the sstablemetadata from all sstables in one node, and at this stage, we 
can see we have 

Re: Cassandra 4.0 hanging on restart

[Paul Chandler]

I changed the the range repair to be full repair, reset the repairedAt for all 
SSTables and deleted the old data out of the system.repairs table. 

This then did not create any new rows in the system.repairs table, and the node 
was able to restart without any problem, so this seems to be a solution

I am concerned about turning on full repair for some of our cluster and what 
that will do to the load, so I am now going to experiment with larger 
incremental range repairs to see if there is a sweet spot where this works ok.

Paul

> On 25 Jan 2022, at 13:36, Bowen Song  wrote:
> 
> That would indicate the "isSuperseded(session)" call returned false. After 
> looking at the source code, it seems the subrange incremental repair is 
> likely causing this.
> 
> Would you mind to try either subrange full repair or full range incremental 
> repair? You may need to reset the "repairedAt" value in all SSTables using 
> the "sstablerepairedset" tool if you decide to move on to use subrange full 
> repairs.
> 
> 
> 
> On 25/01/2022 12:39, Paul Chandler wrote:
>> Hi Bowen,
>> 
>> Yes there are a large number of "Skipping delete of FINALIZED LocalSession” 
>> messages.
>> 
>> We have a script that repairs ranges, stepping through the complete range in 
>> 5 days, this should create 1600 ranges over the 5 days, this runs commands 
>> like this:
>> 
>> nodetool -h localhost -p 7199 repair -pr  -st +09152150533683801432 -et 
>> +09154639946886262655
>> 
>> I am also seeing lots of "Auto deleting repair session LocalSession” 
>> messages - these seem to be deleting the rows with a repairedAt value of 
>> more than 5 days, so it seems like that part is working correctly, but just 
>> taking 5 days to delete them. 
>> 
>> Thanks 
>> 
>> Paul
>> 
>> 
>> 
>> 
>>> On 24 Jan 2022, at 22:12, Bowen Song mailto:bo...@bso.ng>> 
>>> wrote:
>>> 
>>> From the source code I've read, by default Cassandra will run a clean up 
>>> for the system.repairs table every 10 minutes, any row related to a repair 
>>> that has completed over 1 day ago will be automatically removed. I highly 
>>> doubt that you have ran 75,000 repairs in the 24 hours prior to shutting 
>>> down that node, because that's nearly one repair every second.
>>> 
>>> Do you see any logs like these?
>>> 
>>> Auto failing timed out repair session...
>>> Skipping delete of FINALIZED LocalSession ... because it has not been 
>>> superseded by a more recent session
>>> Skipping delete of LocalSession ... because it still contains sstables
>>> They are the logs from the cleanup() method in 
>>> https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416
>>>  
>>> 
>>>  which indicates a record was not deleted during the cleaned up for a 
>>> number of reasons.
>>> 
>>> On 24/01/2022 19:45, Paul Chandler wrote:
 Hi Bowen,
 
 Yes, there does seem to be a lot of rows, on one of the upgraded clusters 
 there 75,000 rows.
 
 I have been experimenting on a test cluster, this has about a 5 minute 
 pause, and around 15,000 rows. 
 
 If I clear the system.repairs table ( by deleting the sstables ) then this 
 does not pause at all, so seems to fix the problem. However I don’t really 
 understand what the implications are of just removing that data.
 
 Thanks 
 
 Paul
 
> On 24 Jan 2022, at 18:50, Bowen Song mailto:bo...@bso.ng>> 
> wrote:
> 
> Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a 
> slow starting node, do you get a lots of rows? This is the most obvious 
> loop run (indirectly) by the ActiveRepairService.start(). 
> 
> On 24/01/2022 13:30, Romain Anselin wrote:
>> 
>> Hi everyone,
>> 
>> We generated a JFR profile of the startup phase of Cassandra with Paul, 
>> and it would appear that the time is spent in the ActiveRepairSession 
>> within the main thread (11mn of execution of the "main" thread in his 
>> environment, vs 15s in mine), which has been introduced in 
>> CASSANDRA-9143 based on a "view git blame" of the source code
>> https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381
>>  
>> 
>> 
>> That seem to match with the gap we see in the logs, where the time is 
>> spent just before the "Preloaded" statement in the logs which comes just 
>> after in the CassandraDaemon code.
>> 
>> INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token 
>> metadata: Normal Tokens:
>> ...  ...

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

That would indicate the "isSuperseded(session)" call returned false. 
After looking at the source code, it seems the subrange incremental 
repair is likely causing this.


Would you mind to try either subrange full repair or full range 
incremental repair? You may need to reset the "repairedAt" value in all 
SSTables using the "sstablerepairedset" tool if you decide to move on to 
use subrange full repairs.



On 25/01/2022 12:39, Paul Chandler wrote:

Hi Bowen,

Yes there are a large number of "Skipping delete of FINALIZED 
LocalSession” messages.


We have a script that repairs ranges, stepping through the complete 
range in 5 days, this should create 1600 ranges over the 5 days, this 
runs commands like this:


nodetool -h localhost -p 7199 repair -pr  -st +09152150533683801432 
-et +09154639946886262655


I am also seeing lots of "Auto deleting repair session LocalSession” 
messages - these seem to be deleting the rows with a repairedAt value 
of more than 5 days, so it seems like that part is working correctly, 
but just taking 5 days to delete them.


Thanks

Paul





On 24 Jan 2022, at 22:12, Bowen Song  wrote:

From the source code I've read, by default Cassandra will run a clean 
up for the system.repairs table every 10 minutes, any row related to 
a repair that has completed over 1 day ago will be automatically 
removed. I highly doubt that you have ran 75,000 repairs in the 24 
hours prior to shutting down that node, because that's nearly one 
repair every second.


Do you see any logs like these?

  * Auto failing timed out repair session...
  * Skipping delete of FINALIZED LocalSession ... because it has not
been superseded by a more recent session
  * Skipping delete of LocalSession ... because it still contains
sstables

They are the logs from the cleanup() method in 
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416 
which indicates a record was not deleted during the cleaned up for a 
number of reasons.


On 24/01/2022 19:45, Paul Chandler wrote:

Hi Bowen,

Yes, there does seem to be a lot of rows, on one of the upgraded 
clusters there 75,000 rows.


I have been experimenting on a test cluster, this has about a 5 
minute pause, and around 15,000 rows.


If I clear the system.repairs table ( by deleting the sstables ) 
then this does not pause at all, so seems to fix the problem. 
However I don’t really understand what the implications are of just 
removing that data.


Thanks

Paul


On 24 Jan 2022, at 18:50, Bowen Song  wrote:

Hmm, interesting... Try "select * from system.repairs;" in cqlsh on 
a slow starting node, do you get a lots of rows? This is the most 
obvious loop run (indirectly) by the ActiveRepairService.start().


On 24/01/2022 13:30, Romain Anselin wrote:


Hi everyone,

We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that 
the time is spent in the ActiveRepairSession within the main thread (11mn of execution of the 
"main" thread in his environment, vs 15s in mine), which has been introduced in 
CASSANDRA-9143 based on a "view git blame" of the source code
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381

That seem to match with the gap we see in the logs, where the time is spent just before 
the "Preloaded" statement in the logs which comes just after in the 
CassandraDaemon code.

INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token metadata: 
Normal Tokens:
...  ...
WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  NoSpamLogger.java:95 - 
/IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] dropping message of type 
SCHEMA_VERSION_REQ whose timeout expired before reaching the network
INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 0 
prepared statements

Remains to determine what that is doing more in details and why it's taking 
longer and longer on startup.

We also exported the sstablemetadata from all sstables in one node, and at this stage, we 
can see we have 300 sstables out of 577 with "Repaired at:" set.

|cd /var/lib/cassandra/data find . -name '*Data*' | while read 
datf; do echo  $datf ; sudo -u cassandra sstablemetadata 
$datf; done >> ~/sstablemetadata.txt cqlsh -e "paging off; select 
* from system.repairs" >> ~/repairs.out $ egrep 'Repaired at: 1' 
sstablemetadata.txt | wc -l 300 $ egrep 'Repaired at:' 
sstablemetadata.txt | wc -l 577|


More info to come

Regards - Romain

On 19/01/2022 13:10, Paul Chandler wrote:

Hi Bowen,

Thanks for the reply, these have been our normal shutdowns, so we do a nodetool 
drain before restarting the service, so I would have thought there should not 
be any commtlogs

However there is these messages for one commit log, But looks like it has 
finished quickly and correctly:

INFO  [main] 2022-01-19 10:08:22,811  

Re: Cassandra 4.0 hanging on restart

[Paul Chandler]

Hi Bowen,

Yes there are a large number of "Skipping delete of FINALIZED LocalSession” 
messages.

We have a script that repairs ranges, stepping through the complete range in 5 
days, this should create 1600 ranges over the 5 days, this runs commands like 
this:

nodetool -h localhost -p 7199 repair -pr  -st +09152150533683801432 -et 
+09154639946886262655

I am also seeing lots of "Auto deleting repair session LocalSession” messages - 
these seem to be deleting the rows with a repairedAt value of more than 5 days, 
so it seems like that part is working correctly, but just taking 5 days to 
delete them. 

Thanks 

Paul




> On 24 Jan 2022, at 22:12, Bowen Song  wrote:
> 
> From the source code I've read, by default Cassandra will run a clean up for 
> the system.repairs table every 10 minutes, any row related to a repair that 
> has completed over 1 day ago will be automatically removed. I highly doubt 
> that you have ran 75,000 repairs in the 24 hours prior to shutting down that 
> node, because that's nearly one repair every second.
> 
> Do you see any logs like these?
> 
> Auto failing timed out repair session...
> Skipping delete of FINALIZED LocalSession ... because it has not been 
> superseded by a more recent session
> Skipping delete of LocalSession ... because it still contains sstables
> They are the logs from the cleanup() method in 
> https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416
>  
> 
>  which indicates a record was not deleted during the cleaned up for a number 
> of reasons.
> 
> On 24/01/2022 19:45, Paul Chandler wrote:
>> Hi Bowen,
>> 
>> Yes, there does seem to be a lot of rows, on one of the upgraded clusters 
>> there 75,000 rows.
>> 
>> I have been experimenting on a test cluster, this has about a 5 minute 
>> pause, and around 15,000 rows. 
>> 
>> If I clear the system.repairs table ( by deleting the sstables ) then this 
>> does not pause at all, so seems to fix the problem. However I don’t really 
>> understand what the implications are of just removing that data.
>> 
>> Thanks 
>> 
>> Paul
>> 
>>> On 24 Jan 2022, at 18:50, Bowen Song mailto:bo...@bso.ng>> 
>>> wrote:
>>> 
>>> Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a slow 
>>> starting node, do you get a lots of rows? This is the most obvious loop run 
>>> (indirectly) by the ActiveRepairService.start(). 
>>> 
>>> On 24/01/2022 13:30, Romain Anselin wrote:
 
 Hi everyone,
 
 We generated a JFR profile of the startup phase of Cassandra with Paul, 
 and it would appear that the time is spent in the ActiveRepairSession 
 within the main thread (11mn of execution of the "main" thread in his 
 environment, vs 15s in mine), which has been introduced in CASSANDRA-9143 
 based on a "view git blame" of the source code
 https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381
  
 
 
 That seem to match with the gap we see in the logs, where the time is 
 spent just before the "Preloaded" statement in the logs which comes just 
 after in the CassandraDaemon code.
 
 INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token 
 metadata: Normal Tokens:
 ...  ...
 WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  
 NoSpamLogger.java:95 - /IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] 
 dropping message of type SCHEMA_VERSION_REQ whose timeout expired before 
 reaching the network
 INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 
 0 prepared statements
 
 Remains to determine what that is doing more in details and why it's 
 taking longer and longer on startup. 
 
 We also exported the sstablemetadata from all sstables in one node, and at 
 this stage, we can see we have 300 sstables out of 577 with "Repaired at:" 
 set.
 
 cd /var/lib/cassandra/data
 find . -name '*Data*' | while read datf; do echo  $datf ; sudo -u 
 cassandra sstablemetadata $datf; done >> ~/sstablemetadata.txt
 cqlsh -e "paging off; select * from system.repairs" >> ~/repairs.out
 
 $ egrep 'Repaired at: 1' sstablemetadata.txt | wc -l
 300
 $ egrep 'Repaired at:' sstablemetadata.txt | wc -l
 577
 
 More info to come
 
 Regards - Romain
 
 On 19/01/2022 13:10, Paul Chandler wrote:
> Hi Bowen,
> 
> Thanks for the reply, these have been our normal shutdowns, so we do a 
> nodetool drain before 

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

From the source code I've read, by default Cassandra will run a clean 
up for the system.repairs table every 10 minutes, any row related to a 
repair that has completed over 1 day ago will be automatically removed. 
I highly doubt that you have ran 75,000 repairs in the 24 hours prior to 
shutting down that node, because that's nearly one repair every second.


Do you see any logs like these?

 * Auto failing timed out repair session...
 * Skipping delete of FINALIZED LocalSession ... because it has not
   been superseded by a more recent session
 * Skipping delete of LocalSession ... because it still contains sstables

They are the logs from the cleanup() method in 
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L416 
which indicates a record was not deleted during the cleaned up for a 
number of reasons.


On 24/01/2022 19:45, Paul Chandler wrote:

Hi Bowen,

Yes, there does seem to be a lot of rows, on one of the upgraded 
clusters there 75,000 rows.


I have been experimenting on a test cluster, this has about a 5 minute 
pause, and around 15,000 rows.


If I clear the system.repairs table ( by deleting the sstables ) then 
this does not pause at all, so seems to fix the problem. However I 
don’t really understand what the implications are of just removing 
that data.


Thanks

Paul


On 24 Jan 2022, at 18:50, Bowen Song  wrote:

Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a 
slow starting node, do you get a lots of rows? This is the most 
obvious loop run (indirectly) by the ActiveRepairService.start().


On 24/01/2022 13:30, Romain Anselin wrote:


Hi everyone,

We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that 
the time is spent in the ActiveRepairSession within the main thread (11mn of execution of the 
"main" thread in his environment, vs 15s in mine), which has been introduced in 
CASSANDRA-9143 based on a "view git blame" of the source code
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381

That seem to match with the gap we see in the logs, where the time is spent just before 
the "Preloaded" statement in the logs which comes just after in the 
CassandraDaemon code.

INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token metadata: 
Normal Tokens:
...  ...
WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  NoSpamLogger.java:95 - 
/IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] dropping message of type 
SCHEMA_VERSION_REQ whose timeout expired before reaching the network
INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 0 
prepared statements

Remains to determine what that is doing more in details and why it's taking 
longer and longer on startup.

We also exported the sstablemetadata from all sstables in one node, and at this stage, we 
can see we have 300 sstables out of 577 with "Repaired at:" set.

|cd /var/lib/cassandra/data find . -name '*Data*' | while read datf; 
do echo  $datf ; sudo -u cassandra sstablemetadata $datf; 
done >> ~/sstablemetadata.txt cqlsh -e "paging off; select * from 
system.repairs" >> ~/repairs.out $ egrep 'Repaired at: 1' 
sstablemetadata.txt | wc -l 300 $ egrep 'Repaired at:' 
sstablemetadata.txt | wc -l 577|


More info to come

Regards - Romain

On 19/01/2022 13:10, Paul Chandler wrote:

Hi Bowen,

Thanks for the reply, these have been our normal shutdowns, so we do a nodetool 
drain before restarting the service, so I would have thought there should not 
be any commtlogs

However there is these messages for one commit log, But looks like it has 
finished quickly and correctly:

INFO  [main] 2022-01-19 10:08:22,811  CommitLog.java:173 - Replaying 
/var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
WARN  [main] 2022-01-19 10:08:22,839  CommitLogReplayer.java:305 - Origin of 2 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
Repeated about 10 times
WARN  [main] 2022-01-19 10:08:22,842  CommitLogReplayer.java:305 - Origin of 3 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
INFO  [main] 2022-01-19 10:08:22,853  CommitLogReader.java:256 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
INFO  [main] 2022-01-19 10:08:22,882  CommitLog.java:175 - Log replay complete, 
0 replayed mutations

Thanks

Paul


On 19 Jan 2022, at 13:03, Bowen Song  wrote:

Nothing obvious from the logs you posted.

Generally speaking, replaying commit log is often the culprit when a node takes a long 
time to start. I have seen many nodes with large memtable and commit log size limit 
spending over half an hour replaying the commit log. I usually do a "nodetool 
flush" before shutting down the node to help speed up the start time if the shutdown 

Re: Cassandra 4.0 hanging on restart

[Paul Chandler]

Hi Bowen,

Yes, there does seem to be a lot of rows, on one of the upgraded clusters there 
75,000 rows.

I have been experimenting on a test cluster, this has about a 5 minute pause, 
and around 15,000 rows. 

If I clear the system.repairs table ( by deleting the sstables ) then this does 
not pause at all, so seems to fix the problem. However I don’t really 
understand what the implications are of just removing that data.

Thanks 

Paul

> On 24 Jan 2022, at 18:50, Bowen Song  wrote:
> 
> Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a slow 
> starting node, do you get a lots of rows? This is the most obvious loop run 
> (indirectly) by the ActiveRepairService.start(). 
> 
> On 24/01/2022 13:30, Romain Anselin wrote:
>> 
>> Hi everyone,
>> 
>> We generated a JFR profile of the startup phase of Cassandra with Paul, and 
>> it would appear that the time is spent in the ActiveRepairSession within the 
>> main thread (11mn of execution of the "main" thread in his environment, vs 
>> 15s in mine), which has been introduced in CASSANDRA-9143 based on a "view 
>> git blame" of the source code
>> https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381
>>  
>> 
>> 
>> That seem to match with the gap we see in the logs, where the time is spent 
>> just before the "Preloaded" statement in the logs which comes just after in 
>> the CassandraDaemon code.
>> 
>> INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token 
>> metadata: Normal Tokens:
>> ...  ...
>> WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  
>> NoSpamLogger.java:95 - /IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] 
>> dropping message of type SCHEMA_VERSION_REQ whose timeout expired before 
>> reaching the network
>> INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 0 
>> prepared statements
>> 
>> Remains to determine what that is doing more in details and why it's taking 
>> longer and longer on startup. 
>> 
>> We also exported the sstablemetadata from all sstables in one node, and at 
>> this stage, we can see we have 300 sstables out of 577 with "Repaired at:" 
>> set.
>> 
>> cd /var/lib/cassandra/data
>> find . -name '*Data*' | while read datf; do echo  $datf ; sudo -u 
>> cassandra sstablemetadata $datf; done >> ~/sstablemetadata.txt
>> cqlsh -e "paging off; select * from system.repairs" >> ~/repairs.out
>> 
>> $ egrep 'Repaired at: 1' sstablemetadata.txt | wc -l
>> 300
>> $ egrep 'Repaired at:' sstablemetadata.txt | wc -l
>> 577
>> 
>> More info to come
>> 
>> Regards - Romain
>> 
>> On 19/01/2022 13:10, Paul Chandler wrote:
>>> Hi Bowen,
>>> 
>>> Thanks for the reply, these have been our normal shutdowns, so we do a 
>>> nodetool drain before restarting the service, so I would have thought there 
>>> should not be any commtlogs
>>> 
>>> However there is these messages for one commit log, But looks like it has 
>>> finished quickly and correctly:
>>> 
>>> INFO  [main] 2022-01-19 10:08:22,811  CommitLog.java:173 - Replaying 
>>> /var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
>>> WARN  [main] 2022-01-19 10:08:22,839  CommitLogReplayer.java:305 - Origin 
>>> of 2 sstables is unknown or doesn't match the local node; 
>>> commitLogIntervals for them were ignored
>>> Repeated about 10 times
>>> WARN  [main] 2022-01-19 10:08:22,842  CommitLogReplayer.java:305 - Origin 
>>> of 3 sstables is unknown or doesn't match the local node; 
>>> commitLogIntervals for them were ignored
>>> INFO  [main] 2022-01-19 10:08:22,853  CommitLogReader.java:256 - Finished 
>>> reading /var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
>>> INFO  [main] 2022-01-19 10:08:22,882  CommitLog.java:175 - Log replay 
>>> complete, 0 replayed mutations 
>>> 
>>> Thanks 
>>> 
>>> Paul
>>> 
 On 19 Jan 2022, at 13:03, Bowen Song   
 wrote:
 
 Nothing obvious from the logs you posted.
 
 Generally speaking, replaying commit log is often the culprit when a node 
 takes a long time to start. I have seen many nodes with large memtable and 
 commit log size limit spending over half an hour replaying the commit log. 
 I usually do a "nodetool flush" before shutting down the node to help 
 speed up the start time if the shutdown was planned. There isn't much you 
 can do about unexpected shutdown, such as server crashes. When that 
 happens, the only reasonable thing to do is wait for the commit log replay 
 to finish. You should see log entries related to replaying commit logs if 
 this is the case.
 
 However, if you don't find any logs related to replaying commit logs, the 
 cause may be completely different.
 
 
 On 19/01/2022 11:54, Paul Chandler wrote:

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

Hmm, interesting... Try "select * from system.repairs;" in cqlsh on a 
slow starting node, do you get a lots of rows? This is the most obvious 
loop run (indirectly) by the ActiveRepairService.start().


On 24/01/2022 13:30, Romain Anselin wrote:


Hi everyone,

We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that 
the time is spent in the ActiveRepairSession within the main thread (11mn of execution of the 
"main" thread in his environment, vs 15s in mine), which has been introduced in 
CASSANDRA-9143 based on a "view git blame" of the source code
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381

That seem to match with the gap we see in the logs, where the time is spent just before 
the "Preloaded" statement in the logs which comes just after in the 
CassandraDaemon code.

INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token metadata: 
Normal Tokens:
...  ...
WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  NoSpamLogger.java:95 - 
/IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] dropping message of type 
SCHEMA_VERSION_REQ whose timeout expired before reaching the network
INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 0 
prepared statements

Remains to determine what that is doing more in details and why it's taking 
longer and longer on startup.

We also exported the sstablemetadata from all sstables in one node, and at this stage, we 
can see we have 300 sstables out of 577 with "Repaired at:" set.

|cd /var/lib/cassandra/data find . -name '*Data*' | while read datf; 
do echo  $datf ; sudo -u cassandra sstablemetadata $datf; done 
>> ~/sstablemetadata.txt cqlsh -e "paging off; select * from 
system.repairs" >> ~/repairs.out $ egrep 'Repaired at: 1' 
sstablemetadata.txt | wc -l 300 $ egrep 'Repaired at:' 
sstablemetadata.txt | wc -l 577|


More info to come

Regards - Romain

On 19/01/2022 13:10, Paul Chandler wrote:

Hi Bowen,

Thanks for the reply, these have been our normal shutdowns, so we do a nodetool 
drain before restarting the service, so I would have thought there should not 
be any commtlogs

However there is these messages for one commit log, But looks like it has 
finished quickly and correctly:

INFO  [main] 2022-01-19 10:08:22,811  CommitLog.java:173 - Replaying 
/var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
WARN  [main] 2022-01-19 10:08:22,839  CommitLogReplayer.java:305 - Origin of 2 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
Repeated about 10 times
WARN  [main] 2022-01-19 10:08:22,842  CommitLogReplayer.java:305 - Origin of 3 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
INFO  [main] 2022-01-19 10:08:22,853  CommitLogReader.java:256 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
INFO  [main] 2022-01-19 10:08:22,882  CommitLog.java:175 - Log replay complete, 
0 replayed mutations

Thanks

Paul


On 19 Jan 2022, at 13:03, Bowen Song  wrote:

Nothing obvious from the logs you posted.

Generally speaking, replaying commit log is often the culprit when a node takes a long 
time to start. I have seen many nodes with large memtable and commit log size limit 
spending over half an hour replaying the commit log. I usually do a "nodetool 
flush" before shutting down the node to help speed up the start time if the shutdown 
was planned. There isn't much you can do about unexpected shutdown, such as server 
crashes. When that happens, the only reasonable thing to do is wait for the commit log 
replay to finish. You should see log entries related to replaying commit logs if this is 
the case.

However, if you don't find any logs related to replaying commit logs, the cause 
may be completely different.


On 19/01/2022 11:54, Paul Chandler wrote:

Hi all,

We have upgraded a couple of clusters from 3.11.6, now we are having issues 
when we restart the nodes.

The node will either hang or take 10-30 minute to restart, these are the last 
messages we have in the system.log:

INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,267  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-184-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,269  LogTransaction.java:240 - 
Unfinished transaction log, deleting 

Re: Cassandra 4.0 hanging on restart

[Romain Anselin]

Hi everyone,

We generated a JFR profile of the startup phase of Cassandra with Paul, and it would appear that 
the time is spent in the ActiveRepairSession within the main thread (11mn of execution of the 
"main" thread in his environment, vs 15s in mine), which has been introduced in 
CASSANDRA-9143 based on a "view git blame" of the source code
https://github.com/apache/cassandra/blob/6709111ed007a54b3e42884853f89cabd38e4316/src/java/org/apache/cassandra/service/CassandraDaemon.java#L381

That seem to match with the gap we see in the logs, where the time is spent just before 
the "Preloaded" statement in the logs which comes just after in the 
CassandraDaemon code.

INFO  [main] 2022-01-20 09:44:40,198  StorageService.java:830 - Token metadata: 
Normal Tokens:
...  ...
WARN  [Messaging-EventLoop-3-1] 2022-01-20 09:45:13,243  NoSpamLogger.java:95 - 
/IP1:7000->/IP1:7000-SMALL_MESSAGES-[no-channel] dropping message of type 
SCHEMA_VERSION_REQ whose timeout expired before reaching the network
INFO  [main] 2022-01-20 09:55:01,134  QueryProcessor.java:150 - Preloaded 0 
prepared statements

Remains to determine what that is doing more in details and why it's taking 
longer and longer on startup.

We also exported the sstablemetadata from all sstables in one node, and at this stage, we 
can see we have 300 sstables out of 577 with "Repaired at:" set.

|cd /var/lib/cassandra/data find . -name '*Data*' | while read datf; do 
echo  $datf ; sudo -u cassandra sstablemetadata $datf; done >> 
~/sstablemetadata.txt cqlsh -e "paging off; select * from 
system.repairs" >> ~/repairs.out $ egrep 'Repaired at: 1' 
sstablemetadata.txt | wc -l 300 $ egrep 'Repaired at:' 
sstablemetadata.txt | wc -l 577|


More info to come

Regards - Romain

On 19/01/2022 13:10, Paul Chandler wrote:

Hi Bowen,

Thanks for the reply, these have been our normal shutdowns, so we do a nodetool 
drain before restarting the service, so I would have thought there should not 
be any commtlogs

However there is these messages for one commit log, But looks like it has 
finished quickly and correctly:

INFO  [main] 2022-01-19 10:08:22,811  CommitLog.java:173 - Replaying 
/var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
WARN  [main] 2022-01-19 10:08:22,839  CommitLogReplayer.java:305 - Origin of 2 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
Repeated about 10 times
WARN  [main] 2022-01-19 10:08:22,842  CommitLogReplayer.java:305 - Origin of 3 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
INFO  [main] 2022-01-19 10:08:22,853  CommitLogReader.java:256 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
INFO  [main] 2022-01-19 10:08:22,882  CommitLog.java:175 - Log replay complete, 
0 replayed mutations

Thanks

Paul


On 19 Jan 2022, at 13:03, Bowen Song  wrote:

Nothing obvious from the logs you posted.

Generally speaking, replaying commit log is often the culprit when a node takes a long 
time to start. I have seen many nodes with large memtable and commit log size limit 
spending over half an hour replaying the commit log. I usually do a "nodetool 
flush" before shutting down the node to help speed up the start time if the shutdown 
was planned. There isn't much you can do about unexpected shutdown, such as server 
crashes. When that happens, the only reasonable thing to do is wait for the commit log 
replay to finish. You should see log entries related to replaying commit logs if this is 
the case.

However, if you don't find any logs related to replaying commit logs, the cause 
may be completely different.


On 19/01/2022 11:54, Paul Chandler wrote:

Hi all,

We have upgraded a couple of clusters from 3.11.6, now we are having issues 
when we restart the nodes.

The node will either hang or take 10-30 minute to restart, these are the last 
messages we have in the system.log:

INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,267  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-184-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,269  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,270  FileUtils.java:545 - 
Deleting file during startup: 

Re: Cassandra 4.0 hanging on restart

[Paul Chandler]

Hi Bowen,

Thanks for the reply, these have been our normal shutdowns, so we do a nodetool 
drain before restarting the service, so I would have thought there should not 
be any commtlogs

However there is these messages for one commit log, But looks like it has 
finished quickly and correctly:

INFO  [main] 2022-01-19 10:08:22,811  CommitLog.java:173 - Replaying 
/var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
WARN  [main] 2022-01-19 10:08:22,839  CommitLogReplayer.java:305 - Origin of 2 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
Repeated about 10 times
WARN  [main] 2022-01-19 10:08:22,842  CommitLogReplayer.java:305 - Origin of 3 
sstables is unknown or doesn't match the local node; commitLogIntervals for 
them were ignored
INFO  [main] 2022-01-19 10:08:22,853  CommitLogReader.java:256 - Finished 
reading /var/lib/cassandra/commitlog/CommitLog-7-1642094921295.log
INFO  [main] 2022-01-19 10:08:22,882  CommitLog.java:175 - Log replay complete, 
0 replayed mutations 

Thanks 

Paul

> On 19 Jan 2022, at 13:03, Bowen Song  wrote:
> 
> Nothing obvious from the logs you posted.
> 
> Generally speaking, replaying commit log is often the culprit when a node 
> takes a long time to start. I have seen many nodes with large memtable and 
> commit log size limit spending over half an hour replaying the commit log. I 
> usually do a "nodetool flush" before shutting down the node to help speed up 
> the start time if the shutdown was planned. There isn't much you can do about 
> unexpected shutdown, such as server crashes. When that happens, the only 
> reasonable thing to do is wait for the commit log replay to finish. You 
> should see log entries related to replaying commit logs if this is the case.
> 
> However, if you don't find any logs related to replaying commit logs, the 
> cause may be completely different.
> 
> 
> On 19/01/2022 11:54, Paul Chandler wrote:
>> Hi all,
>> 
>> We have upgraded a couple of clusters from 3.11.6, now we are having issues 
>> when we restart the nodes.
>> 
>> The node will either hang or take 10-30 minute to restart, these are the 
>> last messages we have in the system.log:
>> 
>> INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,267  FileUtils.java:545 - 
>> Deleting file during startup: 
>> /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-184-big-Summary.db
>> INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  LogTransaction.java:240 
>> - Unfinished transaction log, deleting 
>> /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Data.db
>> INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  FileUtils.java:545 - 
>> Deleting file during startup: 
>> /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Summary.db
>> INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,269  LogTransaction.java:240 
>> - Unfinished transaction log, deleting 
>> /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Data.db
>> INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,270  FileUtils.java:545 - 
>> Deleting file during startup: 
>> /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Summary.db
>> INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,272  LogTransaction.java:240 
>> - Unfinished transaction log, deleting 
>> /var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb_txn_unknowncompactiontype_bc501d00-790f-11ec-9f80-85
>> 8854746758.log
>> INFO  [MemtableFlushWriter:2] 2022-01-19 10:08:23,289  
>> LogTransaction.java:240 - Unfinished transaction log, deleting 
>> /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_bc52dc20-790f-11ec-9f80-858854746758.log
>> 
>> The debug log has messages from DiskBoundaryManager.java at the same time, 
>> then it just has the following messages:||
>> 
>> DEBUG [ScheduledTasks:1] 2022-01-19 10:28:09,430  SSLFactory.java:354 - 
>> Checking whether certificates have been updated []
>> DEBUG [ScheduledTasks:1] 2022-01-19 10:38:09,431  SSLFactory.java:354 - 
>> Checking whether certificates have been updated []
>> DEBUG [ScheduledTasks:1] 2022-01-19 10:48:09,431  SSLFactory.java:354 - 
>> Checking whether certificates have been updated []
>> DEBUG [ScheduledTasks:1] 2022-01-19 10:58:09,431  SSLFactory.java:354 - 
>> Checking whether certificates have been updated []
>> 
>> 
>> It seems to get worse after each restart, and then it gets to the state 
>> where it just hangs, then the only thing to do is to re bootstrap the node.
>> 
>> Once I had re bootstrapped all the nodes in the cluster, I thought the 
>> cluster was stable, but I have now got the case where the one of the nodes 
>> is hanging again.
>> 
>> Does anyone have an ideas what is causing the problems ?
>> 
>> 
>> Thanks
>> 
>> Paul Chandler
>> 

Re: Cassandra 4.0 hanging on restart

[Bowen Song]

Nothing obvious from the logs you posted.

Generally speaking, replaying commit log is often the culprit when a 
node takes a long time to start. I have seen many nodes with large 
memtable and commit log size limit spending over half an hour replaying 
the commit log. I usually do a "nodetool flush" before shutting down the 
node to help speed up the start time if the shutdown was planned. There 
isn't much you can do about unexpected shutdown, such as server crashes. 
When that happens, the only reasonable thing to do is wait for the 
commit log replay to finish. You should see log entries related to 
replaying commit logs if this is the case.


However, if you don't find any logs related to replaying commit logs, 
the cause may be completely different.



On 19/01/2022 11:54, Paul Chandler wrote:

Hi all,

We have upgraded a couple of clusters from 3.11.6, now we are having issues 
when we restart the nodes.

The node will either hang or take 10-30 minute to restart, these are the last 
messages we have in the system.log:

INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,267  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-184-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,269  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,270  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,272  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb_txn_unknowncompactiontype_bc501d00-790f-11ec-9f80-85
8854746758.log
INFO  [MemtableFlushWriter:2] 2022-01-19 10:08:23,289  LogTransaction.java:240 
- Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_bc52dc20-790f-11ec-9f80-858854746758.log

The debug log has messages from DiskBoundaryManager.java at the same time, then 
it just has the following messages:||

DEBUG [ScheduledTasks:1] 2022-01-19 10:28:09,430  SSLFactory.java:354 - 
Checking whether certificates have been updated []
DEBUG [ScheduledTasks:1] 2022-01-19 10:38:09,431  SSLFactory.java:354 - 
Checking whether certificates have been updated []
DEBUG [ScheduledTasks:1] 2022-01-19 10:48:09,431  SSLFactory.java:354 - 
Checking whether certificates have been updated []
DEBUG [ScheduledTasks:1] 2022-01-19 10:58:09,431  SSLFactory.java:354 - 
Checking whether certificates have been updated []


It seems to get worse after each restart, and then it gets to the state where 
it just hangs, then the only thing to do is to re bootstrap the node.

Once I had re bootstrapped all the nodes in the cluster, I thought the cluster 
was stable, but I have now got the case where the one of the nodes is hanging 
again.

Does anyone have an ideas what is causing the problems ?


Thanks

Paul Chandler

Cassandra 4.0 hanging on restart

[Paul Chandler]

Hi all,

We have upgraded a couple of clusters from 3.11.6, now we are having issues 
when we restart the nodes.

The node will either hang or take 10-30 minute to restart, these are the last 
messages we have in the system.log:

INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,267  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-184-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,268  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-185-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,269  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Data.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,270  FileUtils.java:545 - 
Deleting file during startup: 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb-186-big-Summary.db
INFO  [NonPeriodicTasks:1] 2022-01-19 10:08:23,272  LogTransaction.java:240 - 
Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/table_estimates-176c39cdb93d33a5a2188eb06a56f66e/nb_txn_unknowncompactiontype_bc501d00-790f-11ec-9f80-85
8854746758.log
INFO  [MemtableFlushWriter:2] 2022-01-19 10:08:23,289  LogTransaction.java:240 
- Unfinished transaction log, deleting 
/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_bc52dc20-790f-11ec-9f80-858854746758.log

The debug log has messages from DiskBoundaryManager.java at the same time, then 
it just has the following messages:||

DEBUG [ScheduledTasks:1] 2022-01-19 10:28:09,430  SSLFactory.java:354 - 
Checking whether certificates have been updated []
DEBUG [ScheduledTasks:1] 2022-01-19 10:38:09,431  SSLFactory.java:354 - 
Checking whether certificates have been updated []
DEBUG [ScheduledTasks:1] 2022-01-19 10:48:09,431  SSLFactory.java:354 - 
Checking whether certificates have been updated []
DEBUG [ScheduledTasks:1] 2022-01-19 10:58:09,431  SSLFactory.java:354 - 
Checking whether certificates have been updated []


It seems to get worse after each restart, and then it gets to the state where 
it just hangs, then the only thing to do is to re bootstrap the node. 

Once I had re bootstrapped all the nodes in the cluster, I thought the cluster 
was stable, but I have now got the case where the one of the nodes is hanging 
again. 

Does anyone have an ideas what is causing the problems ? 


Thanks 

Paul Chandler

Some more tooling around Cassandra 4.0

[Stefan Miklosovic]

Hi users,

I would like to highlight some tooling we put together at Instaclustr
focusing / updating it to the recent Cassandra 4.0 release.

We wrote a short and descriptive blog about that here (1).

All these tools are completely free of charge and Apache 2.0 licensed.

We hope you find them useful, in case you have any questions, feel
free to reach us on GitHub issues.

(1) https://www.instaclustr.com/cassandra-tools-updated-cassandra-4-0/

Regards

Stefan Miklosovic

Re: Validation of NetworkTopologyStrategy data center name in Cassandra 4.0

[Jens Fischer]

Thanks for providing the links Erick, very helpful. Although it is slightly 
inconvenient for me I now better understand the motivation.

On 10. Aug 2021, at 10:27, Erick Ramirez 
mailto:erick.rami...@datastax.com>> wrote:

You are correct. Cassandra no longer allows invalid DC names for 
NetworkTopologyStrategy in CREATE KEYSPACE or ALTER KEYSPACE from 4.0 
(CASSANDRA-12681). FWIW, 
here is the 
NEWS.txt
 entry for reference. I'm not aware of a hack that would circumvent the 
validation. Cheers!


Gesch?ftsf?hrer: Oliver Koch, Bianca Swanston
Amtsgericht Kempten/Allg?u, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908

Re: Validation of NetworkTopologyStrategy data center name in Cassandra 4.0

[Erick Ramirez]

You are correct. Cassandra no longer allows invalid DC names for
NetworkTopologyStrategy in CREATE KEYSPACE or ALTER KEYSPACE from 4.0 (
CASSANDRA-12681 ).
FWIW, here is the NEWS.txt

entry for reference. I'm not aware of a hack that would circumvent the
validation. Cheers!

>

Validation of NetworkTopologyStrategy data center name in Cassandra 4.0

[Jens Fischer]

Hi,

in Cassandra 3.11.x I was able to create key spaces with basically arbitrary 
names for the data center. When I do this with Cassandra 4.0 I get a 
"ConfigurationException: Unrecognized strategy option {} 
passed to NetworkTopologyStrategy for keyspace ”.

This breaks some unit tests in our CI where we test CREATE KEYSPACE statements 
for different clusters on a single node test instance.

The only documentation I found is an issue from ScyllaDB: 
https://github.com/scylladb/scylla/issues/7595. It seems Cassandra 4.0 added 
some validation on the data center name. I verified that I can get rid of the 
error by configuring a DC in the cassandra-rackdc.properties and enabling 
endpoint_snitch: GossipingPropertyFileSnitch in cassandra.yaml. This, of 
course, is not very practical for unit tests because we would need to change 
the configuration of Cassandra (and restart) before any of the unit tests. No 
problem for production of course, there the cluster is configured accordingly.

Is there a way to disable the validations for testing purposes? Or to change 
them dynamically?

Any help is appreciated!
Jens


Geschäftsführer: Oliver Koch, Bianca Swanston
Amtsgericht Kempten/Allgäu, Registernummer: 10655, Steuernummer 127/137/50792, 
USt.-IdNr. DE272208908

Re: cassandra 4.0 java 11 support

[Scott Andreas]

Anecdotally, I have no issues to report running Cassandra 4.0 on JDK11 and 
would be supportive of removing the "experimental" classification in an 
upcoming release.


From: Erick Ramirez 
Sent: Tuesday, July 27, 2021 3:57 PM
To: user@cassandra.apache.org
Subject: Re: cassandra 4.0 java 11 support

There's been some discussion around removing the "experimental" tag for C* 4.0 
+ Java 11 so by all means, we encourage everyone to try it and report back to 
the community if you run into issues. Java 11 support was added 2 years ago so 
I think most of the issues have been ironed out. Now that 4.0 is out, we're 
hoping there would be more users testing it out. Cheers!

Re: cassandra 4.0 java 11 support

[Erick Ramirez]

There's been some discussion around removing the "experimental" tag for C*
4.0 + Java 11 so by all means, we encourage everyone to try it and report
back to the community if you run into issues. Java 11 support was added 2
years ago so I think most of the issues have been ironed out. Now that 4.0
is out, we're hoping there would be more users testing it out. Cheers!

Re: cassandra 4.0 java 11 support

[Bowen Song]

Experimental means anything can happen - dragons, unicorns, ...

On 27/07/2021 21:32, CPC wrote:

Hi ,

At cassandra site 
https://cassandra.apache.org/doc/latest/cassandra/new/java11.html 
 , 
it says java 11 support is experimental and not recommended for 
production. What is the reason for that? I mean performance or bugs?


Thank you...

Re: [RELEASE] Apache Cassandra 4.0-rc2 released

[Abdul Patel]

Thanks Patrick!!

On Wed, Jun 30, 2021, 7:54 PM Patrick McFadin  wrote:

> Hi Abdul,
>
> This is a release candidate so what the project is proposing as production
> ready. We're asking the community to test however possible and provide any
> feedback you have. If there are no issues found, this will turn into the GA
> release.
>
> Patrick
>
> On Wed, Jun 30, 2021 at 1:09 PM Abdul Patel  wrote:
>
>> Great news!!
>> Is this production ready?
>>
>> On Wed, Jun 30, 2021, 3:56 PM Patrick McFadin  wrote:
>>
>>> Congrats to everyone that worked on this iteration. If you haven't
>>> looked at the CHANGES.txt there were some great catches in RC1. Just like
>>> it should happen!
>>>
>>> On Wed, Jun 30, 2021 at 12:29 PM Mick Semb Wever  wrote:
>>>
>>>>
>>>> The Cassandra team is pleased to announce the release of Apache
>>>> Cassandra version 4.0-rc2.
>>>>
>>>> Apache Cassandra is a fully distributed database. It is the right
>>>> choice when you need scalability and high availability without compromising
>>>> performance.
>>>>  http://cassandra.apache.org/
>>>>
>>>> Downloads of source and binary distributions are listed in our download
>>>> section:
>>>>  http://cassandra.apache.org/download/
>>>>
>>>> This version is a release candidate[1] on the 4.0 series. As always,
>>>> please pay attention to the release notes[2] and let us know[3] if you were
>>>> to encounter any problem.
>>>>
>>>> Please note, the bintray location is now replaced with the ASF's JFrog
>>>> Artifactory location: https://apache.jfrog.io/artifactory/cassandra/
>>>>
>>>> Enjoy!
>>>>
>>>> [1]: CHANGES.txt
>>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc2
>>>> [2]: NEWS.txt
>>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc2
>>>> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>>>>
>>>

Re: [RELEASE] Apache Cassandra 4.0-rc2 released

[Patrick McFadin]

Hi Abdul,

This is a release candidate so what the project is proposing as production
ready. We're asking the community to test however possible and provide any
feedback you have. If there are no issues found, this will turn into the GA
release.

Patrick

On Wed, Jun 30, 2021 at 1:09 PM Abdul Patel  wrote:

> Great news!!
> Is this production ready?
>
> On Wed, Jun 30, 2021, 3:56 PM Patrick McFadin  wrote:
>
>> Congrats to everyone that worked on this iteration. If you haven't looked
>> at the CHANGES.txt there were some great catches in RC1. Just like it
>> should happen!
>>
>> On Wed, Jun 30, 2021 at 12:29 PM Mick Semb Wever  wrote:
>>
>>>
>>> The Cassandra team is pleased to announce the release of Apache
>>> Cassandra version 4.0-rc2.
>>>
>>> Apache Cassandra is a fully distributed database. It is the right choice
>>> when you need scalability and high availability without compromising
>>> performance.
>>>  http://cassandra.apache.org/
>>>
>>> Downloads of source and binary distributions are listed in our download
>>> section:
>>>  http://cassandra.apache.org/download/
>>>
>>> This version is a release candidate[1] on the 4.0 series. As always,
>>> please pay attention to the release notes[2] and let us know[3] if you were
>>> to encounter any problem.
>>>
>>> Please note, the bintray location is now replaced with the ASF's JFrog
>>> Artifactory location: https://apache.jfrog.io/artifactory/cassandra/
>>>
>>> Enjoy!
>>>
>>> [1]: CHANGES.txt
>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc2
>>> [2]: NEWS.txt
>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc2
>>> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>>>
>>

Re: [RELEASE] Apache Cassandra 4.0-rc2 released

[Joe Obernberger]

Downloading now!  Thank you!  yum update cassandra!

-Joe

On 6/30/2021 3:55 PM, Patrick McFadin wrote:
Congrats to everyone that worked on this iteration. If you haven't 
looked at the CHANGES.txt there were some great catches in RC1. Just 
like it should happen!


On Wed, Jun 30, 2021 at 12:29 PM Mick Semb Wever  wrote:


The Cassandra team is pleased to announce the release of Apache
Cassandra version 4.0-rc2.

Apache Cassandra is a fully distributed database. It is the right
choice when you need scalability and high availability without
compromising performance.
http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our
download section:
http://cassandra.apache.org/download/

This version is a release candidate[1] on the 4.0 series. As
always, please pay attention to the release notes[2] and let us
know[3] if you were to encounter any problem.

Please note, the bintray location is now replaced with the ASF's
JFrog Artifactory location:
https://apache.jfrog.io/artifactory/cassandra/

Enjoy!

[1]: CHANGES.txt

https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc2
[2]: NEWS.txt

https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc2
[3]: https://issues.apache.org/jira/browse/CASSANDRA


<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=emailclient> 
	Virus-free. www.avg.com 
<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=emailclient> 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: [RELEASE] Apache Cassandra 4.0-rc2 released

[Abdul Patel]

Great news!!
Is this production ready?

On Wed, Jun 30, 2021, 3:56 PM Patrick McFadin  wrote:

> Congrats to everyone that worked on this iteration. If you haven't looked
> at the CHANGES.txt there were some great catches in RC1. Just like it
> should happen!
>
> On Wed, Jun 30, 2021 at 12:29 PM Mick Semb Wever  wrote:
>
>>
>> The Cassandra team is pleased to announce the release of Apache Cassandra
>> version 4.0-rc2.
>>
>> Apache Cassandra is a fully distributed database. It is the right choice
>> when you need scalability and high availability without compromising
>> performance.
>>  http://cassandra.apache.org/
>>
>> Downloads of source and binary distributions are listed in our download
>> section:
>>  http://cassandra.apache.org/download/
>>
>> This version is a release candidate[1] on the 4.0 series. As always,
>> please pay attention to the release notes[2] and let us know[3] if you were
>> to encounter any problem.
>>
>> Please note, the bintray location is now replaced with the ASF's JFrog
>> Artifactory location: https://apache.jfrog.io/artifactory/cassandra/
>>
>> Enjoy!
>>
>> [1]: CHANGES.txt
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc2
>> [2]: NEWS.txt
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc2
>> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>>
>

Re: [RELEASE] Apache Cassandra 4.0-rc2 released

[Patrick McFadin]

Congrats to everyone that worked on this iteration. If you haven't looked
at the CHANGES.txt there were some great catches in RC1. Just like it
should happen!

On Wed, Jun 30, 2021 at 12:29 PM Mick Semb Wever  wrote:

>
> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 4.0-rc2.
>
> Apache Cassandra is a fully distributed database. It is the right choice
> when you need scalability and high availability without compromising
> performance.
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>  http://cassandra.apache.org/download/
>
> This version is a release candidate[1] on the 4.0 series. As always,
> please pay attention to the release notes[2] and let us know[3] if you were
> to encounter any problem.
>
> Please note, the bintray location is now replaced with the ASF's JFrog
> Artifactory location: https://apache.jfrog.io/artifactory/cassandra/
>
> Enjoy!
>
> [1]: CHANGES.txt
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc2
> [2]: NEWS.txt
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc2
> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>

[RELEASE] Apache Cassandra 4.0-rc2 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 4.0-rc2.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.
 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:
 http://cassandra.apache.org/download/

This version is a release candidate[1] on the 4.0 series. As always, please
pay attention to the release notes[2] and let us know[3] if you were to
encounter any problem.

Please note, the bintray location is now replaced with the ASF's JFrog
Artifactory location: https://apache.jfrog.io/artifactory/cassandra/

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc2
[2]: NEWS.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc2
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Cassandra 4.0 GA

[Jai Bheemsen Rao Dhanwada]

Thank you for the information

On Wed, May 26, 2021 at 5:40 PM Kane Wilson  wrote:

> On Tue, May 25, 2021 at 6:36 AM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello All,
>>
>> I see that Cassandra 4.0 RC1 is released in April, is there going to be
>> an official 4.0 GA release or is RC1 considered as an official GA release
>> with Production use? If not is there a tentative timeline for the GA
>> release?
>>
>
> Eventually a release candidate will be promoted to a GA release. I think
> it's unlikely RC1 will be promoted however. There is no timeline, it's a
> "when it's done it's done" thing. We're certainly close now though. A few
> months max, one would hope.
>
>
> --
> raft.so - Cassandra consulting, support, and managed services
>

Re: Cassandra 4.0 GA

[Kane Wilson]

On Tue, May 25, 2021 at 6:36 AM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello All,
>
> I see that Cassandra 4.0 RC1 is released in April, is there going to be an
> official 4.0 GA release or is RC1 considered as an official GA release with
> Production use? If not is there a tentative timeline for the GA release?
>

Eventually a release candidate will be promoted to a GA release. I think
it's unlikely RC1 will be promoted however. There is no timeline, it's a
"when it's done it's done" thing. We're certainly close now though. A few
months max, one would hope.


-- 
raft.so - Cassandra consulting, support, and managed services

Cassandra 4.0 GA

[Jai Bheemsen Rao Dhanwada]

Hello All,

I see that Cassandra 4.0 RC1 is released in April, is there going to be an
official 4.0 GA release or is RC1 considered as an official GA release with
Production use? If not is there a tentative timeline for the GA release?

Re: Compatibility between Cassandra 3.11 and cqlsh from Cassandra 4.0 RC1

[Bowen Song]

Okay, CASSANDRA-16652 
<https://issues.apache.org/jira/browse/CASSANDRA-16652> is created.



On 04/05/2021 22:30, Paulo Motta wrote:

Hi Bowen,

This seems like a bug to me, please kindly file an issue on 
https://issues.apache.org/jira/projects/CASSANDRA/summary 
<https://issues.apache.org/jira/projects/CASSANDRA/summary> with the 
provided reproduction steps.


Thanks,

Paulo

Em ter., 4 de mai. de 2021 às 18:22, Bowen Song  escreveu:

Hi all,


I was using the cqlsh from Cassandra 4.0 RC1 and trying to connect
to a Cassandra 3.11 cluster, and it does not appear to be working
correctly. Specificity, the "desc" command does not work at all.

Steps to reproduce:

# ensure you have docker installed and running
# run the following docker commands to create and start a
Cassandra 3.11 container
~ $ docker create --name cassandra3 cassandra:3.11.10
5d903e48e0661e39080198de5e8752356a5a666132211a500ea38af0fc2a0356
~ $ docker start cassandra3
cassandra3
~ $ docker exec -ti cassandra3 bash

# inside the docker container, try the default cqlsh version
root@5d903e48e066:/# cqlsh
Connected to Test Cluster at 127.0.0.1:9042
<http://127.0.0.1:9042>.
[cqlsh 5.0.1 | Cassandra 3.11.10 | CQL spec 3.4.4 | Native
protocol v4]
Use HELP for help.
cqlsh> desc keyspaces;

system_traces  system_schema  system_auth  system
system_distributed

cqlsh> use system_auth;
cqlsh:system_auth> desc tables;

resource_role_permissons_index  role_permissions
role_members  roles

cqlsh:system_auth> select * from roles;

 role  | can_login | is_superuser | member_of |
salted_hash

---+---+--+---+--
 cassandra |  True | True |  null |
$2a$10$8UNyioBF41/OZfcCa2aqXOHvXiNXArBHKaUUhMyPAFKpfN8byXonm

(1 rows)
cqlsh:system_auth> exit

# obviously, everything worked as expected

    # now install git to clone the Cassandra 4.0 RC1 source
code, and python3-six, which is a dependency of cqlsh
root@5d903e48e066:/# apt-get update -qq && apt-get install
-qq git python3-six
..

# clone the Cassandra repository and checkout the
cassandra-4.0-rc1 tag
    root@5d903e48e066:/# git clone -b cassandra-4.0-rc1
--depth 1 https://github.com/apache/cassandra.git
    <https://github.com/apache/cassandra.git> cassandra-4.0-rc1
Cloning into 'cassandra-4.0-rc1'...
..

# run the cqlsh from the Git repository, and repeat all
the cqlsh statements above
root@5d903e48e066:/# cd cassandra-4.0-rc1/bin
root@5d903e48e066:/cassandra-4.0-rc1/bin#
<mailto:root@5d903e48e066:/cassandra-4.0-rc1/bin#> ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042
<http://127.0.0.1:9042>
[cqlsh 6.0.0 | Cassandra 3.11.10 | CQL spec 3.4.4 | Native
protocol v4]
Use HELP for help.
cqlsh> desc keyspaces;

cqlsh> use system_auth;
cqlsh:system_auth> desc tables;

cqlsh:system_auth> select * from roles;

 role  | can_login | is_superuser | member_of |
salted_hash

---+---+--+---+--
 cassandra |  True | True |  null |
$2a$10$8UNyioBF41/OZfcCa2aqXOHvXiNXArBHKaUUhMyPAFKpfN8byXonm

    (1 rows)
cqlsh:system_auth> exit
    root@5d903e48e066:/cassandra-4.0-rc1/bin#
<mailto:root@5d903e48e066:/cassandra-4.0-rc1/bin#>

# "desc" did not work, but "use" and "select" worked


My question is, are they suppose to be compatible? If so, this
almost certainly is a bug. If not, I'd strongly recommend
Cassandra to add a compatibility warning in the 4.0 version
cqlsh when it's connected to a 3.x Cassandra server. Because I
can foresee many users will run into this issue, especially
when they are in the middle of upgrading from 3.x to 4.0.


Regards,

Bowen

Re: Compatibility between Cassandra 3.11 and cqlsh from Cassandra 4.0 RC1

[Paulo Motta]

Hi Bowen,

This seems like a bug to me, please kindly file an issue on
https://issues.apache.org/jira/projects/CASSANDRA/summary with the provided
reproduction steps.

Thanks,

Paulo

Em ter., 4 de mai. de 2021 às 18:22, Bowen Song  escreveu:

> Hi all,
>
>
> I was using the cqlsh from Cassandra 4.0 RC1 and trying to connect to a
> Cassandra 3.11 cluster, and it does not appear to be working correctly.
> Specificity, the "desc" command does not work at all.
>
> Steps to reproduce:
>
> # ensure you have docker installed and running
> # run the following docker commands to create and start a Cassandra 3.11
> container
> ~ $ docker create --name cassandra3 cassandra:3.11.10
> 5d903e48e0661e39080198de5e8752356a5a666132211a500ea38af0fc2a0356
> ~ $ docker start cassandra3
> cassandra3
> ~ $ docker exec -ti cassandra3 bash
>
> # inside the docker container, try the default cqlsh version
> root@5d903e48e066:/# cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.11.10 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> desc keyspaces;
>
> system_traces  system_schema  system_auth  system  system_distributed
>
> cqlsh> use system_auth;
> cqlsh:system_auth> desc tables;
>
> resource_role_permissons_index  role_permissions  role_members  roles
>
> cqlsh:system_auth> select * from roles;
>
>  role  | can_login | is_superuser | member_of | salted_hash
>
> ---+---+--+---+--
>  cassandra |  True | True |  null |
> $2a$10$8UNyioBF41/OZfcCa2aqXOHvXiNXArBHKaUUhMyPAFKpfN8byXonm
>
> (1 rows)
> cqlsh:system_auth> exit
>
> # obviously, everything worked as expected
>
> # now install git to clone the Cassandra 4.0 RC1 source code, and
> python3-six, which is a dependency of cqlsh
> root@5d903e48e066:/# apt-get update -qq && apt-get install -qq git
> python3-six
> ..
>
> # clone the Cassandra repository and checkout the cassandra-4.0-rc1 tag
> root@5d903e48e066:/# git clone -b cassandra-4.0-rc1 --depth 1
> https://github.com/apache/cassandra.git cassandra-4.0-rc1
> Cloning into 'cassandra-4.0-rc1'...
> ..
>
> # run the cqlsh from the Git repository, and repeat all the cqlsh
> statements above
> root@5d903e48e066:/# cd cassandra-4.0-rc1/bin
> root@5d903e48e066:/cassandra-4.0-rc1/bin# ./cqlsh
> Connected to Test Cluster at 127.0.0.1:9042
> [cqlsh 6.0.0 | Cassandra 3.11.10 | CQL spec 3.4.4 | Native protocol v4]
> Use HELP for help.
> cqlsh> desc keyspaces;
>
> cqlsh> use system_auth;
> cqlsh:system_auth> desc tables;
>
> cqlsh:system_auth> select * from roles;
>
>  role  | can_login | is_superuser | member_of | salted_hash
>
> ---+---+------+---+--
>  cassandra |  True | True |  null |
> $2a$10$8UNyioBF41/OZfcCa2aqXOHvXiNXArBHKaUUhMyPAFKpfN8byXonm
>
> (1 rows)
> cqlsh:system_auth> exit
> root@5d903e48e066:/cassandra-4.0-rc1/bin#
>
> # "desc" did not work, but "use" and "select" worked
>
>
> My question is, are they suppose to be compatible? If so, this almost
> certainly is a bug. If not, I'd strongly recommend Cassandra to add a
> compatibility warning in the 4.0 version cqlsh when it's connected to a 3.x
> Cassandra server. Because I can foresee many users will run into this
> issue, especially when they are in the middle of upgrading from 3.x to 4.0.
>
>
> Regards,
>
> Bowen
>
>

Compatibility between Cassandra 3.11 and cqlsh from Cassandra 4.0 RC1

[Bowen Song]

Hi all,


I was using the cqlsh from Cassandra 4.0 RC1 and trying to connect to a 
Cassandra 3.11 cluster, and it does not appear to be working correctly. 
Specificity, the "desc" command does not work at all.


Steps to reproduce:

   # ensure you have docker installed and running
   # run the following docker commands to create and start a
   Cassandra 3.11 container
   ~ $ docker create --name cassandra3 cassandra:3.11.10
   5d903e48e0661e39080198de5e8752356a5a666132211a500ea38af0fc2a0356
   ~ $ docker start cassandra3
   cassandra3
   ~ $ docker exec -ti cassandra3 bash

   # inside the docker container, try the default cqlsh version
   root@5d903e48e066:/# cqlsh
   Connected to Test Cluster at 127.0.0.1:9042.
   [cqlsh 5.0.1 | Cassandra 3.11.10 | CQL spec 3.4.4 | Native
   protocol v4]
   Use HELP for help.
   cqlsh> desc keyspaces;

   system_traces  system_schema  system_auth  system system_distributed

   cqlsh> use system_auth;
   cqlsh:system_auth> desc tables;

   resource_role_permissons_index  role_permissions role_members  roles

   cqlsh:system_auth> select * from roles;

 role  | can_login | is_superuser | member_of | salted_hash
   
---+---+--+---+--
 cassandra |  True | True |  null |
   $2a$10$8UNyioBF41/OZfcCa2aqXOHvXiNXArBHKaUUhMyPAFKpfN8byXonm

   (1 rows)
   cqlsh:system_auth> exit

   # obviously, everything worked as expected

   # now install git to clone the Cassandra 4.0 RC1 source code,
   and python3-six, which is a dependency of cqlsh
   root@5d903e48e066:/# apt-get update -qq && apt-get install -qq
   git python3-six
   ..

   # clone the Cassandra repository and checkout the
   cassandra-4.0-rc1 tag
   root@5d903e48e066:/# git clone -b cassandra-4.0-rc1 --depth 1
   https://github.com/apache/cassandra.git cassandra-4.0-rc1
   Cloning into 'cassandra-4.0-rc1'...
   ..

   # run the cqlsh from the Git repository, and repeat all the
   cqlsh statements above
   root@5d903e48e066:/# cd cassandra-4.0-rc1/bin
       root@5d903e48e066:/cassandra-4.0-rc1/bin# ./cqlsh
   Connected to Test Cluster at 127.0.0.1:9042
   [cqlsh 6.0.0 | Cassandra 3.11.10 | CQL spec 3.4.4 | Native
   protocol v4]
   Use HELP for help.
   cqlsh> desc keyspaces;

   cqlsh> use system_auth;
   cqlsh:system_auth> desc tables;

   cqlsh:system_auth> select * from roles;

 role  | can_login | is_superuser | member_of | salted_hash
   
---+---+--+---+--
 cassandra |  True | True |  null |
   $2a$10$8UNyioBF41/OZfcCa2aqXOHvXiNXArBHKaUUhMyPAFKpfN8byXonm

   (1 rows)
   cqlsh:system_auth> exit
   root@5d903e48e066:/cassandra-4.0-rc1/bin#

   # "desc" did not work, but "use" and "select" worked


   My question is, are they suppose to be compatible? If so, this
   almost certainly is a bug. If not, I'd strongly recommend Cassandra
   to add a compatibility warning in the 4.0 version cqlsh when it's
   connected to a 3.x Cassandra server. Because I can foresee many
   users will run into this issue, especially when they are in the
   middle of upgrading from 3.x to 4.0.


   Regards,

   Bowen

Re: Cassandra 4.0 and python

[Paul Chandler]

Thanks Kane,

If anyone else is interested in this, I created a Jira ticket : 
https://issues.apache.org/jira/browse/CASSANDRA-16641 
 but the response is 
that 3.6 is the minimum official supported version, although 2.7 still should 
work. This is why the Debian packaging was changed. 

Thanks 

Paul  



> On 29 Apr 2021, at 00:27, Kane Wilson  wrote:
> 
> No, I suspect the deb package dependencies haven't been updated correctly, as 
> 2.7 should definitely still work. Could you raise a JIRA for this issue?
> 
> Not sure if apt has some way to force install/ignore dependencies, however if 
> you do that it may work, otherwise your only workaround would be to install 
> from the tarball.
> 
> raft.so  - Cassandra consulting, support, and managed 
> services
> 
> 
> On Thu, Apr 29, 2021 at 2:24 AM Paul Chandler  > wrote:
> Hi all,
> 
> We have been testing with 4.0~beta2 in our setup for a few weeks and all has 
> gone very smoothly, however when tried to install 4.0~rc1 we ran into 
> problems with python versions.
> 
> We are on Ubuntu 16.04.7 LTS so use apt to install Cassandra, and this now 
> gives the following error:
> 
> The following packages have unmet dependencies:
>  cassandra : Depends: python3 (>= 3.6) but 3.5.1-3 is to be installed
> E: Unable to correct problems, you have held broken packages.
> 
> Looking at the apt packaging the requirement for python has changed from 2.7 
> to 3.6 between beta4 and rc1. 
> 
> I have found https://issues.apache.org/jira/browse/CASSANDRA-16396 
>  which says it needed 
> to be python 3.6, however reading this ticket this seems to imply 2.7 is 
> still supported https://issues.apache.org/jira/browse/CASSANDRA-15659 
> 
> 
> Also the code for for cqlsh says it supports 2.7 as well:  
> https://github.com/apache/cassandra/blob/b0c50c10dbc443a05662b111a971a65cafa258d5/bin/cqlsh#L65
>  
> 
> 
> All our clusters are currently on Ubuntu 16.04 which does not come with 
> python 3.6, so this is going to be a major pain to upgrade them to 4.0.
> 
> Does the apt packaging really need to specify 3.6 ?
> 
> Thanks 
> 
> Paul Chandler

Re: Cassandra 4.0 and python

[Kane Wilson]

No, I suspect the deb package dependencies haven't been updated
correctly, as 2.7 should definitely still work. Could you raise a JIRA for
this issue?

Not sure if apt has some way to force install/ignore dependencies, however
if you do that it may work, otherwise your only workaround would be to
install from the tarball.

raft.so - Cassandra consulting, support, and managed services


On Thu, Apr 29, 2021 at 2:24 AM Paul Chandler  wrote:

> Hi all,
>
> We have been testing with 4.0~beta2 in our setup for a few weeks and all
> has gone very smoothly, however when tried to install 4.0~rc1 we ran into
> problems with python versions.
>
> We are on Ubuntu 16.04.7 LTS so use apt to install Cassandra, and this now
> gives the following error:
>
> The following packages have unmet dependencies:
>  cassandra : Depends: python3 (>= 3.6) but 3.5.1-3 is to be installed
> E: Unable to correct problems, you have held broken packages.
>
> Looking at the apt packaging the requirement for python has changed from
> 2.7 to 3.6 between beta4 and rc1.
>
> I have found https://issues.apache.org/jira/browse/CASSANDRA-16396 which
> says it needed to be python 3.6, however reading this ticket this seems to
> imply 2.7 is still supported
> https://issues.apache.org/jira/browse/CASSANDRA-15659
>
> Also the code for for cqlsh says it supports 2.7 as well:
> https://github.com/apache/cassandra/blob/b0c50c10dbc443a05662b111a971a65cafa258d5/bin/cqlsh#L65
>
> All our clusters are currently on Ubuntu 16.04 which does not come with
> python 3.6, so this is going to be a major pain to upgrade them to 4.0.
>
> Does the apt packaging really need to specify 3.6 ?
>
> Thanks
>
> Paul Chandler
>

Cassandra 4.0 and python

[Paul Chandler]

Hi all,

We have been testing with 4.0~beta2 in our setup for a few weeks and all has 
gone very smoothly, however when tried to install 4.0~rc1 we ran into problems 
with python versions.

We are on Ubuntu 16.04.7 LTS so use apt to install Cassandra, and this now 
gives the following error:

The following packages have unmet dependencies:
 cassandra : Depends: python3 (>= 3.6) but 3.5.1-3 is to be installed
E: Unable to correct problems, you have held broken packages.

Looking at the apt packaging the requirement for python has changed from 2.7 to 
3.6 between beta4 and rc1. 

I have found https://issues.apache.org/jira/browse/CASSANDRA-16396 
 which says it needed to 
be python 3.6, however reading this ticket this seems to imply 2.7 is still 
supported https://issues.apache.org/jira/browse/CASSANDRA-15659 


Also the code for for cqlsh says it supports 2.7 as well:  
https://github.com/apache/cassandra/blob/b0c50c10dbc443a05662b111a971a65cafa258d5/bin/cqlsh#L65
 


All our clusters are currently on Ubuntu 16.04 which does not come with python 
3.6, so this is going to be a major pain to upgrade them to 4.0.

Does the apt packaging really need to specify 3.6 ?

Thanks 

Paul Chandler

Re: [RELEASE] Apache Cassandra 4.0-rc1 released

[Joe Obernberger]

Can't wait to try it!

-joe

On 4/25/2021 11:44 AM, Patrick McFadin wrote:
This is pretty exciting and a huge milestone for the project. 
Congratulations to all the contributors who worked hard at making this 
the release it needed to be and honoring the database that powers the 
world.Â


Patrick

On Sun, Apr 25, 2021 at 4:10 AM Mick Semb Wever  wrote:

The Cassandra team is pleased to announce the release of Apache
Cassandra version 4.0-rc1.

Apache Cassandra is a fully distributed database. It is the right
choice when you need scalability and high availability without
compromising performance.

� http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our
download section:

� http://cassandra.apache.org/download/

This version is a release candidate[1] on the 4.0 series. As always,
please pay attention to the release notes[2] and Let us know[3] if you
were to encounter any problem.

Debian users shall note, as the docs are not yet updated, the bintray
location is now replaced with the ASF's JFrog Artifactory location:
� https://apache.jfrog.io/artifactory/cassandra/

Enjoy!

[1]: CHANGES.txt

https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc1
[2]: NEWS.txt

https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc1
[3]: https://issues.apache.org/jira/browse/CASSANDRA


<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=emailclient> 
	Virus-free. www.avg.com 
<http://www.avg.com/email-signature?utm_medium=email_source=link_campaign=sig-email_content=emailclient> 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: [RELEASE] Apache Cassandra 4.0-rc1 released

[Patrick McFadin]

This is pretty exciting and a huge milestone for the project.
Congratulations to all the contributors who worked hard at making this the
release it needed to be and honoring the database that powers the world.

Patrick

On Sun, Apr 25, 2021 at 4:10 AM Mick Semb Wever  wrote:

> The Cassandra team is pleased to announce the release of Apache
> Cassandra version 4.0-rc1.
>
> Apache Cassandra is a fully distributed database. It is the right
> choice when you need scalability and high availability without
> compromising performance.
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>
>  http://cassandra.apache.org/download/
>
> This version is a release candidate[1] on the 4.0 series. As always,
> please pay attention to the release notes[2] and Let us know[3] if you
> were to encounter any problem.
>
> Debian users shall note, as the docs are not yet updated, the bintray
> location is now replaced with the ASF's JFrog Artifactory location:
>  https://apache.jfrog.io/artifactory/cassandra/
>
> Enjoy!
>
> [1]: CHANGES.txt
>
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc1
> [2]: NEWS.txt
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc1
> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>

[RELEASE] Apache Cassandra 4.0-rc1 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache
Cassandra version 4.0-rc1.

Apache Cassandra is a fully distributed database. It is the right
choice when you need scalability and high availability without
compromising performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download section:

 http://cassandra.apache.org/download/

This version is a release candidate[1] on the 4.0 series. As always,
please pay attention to the release notes[2] and Let us know[3] if you
were to encounter any problem.

Debian users shall note, as the docs are not yet updated, the bintray
location is now replaced with the ASF's JFrog Artifactory location:
 https://apache.jfrog.io/artifactory/cassandra/

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-rc1
[2]: NEWS.txt 
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-rc1
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Center for Internet Security Benchmark for Cassandra 4.0

[Joseph Testa]

Is anyone here on the list interested in helping out in working on the next
version of the benchmark?

Would love some assistance and you can potentially get your name on the
document as an author :)

Feel free to reach out, we're always looking for new contributors, you can
check them out here:

https://www.cisecurity.org/

Thanks, Joe

Re: Cassandra 4.0 and changing DC setting

[Kane Wilson]

Great thanks for testing this Paul. You're likely the first one to actually
try it on 4.0. Albeit I still think we should have a flag to ALTER to do
this for you... not ideal to have users messing with system tables.

raft.so - Cassandra consulting, support, and managed services


On Mon, Feb 22, 2021 at 10:58 PM Paul Chandler  wrote:

> Yes, I am only running this on test clusters, I don’t run anything like
> this without lots of tests first.
>
> Anyway this worked well, so thanks for the info.
>
> For anyone else who needs this, the cql statement to do this was:
>
> insert into system_schema.keyspaces ( keyspace_name , durable_writes,
> replication ) values ( 'system_auth', true, {'OLD_DC': '3', ’NEW_DC': '3',
> 'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} );
>
> This will then allow you to log in after the nodes come back up, and the
> other keyspaces can be changed as normal afterwards.
>
> Thanks for you help.
>
> Paul
>
>
> On 21 Feb 2021, at 22:30, Kane Wilson  wrote:
>
> Make sure you test it on a practice cluster. Messing with the system
> tables is risky business!
>
> raft.so - Cassandra consulting, support, and managed services
>
>
> On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler  wrote:
>
>> Hi Kane,
>>
>> That sounds a good idea, I will give it a try on Monday.
>>
>> Thanks
>>
>> Paul
>>
>> On 21 Feb 2021, at 11:33, Kane Wilson  wrote:
>>
>> There has been proposals to add a force/unsafe flag to alter DC but it
>> hasn't been actioned and at this rate seems unlikely to make it into 4.0.
>> There is however a workaround, albeit not very user friendly. You should be
>> able to modify the system_schema tables directly to do your DC updates. I
>> am yet to test it out myself so don't know the updates you'd need to make
>> but you should be able to get a good idea by querying, doing updates and
>> observing the effect.
>>
>> raft.so - Cassandra consulting, support, managed services
>>
>> On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  wrote:
>>
>>> All,
>>>
>>> We have a use case where we need to change the datacenter name for a
>>> cassandra cluster, we have a script to do this that involves a short
>>> downtime. This does the following
>>>
>>>
>>> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’
>>> : ‘3’, ’NEW_DC”: ‘3’  }
>>> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each
>>> node
>>> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
>>> 4) Stop all nodes
>>> 5) Start each seed node, then start the rest of the nodes
>>> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”:
>>> ‘3’  }
>>>
>>> In 3.11.x this all works fine and the cluster is good to be used again
>>> after step 6.
>>>
>>> However in 4.0 step 1 is now blocked by the following change "Cassandra
>>> will no longer allow invalid keyspace replication options, such as invalid
>>> datacenter names for NetworkTopologyStrategy”
>>>
>>> If you skip step 1) then when the nodes come back up, you cannot login
>>> because the system_auth keyspace still has a replication factor of ‘OLD_DC’
>>> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be
>>> accessed.
>>>
>>> Is there a way around this to change the name of the datacenter?
>>>
>>>
>>> Thanks
>>>
>>> Paul
>>>
>>>
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>

Re: Cassandra 4.0 and changing DC setting

[Paul Chandler]

Yes, I am only running this on test clusters, I don’t run anything like this 
without lots of tests first.

Anyway this worked well, so thanks for the info.

For anyone else who needs this, the cql statement to do this was:

insert into system_schema.keyspaces ( keyspace_name , durable_writes, 
replication ) values ( 'system_auth', true, {'OLD_DC': '3', ’NEW_DC': '3', 
'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy'} );

This will then allow you to log in after the nodes come back up, and the other 
keyspaces can be changed as normal afterwards.

Thanks for you help.

Paul


> On 21 Feb 2021, at 22:30, Kane Wilson  wrote:
> 
> Make sure you test it on a practice cluster. Messing with the system tables 
> is risky business!
> 
> raft.so  - Cassandra consulting, support, and managed 
> services
> 
> 
> On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler  > wrote:
> Hi Kane,
> 
> That sounds a good idea, I will give it a try on Monday.
> 
> Thanks 
> 
> Paul
> 
>> On 21 Feb 2021, at 11:33, Kane Wilson mailto:k...@raft.so>> 
>> wrote:
>> 
>> There has been proposals to add a force/unsafe flag to alter DC but it 
>> hasn't been actioned and at this rate seems unlikely to make it into 4.0. 
>> There is however a workaround, albeit not very user friendly. You should be 
>> able to modify the system_schema tables directly to do your DC updates. I am 
>> yet to test it out myself so don't know the updates you'd need to make but 
>> you should be able to get a good idea by querying, doing updates and 
>> observing the effect.
>> 
>> raft.so - Cassandra consulting, support, managed services
>> 
>> On Sat., 20 Feb. 2021, 02:29 Paul Chandler, > > wrote:
>> All,
>> 
>> We have a use case where we need to change the datacenter name for a 
>> cassandra cluster, we have a script to do this that involves a short 
>> downtime. This does the following 
>> 
>> 
>> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ : 
>> ‘3’, ’NEW_DC”: ‘3’  }
>> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each 
>> node 
>> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
>> 4) Stop all nodes
>> 5) Start each seed node, then start the rest of the nodes
>> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”: ‘3’  }
>> 
>> In 3.11.x this all works fine and the cluster is good to be used again after 
>> step 6.
>> 
>> However in 4.0 step 1 is now blocked by the following change "Cassandra will 
>> no longer allow invalid keyspace replication options, such as invalid 
>> datacenter names for NetworkTopologyStrategy”
>> 
>> If you skip step 1) then when the nodes come back up, you cannot login 
>> because the system_auth keyspace still has a replication factor of ‘OLD_DC’ 
>> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be 
>> accessed.
>> 
>> Is there a way around this to change the name of the datacenter?
>> 
>> 
>> Thanks 
>> 
>> Paul
>> 
>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>> 
>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>> 
>> 
> 

Re: Cassandra 4.0 and changing DC setting

[Kane Wilson]

Make sure you test it on a practice cluster. Messing with the system tables
is risky business!

raft.so - Cassandra consulting, support, and managed services


On Sun, Feb 21, 2021 at 11:12 PM Paul Chandler  wrote:

> Hi Kane,
>
> That sounds a good idea, I will give it a try on Monday.
>
> Thanks
>
> Paul
>
> On 21 Feb 2021, at 11:33, Kane Wilson  wrote:
>
> There has been proposals to add a force/unsafe flag to alter DC but it
> hasn't been actioned and at this rate seems unlikely to make it into 4.0.
> There is however a workaround, albeit not very user friendly. You should be
> able to modify the system_schema tables directly to do your DC updates. I
> am yet to test it out myself so don't know the updates you'd need to make
> but you should be able to get a good idea by querying, doing updates and
> observing the effect.
>
> raft.so - Cassandra consulting, support, managed services
>
> On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  wrote:
>
>> All,
>>
>> We have a use case where we need to change the datacenter name for a
>> cassandra cluster, we have a script to do this that involves a short
>> downtime. This does the following
>>
>>
>> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ :
>> ‘3’, ’NEW_DC”: ‘3’  }
>> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each
>> node
>> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
>> 4) Stop all nodes
>> 5) Start each seed node, then start the rest of the nodes
>> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”:
>> ‘3’  }
>>
>> In 3.11.x this all works fine and the cluster is good to be used again
>> after step 6.
>>
>> However in 4.0 step 1 is now blocked by the following change "Cassandra
>> will no longer allow invalid keyspace replication options, such as invalid
>> datacenter names for NetworkTopologyStrategy”
>>
>> If you skip step 1) then when the nodes come back up, you cannot login
>> because the system_auth keyspace still has a replication factor of ‘OLD_DC’
>> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be
>> accessed.
>>
>> Is there a way around this to change the name of the datacenter?
>>
>>
>> Thanks
>>
>> Paul
>>
>>
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Cassandra 4.0 and changing DC setting

[Paul Chandler]

Hi Kane,

That sounds a good idea, I will give it a try on Monday.

Thanks 

Paul

> On 21 Feb 2021, at 11:33, Kane Wilson  wrote:
> 
> There has been proposals to add a force/unsafe flag to alter DC but it hasn't 
> been actioned and at this rate seems unlikely to make it into 4.0. There is 
> however a workaround, albeit not very user friendly. You should be able to 
> modify the system_schema tables directly to do your DC updates. I am yet to 
> test it out myself so don't know the updates you'd need to make but you 
> should be able to get a good idea by querying, doing updates and observing 
> the effect.
> 
> raft.so - Cassandra consulting, support, managed services
> 
> On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  > wrote:
> All,
> 
> We have a use case where we need to change the datacenter name for a 
> cassandra cluster, we have a script to do this that involves a short 
> downtime. This does the following 
> 
> 
> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ : 
> ‘3’, ’NEW_DC”: ‘3’  }
> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each node 
> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
> 4) Stop all nodes
> 5) Start each seed node, then start the rest of the nodes
> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”: ‘3’  }
> 
> In 3.11.x this all works fine and the cluster is good to be used again after 
> step 6.
> 
> However in 4.0 step 1 is now blocked by the following change "Cassandra will 
> no longer allow invalid keyspace replication options, such as invalid 
> datacenter names for NetworkTopologyStrategy”
> 
> If you skip step 1) then when the nodes come back up, you cannot login 
> because the system_auth keyspace still has a replication factor of ‘OLD_DC’ : 
> ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be 
> accessed.
> 
> Is there a way around this to change the name of the datacenter?
> 
> 
> Thanks 
> 
> Paul
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
> 
> For additional commands, e-mail: user-h...@cassandra.apache.org 
> 
> 

Re: Cassandra 4.0 and changing DC setting

[Kane Wilson]

There has been proposals to add a force/unsafe flag to alter DC but it
hasn't been actioned and at this rate seems unlikely to make it into 4.0.
There is however a workaround, albeit not very user friendly. You should be
able to modify the system_schema tables directly to do your DC updates. I
am yet to test it out myself so don't know the updates you'd need to make
but you should be able to get a good idea by querying, doing updates and
observing the effect.

raft.so - Cassandra consulting, support, managed services

On Sat., 20 Feb. 2021, 02:29 Paul Chandler,  wrote:

> All,
>
> We have a use case where we need to change the datacenter name for a
> cassandra cluster, we have a script to do this that involves a short
> downtime. This does the following
>
>
> 1) Change replication factor for the system key spaces to be { ‘OLD_DC’ :
> ‘3’, ’NEW_DC”: ‘3’  }
> 2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each
> node
> 3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
> 4) Stop all nodes
> 5) Start each seed node, then start the rest of the nodes
> 6) change the replication factor for all the keyspaces to {  ’NEW_DC”:
> ‘3’  }
>
> In 3.11.x this all works fine and the cluster is good to be used again
> after step 6.
>
> However in 4.0 step 1 is now blocked by the following change "Cassandra
> will no longer allow invalid keyspace replication options, such as invalid
> datacenter names for NetworkTopologyStrategy”
>
> If you skip step 1) then when the nodes come back up, you cannot login
> because the system_auth keyspace still has a replication factor of ‘OLD_DC’
> : ‘3’ but there are no nodes in the dc OLD_DC so the keyspace cannot be
> accessed.
>
> Is there a way around this to change the name of the datacenter?
>
>
> Thanks
>
> Paul
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Cassandra 4.0 and changing DC setting

[Paul Chandler]

All,

We have a use case where we need to change the datacenter name for a cassandra 
cluster, we have a script to do this that involves a short downtime. This does 
the following 


1) Change replication factor for the system key spaces to be { ‘OLD_DC’ : ‘3’, 
’NEW_DC”: ‘3’  }
2) Change the dc value in cassandra-rackdc.properties to NEW_DC for each node 
3) Add -Dcassandra.ignore_dc=true in cassandra-env.sh for each node
4) Stop all nodes
5) Start each seed node, then start the rest of the nodes
6) change the replication factor for all the keyspaces to {  ’NEW_DC”: ‘3’  }

In 3.11.x this all works fine and the cluster is good to be used again after 
step 6.

However in 4.0 step 1 is now blocked by the following change "Cassandra will no 
longer allow invalid keyspace replication options, such as invalid datacenter 
names for NetworkTopologyStrategy”

If you skip step 1) then when the nodes come back up, you cannot login because 
the system_auth keyspace still has a replication factor of ‘OLD_DC’ : ‘3’ but 
there are no nodes in the dc OLD_DC so the keyspace cannot be accessed.

Is there a way around this to change the name of the datacenter?


Thanks 

Paul




-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

[RELEASE] Apache Cassandra 4.0-beta4 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 4.0-beta4.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a beta release[1] on the 4.0 series. As always, please pay
attention to the release notes[2] and Let us know[3] if you were to
encounter any problem.

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-beta4
[2]: NEWS.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-beta4
[3]: https://issues.apache.org/jira/browse/CASSANDRA

[RELEASE] Apache Cassandra 4.0-beta3 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 4.0-beta3.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 4.0 series. As always, please
pay attention to the release notes[2] and Let us know[3] if you were to
encounter any problem.

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-beta3
[2]: NEWS.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-beta3
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: Cassandra 4.0-beta1 available on FreeBSD

[Lapo Luchini]

Angelo Polo wrote:

Cassandra 4.0-beta1 is now available on FreeBSD.


By the way I'm runinning a 6 nodes production cluster using 3.11.6 on 
FreeBSD 12.1/amd64 and I'm very happy about it (thanks Angelo for 
maintaining the FreeBSD Port!).


I hope your 4.0-beta2 patch will be accepted soon (and that 4.0 goes out 
of beta soon to) so that I'll be able to upgrade that cluster.


--
Lapo Luchini - http://lapo.it/


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

[RELEASE] Apache Cassandra 4.0-beta2 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 4.0-beta2.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 4.0 series. As always, please
pay attention to the release notes[2] and Let us know[3] if you were to
encounter any problem.

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-beta2
[2]: NEWS.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-beta2
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Cassandra 4.0-beta1 available on FreeBSD

[Angelo Polo]

Cassandra 4.0-beta1 is now available on FreeBSD.

You can find information about the port here:
https://www.freshports.org/databases/cassandra4/

The beta can be installed from an up-to-date ports tree under
databases/cassandra4.

Best,
Angelo

Re: [RELEASE] Apache Cassandra 4.0-beta1 released

[Mick Semb Wever]

> This version is a beta release[1] on the 4.0 series. As always, please
> pay attention to the release notes[2] and let us know[3] if you were
> to encounter any problem.



A quick followup note to both user and dev groups.

Our Beta release guidelines¹ states that there will be no further API
changes leading up to 4.0 GA.
But we do currently have in place three planned exceptions to this, found in
the following tickets:

- CASSANDRA-15299 –  "CASSANDRA-13304 follow-up: improve checksumming and
compression in protocol v5-beta"
- CASSANDRA-15234 – Standardise config and JVM parameters
- CASSANDRA-13701 – Lower default num_tokens


The API changes to these are minimal, and part of the reason these were
allowed to slip into the beta phase.
For example…
* CASSANDRA-15299 only affects those that are using the not yet stabilised
v5 native protocol.
* CASSANDRA-15234 will introduce cleaner, easier to use, cassandra.yaml
settings, but all existing yaml setting names will still work.
* CASSANDRA-13701 will change the default num_tokens setting to 16 (from
256), a change that will only impact provisioning of new clusters (because
existing clusters must configure any new nodes to use the existing
num_tokens value).

All three tickets have fixVersion still set to `4.0-alpha` because of this
situation.


References:
 [1] https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle

Re: [RELEASE] Apache Cassandra 4.0-beta1 released

[Michael Semb Wever]

> From NEWS.txt:
> >  Release builds require Java 11 + Java 8. Development builds can use Java
> 8 without 11.
> 
> Did it mean to read "Development builds can use *Java 11 without 8*."?


Cassandra can be built with either JDK 8 or JDK 11.
Our formal releases are built with JDK 8. 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [RELEASE] Apache Cassandra 4.0-beta1 released

[Oleksandr Shulgin]

On Tue, Jul 21, 2020 at 12:06 AM Mick Semb Wever  wrote:

> The Cassandra team is pleased to announce the release of Apache
> Cassandra version 4.0-beta1.
>
> Apache Cassandra is a fully distributed database. It is the right
> choice when you need scalability and high availability without
> compromising performance.
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>
>  http://cassandra.apache.org/download/
>
> This version is a beta release[1] on the 4.0 series. As always, please
> pay attention to the release notes[2] and let us know[3] if you were
> to encounter any problem.
>
> Enjoy!
>  And check out our blog post on Cassandra 4.0 beta1
>
> https://cassandra.apache.org/blog/2020/07/20/apache-cassandra-4-0-beta1.html
>

Congratulations on moving to the beta phase!

[1]: CHANGES.txt
>
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-beta1
> [2]: NEWS.txt
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-beta1


>From NEWS.txt:
>  Release builds require Java 11 + Java 8. Development builds can use Java
8 without 11.

Did it mean to read "Development builds can use *Java 11 without 8*."?

Cheers,
--
Alex

[RELEASE] Apache Cassandra 4.0-beta1 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache
Cassandra version 4.0-beta1.

Apache Cassandra is a fully distributed database. It is the right
choice when you need scalability and high availability without
compromising performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download section:

 http://cassandra.apache.org/download/

This version is a beta release[1] on the 4.0 series. As always, please
pay attention to the release notes[2] and let us know[3] if you were
to encounter any problem.

Enjoy!
 And check out our blog post on Cassandra 4.0 beta1
https://cassandra.apache.org/blog/2020/07/20/apache-cassandra-4-0-beta1.html


[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-beta1
[2]: NEWS.txt 
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-beta1
[3]: https://issues.apache.org/jira/browse/CASSANDRA

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

[RELEASE] Apache Cassandra 4.0-alpha4 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 4.0-alpha4.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.
 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:
 http://cassandra.apache.org/download/


Downloads of source and binary distributions:
http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha4/apache-cassandra-4.0-alpha4-bin.tar.gz
http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha4/apache-cassandra-4.0-alpha4-src.tar.gz

Debian and Redhat configurations.

  sources.list:
  deb http://www.apache.org/dist/cassandra/debian 40x main

  yum config:
  baseurl=https://www.apache.org/dist/cassandra/redhat/40x/

See http://cassandra.apache.org/download/ for full install instructions.

This is an ALPHA version! It is not intended for production use, however
the project would appreciate your testing and feedback to make the final
release better. As always, please pay attention to the release notes[2] and
let us know[3] if you encounter any problems.

Enjoy!

[1]: CHANGES.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-alpha4
[2]: NEWS.txt
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha4
[3]: https://issues.apache.org/jira/browse/CASSANDRA

Re: [RELEASE] Apache Cassandra 4.0-alpha3 released

[Erick Ramirez]

Congratulations! 

For those who may not be familiar with the behind-the-scenes, this is a
major milestone for the project and another step closer to the release of
Apache Cassandra 4.0. I'm so excited about this news and you should be
too! 

Re: [RELEASE] Apache Cassandra 4.0-alpha3 released

[Jon Haddad]

Thanks for handling this, Mick!

On Fri, Feb 7, 2020 at 12:02 PM Mick Semb Wever  wrote:

>
>
> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 4.0-alpha3.
>
> Apache Cassandra is a fully distributed database. It is the right choice
> when you need scalability and high availability without compromising
> performance.
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions are listed in our download
> section:
>  http://cassandra.apache.org/download/
>
>
> Downloads of source and binary distributions:
>
> http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha3/apache-cassandra-4.0-alpha3-bin.tar.gz
>
> http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha3/apache-cassandra-4.0-alpha3-src.tar.gz
>
> Debian and Redhat configurations.
>
>   sources.list:
>   deb http://www.apache.org/dist/cassandra/debian 40x main
>
>   yum config:
>   baseurl=https://www.apache.org/dist/cassandra/redhat/40x/
>
> See http://cassandra.apache.org/download/ for full install instructions.
>
> This is an ALPHA version! It is not intended for production use, however
> the project would appreciate your testing and feedback to make the final
> release better. As always, please pay attention to the release notes[2]
> and let us know[3] if you encounter any problems.
>
> Enjoy!
>
> [1]: CHANGES.txt
> ?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-alpha3
> [2]: NEWS.txt
> ?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha3
> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

[RELEASE] Apache Cassandra 4.0-alpha3 released

[Mick Semb Wever]

The Cassandra team is pleased to announce the release of Apache Cassandra 
version 4.0-alpha3.

Apache Cassandra is a fully distributed database. It is the right choice when 
you need scalability and high availability without compromising performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download section:
 http://cassandra.apache.org/download/


Downloads of source and binary distributions:
http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha3/apache-cassandra-4.0-alpha3-bin.tar.gz
http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha3/apache-cassandra-4.0-alpha3-src.tar.gz

Debian and Redhat configurations.

  sources.list:
  deb http://www.apache.org/dist/cassandra/debian 40x main

  yum config:
  baseurl=https://www.apache.org/dist/cassandra/redhat/40x/

See http://cassandra.apache.org/download/ for full install instructions.

This is an ALPHA version! It is not intended for production use, however
the project would appreciate your testing and feedback to make the final
release better. As always, please pay attention to the release notes[2]
and let us know[3] if you encounter any problems.

Enjoy!

[1]: CHANGES.txt 
?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-alpha3
[2]: NEWS.txt 
?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha3
[3]: https://issues.apache.org/jira/browse/CASSANDRA

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [Discuss] num_tokens default in Cassandra 4.0

[Carl Mueller]

Your case seems to argue for completely eliminating vnodes. Which the Priam
people have been preaching for a long time.

There is not, certainly to a cassandra user-level person, good
documentation on the pros and cons of vnodes vs single tokens, and as we
see here the impacts of various vnode counts isn't an obvious/trivial
concept in a database system already loaded with nontrivial concepts.

I've never seen a good honest discussion on why vnodes made their way into
cassandra as the default, and 256 as the node count. They immediately broke
secondary indexes, and as max says they muddle the data distribution for
resiliency and scaling more than one node at once.

The only advantage seems to be "no manual token management" which to me was
just laziness at the tooling/nodetool level, and better streaming impacts
on node standup, although you seem completely limited to single node at a
time scaling which is a huge restriction.

Also, there is an inability to change the node count, which seems really
strange given than vnodes are supposed to be able to address heterogenous
node hardware and subsegment the data to a finer grain. I get that changing
node count would be a "hard problem", but the current solution is basically
"spin up a new datacenter with a new node count" which is a sucky solution.

RE: the racks point by Jeremiah, we do have rack alignment, and although I
understand theoretically that I should be able to do things with rack
aligned quorum safety (And did in some extreme instances of an LCS --> STCS
--> LCS local recompaction to force tombstone purges that were in
inaccessible sections of the LCS tree), the current warnings about scaling
simultaneously and lack of discussion on how we can use racks in the case
of vnodes to do so, and given all the tickets about problems with multiple
node scaling, we're kind of stuck.

I get that the ideal case of cassandra is gradually growing data with
balanced load growth.

But for more chaotic loads, which things like IoT fleets coming online at
once and misbehaving networks of IoT devices, it would be really nice to
increase our load scaling abilities. We are kind of stuck with vertical
node scaling, which has rapidly diminishing returns, and spinning up nodes
one at a time.

Vnode count seems to impact all of this in some ways, and in opaque ways.

Anyway, I'm fine with 16, agree that token selection should be improved,
but think a priority for adding the ability to change node counts online
should be explored, even if it involves slowly picking off 1 vnode at a
time from one machine. VNode evolution would be very rare, more rare than
version upgrades.

On Mon, Feb 3, 2020 at 11:07 PM Max C.  wrote:

> Let’s say you have a 6 node cluster, with RF=3, and no vnodes.  In that
> case each piece of data is stored as follows:
>
> : 
> *N1: N2 N3*
> N2: N3 N4
> N3: N4 N5
> *N4: N5 N6*
> N5: N6 N1
> N6: N1 N2
>
> With this setup, there are some circumstances where you could lose 2 nodes
> (ex: N1 & N4) and still be able to maintain CL=quorum.  If your cluster is
> very large, then you could lose even more — and that’s a good thing,
> because if you have hundreds/thousands of nodes then you don’t want the
> world to come tumbling down if  > 1 node is down.  Or maybe you want to
> upgrade the OS on your nodes, and want to (with very careful planning!) do
> it by taking down more than 1 node at a time.
>
> … but if you have a large number of vnodes, then a given node will share a
> small segment of data with LOTS of other nodes, which destroys this
> property.  The more vnodes, the less likely you’re able to handle > 1 node
> down.
>
> For example, see this diagram in the Datastax docs —
>
>
> https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbArch/archDataDistributeVnodesUsing.html#Distributingdatausingvnodes
>
> In that bottom picture, you can’t knock out 2 nodes and still maintain
> CL=quorum.  Ex:  If you knock out node 1 & 4, then ranges B & L would no
> longer meet CL=quorum;  but you can do that in the top diagram, since there
> are no ranges shared between node 1 & 4.
>
> Hope that helps.
>
> - Max
>
>
> On Feb 3, 2020, at 8:39 pm, onmstester onmstester <
> onmstes...@zoho.com.INVALID> wrote:
>
> Sorry if its trivial, but i do not understand how num_tokens affects
> availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost
> at most one node and all of the tokens assigned to that node would be also
> assigned to two other nodes no matter what num_tokens is, right?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>  Forwarded message 
> From: Jon Haddad 
> To: 
> Date: Tue, 04 Feb 2020 01:15:21 +0330
> Subject: Re: [Discuss] num_tokens default in Cassandra

Re: [Discuss] num_tokens default in Cassandra 4.0

[Jeremiah D Jordan]

JustFYI if being able to operationally do things many nodes at a time, you 
should look at setting up racks.  With num racks = RF you can take down all 
nodes in a given rack at once without affecting LOCAL_QUORUM.  Your single 
token example has the same functionality in this respect as a vnodes cluster 
using racks (and actually if you setup a single token cluster using racks you 
would have setup nodes N1 and N4 to be in the same rack).

> On Feb 3, 2020, at 11:07 PM, Max C.  wrote:
> 
> Let’s say you have a 6 node cluster, with RF=3, and no vnodes.  In that case 
> each piece of data is stored as follows:
> 
> : 
> N1: N2 N3
> N2: N3 N4
> N3: N4 N5
> N4: N5 N6
> N5: N6 N1
> N6: N1 N2
> 
> With this setup, there are some circumstances where you could lose 2 nodes 
> (ex: N1 & N4) and still be able to maintain CL=quorum.  If your cluster is 
> very large, then you could lose even more — and that’s a good thing, because 
> if you have hundreds/thousands of nodes then you don’t want the world to come 
> tumbling down if  > 1 node is down.  Or maybe you want to upgrade the OS on 
> your nodes, and want to (with very careful planning!) do it by taking down 
> more than 1 node at a time.
> 
> … but if you have a large number of vnodes, then a given node will share a 
> small segment of data with LOTS of other nodes, which destroys this property. 
>  The more vnodes, the less likely you’re able to handle > 1 node down.
> 
> For example, see this diagram in the Datastax docs —
> 
> https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbArch/archDataDistributeVnodesUsing.html#Distributingdatausingvnodes
>  
> <https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbArch/archDataDistributeVnodesUsing.html#Distributingdatausingvnodes>
> 
> In that bottom picture, you can’t knock out 2 nodes and still maintain 
> CL=quorum.  Ex:  If you knock out node 1 & 4, then ranges B & L would no 
> longer meet CL=quorum;  but you can do that in the top diagram, since there 
> are no ranges shared between node 1 & 4.
> 
> Hope that helps.
> 
> - Max
> 
> 
>> On Feb 3, 2020, at 8:39 pm, onmstester onmstester 
>> mailto:onmstes...@zoho.com.INVALID>> wrote:
>> 
>> Sorry if its trivial, but i do not understand how num_tokens affects 
>> availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost 
>> at most one node and all of the tokens assigned to that node would be also 
>> assigned to two other nodes no matter what num_tokens is, right?
>> 
>> Sent using Zoho Mail <https://www.zoho.com/mail/>
>> 
>> 
>> ==== Forwarded message 
>> From: Jon Haddad mailto:j...@jonhaddad.com>>
>> To: mailto:d...@cassandra.apache.org>>
>> Date: Tue, 04 Feb 2020 01:15:21 +0330
>> Subject: Re: [Discuss] num_tokens default in Cassandra 4.0
>>  Forwarded message 
>> 
>> I think it's a good idea to take a step back and get a high level view of 
>> the problem we're trying to solve. 
>> 
>> First, high token counts result in decreased availability as each node has 
>> data overlap with with more nodes in the cluster. Specifically, a node can 
>> share data with RF-1 * 2 * num_tokens. So a 256 token cluster at RF=3 is 
>> going to almost always share data with every other node in the cluster that 
>> isn't in the same rack, unless you're doing something wild like using more 
>> than a thousand nodes in a cluster. We advertise 
>> 
>> With 16 tokens, that is vastly improved, but you still have up to 64 nodes 
>> each node needs to query against, so you're again, hitting every node 
>> unless you go above ~96 nodes in the cluster (assuming 3 racks / AZs). I 
>> wouldn't use 16 here, and I doubt any of you would either. I've advocated 
>> for 4 tokens because you'd have overlap with only 16 nodes, which works 
>> well for small clusters as well as large. Assuming I was creating a new 
>> cluster for myself (in a hypothetical brand new application I'm building) I 
>> would put this in production. I have worked with several teams where I 
>> helped them put 4 token clusters in prod and it has worked very well. We 
>> didn't see any wild imbalance issues. 
>> 
>> As Mick's pointed out, our current method of using random token assignment 
>> for the default number of problematic for 4 tokens. I fully agree with 
>> this, and I think if we were to try to use 4 tokens, we'd want to address 
>> this in tandem. We can discuss how to better allocate tokens by default 
>> (something more predictable than random), but I'd like to avoid the 
>&g

Fwd: Re: [Discuss] num_tokens default in Cassandra 4.0

[onmstester onmstester]

Thank you so much



Sent using https://www.zoho.com/mail/






 Forwarded message 
From: Max C. 
To: 
Date: Tue, 04 Feb 2020 08:37:21 +0330
Subject: Re: [Discuss] num_tokens default in Cassandra 4.0
 Forwarded message 



Let’s say you have a 6 node cluster, with RF=3, and no vnodes.  In that case 
each piece of data is stored as follows:



: 

N1: N2 N3

N2: N3 N4

N3: N4 N5

N4: N5 N6

N5: N6 N1

N6: N1 N2



With this setup, there are some circumstances where you could lose 2 nodes (ex: 
N1 & N4) and still be able to maintain CL=quorum.  If your cluster is very 
large, then you could lose even more — and that’s a good thing, because if you 
have hundreds/thousands of nodes then you don’t want the world to come tumbling 
down if  > 1 node is down.  Or maybe you want to upgrade the OS on your nodes, 
and want to (with very careful planning!) do it by taking down more than 1 node 
at a time.



… but if you have a large number of vnodes, then a given node will share a 
small segment of data with LOTS of other nodes, which destroys this property.  
The more vnodes, the less likely you’re able to handle > 1 node down.



For example, see this diagram in the Datastax docs —



https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbArch/archDataDistributeVnodesUsing.html#Distributingdatausingvnodes



In that bottom picture, you can’t knock out 2 nodes and still maintain 
CL=quorum.  Ex:  If you knock out node 1 & 4, then ranges B & L would no longer 
meet CL=quorum;  but you can do that in the top diagram, since there are no 
ranges shared between node 1 & 4.



Hope that helps.



- Max





On Feb 3, 2020, at 8:39 pm, onmstester onmstester 
<mailto:onmstes...@zoho.com.INVALID> wrote:



Sorry if its trivial, but i do not understand how num_tokens affects 
availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost at 
most one node and all of the tokens assigned to that node would be also 
assigned to two other nodes no matter what num_tokens is, right?



Sent using https://www.zoho.com/mail/






 Forwarded message 
From: Jon Haddad <mailto:j...@jonhaddad.com>
To: <mailto:d...@cassandra.apache.org>
Date: Tue, 04 Feb 2020 01:15:21 +0330
Subject: Re: [Discuss] num_tokens default in Cassandra 4.0
 Forwarded message 



I think it's a good idea to take a step back and get a high level view of 
the problem we're trying to solve. 
 
First, high token counts result in decreased availability as each node has 
data overlap with with more nodes in the cluster.  Specifically, a node can 
share data with RF-1 * 2 * num_tokens.  So a 256 token cluster at RF=3 is 
going to almost always share data with every other node in the cluster that 
isn't in the same rack, unless you're doing something wild like using more 
than a thousand nodes in a cluster.  We advertise 
 
With 16 tokens, that is vastly improved, but you still have up to 64 nodes 
each node needs to query against, so you're again, hitting every node 
unless you go above ~96 nodes in the cluster (assuming 3 racks / AZs).  I 
wouldn't use 16 here, and I doubt any of you would either.  I've advocated 
for 4 tokens because you'd have overlap with only 16 nodes, which works 
well for small clusters as well as large.  Assuming I was creating a new 
cluster for myself (in a hypothetical brand new application I'm building) I 
would put this in production.  I have worked with several teams where I 
helped them put 4 token clusters in prod and it has worked very well.  We 
didn't see any wild imbalance issues. 
 
As Mick's pointed out, our current method of using random token assignment 
for the default number of problematic for 4 tokens.  I fully agree with 
this, and I think if we were to try to use 4 tokens, we'd want to address 
this in tandem.  We can discuss how to better allocate tokens by default 
(something more predictable than random), but I'd like to avoid the 
specifics of that for the sake of this email. 
 
To Alex's point, repairs are problematic with lower token counts due to 
over streaming.  I think this is a pretty serious issue and I we'd have to 
address it before going all the way down to 4.  This, in my opinion, is a 
more complex problem to solve and I think trying to fix it here could make 
shipping 4.0 take even longer, something none of us want. 
 
For the sake of shipping 4.0 without adding extra overhead and time, I'm ok 
with moving to 16 tokens, and in the process adding extensive documentation 
outlining what we recommend for production use.  I think we should also try 
to figure out something better than random as the default to fix the data 
imbalance issues.  I've got a few ideas here I've been noodling on. 
 
As long as folks are fine with potentially changing the default again in C* 
5.0 (after another discussion / debate), 16 is enough of an improvement 
that I'm O

Re: [Discuss] num_tokens default in Cassandra 4.0

[Max C.]

Let’s say you have a 6 node cluster, with RF=3, and no vnodes.  In that case 
each piece of data is stored as follows:

: 
N1: N2 N3
N2: N3 N4
N3: N4 N5
N4: N5 N6
N5: N6 N1
N6: N1 N2

With this setup, there are some circumstances where you could lose 2 nodes (ex: 
N1 & N4) and still be able to maintain CL=quorum.  If your cluster is very 
large, then you could lose even more — and that’s a good thing, because if you 
have hundreds/thousands of nodes then you don’t want the world to come tumbling 
down if  > 1 node is down.  Or maybe you want to upgrade the OS on your nodes, 
and want to (with very careful planning!) do it by taking down more than 1 node 
at a time.

… but if you have a large number of vnodes, then a given node will share a 
small segment of data with LOTS of other nodes, which destroys this property.  
The more vnodes, the less likely you’re able to handle > 1 node down.

For example, see this diagram in the Datastax docs —

https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbArch/archDataDistributeVnodesUsing.html#Distributingdatausingvnodes
 
<https://docs.datastax.com/en/dse/5.1/dse-arch/datastax_enterprise/dbArch/archDataDistributeVnodesUsing.html#Distributingdatausingvnodes>

In that bottom picture, you can’t knock out 2 nodes and still maintain 
CL=quorum.  Ex:  If you knock out node 1 & 4, then ranges B & L would no longer 
meet CL=quorum;  but you can do that in the top diagram, since there are no 
ranges shared between node 1 & 4.

Hope that helps.

- Max


> On Feb 3, 2020, at 8:39 pm, onmstester onmstester 
>  wrote:
> 
> Sorry if its trivial, but i do not understand how num_tokens affects 
> availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost 
> at most one node and all of the tokens assigned to that node would be also 
> assigned to two other nodes no matter what num_tokens is, right?
> 
> Sent using Zoho Mail <https://www.zoho.com/mail/>
> 
> 
>  Forwarded message 
> From: Jon Haddad mailto:j...@jonhaddad.com>>
> To: mailto:d...@cassandra.apache.org>>
> Date: Tue, 04 Feb 2020 01:15:21 +0330
> Subject: Re: [Discuss] num_tokens default in Cassandra 4.0
>  Forwarded message 
> 
> I think it's a good idea to take a step back and get a high level view of 
> the problem we're trying to solve. 
> 
> First, high token counts result in decreased availability as each node has 
> data overlap with with more nodes in the cluster. Specifically, a node can 
> share data with RF-1 * 2 * num_tokens. So a 256 token cluster at RF=3 is 
> going to almost always share data with every other node in the cluster that 
> isn't in the same rack, unless you're doing something wild like using more 
> than a thousand nodes in a cluster. We advertise 
> 
> With 16 tokens, that is vastly improved, but you still have up to 64 nodes 
> each node needs to query against, so you're again, hitting every node 
> unless you go above ~96 nodes in the cluster (assuming 3 racks / AZs). I 
> wouldn't use 16 here, and I doubt any of you would either. I've advocated 
> for 4 tokens because you'd have overlap with only 16 nodes, which works 
> well for small clusters as well as large. Assuming I was creating a new 
> cluster for myself (in a hypothetical brand new application I'm building) I 
> would put this in production. I have worked with several teams where I 
> helped them put 4 token clusters in prod and it has worked very well. We 
> didn't see any wild imbalance issues. 
> 
> As Mick's pointed out, our current method of using random token assignment 
> for the default number of problematic for 4 tokens. I fully agree with 
> this, and I think if we were to try to use 4 tokens, we'd want to address 
> this in tandem. We can discuss how to better allocate tokens by default 
> (something more predictable than random), but I'd like to avoid the 
> specifics of that for the sake of this email. 
> 
> To Alex's point, repairs are problematic with lower token counts due to 
> over streaming. I think this is a pretty serious issue and I we'd have to 
> address it before going all the way down to 4. This, in my opinion, is a 
> more complex problem to solve and I think trying to fix it here could make 
> shipping 4.0 take even longer, something none of us want. 
> 
> For the sake of shipping 4.0 without adding extra overhead and time, I'm ok 
> with moving to 16 tokens, and in the process adding extensive documentation 
> outlining what we recommend for production use. I think we should also try 
> to figure out something better than random as the default to fix the data 
> imbalance issues. I've got a few ideas here I've been noodling on. 
> 
> As long as folks are fine with potentially changing the default again 

Re: Re: [Discuss] num_tokens default in Cassandra 4.0

[Jeff Jirsa]

The more vnodes you have on each host, the more likely it becomes that any
2 hosts are adjacent/neighbors/replicas.


On Mon, Feb 3, 2020 at 8:39 PM onmstester onmstester
 wrote:

> Sorry if its trivial, but i do not understand how num_tokens affects
> availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost
> at most one node and all of the tokens assigned to that node would be also
> assigned to two other nodes no matter what num_tokens is, right?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>  Forwarded message 
> From: Jon Haddad 
> To: 
> Date: Tue, 04 Feb 2020 01:15:21 +0330
> Subject: Re: [Discuss] num_tokens default in Cassandra 4.0
>  Forwarded message 
>
> I think it's a good idea to take a step back and get a high level view of
> the problem we're trying to solve.
>
> First, high token counts result in decreased availability as each node has
> data overlap with with more nodes in the cluster. Specifically, a node can
> share data with RF-1 * 2 * num_tokens. So a 256 token cluster at RF=3 is
> going to almost always share data with every other node in the cluster
> that
> isn't in the same rack, unless you're doing something wild like using more
> than a thousand nodes in a cluster. We advertise
>
> With 16 tokens, that is vastly improved, but you still have up to 64 nodes
> each node needs to query against, so you're again, hitting every node
> unless you go above ~96 nodes in the cluster (assuming 3 racks / AZs). I
> wouldn't use 16 here, and I doubt any of you would either. I've advocated
> for 4 tokens because you'd have overlap with only 16 nodes, which works
> well for small clusters as well as large. Assuming I was creating a new
> cluster for myself (in a hypothetical brand new application I'm building)
> I
> would put this in production. I have worked with several teams where I
> helped them put 4 token clusters in prod and it has worked very well. We
> didn't see any wild imbalance issues.
>
> As Mick's pointed out, our current method of using random token assignment
> for the default number of problematic for 4 tokens. I fully agree with
> this, and I think if we were to try to use 4 tokens, we'd want to address
> this in tandem. We can discuss how to better allocate tokens by default
> (something more predictable than random), but I'd like to avoid the
> specifics of that for the sake of this email.
>
> To Alex's point, repairs are problematic with lower token counts due to
> over streaming. I think this is a pretty serious issue and I we'd have to
> address it before going all the way down to 4. This, in my opinion, is a
> more complex problem to solve and I think trying to fix it here could make
> shipping 4.0 take even longer, something none of us want.
>
> For the sake of shipping 4.0 without adding extra overhead and time, I'm
> ok
> with moving to 16 tokens, and in the process adding extensive
> documentation
> outlining what we recommend for production use. I think we should also try
> to figure out something better than random as the default to fix the data
> imbalance issues. I've got a few ideas here I've been noodling on.
>
> As long as folks are fine with potentially changing the default again in
> C*
> 5.0 (after another discussion / debate), 16 is enough of an improvement
> that I'm OK with the change, and willing to author the docs to help people
> set up their first cluster. For folks that go into production with the
> defaults, we're at least not setting them up for total failure once their
> clusters get large like we are now.
>
> In future versions, we'll probably want to address the issue of data
> imbalance by building something in that shifts individual tokens around. I
> don't think we should try to do this in 4.0 either.
>
> Jon
>
>
>
> On Fri, Jan 31, 2020 at 2:04 PM Jeremy Hanna 
> wrote:
>
> > I think Mick and Anthony make some valid operational and skew points for
> > smaller/starting clusters with 4 num_tokens. There’s an arbitrary line
> > between small and large clusters but I think most would agree that most
> > clusters are on the small to medium side. (A small nuance is afaict the
> > probabilities have to do with quorum on a full token range, ie it has to
> do
> > with the size of a datacenter not the full cluster
> >
> > As I read this discussion I’m personally more inclined to go with 16 for
> > now. It’s true that if we could fix the skew and topology gotchas for
> those
> > starting things up, 4 would be ideal from an availability perspective.
> > However we’re still in the brainstorming stage for how to address those
> > challenges. I think we should

Fwd: Re: [Discuss] num_tokens default in Cassandra 4.0

[onmstester onmstester]

Sorry if its trivial, but i do not understand how num_tokens affects 
availability, with RF=3, CLW,CLR=quorum, the cluster could tolerate to lost at 
most one node and all of the tokens assigned to that node would be also 
assigned to two other nodes no matter what num_tokens is, right?


Sent using https://www.zoho.com/mail/




 Forwarded message 
From: Jon Haddad <mailto:j...@jonhaddad.com>
To: <mailto:d...@cassandra.apache.org>
Date: Tue, 04 Feb 2020 01:15:21 +0330
Subject: Re: [Discuss] num_tokens default in Cassandra 4.0
 Forwarded message 


I think it's a good idea to take a step back and get a high level view of 
the problem we're trying to solve. 
 
First, high token counts result in decreased availability as each node has 
data overlap with with more nodes in the cluster.  Specifically, a node can 
share data with RF-1 * 2 * num_tokens.  So a 256 token cluster at RF=3 is 
going to almost always share data with every other node in the cluster that 
isn't in the same rack, unless you're doing something wild like using more 
than a thousand nodes in a cluster.  We advertise 
 
With 16 tokens, that is vastly improved, but you still have up to 64 nodes 
each node needs to query against, so you're again, hitting every node 
unless you go above ~96 nodes in the cluster (assuming 3 racks / AZs).  I 
wouldn't use 16 here, and I doubt any of you would either.  I've advocated 
for 4 tokens because you'd have overlap with only 16 nodes, which works 
well for small clusters as well as large.  Assuming I was creating a new 
cluster for myself (in a hypothetical brand new application I'm building) I 
would put this in production.  I have worked with several teams where I 
helped them put 4 token clusters in prod and it has worked very well.  We 
didn't see any wild imbalance issues. 
 
As Mick's pointed out, our current method of using random token assignment 
for the default number of problematic for 4 tokens.  I fully agree with 
this, and I think if we were to try to use 4 tokens, we'd want to address 
this in tandem.  We can discuss how to better allocate tokens by default 
(something more predictable than random), but I'd like to avoid the 
specifics of that for the sake of this email. 
 
To Alex's point, repairs are problematic with lower token counts due to 
over streaming.  I think this is a pretty serious issue and I we'd have to 
address it before going all the way down to 4.  This, in my opinion, is a 
more complex problem to solve and I think trying to fix it here could make 
shipping 4.0 take even longer, something none of us want. 
 
For the sake of shipping 4.0 without adding extra overhead and time, I'm ok 
with moving to 16 tokens, and in the process adding extensive documentation 
outlining what we recommend for production use.  I think we should also try 
to figure out something better than random as the default to fix the data 
imbalance issues.  I've got a few ideas here I've been noodling on. 
 
As long as folks are fine with potentially changing the default again in C* 
5.0 (after another discussion / debate), 16 is enough of an improvement 
that I'm OK with the change, and willing to author the docs to help people 
set up their first cluster.  For folks that go into production with the 
defaults, we're at least not setting them up for total failure once their 
clusters get large like we are now. 
 
In future versions, we'll probably want to address the issue of data 
imbalance by building something in that shifts individual tokens around.  I 
don't think we should try to do this in 4.0 either. 
 
Jon 
 
 
 
On Fri, Jan 31, 2020 at 2:04 PM Jeremy Hanna 
<mailto:jeremy.hanna1...@gmail.com> 
wrote: 
 
> I think Mick and Anthony make some valid operational and skew points for 
> smaller/starting clusters with 4 num_tokens. There’s an arbitrary line 
> between small and large clusters but I think most would agree that most 
> clusters are on the small to medium side. (A small nuance is afaict the 
> probabilities have to do with quorum on a full token range, ie it has to do 
> with the size of a datacenter not the full cluster 
> 
> As I read this discussion I’m personally more inclined to go with 16 for 
> now. It’s true that if we could fix the skew and topology gotchas for those 
> starting things up, 4 would be ideal from an availability perspective. 
> However we’re still in the brainstorming stage for how to address those 
> challenges. I think we should create tickets for those issues and go with 
> 16 for 4.0. 
> 
> This is about an out of the box experience. It balances availability, 
> operations (such as skew and general bootstrap friendliness and 
> streaming/repair), and cluster sizing. Balancing all of those, I think for 
> now I’m more comfortable with 16 as the default with docs on considerations 
> and tickets to unblock 4 as the default for all users

Testing Cassandra 4.0 Pre-releases on Windows

[Jordan West]

Hi Cassandra Users,

The Cassandra dev community is looking for users of Cassandra running on
windows to test compatibility of the 4.0 alpha / beta / rc releases.

If you run on windows and have time / resources to test, please report any
compatibility issues you find to the dev mailing list or JIRA (
https://issues.apache.org/jira/projects/CASSANDRA/summary)

Thanks!
Jordan

[RELEASE] Apache Cassandra 4.0-alpha2 released

[Michael Shuler]

The Cassandra team is pleased to announce the release of Apache 
Cassandra version 4.0-alpha2.


Apache Cassandra is a fully distributed database. It is the right choice 
when you need scalability and high availability without compromising 
performance.


 http://cassandra.apache.org/

Downloads of source and binary distributions:

http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha2/apache-cassandra-4.0-alpha2-bin.tar.gz

http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha2/apache-cassandra-4.0-alpha2-src.tar.gz

Debian and Redhat configurations

 sources.list:
 deb http://www.apache.org/dist/cassandra/debian 40x main

 yum config:
 baseurl=https://www.apache.org/dist/cassandra/redhat/40x/

See http://cassandra.apache.org/download/ for full install instructions.

This is an ALPHA version! It is not intended for production use, however 
the project would appreciate your testing and feedback to make the final 
release better. As always, please pay attention to the release notes[2] 
and let us know[3] if you encounter any problems.


Enjoy!

[1]: CHANGES.txt 
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-alpha2
[2]: NEWS.txt 
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha2

[3]: https://issues.apache.org/jira/browse/CASSANDRA

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: [RELEASE] Apache Cassandra 4.0-alpha1 released

[Abdul Patel]

Have anyone used this yet?
Any intial thoughts?
When will be full and final ready fornprod version coming in?

On Sunday, September 8, 2019, Michael Shuler  wrote:

> The Cassandra team is pleased to announce the release of Apache Cassandra
> version 4.0-alpha1.
>
> Apache Cassandra is a fully distributed database. It is the right choice
> when you need scalability and high availability without compromising
> performance.
>
>  http://cassandra.apache.org/
>
> Downloads of source and binary distributions for 4.0-alpha1:
>
>
> http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha1/
> apache-cassandra-4.0-alpha1-bin.tar.gz
>
> http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha1/
> apache-cassandra-4.0-alpha1-src.tar.gz
>
> Debian and Redhat configurations
>
>  sources.list:
>  deb http://www.apache.org/dist/cassandra/debian 40x main
>
>  yum config:
>  baseurl=https://www.apache.org/dist/cassandra/redhat/40x/
>
> See http://cassandra.apache.org/download/ for full install instructions.
> Since this is the first alpha release, it will not be present on the
> download page.
>
> This is an ALPHA version! It is not intended for production use, however
> the project would appreciate your testing and feedback to make the final
> release better. As always, please pay attention to the release notes[2] and
> Let us know[3] if you were to encounter any problem.
>
> Enjoy!
>
> [1]: CHANGES.txt: https://gitbox.apache.org/repo
> s/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/
> tags/cassandra-4.0-alpha1
> [2]: NEWS.txt: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_
> plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha1
> [3]: JIRA: https://issues.apache.org/jira/browse/CASSANDRA
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

[RELEASE] Apache Cassandra 4.0-alpha1 released

[Michael Shuler]

The Cassandra team is pleased to announce the release of Apache 
Cassandra version 4.0-alpha1.


Apache Cassandra is a fully distributed database. It is the right choice 
when you need scalability and high availability without compromising 
performance.


 http://cassandra.apache.org/

Downloads of source and binary distributions for 4.0-alpha1:


http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha1/apache-cassandra-4.0-alpha1-bin.tar.gz

http://www.apache.org/dyn/closer.lua/cassandra/4.0-alpha1/apache-cassandra-4.0-alpha1-src.tar.gz

Debian and Redhat configurations

 sources.list:
 deb http://www.apache.org/dist/cassandra/debian 40x main

 yum config:
 baseurl=https://www.apache.org/dist/cassandra/redhat/40x/

See http://cassandra.apache.org/download/ for full install instructions. 
Since this is the first alpha release, it will not be present on the 
download page.


This is an ALPHA version! It is not intended for production use, however 
the project would appreciate your testing and feedback to make the final 
release better. As always, please pay attention to the release notes[2] 
and Let us know[3] if you were to encounter any problem.


Enjoy!

[1]: CHANGES.txt: 
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0-alpha1
[2]: NEWS.txt: 
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0-alpha1

[3]: JIRA: https://issues.apache.org/jira/browse/CASSANDRA

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: When Apache Cassandra 4.0 will release?

[Pandey Bhaskar]

Thanks Simon. Really good to know about it. I was trying to configure and
its working.

On Fri, Jul 26, 2019 at 9:56 PM Simon Fontana Oscarsson <
simon.fontana.oscars...@ericsson.com> wrote:

> Hi,
>
> To my knowledge there is no set date for 4.0, the community is
> prioritizing QA over fast release which I think is great! Anytime during Q4
> is suggested. See the latest information from the dev mailing list:
> <https://lists.apache.org/thread.html/1a768d057d1af5a0f373c4c399a23e65cb04c61bbfff612634b9437c@%3Cdev.cassandra.apache.org%3E>
> https://lists.apache.org/thread.html/1a768d057d1af5a0f373c4c399a23e65cb04c61bbfff612634b9437c@%3Cdev.cassandra.apache.org%3E
> Maybe we will get some more information on the summit in september.
>
> There is a plugin available for audit logging on Cassandra 3.0 and 3.11
> called ecAudit:  <https://github.com/Ericsson/ecaudit>
> https://github.com/Ericsson/ecaudit
> You can find see what compatibility it has with the audit logging for 4.0
> here:
> <https://github.com/Ericsson/ecaudit/blob/release/c3.0.11/doc/cassandra_compatibility.md#apache-cassandra-40>
> https://github.com/Ericsson/ecaudit/blob/release/c3.0.11/doc/cassandra_compatibility.md#apache-cassandra-40
>
>
> --
>
> SIMON FONTANA OSCARSSON
> Software Developer
>
> Ericsson
> Ölandsgatan 1
> 37133 Karlskrona, swedensimon.fontana.oscars...@ericsson.comwww.ericsson.com
>
>
> On fre, 2019-07-26 at 16:01 +, bhaskar pandey wrote:
>
> Hi,
>
> When Apache Cassandra 4.0 will release? Any specific time line? I am eagerly 
> waiting for audit log.
>
> Regards,
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: When Apache Cassandra 4.0 will release?

[Simon Fontana Oscarsson]

Hi,
To my knowledge there is no set date for 4.0, the community is
prioritizing QA over fast release which I think is great! Anytime
during Q4 is suggested. See the latest information from the dev mailing
list: https://lists.apache.org/thread.html/1a768d057d1af5a0f373c4c399a2
3e65cb04c61bbfff612634b9437c@%3Cdev.cassandra.apache.org%3E
Maybe we will get some more information on the summit in september.
There is a plugin available for audit logging on Cassandra 3.0 and 3.11
called ecAudit: https://github.com/Ericsson/ecaudit
You can find see what compatibility it has with the audit logging for
4.0 here: https://github.com/Ericsson/ecaudit/blob/release/c3.0.11/doc/
cassandra_compatibility.md#apache-cassandra-40 

-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com
On fre, 2019-07-26 at 16:01 +, bhaskar pandey wrote:
> Hi,
> 
> When Apache Cassandra 4.0 will release? Any specific time line? I am
> eagerly waiting for audit log. 
> 
> Regards,
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

smime.p7s
Description: S/MIME cryptographic signature

When Apache Cassandra 4.0 will release?

[bhaskar pandey]

Hi,

When Apache Cassandra 4.0 will release? Any specific time line? I am eagerly 
waiting for audit log. 

Regards,

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Cassandra 4.0

[Jacques-Henri Berthemet]

Are there binary builds available for testing or is it source only?

--
Jacques-Henri Berthemet

-Original Message-
From: Nate McCall  
Sent: Wednesday, October 24, 2018 10:02 PM
To: Cassandra Users 
Subject: Re: Cassandra 4.0

When it's ready :)

In all seriousness, the past two blog posts include some discussion on our 
motivations and current goals with regard to 4.0:
http://cassandra.apache.org/blog/
On Wed, Oct 24, 2018 at 4:49 AM Abdul Patel  wrote:
>
> Hi all,
>
> Any idea when 4.0 is planned to release?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Cassandra 4.0

[Nate McCall]

When it's ready :)

In all seriousness, the past two blog posts include some discussion on
our motivations and current goals with regard to 4.0:
http://cassandra.apache.org/blog/
On Wed, Oct 24, 2018 at 4:49 AM Abdul Patel  wrote:
>
> Hi all,
>
> Any idea when 4.0 is planned to release?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra 4.0

[Abdul Patel]

Hi all,

Any idea when 4.0 is planned to release?

RE: Java 11 support in Cassandra 4.0 + Early Testing and Feedback

[Kyrylo Lebediev]

As many people use Oracle JDK, I think it worth mentioning that according 
Oracle Support Roadmap there are some changes in their policies for Java 11 and 
above (http://www.oracle.com/technetwork/java/eol-135779.html).

In particular:
“Starting with Java SE 9, in addition to providing Oracle JDK for free under 
the BCL<http://www.oracle.com/technetwork/java/javase/terms/license>, Oracle 
also started providing builds of OpenJDK<http://openjdk.java.net/> under an 
open source license<http://openjdk.java.net/legal/gplv2+ce.html> (similar to 
that of Linux). Oracle is working to make the Oracle JDK and OpenJDK builds 
from Oracle 
interchangeable<https://blogs.oracle.com/java-platform-group/faster-and-easier-use-and-redistribution-of-java-se>
 - targeting developers and organizations that do not want commercial support 
or enterprise management tools. Beginning with Oracle Java SE 11 (18.9 LTS), 
the Oracle JDK will continue to be available royalty-free for development, 
testing, prototyping or demonstrating purposes. As announced in September 
2017<https://blogs.oracle.com/java-platform-group/faster-and-easier-use-and-redistribution-of-java-se>,
 with the OracleJDK and builds of Oracle OpenJDK being interchangeable for 
releases of Java SE 11 and later, the Oracle JDK will primarily be for 
commercial and support customers and OpenJDK builds from Oracle are for those 
who do not want commercial support or enterprise management tools.”

What these statements mean for Cassandra users in terms of which JDK (there are 
several OSS alternatives available) is the best to use in case of absence of 
active Oracle/Java Subscription?

Regards,
Kyrill

From: Jonathan Haddad 
Sent: Thursday, August 16, 2018 9:02 PM
To: user 
Subject: Java 11 support in Cassandra 4.0 + Early Testing and Feedback

Hey folks,

As we start to get ready to feature freeze trunk for 4.0, it's going to be 
important to get a lot of community feedback.  This is going to be a big 
release for a number of reasons.

* Virtual tables.  Finally a nice way of querying for system metrics & status
* Streaming optimizations 
(https://cassandra.apache.org/blog/2018/08/07/faster_streaming_in_cassandra.html)
* Groundwork for strongly consistent schema changes
* Optimizations to internode communcation
* Experimental support for Java 11

I (and many others) would like Cassandra to be rock solid on day one if its 
release.  The best way to ensure that happens is if people provide feedback.  
One of the areas we're going to need a lot of feedback on is on how things work 
with Java 11, especially if you have a way of simulating a real world workload 
on a staging cluster.  I've written up instructions here on how to start 
testing: http://thelastpickle.com/blog/2018/08/16/java11.html

Java 11 hasn't been released yet, but that doesn't mean it's not a good time to 
test.  Any bugs we can identify now will help us get to a stable release 
faster.  If you rely on Cassandra for your business, please take some time to 
participate in the spirit of OSS by helping test & provide feedback to the team.

Thanks everyone!
---
Jon Haddad
Principal Consultant, The Last Pickle

Re: bigger data density with Cassandra 4.0?

[dinesh.jo...@yahoo.com.INVALID]

With LCS, 6696 you can maximize the percentage of SSTables that use the new 
streaming path. With LCS and relatively small SSTables you should see good 
gains. Bootstrap is a use-case that should see the maximum benefits. This 
feature will get better with time.
Dinesh 

On Wednesday, August 29, 2018, 12:34:32 AM PDT, kurt greaves 
 wrote:  
 
 My reasoning was if you have a small cluster with vnodes you're more likely to 
have enough overlap between nodes that whole SSTables will be streamed on major 
ops. As  N gets >RF you'll have less common ranges and thus less likely to be 
streaming complete SSTables. Correct me if I've misunderstood.
On 28 August 2018 at 01:37, Dinesh Joshi  wrote:

Although the extent of benefits depend on the specific use case, the cluster 
size is definitely not a limiting factor.

Dinesh
On Aug 27, 2018, at 5:05 AM, kurt greaves  wrote:


I believe there are caveats that it will only really help if you're not using 
vnodes, or you have a very small cluster, and also internode encryption is not 
enabled. Alternatively if you're using JBOD vnodes will be marginally better, 
but JBOD is not a great idea (and doesn't guarantee a massive improvement).
On 27 August 2018 at 15:46, dinesh.jo...@yahoo.com.INVALID 
 wrote:

Yes, this feature will help with operating nodes with higher data density.
Dinesh 

On Saturday, August 25, 2018, 9:01:27 PM PDT, onmstester onmstester 
 wrote:  
 
 I've noticed this new feature of 4.0:
Streaming optimizations (https://cassandra.apache.org/ 
blog/2018/08/07/faster_streami ng_in_cassandra.html)
Is this mean that we could have much more data density with Cassandra 4.0 (less 
problems than 3.X)? I mean > 10 TB of data on each node without worrying about 
node join/remove?
This is something needed for Write-Heavy applications that do not read a lot. 
When you have like 2 TB of data per day and need to keep it for 6 month, it 
would be waste of money to purchase 180 servers (even Commodity or Cloud). 
IMHO, even if 4.0 fix problem with streaming/joining a new node, still 
Compaction is another evil for a big node, but we could tolerate that somehow


Sent using Zoho Mail



  




  

RE: [EXTERNAL] Re: Re: bigger data density with Cassandra 4.0?

[Rahul Singh]

YugaByte is also another new dancer in the Cassandra dance. The data store is 
based on RocksDB — and it’s written in C++. Although they ar wire compliant 
with c* I’m pretty are everything under the hood is NOT a port like Scylla was 
initially.

Rahul Singh
Chief Executive Officer
m 202.905.2818

Anant Corporation
1010 Wisconsin Ave NW, Suite 250
Washington, D.C. 20007

We build and manage digital business technology platforms.
On Aug 29, 2018, 10:05 AM -0400, Durity, Sean R , 
wrote:
> If you are going to compare vs commercial offerings like Scylla and CosmosDB, 
> you should be looking at DataStax Enterprise. They are moving more quickly 
> than open source (IMO) on adding features and tools that enterprises really 
> need. I think they have some emerging tech for large/dense nodes, in 
> particular. The ability to handle different data model types (Graph and 
> Search) and embedded analytics sets it apart from plain Cassandra. Plus, they 
> have replaced Cassandra’s SEDA architecture to give it a significant boost in 
> performance. As a customer, I see the value in what they are doing.
>
>
> Sean Durity
> From: onmstester onmstester 
> Sent: Wednesday, August 29, 2018 7:43 AM
> To: user 
> Subject: [EXTERNAL] Re: Re: bigger data density with Cassandra 4.0?
>
> Could you please explain more about (you mean slower performance in compare 
> to Cassandra?)
> ---Hbase tends to be quite average for transactional data
>
> and about:
> ScyllaDB IDK, I'd assume they just sorted out streaming by learning from 
> C*'s mistakes.
> While ScyllaDB is a much younger project than Cassandra with so much less 
> usage and attention, Currently I encounter a dilemma on launching new 
> clusters which is: should i wait for Cassandra community to apply all 
> enhancement's and bug fixes that applied by their main competitors (Scylla DB 
> or Cosmos DB) or just switch to competitors (afraid of the new world!)?
> For example right now is there a motivation to handle more dense nodes in 
> near future?
>
> Again, Thank you for your time
>
> Sent using Zoho Mail
>
>
>  On Wed, 29 Aug 2018 15:16:40 +0430 kurt greaves  
> wrote 
>
> > quote_type
> > Most of the issues around big nodes is related to streaming, which is 
> > currently quite slow (should be a bit better in 4.0). HBase is built on top 
> > of hadoop, which is much better at large files/very dense nodes, and tends 
> > to be quite average for transactional data. ScyllaDB IDK, I'd assume they 
> > just sorted out streaming by learning from C*'s mistakes.
> >
> > On 29 August 2018 at 19:43, onmstester onmstester  
> > wrote:
> >
> > > quote_type
> > >
> > > Thanks Kurt,
> > > Actually my cluster has > 10 nodes, so there is a tiny chance to stream a 
> > > complete SSTable.
> > > While logically any Columnar noSql db like Cassandra, needs always to 
> > > re-sort grouped data for later-fast-reads and having nodes with big 
> > > amount of data (> 2 TB) would be annoying for this background process, 
> > > How is it possible that some of these databases like HBase and Scylla db 
> > > does not emphasis on small nodes (like Cassandra do)?
> > >
> > > Sent using Zoho Mail
> > >
> > >
> > >  Forwarded message 
> > > From : kurt greaves 
> > > To : "User"
> > > Date : Wed, 29 Aug 2018 12:03:47 +0430
> > > Subject : Re: bigger data density with Cassandra 4.0?
> > >  Forwarded message 
> > >
> > > > quote_type
> > > > My reasoning was if you have a small cluster with vnodes you're more 
> > > > likely to have enough overlap between nodes that whole SSTables will be 
> > > > streamed on major ops. As  N gets >RF you'll have less common ranges 
> > > > and thus less likely to be streaming complete SSTables. Correct me if 
> > > > I've misunderstood.
> > >
>
>
>
>
> The information in this Internet Email is confidential and may be legally 
> privileged. It is intended solely for the addressee. Access to this Email by 
> anyone else is unauthorized. If you are not the intended recipient, any 
> disclosure, copying, distribution or any action taken or omitted to be taken 
> in reliance on it, is prohibited and may be unlawful. When addressed to our 
> clients any opinions or advice contained in this Email are subject to the 
> terms and conditions expressed in any applicable governing The Home Depot 
> terms of business or client engagement letter. The Home Depot disclaims all 
> responsibility and liability for the accuracy and content of this attachment 
> and for any damages or losses arising from any inaccuracies, errors, viruses, 
> e.g., worms, trojan horses, etc., or other items of a destructive nature, 
> which may be contained in this attachment and shall not be liable for direct, 
> indirect, consequential or special damages in connection with this e-mail 
> message or its attachment.