Re: storage engine series

[Aaron Ploetz]

Nice! This sounds awesome, Jon.

On Mon, Apr 29, 2024 at 6:25 PM Jon Haddad  wrote:

> Hey everyone,
>
> I'm doing a 4 week YouTube series on the C* storage engine.  My first
> video was last week where I gave an overview into some of the storage
> engine internals [1].
>
> The next 3 weeks are looking at the new Trie indexes coming in 5.0 [2],
> running Cassandra on EBS [3], and finally looking at some potential
> optimizations [4] that could be done to improve things even further in the
> future.
>
> I hope these videos are useful to the community, and I welcome feedback!
>
> Jon
>
> [1] https://www.youtube.com/live/yj0NQw9DgcE?si=ra1zqusMdSs6vl4T
> [2] https://www.youtube.com/live/ZdzwtH0cJDE?si=CumcPny2UG8zwtsw
> [3] https://www.youtube.com/live/kcq1TC407U4?si=pZ8AkXkMzIylQgB6
> [4] https://www.youtube.com/live/yj0NQw9DgcE?si=ra1zqusMdSs6vl4T
>

Re: Upgrade from C* 3 to C* 4 per datacenter

[Aaron Ploetz]

Just a heads-up, but there have been issues (at least one) reported when
upgrading a multi-DC cluster from 3.x to 4.x when the cluster uses
node-to-node SSL/TLS encryption. This is largely attributed to the fact
that the secure port in 4.x changes to 9142, whereas in 3.x it continues to
run on 9042 (same as non-SSL/TLS).

On Thu, Oct 26, 2023 at 2:03 PM Sebastian Marsching 
wrote:

> Hi,
>
> as we are currently facing the same challenge (upgrading an existing
> cluster from C* 3 to C* 4), I wanted to share our strategy with you. It
> largely is what Scott already suggested, but I have some extra details, so
> I thought it might still be useful.
>
> We duplicated our cluster using the strategy described at
> http://adamhutson.com/cloning-cassandra-clusters-the-fast-way/. Of course
> it is possible to figure out all the steps on your own, but I feel like
> this detailed guide saved me at least a few hours, if not days. Instead of
> restoring from a backup, we chose to create a snapshot on the live nodes
> and copy the data from there, but this does not really change the overall
> process.
>
> We only run a single data-center cluster, but I think that this process
> easily translates to a multi data-center setup. In this case, you can
> choose to only clone a single data center or you can clone a few or all of
> them, if you deem this to be necessary for your tests. The only
> “limitation” is that for each data center that you clone, you need exactly
> the same number of nodes in your test cluster that you have in the
> respective data center of your production cluster.
>
> Once the cluster is cloned, you can test whatever you like (e.g. upgrade
> to C* 4, test operations in a mixed-version cluster, etc.).
>
> Our experience with the upgrade from C* 3.11 to C* 4.1 on the test cluster
> was quite smooth. The only problem that we saw was that when later adding a
> second data center to the test cluster, we got a lot of
> CorruptSSTableExceptions on one of the nodes in the existing data center.
> We first attributed this to the upgrade, but later we found out that this
> also happens when running on C* 3.11.
>
> We now believe that the hardware of one of the nodes that we used for the
> test cluster has a defect, because the exceptions were limited to this
> exact node, even after moving data around. It just took us a while to
> figure this out, because the hardware for the test cluster was brand new,
> so “broken hardware” wasn’t our first guess. We are still in the process of
> definitely proving that this specific piece of hardware is broken, but we
> are now sufficiently confident in the stability of C* 4, that we are soon
> going to move forward with upgrading the production cluster.
>
> -Sebastian
>
>

Re: Cassandra 5 & Support for JDK 17

[Aaron Ploetz]

Hello Sean,

Cassandra 5.0 is indeed intended to run on Java 17.  In fact, I just tried
running 5.0-alpha
, and it does
indeed run on Java 17.0.8.

There is a Jira ticket out there for tracking Java 17 issues with 5.0:
CASSANDRA-16895 .
Its current status is "In Progress," which suggests that there may be some
issues that haven't been resolved.  I'd suggest keeping an eye on that
ticket, as that should give you a pulse on this issue.

Thanks,

Aaron


On Tue, Aug 22, 2023 at 7:55 AM McIntyre, Sean  wrote:

> Hello.  I have several applications that I am looking to move to JDK 17
> and want to understand the plans for Cassandra to support this version of
> Java.  I’ve read that version 5 may well support JDK17 and that it may be
> available by the end of this year.  Are these firm plans?  Are there
> tickets I could watch to understand the progress?  There are a few upgrades
> I need to do that are blocked by the lack of JDK 17, so am very interested
> in all this coming together sooner, rather than later.
>
>
>
> Regards,
>
>
>
> Sean McIntyre
>
> Engineering Manager
>
> Akamai Technologies
>

Re: Reads not returning data after adding node

[Aaron Ploetz]

>
> I just asked that question on this list and the answer was that adding the
> new nodes as rack4, rack5 and rack6 is fine. They are all on
> separate physical racks. Is that ok?
>

Yes, Jeff is right, all 6 nodes each on their own rack will work just fine.

Should I do a full repair first or is the remove node operation basically
> doing that?
>

I don't think you'll need a full repair.  Removenode should be taking care
of streaming that node's data to where it needs to go.

On Mon, Apr 3, 2023 at 10:26 AM David Tinker  wrote:

> Thanks. Yes my big screwup here was to make the new node a seed node so it
> didn't get any data. I am going to add 3 more nodes, one at a time when the
> cluster has finished with the remove and everything seems stable. Should I
> do a full repair first or is the remove node operation basically doing that?
>
> Re the racks. I just asked that question on this list and the answer was
> that adding the new nodes as rack4, rack5 and rack6 is fine. They are all
> on separate physical racks. Is that ok?
>
> On Mon, Apr 3, 2023 at 5:16 PM Aaron Ploetz  wrote:
>
>> The time it takes to stream data off of a node varies by network, cloud
>> region, and other factors.  So it's not unheard of for it to take a bit to
>> finish.
>>
>> Just thought I'd mention that auto_bootstrap is true by default.  So if
>> you're not setting it, the node should bootstrap as long as it's not a seed
>> node.
>>
>> As for the rack issue, yes, it's a good idea to keep your racks in
>> multiples of your RF.  When performing token ownership calculations,
>> Cassandra takes rack designation into consideration.  It tries to ensure
>> that multiple replicas for a row are not placed in the same rack.  TBH -
>> I'd build out two more nodes to have 6 nodes across 3 racks (2 in each),
>> just to ensure even distribution.  Otherwise, you might notice that the
>> nodes sharing a rack will consume disk at a different rate than the nodes
>> which have their own rack.
>>
>> On Mon, Apr 3, 2023 at 8:57 AM David Tinker 
>> wrote:
>>
>>> Thanks. Hmm, the remove has been busy for hours but seems to be
>>> progressing.
>>>
>>> I have been running this on the nodes to monitor progress:
>>> # nodetool netstats | grep Already
>>> Receiving 92 files, 843934103369 bytes total. Already received
>>> 82 files (89.13%), 590204687299 bytes total (69.93%)
>>> Sending 84 files, 860198753783 bytes total. Already sent 56
>>> files (66.67%), 307038785732 bytes total (35.69%)
>>> Sending 78 files, 815573435637 bytes total. Already sent 56
>>> files (71.79%), 313079823738 bytes total (38.39%)
>>>
>>> The percentages are ticking up.
>>>
>>> # nodetool ring | head -20
>>> Datacenter: dc1
>>> ==
>>> Address   RackStatus State   LoadOwns
>>>  Token
>>>
>>>  9189523899826545641
>>> xxx.xxx.xxx..24rack4   Down   Leaving 26.62 GiB   79.95%
>>>  -9194674091837769168
>>> xxx.xxx.xxx.107   rack1   Up Normal  2.68 TiB73.25%
>>>  -9168781258594813088
>>> xxx.xxx.xxx.253   rack2   Up Normal  2.63 TiB73.92%
>>>  -9163037340977721917
>>> xxx.xxx.xxx.105   rack3   Up Normal  2.68 TiB72.88%
>>>  -9148860739730046229
>>>
>>>
>>> On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user <
>>> user@cassandra.apache.org> wrote:
>>>
>>>> Use nodetool removenode is strongly preferred in most circumstances,
>>>> and only resort to assassinate if you do not care about data
>>>> consistency or you know there won't be any consistency issue (e.g. no new
>>>> writes and did not run nodetool cleanup).
>>>>
>>>> Since the size of data on the new node is small, nodetool removenode
>>>> should finish fairly quickly and bring your cluster back.
>>>>
>>>> Next time when you are doing something like this again, please test it
>>>> out on a non-production environment, make sure everything works as expected
>>>> before moving onto the production.
>>>>
>>>>
>>>> On 03/04/2023 06:28, David Tinker wrote:
>>>>
>>>> Should I use assassinate or removenode? Given that there is some data
>>>> on the node. Or will that be found on the other nodes? Sorry for all the
>>>> questions but I really don't w

Re: Reads not returning data after adding node

[Aaron Ploetz]

The time it takes to stream data off of a node varies by network, cloud
region, and other factors.  So it's not unheard of for it to take a bit to
finish.

Just thought I'd mention that auto_bootstrap is true by default.  So if
you're not setting it, the node should bootstrap as long as it's not a seed
node.

As for the rack issue, yes, it's a good idea to keep your racks in
multiples of your RF.  When performing token ownership calculations,
Cassandra takes rack designation into consideration.  It tries to ensure
that multiple replicas for a row are not placed in the same rack.  TBH -
I'd build out two more nodes to have 6 nodes across 3 racks (2 in each),
just to ensure even distribution.  Otherwise, you might notice that the
nodes sharing a rack will consume disk at a different rate than the nodes
which have their own rack.

On Mon, Apr 3, 2023 at 8:57 AM David Tinker  wrote:

> Thanks. Hmm, the remove has been busy for hours but seems to be
> progressing.
>
> I have been running this on the nodes to monitor progress:
> # nodetool netstats | grep Already
> Receiving 92 files, 843934103369 bytes total. Already received 82
> files (89.13%), 590204687299 bytes total (69.93%)
> Sending 84 files, 860198753783 bytes total. Already sent 56 files
> (66.67%), 307038785732 bytes total (35.69%)
> Sending 78 files, 815573435637 bytes total. Already sent 56 files
> (71.79%), 313079823738 bytes total (38.39%)
>
> The percentages are ticking up.
>
> # nodetool ring | head -20
> Datacenter: dc1
> ==
> Address   RackStatus State   LoadOwns
>Token
>
>9189523899826545641
> xxx.xxx.xxx..24rack4   Down   Leaving 26.62 GiB   79.95%
>-9194674091837769168
> xxx.xxx.xxx.107   rack1   Up Normal  2.68 TiB73.25%
>-9168781258594813088
> xxx.xxx.xxx.253   rack2   Up Normal  2.63 TiB73.92%
>-9163037340977721917
> xxx.xxx.xxx.105   rack3   Up Normal  2.68 TiB72.88%
>-9148860739730046229
>
>
> On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> Use nodetool removenode is strongly preferred in most circumstances, and
>> only resort to assassinate if you do not care about data consistency or
>> you know there won't be any consistency issue (e.g. no new writes and did
>> not run nodetool cleanup).
>>
>> Since the size of data on the new node is small, nodetool removenode
>> should finish fairly quickly and bring your cluster back.
>>
>> Next time when you are doing something like this again, please test it
>> out on a non-production environment, make sure everything works as expected
>> before moving onto the production.
>>
>>
>> On 03/04/2023 06:28, David Tinker wrote:
>>
>> Should I use assassinate or removenode? Given that there is some data on
>> the node. Or will that be found on the other nodes? Sorry for all the
>> questions but I really don't want to mess up.
>>
>> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz  wrote:
>>
>>> That's what nodetool assassinte will do.
>>>
>>> On Sun, Apr 2, 2023 at 10:19 PM David Tinker 
>>> wrote:
>>>
 Is it possible for me to remove the node from the cluster i.e. to undo
 this mess and get the cluster operating again?

 On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz  wrote:

> You can leave it in the seed list of the other nodes, just make sure
> it's not included in this node's seed list.  However, if you do decide to
> fix the issue with the racks first assassinate this node (nodetool
> assassinate ), and update the rack name before you restart.
>
> On Sun, Apr 2, 2023 at 10:06 PM David Tinker 
> wrote:
>
>> It is also in the seeds list for the other nodes. Should I remove it
>> from those, restart them one at a time, then restart it?
>>
>> /etc/cassandra # grep -i bootstrap *
>> doesn't show anything so I don't think I have auto_bootstrap false.
>>
>> Thanks very much for the help.
>>
>>
>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz 
>> wrote:
>>
>>> Just remove it from the seed list in the cassandra.yaml file and
>>> restart the node.  Make sure that auto_bootstrap is set to true first
>>> though.
>>>
>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker 
>>> wrote:
>>>
 So likely because I made it a seed node when I added it to the
 cluster it didn't do the bootstrap process. How can I recover this?

 On Mon, Apr 3, 2023 at 6:41 AM David Tinker 
 wrote:

> Yes replication factor is 3.
>
> I ran nodetool repair -pr on all the nodes (one at a time) and am
> still having issues getting data back from queries.
>
> I did make the new node a seed node.
>
> Re "rack4": I assumed that was just an indication as to the
> 

Accord and elector selection

[Aaron Ploetz]

Under the premise of an Accord transaction, how does replica/elector
selection happen?  Does it choose replicas based on the partition keys?  Or
is there another way?

Thanks!

Aaron

Re: Cleanup

[Aaron Ploetz]

So if I remember right, setting compaction_throughput_per_mb to zero
effectively disables throttling, which means cleanup and compaction will
run as fast as the instance will allow.  For normal use, I'd recommend
capping that at 8 or 16.

Aaron


On Thu, Feb 16, 2023 at 9:43 AM Marc Hoppins  wrote:

> Compaction_throughtput_per_mb is 0 in cassandra.yaml. Is setting it in
> nodetool going to provide any increase?
>
>
>
> *From:* Durity, Sean R via user 
> *Sent:* Thursday, February 16, 2023 4:20 PM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Cleanup
>
>
>
> EXTERNAL
>
> Clean-up is constrained/throttled by compactionthroughput. If your system
> can handle it, you can increase that throughput (nodetool
> setcompactionthroughput) for the clean-up in order to reduce the total time.
>
>
>
> It is a node-isolated operation, not cluster-involved. I often run clean
> up on all nodes in a DC at the same time. Think of it as compaction and
> consider your cluster performance/workload/timelines accordingly.
>
>
>
> Sean R. Durity
>
>
>
> *From:* manish khandelwal 
> *Sent:* Thursday, February 16, 2023 5:05 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: Cleanup
>
>
>
> There is no advantage of running cleanup if no new nodes are introduced.
> So cleanup time should remain same when adding new nodes. Cleanup is a
> local to node so network bandwidth should have no effect on reducing
> cleanup time. Dont ignore cleanup
>
>
>
> There is no advantage of running cleanup if no new nodes are introduced.
> So cleanup time should remain same when adding new nodes.
>
>
>
>  Cleanup is a local to node so network bandwidth should have no effect on
> reducing cleanup time.
>
>
>
>  Dont ignore cleanup as it can cause you disks occupied without any use.
>
>
>
>  You should plan to run cleanup in a lean period (low traffic). Also you
> can use suboptions of keyspace and table names to plan it such a way that
> I/O pressure is not much.
>
>
>
>
>
> Regards
>
> Manish
>
>
>
> On Thu, Feb 16, 2023 at 3:12 PM Marc Hoppins 
> wrote:
>
> Hulloa all,
>
>
>
> I read a thing re. adding new nodes where the recommendation was to run
> cleanup on the nodes after adding a new node to remove redundant token
> ranges.
>
>
>
> I timed this way back when we only had ~20G of data per node and it took
> approx. 5 mins per node.  After adding a node on Tuesday, I figured I’d run
> cleanup.
>
>
>
> Per node, it is taking 6+ hours now as we have 2-2.5T per node.
>
>
>
> Should we be running cleanup regularly regardless of whether or not new
> nodes have been added?  Would it reduce cleanup times for when we do add
> new nodes?
>
> If we double the network bandwidth can we effectively reduce this lengthy
> cleanup?
>
> Maybe just ignore cleanup entirely?
>
> I appreciate that cleanup will increase the load but running cleanup on
> one node at a time seems impractical.  How many simultaneous nodes (per
> rack) should we limit cleanup to?
>
>
>
> More experienced suggestions would be most appreciated.
>
>
> Marc
>
>
>
> INTERNAL USE
>
>

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

[Aaron Ploetz]

>
> We are going to have our first gathering next Wednesday at 8AM PT


I'd like to help out, too, if you wouldn't mind forwarding that meeting on
to me.

Thanks,

Aaron

Re: Denylisting with a composite partition key

[Aaron Ploetz]

Works!

So I was running on my *local*, and all of my attempts to add to the
denylist were failing because the denylist_consistency_level was set to
QUORUM:

WARN  [main] 2022-10-25 11:57:27,238 NoSpamLogger.java:108 - Attempting to
load denylist and not enough nodes are available for a QUORUM refresh.
Reload the denylist when unavailable nodes are recovered to ensure your
denylist remains in sync.
INFO  [main] 2022-10-25 11:57:27,238 PartitionDenylist.java:163 -
Insufficient nodes while loading partition denylist cache. Scheduled retry
in 5 seconds.

Adjusted that setting to ONE, and now everything is working.

Thanks again, Cheng!

Aaron


On Fri, Oct 21, 2022 at 9:48 AM Aaron Ploetz  wrote:

> Awesome.  Thank you, Cheng!  I’ll give this a shot and let you know.
>
> Thanks,
>
> Aaron
>
>
> On Oct 21, 2022, at 12:45 AM, Cheng Wang  wrote:
>
> 
> Hi Aaron,
>
> After reading through the code, I finally figured out the issue. So back
> to your original question where you failed to run
> $>run denylistKey stackoverflow weather_sensor_data "'Minneapolis,
> MN',202210"
> #IllegalArgumentException: Operation denylistKey with 4 parameters doesn't
> exist in bean org.apache.cassandra.db:type=StorageProxy
>
> It's not a Cassandra issue since it failed at the JMX parser stage,  even
> before it goes to the Cassandra internal StorageProxy::denylistKey method.
> Yes, you got the right gist. It's because of the extra space between
> "Minneapolis" and "MN". To avoid the error, you need to add "\\" before the
> white space as below :
>
> $>run denylistKey "stackoverflow" "weather_sensor_data" "Minneapolis,\\
> MN:202210"
>
> #calling operation denylistKey of mbean
> org.apache.cassandra.db:type=StorageProxy with params [stackoverflow,
> weather_sensor_data, Minneapolis, MN:202210]
>
> #operation returns:
>
> true
>
> And yes, the composite key is delimited by ":" . You can find more
> examples from the test code as it used composite keys
>
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/service/PartitionDenylistTest.java
>
> And then when you select the key, it will get denylisted:
>
> cqlsh> select ks_name, table_name, key from
> system_distributed.partition_denylist;
>
>
>  ks_name   | table_name  | key
>
>
> ---+-+--
>
>  stackoverflow | weather_sensor_data |
> 0x000f4d696e6e6561706f6c69732c204d4e04000315e200
>
>
> (2 rows)
>
> cqlsh> select * from stackoverflow.weather_sensor_data where city =
> 'Minneapolis, MN' AND month = 202210;
>
> InvalidRequest: Error from server: code=2200 [Invalid query]
> message="Unable to read denylisted partition
> [0xDecoratedKey(8132598509541575594,
> 000f4d696e6e6561706f6c69732c204d4e04000315e200)] in
> stackoverflow/weather_sensor_data"
>
> cqlsh>
>
>
> For your second question why even a single partition key doesn't work, my
> apologies my previous answer on direct insert to the s
> ystem_distributed.partition_denylist like
> insert into system_distributed.partition_denylist (ks_name, table_name,
> key) values ('stackoverflow', 'weather_sensor_data',
> textAsBlob('''Minneapolis, MN'', 202210'));
> is wrong. It's because the way Cassandra stores the key in the system
> table is it converts the string into a hexadecimal literal. You can find
> the source code here
>
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/schema/PartitionDenylist.java
>  And check out the function addKeyToDenylist
> final String insert = String.format("INSERT INTO system_distributed.
> partition_denylist (ks_name, table_name, key) VALUES ('%s', '%s', 0x%s)",
> keyspace, table, ByteBufferUtil.bytesToHex(key));
>
> You can see that it's different from the way of textAsBlob. A better (yet
> general) explanation is here
>
> https://stackoverflow.com/questions/31994880/why-cassandra-cql-query-with-hexadecimal-literal-works-but-textasbinary-does-not
>
> So... the short answer is, I get back my previous answer. My apologies!
> It's not recommended to directly insert rows into
> the system_distributed.partition_denylist table.You'd better go with the
> JMX interface since it will go to the StorageProxy which handles more logic
> like the hex string conversion and other error handling stuff.
>
> Hope the answer is helpful! Pls feel free to let me know if you have any
> questions!
>
> Thanks,
> Cheng
>
>
> On Thu, Oct 20, 2022 at 7:07 AM Aaron Pl

Re: Denylisting with a composite partition key

[Aaron Ploetz]

Awesome.  Thank you, Cheng!  I’ll give this a shot and let you know.

Thanks,

Aaron


> On Oct 21, 2022, at 12:45 AM, Cheng Wang  wrote:
> 
> 
> Hi Aaron,
> 
> After reading through the code, I finally figured out the issue. So back to 
> your original question where you failed to run 
> $>run denylistKey stackoverflow weather_sensor_data "'Minneapolis, MN',202210"
> #IllegalArgumentException: Operation denylistKey with 4 parameters doesn't 
> exist in bean org.apache.cassandra.db:type=StorageProxy
> 
> It's not a Cassandra issue since it failed at the JMX parser stage,  even 
> before it goes to the Cassandra internal StorageProxy::denylistKey method. 
> Yes, you got the right gist. It's because of the extra space between 
> "Minneapolis" and "MN". To avoid the error, you need to add "\\" before the 
> white space as below :
> 
> $>run denylistKey "stackoverflow" "weather_sensor_data" "Minneapolis,\\ 
> MN:202210"
> #calling operation denylistKey of mbean 
> org.apache.cassandra.db:type=StorageProxy with params [stackoverflow, 
> weather_sensor_data, Minneapolis, MN:202210]
> #operation returns: 
> true
> 
> And yes, the composite key is delimited by ":" . You can find more examples 
> from the test code as it used composite keys
> https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/service/PartitionDenylistTest.java
> 
> And then when you select the key, it will get denylisted:
> cqlsh> select ks_name, table_name, key from 
> system_distributed.partition_denylist;
> 
>  ks_name   | table_name  | key
> ---+-+--
>  stackoverflow | weather_sensor_data | 
> 0x000f4d696e6e6561706f6c69732c204d4e04000315e200
> 
> (2 rows)
> cqlsh> select * from stackoverflow.weather_sensor_data where city = 
> 'Minneapolis, MN' AND month = 202210;
> InvalidRequest: Error from server: code=2200 [Invalid query] message="Unable 
> to read denylisted partition [0xDecoratedKey(8132598509541575594, 
> 000f4d696e6e6561706f6c69732c204d4e04000315e200)] in 
> stackoverflow/weather_sensor_data"
> cqlsh> 
> 
> 
> For your second question why even a single partition key doesn't work, my 
> apologies my previous answer on direct insert to the 
> system_distributed.partition_denylist like
> insert into system_distributed.partition_denylist (ks_name, table_name, key) 
> values ('stackoverflow', 'weather_sensor_data', textAsBlob('''Minneapolis, 
> MN'', 202210'));
> is wrong. It's because the way Cassandra stores the key in the system table 
> is it converts the string into a hexadecimal literal. You can find the source 
> code here
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/schema/PartitionDenylist.java
>  And check out the function addKeyToDenylist
>  final String insert = String.format("INSERT INTO 
> system_distributed.partition_denylist (ks_name, table_name, key) VALUES 
> ('%s', '%s', 0x%s)",
> keyspace, table, 
> ByteBufferUtil.bytesToHex(key));
> 
> You can see that it's different from the way of textAsBlob. A better (yet 
> general) explanation is here
> https://stackoverflow.com/questions/31994880/why-cassandra-cql-query-with-hexadecimal-literal-works-but-textasbinary-does-not
> 
> So... the short answer is, I get back my previous answer. My apologies!  It's 
> not recommended to directly insert rows into the 
> system_distributed.partition_denylist table.You'd better go with the JMX 
> interface since it will go to the StorageProxy which handles more logic like 
> the hex string conversion and other error handling stuff. 
> 
> Hope the answer is helpful! Pls feel free to let me know if you have any 
> questions!
> 
> Thanks,
> Cheng
> 
> 
>> On Thu, Oct 20, 2022 at 7:07 AM Aaron Ploetz  wrote:
>> No worries, Cheng!
>> 
>> So I actually pivoted a little and adjusted my example table to use a single 
>> integer-based partition key.
>> 
>> aaron@cqlsh:stackoverflow> SELECT ks_name, table_name, blobAsint(key) FROM 
>> system_distributed.partition_denylist WHERE ks_name='stackoverflow' AND 
>> table_name='weather_sensor_data_by_month';
>> 
>>  ks_name   | table_name   | system.blobasint(key)
>> ---+--+---
>>  stackoverflow | weather_sensor_data_by_month |202210
>> 
>> But even with the yaml changes, it still allows me to SELECT that partition.
>> 
>> aaron@cqlsh:stackove

Re: Denylisting with a composite partition key

[Aaron Ploetz]

No worries, Cheng!

So I actually pivoted a little and adjusted my example table to use a
single integer-based partition key.

aaron@cqlsh:stackoverflow> SELECT ks_name, table_name, blobAsint(key) FROM
system_distributed.partition_denylist WHERE ks_name='stackoverflow' AND
table_name='weather_sensor_data_by_month';

 ks_name   | table_name   | system.blobasint(key)
---+--+---
 stackoverflow | weather_sensor_data_by_month |202210

But even with the yaml changes, it still allows me to SELECT that partition.

aaron@cqlsh:stackoverflow> SELECT * FROM weather_sensor_data_by_month WHERE
month=202210 LIMIT 1;

 month  | city   | recorded_time   | temp
++-+--
 202210 | Minneapolis,MN | 2022-10-17 11:30:00.00+ |1

Just wondering what I should do to get it to deny access?

Thanks,

Aaron



On Wed, Oct 19, 2022 at 5:34 PM Cheng Wang  wrote:

> Hi Aaron,
>
> Sorry for the late reply, was dealing with a production issue (maybe
> another topic for Cassandra Summit :-)). Are you running on your local
> machine? Then yes, you do need to enable the config for all the following
>
> enable_partition_denylist: true
>
> enable_denylist_writes: true
> enable_denylist_reads: true
> enable_denylist_range_reads: true
>
> I am not sure enable_partition_denylist will enable for the rest of three 
> @Jordan
> West   maybe you can confirm for that? But it's
> better to enable for all just to be safe.
> I will play by my side and get back to you soon about the composite keys.
>
> Thanks
> Cheng
>
> On Wed, Oct 19, 2022 at 1:42 PM Aaron Ploetz 
> wrote:
>
>> Just checking, but for this to work, do I have to mess with these
>> settings in the YAML at all?
>>
>> partition_denylist_enabled: true
>> denylist_reads_enabled: true
>>
>> They're commented out by default.
>>
>> Thanks,
>>
>> Aaron
>>
>>
>> On Mon, Oct 17, 2022 at 4:53 PM Aaron Ploetz 
>> wrote:
>>
>>> Thanks for the help with the INSERT, Cheng!  I'm further along than
>>> before.  But it still must not be matching up quite right, because I can
>>> still select that partition.
>>>
>>> I have several different combinations of the two keys (and I removed the
>>> space) of "Minneapolis,MN" and 202210.  Here's what I've got out there so
>>> far:
>>>
>>> aaron@cqlsh:stackoverflow> select ks_name, table_name, blobAsText(key)
>>> from system_distributed.partition_denylist;
>>>
>>>  ks_name   | table_name  | system.blobastext(key)
>>> ---+-+--
>>>  stackoverflow | weather_sensor_data | 'Minneapolis,MN', 202210
>>>  stackoverflow | weather_sensor_data |  'Minneapolis,MN',202210
>>>  stackoverflow | weather_sensor_data |  'Minneapolis,MN':202210
>>>  stackoverflow | weather_sensor_data |   Minneapolis,MN, 202210
>>>  stackoverflow | weather_sensor_data | Minneapolis,MN202210
>>>  stackoverflow | weather_sensor_data |Minneapolis,MN:202210
>>>
>>> (6 rows)
>>>
>>> aaron@cqlsh:stackoverflow> SELECT * FROM weather_sensor_data WHERE
>>> city='Minneapolis,MN' AND month=202210;
>>>
>>>  city   | month  | recorded_time   | temp
>>> ++-+--
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:30:00.00+ |1
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:25:00.00+ |1
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:20:00.00+ |1
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:15:00.00+ |1
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:10:00.00+ |2
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:05:00.00+ |2
>>>  Minneapolis,MN | 202210 | 2022-10-17 11:00:00.00+ |2
>>>
>>> (7 rows)
>>>
>>> As you can see, I can still select the partition.  I was really hoping
>>> one of those combinations would do it.
>>>
>>> Looking at the StorageProxyTest.java in the project, I saw that it was
>>> delimited by a colon ":", which is why I tried that, too.
>>>
>>> Still looking for the right way to enter both of those keys.
>>>
>>> Thanks,
>>>
>>> Aaron
>>>
>>>
>>> On Mon, Oct 17, 2022 at 4:40 PM Cheng Wang  wrote:
>>>
>>>> Another approach is, 

Re: Denylisting with a composite partition key

[Aaron Ploetz]

Just checking, but for this to work, do I have to mess with these settings
in the YAML at all?

partition_denylist_enabled: true
denylist_reads_enabled: true

They're commented out by default.

Thanks,

Aaron


On Mon, Oct 17, 2022 at 4:53 PM Aaron Ploetz  wrote:

> Thanks for the help with the INSERT, Cheng!  I'm further along than
> before.  But it still must not be matching up quite right, because I can
> still select that partition.
>
> I have several different combinations of the two keys (and I removed the
> space) of "Minneapolis,MN" and 202210.  Here's what I've got out there so
> far:
>
> aaron@cqlsh:stackoverflow> select ks_name, table_name, blobAsText(key)
> from system_distributed.partition_denylist;
>
>  ks_name   | table_name  | system.blobastext(key)
> ---+-+--
>  stackoverflow | weather_sensor_data | 'Minneapolis,MN', 202210
>  stackoverflow | weather_sensor_data |  'Minneapolis,MN',202210
>  stackoverflow | weather_sensor_data |  'Minneapolis,MN':202210
>  stackoverflow | weather_sensor_data |   Minneapolis,MN, 202210
>  stackoverflow | weather_sensor_data | Minneapolis,MN202210
>  stackoverflow | weather_sensor_data |Minneapolis,MN:202210
>
> (6 rows)
>
> aaron@cqlsh:stackoverflow> SELECT * FROM weather_sensor_data WHERE
> city='Minneapolis,MN' AND month=202210;
>
>  city   | month  | recorded_time   | temp
> ++-+--
>  Minneapolis,MN | 202210 | 2022-10-17 11:30:00.00+ |1
>  Minneapolis,MN | 202210 | 2022-10-17 11:25:00.00+ |1
>  Minneapolis,MN | 202210 | 2022-10-17 11:20:00.00+ |1
>  Minneapolis,MN | 202210 | 2022-10-17 11:15:00.00+ |1
>  Minneapolis,MN | 202210 | 2022-10-17 11:10:00.00+ |2
>  Minneapolis,MN | 202210 | 2022-10-17 11:05:00.00+ |2
>  Minneapolis,MN | 202210 | 2022-10-17 11:00:00.00+ |2
>
> (7 rows)
>
> As you can see, I can still select the partition.  I was really hoping one
> of those combinations would do it.
>
> Looking at the StorageProxyTest.java in the project, I saw that it was
> delimited by a colon ":", which is why I tried that, too.
>
> Still looking for the right way to enter both of those keys.
>
> Thanks,
>
> Aaron
>
>
> On Mon, Oct 17, 2022 at 4:40 PM Cheng Wang  wrote:
>
>> Another approach is, instead of using $$, you can put additional pair of
>> single quote around the 'Minneapolis, MN'
>>
>> cqlsh> insert into system_distributed.partition_denylist (ks_name,
>> table_name, key) values ('stackoverflow', 'weather_sensor_data',
>> textAsBlob('''Minneapolis, MN'', 202210'));
>>
>> cqlsh> select ks_name, table_name, blobAsText(key) from
>> system_distributed.partition_denylist;
>>
>>
>>  ks_name   | table_name  | system.blobastext(key)
>>
>> ---+-+---
>>
>>  stackoverflow | weather_sensor_data | 'Minneapolis, MN', 202210
>>
>> On Mon, Oct 17, 2022 at 2:30 PM Cheng Wang  wrote:
>>
>>> Hi Aaron,
>>>
>>> Yes, you can directly insert into the system_distributed.partition_denylist
>>> instead of using JMX. Jordan wrote a blog post for denylist
>>>
>>> https://cassandra.apache.org/_/blog/Apache-Cassandra-4.1-Denylisting-Partitions.html
>>>
>>> And the syntax error, one way around is to put $$ around like below
>>>
>>> cqlsh> insert into system_distributed.partition_denylist (ks_name,
>>> table_name, key) values ('stackoverflow', 'weather_sensor_data',
>>> textAsBlob($$'Minneapolis, MN', 202210$$));
>>>
>>> There is post about this
>>>
>>> https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/escape_char_r.html#:~:text=Column%20names%20that%20contain%20characters,using%20a%20single%20quotation%20mark
>>> .
>>>
>>> and then you can verify the insert by doing
>>>
>>> cqlsh> select ks_name, table_name, blobAsText(key) from
>>> system_distributed.partition_denylist;
>>>
>>>
>>>  ks_name   | table_name  | system.blobastext(key)
>>>
>>> ---+-+---
>>>
>>>  stackoverflow | weather_sensor_data | 'Minneapolis, MN', 202210
>>> Pls let me know if it works for you.
>>>
>>>
>>> On Mon, Oct 17, 2022 at 1:35 PM Aaron Ploetz 
>>> wrote:
>>>
>>>> I have this table definition:
>

Re: Denylisting with a composite partition key

[Aaron Ploetz]

Thanks for the help with the INSERT, Cheng!  I'm further along than
before.  But it still must not be matching up quite right, because I can
still select that partition.

I have several different combinations of the two keys (and I removed the
space) of "Minneapolis,MN" and 202210.  Here's what I've got out there so
far:

aaron@cqlsh:stackoverflow> select ks_name, table_name, blobAsText(key) from
system_distributed.partition_denylist;

 ks_name   | table_name  | system.blobastext(key)
---+-+--
 stackoverflow | weather_sensor_data | 'Minneapolis,MN', 202210
 stackoverflow | weather_sensor_data |  'Minneapolis,MN',202210
 stackoverflow | weather_sensor_data |  'Minneapolis,MN':202210
 stackoverflow | weather_sensor_data |   Minneapolis,MN, 202210
 stackoverflow | weather_sensor_data | Minneapolis,MN202210
 stackoverflow | weather_sensor_data |Minneapolis,MN:202210

(6 rows)

aaron@cqlsh:stackoverflow> SELECT * FROM weather_sensor_data WHERE
city='Minneapolis,MN' AND month=202210;

 city   | month  | recorded_time   | temp
++-+--
 Minneapolis,MN | 202210 | 2022-10-17 11:30:00.00+ |1
 Minneapolis,MN | 202210 | 2022-10-17 11:25:00.00+ |1
 Minneapolis,MN | 202210 | 2022-10-17 11:20:00.00+ |1
 Minneapolis,MN | 202210 | 2022-10-17 11:15:00.00+ |1
 Minneapolis,MN | 202210 | 2022-10-17 11:10:00.00+ |2
 Minneapolis,MN | 202210 | 2022-10-17 11:05:00.00+ |2
 Minneapolis,MN | 202210 | 2022-10-17 11:00:00.00+ |2

(7 rows)

As you can see, I can still select the partition.  I was really hoping one
of those combinations would do it.

Looking at the StorageProxyTest.java in the project, I saw that it was
delimited by a colon ":", which is why I tried that, too.

Still looking for the right way to enter both of those keys.

Thanks,

Aaron


On Mon, Oct 17, 2022 at 4:40 PM Cheng Wang  wrote:

> Another approach is, instead of using $$, you can put additional pair of
> single quote around the 'Minneapolis, MN'
>
> cqlsh> insert into system_distributed.partition_denylist (ks_name,
> table_name, key) values ('stackoverflow', 'weather_sensor_data',
> textAsBlob('''Minneapolis, MN'', 202210'));
>
> cqlsh> select ks_name, table_name, blobAsText(key) from
> system_distributed.partition_denylist;
>
>  ks_name   | table_name  | system.blobastext(key)
>
> ---+-+---
>
>  stackoverflow | weather_sensor_data | 'Minneapolis, MN', 202210
>
> On Mon, Oct 17, 2022 at 2:30 PM Cheng Wang  wrote:
>
>> Hi Aaron,
>>
>> Yes, you can directly insert into the system_distributed.partition_denylist
>> instead of using JMX. Jordan wrote a blog post for denylist
>>
>> https://cassandra.apache.org/_/blog/Apache-Cassandra-4.1-Denylisting-Partitions.html
>>
>> And the syntax error, one way around is to put $$ around like below
>>
>> cqlsh> insert into system_distributed.partition_denylist (ks_name,
>> table_name, key) values ('stackoverflow', 'weather_sensor_data',
>> textAsBlob($$'Minneapolis, MN', 202210$$));
>>
>> There is post about this
>>
>> https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/escape_char_r.html#:~:text=Column%20names%20that%20contain%20characters,using%20a%20single%20quotation%20mark
>> .
>>
>> and then you can verify the insert by doing
>>
>> cqlsh> select ks_name, table_name, blobAsText(key) from
>> system_distributed.partition_denylist;
>>
>>
>>  ks_name   | table_name  | system.blobastext(key)
>>
>> ---+-+---
>>
>>  stackoverflow | weather_sensor_data | 'Minneapolis, MN', 202210
>> Pls let me know if it works for you.
>>
>>
>> On Mon, Oct 17, 2022 at 1:35 PM Aaron Ploetz 
>> wrote:
>>
>>> I have this table definition:
>>>
>>> CREATE TABLE stackoverflow.weather_sensor_data (
>>> city text,
>>> month int,
>>> recorded_time timestamp,
>>> temp float,
>>> PRIMARY KEY ((city, month), recorded_time)
>>> ) WITH CLUSTERING ORDER BY (recorded_time DESC)
>>>
>>> Sample data looks like this:
>>>
>>> > SELECT * FROM weather_sensor_data WHERE city='Minneapolis, MN' AND
>>> month=202111;
>>>
>>>  city| month  | recorded_time   | temp
>>> -++-+--
>>>  Minneapolis, MN | 202111 | 2021-11-01 

Denylisting with a composite partition key

[Aaron Ploetz]

I have this table definition:

CREATE TABLE stackoverflow.weather_sensor_data (
city text,
month int,
recorded_time timestamp,
temp float,
PRIMARY KEY ((city, month), recorded_time)
) WITH CLUSTERING ORDER BY (recorded_time DESC)

Sample data looks like this:

> SELECT * FROM weather_sensor_data WHERE city='Minneapolis, MN' AND
month=202111;

 city| month  | recorded_time   | temp
-++-+--
 Minneapolis, MN | 202111 | 2021-11-01 08:35:00.00+ |3
 Minneapolis, MN | 202111 | 2021-11-01 08:30:00.00+ |3
 Minneapolis, MN | 202111 | 2021-11-01 08:25:00.00+ |2
 Minneapolis, MN | 202111 | 2021-11-01 08:20:00.00+ |2
 Minneapolis, MN | 202111 | 2021-11-01 08:15:00.00+ |2

(5 rows)

Using JMX Term, I've tried to denylist that partition, but I must have the
syntax for composite keys incorrect:

$>bean org.apache.cassandra.db:type=StorageProxy
$>run denylistKey stackoverflow weather_sensor_data "'Minneapolis,
MN',202210"
#IllegalArgumentException: Operation denylistKey with 4 parameters doesn't
exist in bean org.apache.cassandra.db:type=StorageProxy

Obviously, it's reading the space between "Minneapolis," and "MN" as a
delimiter.  What's the right way to handle commas, spaces, and composite
keys for this?

Also, is there another way to accomplish this without using JMX?

Thanks,

Aaron

Apache Cassandra(R) Corner podcast - Call for guests

[Aaron Ploetz]

Just wanted to reach out to the folks on this list.  The Apache Cassandra
Corner® podcast is all about sharing and discussing some of the great use
cases out there, as well as the infrastructure engineering which happens
around it.  If you are using or supporting Apache Cassandra®, I'd love to
have you on as a guest.

Feel free to reach out to me at either address:

aaronplo...@gmail.com
aaron.plo...@datastax.com

Thanks!

Aaron

The Apache Cassandra(R) Corner Podcast

[Aaron Ploetz]

How does open source marketing work and how does it help the Apache
Cassandra® project? Listen in as Constantia's Melissa Logan and I talk
about open source, Cassandra, and the Data on Kubernetes community.

https://anchor.fm/cassandra-corner/episodes/ep5---Melissa-Logan-Constantia-e1kgp40

The Apache Cassandra® Corner podcast is available on most major podcast
platforms. It is intended to highlight cool use cases for Cassandra as well
as the folks in the community who implement them. If you would like to be a
guest on the podcast, please reach out to me!

Thanks,

Aaron

Re: Guardrails in Cassandra 4.1 Alpha

[Aaron Ploetz]

Ahh...yes, my default "aaron" user is indeed a SUPERUSER.

Ok, so I created a new, non-superuser and tried again...

> SELECT * FROm stackoverflow.movies WHERE title='Sneakers (1992)' ALLOW
FILTERING;
InvalidRequest: Error from server: code=2200 [Invalid query]
message="Guardrail allow_filtering violated: Querying with ALLOW FILTERING
is not allowed"

Thank you for the quick response, Andres!

On Thu, Jun 23, 2022 at 2:14 PM Andrés de la Peña 
wrote:

> Hi Aaron,
>
> Guardrails are not applied to superusers. The default user is a superuser,
> so to see guardrails in action you need to create and use a user that is
> not a superuser.
>
> You can do that by setting, for example, these properties on
> cassandra.yaml:
>
> authenticator: PasswordAuthenticator
> authorizer: CassandraAuthorizer
>
> Then you can login with cqlsh using the default superuser and create a
> regular user with the adequate permissions. For example:
>
> bin/cqlsh -u cassandra -p cassandra
> > CREATE USER test WITH PASSWORD 'test';
> > GRANT SELECT ON ALL KEYSPACES TO test;
> bin/cqlsh -u test -p test
> > SELECT * FROM stackoverflow.movies WHERE title='Sneakers (1992)' ALLOW
> FILTERING;
> InvalidRequest: Error from server: code=2200 [Invalid query]
> message="Guardrail allow_filtering violated: Querying with ALLOW FILTERING
> is not allowed"
>
> Finally, that particular guardrail isn't applied to system tables, so it
> would still allow filtering on the system.local and system_views.settings
> tables, but not in stackoverflow.movies.
>
> I hope this helps.
>
> On Thu, 23 Jun 2022 at 19:51, Aaron Ploetz  wrote:
>
>> So I'm trying to test out the guardrails in 4.1-alpha.  I've set
>> allow_filtering_enabled: false, but it doesn't seem to care (I can still
>> use it).
>>
>> > SELECT release_version FROM system.local;
>>  release_version
>> -
>>  4.1-alpha1-SNAPSHOT
>>
>> (1 rows)
>>
>> > SELECT * FROM system_views.settings WHERE
>> name='allow_filtering_enabled';
>>  name| value
>> -+---
>>  allow_filtering_enabled | false
>>
>> (1 rows)
>>
>> > SELECT * FROm stackoverflow.movies WHERE title='Sneakers (1992)' ALLOW
>> FILTERING;
>>  id   | genre  | title
>> --++-
>>  1396 | Crime|Drama|Sci-Fi | Sneakers (1992)
>>
>> (1 rows)
>>
>> Is there like some main "guardrails enabled" setting that I missed?
>>
>> Thanks,
>>
>> Aaron
>>
>>

Guardrails in Cassandra 4.1 Alpha

[Aaron Ploetz]

So I'm trying to test out the guardrails in 4.1-alpha.  I've set
allow_filtering_enabled: false, but it doesn't seem to care (I can still
use it).

> SELECT release_version FROM system.local;
 release_version
-
 4.1-alpha1-SNAPSHOT

(1 rows)

> SELECT * FROM system_views.settings WHERE name='allow_filtering_enabled';
 name| value
-+---
 allow_filtering_enabled | false

(1 rows)

> SELECT * FROm stackoverflow.movies WHERE title='Sneakers (1992)' ALLOW
FILTERING;
 id   | genre  | title
--++-
 1396 | Crime|Drama|Sci-Fi | Sneakers (1992)

(1 rows)

Is there like some main "guardrails enabled" setting that I missed?

Thanks,

Aaron

new Cassandra Corner episode w/ Patrick McFadin

[Aaron Ploetz]

Nothing to see here...because well, it's a podcast, right? But I did get to
spend some time with Patrick McFadin discussing our favorite database -
Apache Cassandra! Listen in as we talk a little about the early days of
Cassandra as well as the new, upcoming Cassandra 4.1 release.

https://anchor.fm/cassandra-corner/episodes/ep4---Patrick-McFadin-Apache-Cassandra-4-1-e1k8k1l

The Cassandra Corner podcast is available on Anchor, Apple Podcasts,
Spotify, Google Podcasts and Stitcher. It is intended to highlight cool use
cases for Cassandra as well as the folks in the community who implement
them. If you would like to be a guest on the podcast, please reach out to
me!

Re: Topology vs RackDC

[Aaron Ploetz]

Agree with Paulo on this one.

If you're running a cluster on K8s, in the cloud, or under some other
conditions in which IPs change, the cassandra-topology.properties file is
going to quickly become a burden.  Especially if that cluster has more than
just a few nodes.

Increasing your cluster from 99 nodes to 120?  Need to make sure that
updated file ends up on all 120 nodes, with all 120 IPs.  Also, I'm pretty
sure that a node can't read the new file without a restart.  So now you're
talking about bouncing a 100+ node cluster just to add new nodes.  Same
with replacing a single node or IP.  Update the file, push it out to all
nodes, and restart all nodes.

Basically, the cassandra-rackdc.properties file is preferred because it
makes operations easier at scale.  Config files that change less make
everyone happier!

At my last gig, we built the propagation of the cassandra-rackdc.properties
file into our node-add automation.  That way it looks at the target
instance, and already knows which DC and rack to inject into the file.

Thanks,

Aaron

P.S. - When using the cassandra-rackdc.properties file, the prevailing
production wisdom has always been to delete the
cassandra-topology.properties file to keep it from interfering.  Not sure
if that's still an issue or not, but it was in the 3.x versions.


On Thu, Jun 2, 2022 at 7:59 AM Paulo Motta  wrote:

> It think topology file is better for static clusters, while rackdc for
> dynamic clusters where users can add/remove hosts without needing to update
> the topology file on all hosts.
>
> On Thu, 2 Jun 2022 at 09:13 Marc Hoppins  wrote:
>
>> Hi all,
>>
>> Why is RACKDC preferred for production than TOPOLOGY?
>>
>> Surely one common file is far simpler to distribute than deal with the
>> mucky-muck of various configs for each host if they are in one rack or
>> another and/or one datacentre or another?  It is also fairly
>> self-documenting of the setup with the entire cluster there in one file.
>>
>> From what I read in the documentation, regardless of which snitch one
>> implements, cassandra-topology.properties will get read, either as a
>> primary or as a backup...so why not just use topology for ALL cases?
>>
>> Thanks
>>
>> Marc
>>
>

Re: The "Cassandra Corner" podcast

[Aaron Ploetz]

Maxn,

Ah!  That's because I had to explicitly enable RSS in the settings.  Doh!

Give that URL another shot and let me know if it doesn't work.

Thanks,

Aaron


On Wed, Jun 1, 2022 at 12:54 PM Max Campos  wrote:

> Hi Aaron,
>
> Do you have the RSS feed URL handy?  I looked at the source of the page,
> and it shows this:
>
>  title="The Cassandra Corner"  
> href="https://anchor.fm/s/99c5e268/podcast/rss;/>
>
> … which leads to a 404.  Thanks.
>
> - Max
>
> On Jun 1, 2022, at 6:55 am, Aaron Ploetz  wrote:
>
> We have our first episode of the "Cassandra Corner" available on the
> Anchor/Spotify platform!  A big thanks to Lakhan Samani of the Authorizer
> <https://docs.authorizer.dev/> project for being our first guest.
>
> If interested, you can listen to it here:
>
>
> https://anchor.fm/cassandra-corner/episodes/ep1---Lakhan-Samani-Authorizer-e1j10hv
>
> This podcast is intended to highlight cool use cases for Cassandra and the
> folks in the Apache Cassandra community who implement them.  If you would
> like to be a guest on the podcast, please reach out to me.
>
> Thanks everyone,
>
> Aaron Ploetz
> aaron.plo...@datastax.com
>
>
>

The "Cassandra Corner" podcast

[Aaron Ploetz]

We have our first episode of the "Cassandra Corner" available on the
Anchor/Spotify platform!  A big thanks to Lakhan Samani of the Authorizer
<https://docs.authorizer.dev/> project for being our first guest.

If interested, you can listen to it here:

https://anchor.fm/cassandra-corner/episodes/ep1---Lakhan-Samani-Authorizer-e1j10hv

This podcast is intended to highlight cool use cases for Cassandra and the
folks in the Apache Cassandra community who implement them.  If you would
like to be a guest on the podcast, please reach out to me.

Thanks everyone,

Aaron Ploetz
aaron.plo...@datastax.com

Re: Cassandra tarball install and systemd

[Aaron Ploetz]

> I am starting Cassandra with $CASSANDRA_HOME/bin/cassandra

When starting Cassandra, it accepts a PID file location with the -p flag:

$CASSANDRA_HOME/bin/cassandra -p /var/run/cassandra/cassandra.pid

Start Cassandra with that, and then the PID file will be there.  Assuming
of course, that the user starting Cassandra has access to /var/run/casandra/
.

Aaron


On Wed, Feb 9, 2022 at 9:53 AM Bowen Song  wrote:

> Isn't the /etc/init.d/cassandra script supposed to create the PID file if
> it doesn't exist? See:
> https://github.com/apache/cassandra/blob/cb1c8f9d34edfa639096d2d122dfd0ee6d23b479/debian/init#L83
>
>
> On 09/02/2022 15:17, Saha, Sushanta K wrote:
>
> I picked up the script */etc/init.d/cassandra* from the net. Not sure if
> tarball installation includes such a script. It should.
>
> This script is using the following line:
> *pid_file=/var/run/cassandra/cassandra.pid*
>
> But, there is no such *.pid* file that I can find. I am starting
> Cassandra with $CASSANDRA_HOME/bin/cassandra.
>
> Appreciate any pointer for this auto stop and start of Cassandra.
>
> Thanks & Regards
>  Sushanta
>
> --
>
> *Sushanta Saha|*MTS IV-Cslt-Sys Engrg|WebIaaS_DB Group|HQ -
> * VerizonWireless O 770.797.1260  C 770.714.6555 Iaas Support Line
> 949-286-8810*
>
>

Re: update cassandra.yaml file on number of cluster nodes

[Aaron Ploetz]

When I was at Target, we had a lot of success with Rundeck.  It was a quick
way to push out changes to hundreds of clusters.
https://www.rundeck.com/open-source


On Mon, Oct 18, 2021 at 12:19 PM Yakir Gibraltar  wrote:

> We manage via Puppet:  https://github.com/voxpupuli/puppet-cassandra
>
> On Mon, Oct 18, 2021 at 7:19 PM Elliott Sims 
> wrote:
>
>> Ansible here as well with a similar setup.  A play at the end of the
>> playbook that waits until all nodes in the cluster are "UN" before moving
>> on to the next node to change.
>>
>> On Mon, Oct 18, 2021 at 10:01 AM vytenis silgalis 
>> wrote:
>>
>>> Yep, also use Ansible with configs living in git here.
>>>
>>> On Fri, Oct 15, 2021 at 5:19 PM Bowen Song  wrote:
>>>
 We have Cassandra on bare-metal servers, and we manage our servers via
 Ansible. In this use case, we create an Ansible playbook to update the
 servers one by one, change the cassandra.yaml file, restart Cassandra, and
 wait for it to finish the restart, and then wait for a few minutes before
 moving on to the next server.
 On 15/10/2021 22:42, ZAIDI, ASAD wrote:



 Hello Folks,



 Can you guys please suggest tool or approach  to update  cassandra.yaml
 file in multi-dc environment with large number of nodes efficiently.



 Thank you.

 Asad






>
> --
> *בברכה,*
> *יקיר גיברלטר*
>

Re: Vulnerability in libthrift library (CVE-2019-0205)

[Aaron Ploetz]

In reading the Jira ticket, I see this line:

"a server or client may run into an endless loop *when fed with specific
input data.*"

That seems to suggest that if the cassandra.yaml contains start_rpc: false,
you should be ok.

On Mon, Oct 4, 2021 at 8:12 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:

> We are not using thrift but the lib is there with Cassandra binary right?
> Does it cause any risk?
>
> On Mon, Oct 4, 2021, 5:53 PM Erick Ramirez 
> wrote:
>
>> See https://issues.apache.org/jira/browse/CASSANDRA-15420. It only
>> applies if you're still using Thrift in 2021. Cheers!
>>
>

Re: Use NetworkTopologyStrategy for single data center and add data centers later

[Aaron Ploetz]

Yes, you absolutely can (and should) use NetworkTopologyStrategy with a
single data center.  In fact, I would argue that SimpleStrategy should
almost never be used.  But that's just me.

Thanks,

Aaron


On Sat, Dec 19, 2020 at 3:21 AM Manu Chadha  wrote:

> Is it possible to use NetworkTopologyStrategy when creating a keyspace and
> add data centers later?
>
> I am just starting with an MVP application and I don't expect much
> traffic or data. Thus I have created only one data center. However, I'll
> like to add more data centers later if needed
>
> I notice that the replication factor for each data center needs to be
> specified at the time of keyspace creation
>
> CREATE KEYSPACE "Excalibur"
>
>   WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 
> 2};
>
> As I only have dc1 at the moment, could I just do
>
> CREATE KEYSPACE "Excalibur"
>
>   WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1' : 3};
>
> and when I have another datacenter say dc2, could I edit the Excalibur
>  keyspace?
>
> ALTER KEYSPACE "Excalibur"
>
>   WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc2' : 2};
>
>
>
> or can I start with SimpleStrategy now and change to
> NetworkTopologyStrategy later? I suspect this might not work as I think
> this needs changing snitch etc.
>
>
>
>
>
> Sent from Mail  for
> Windows 10
>
>
>

Re: Cassandra timeout during read query

[Aaron Ploetz]

Deepak,

Can you reply with:

1) The query you are trying to run.
2) The table definition (PRIMARY KEY, specifically).
3) Maybe a little description of what the table is designed to do.
4) How much data you're expecting returned (both # of rows and data size).

Thanks,

Aaron


On Mon, Sep 14, 2020 at 10:58 AM Deepak Sharma
 wrote:

> Hi There,
>
> We are running into a strange issue in our Cassandra Cluster where one
> specific query is failing with following error:
>
> Cassandra timeout during read query at consistency QUORUM (3 responses
> were required but only 0 replica responded)
>
> This is not a typical query read timeout that we know for sure. This error
> is getting spit out within 5 seconds and the query timeout we have set is
> around 30 seconds
>
> Can we know what is happening here and how can we reproduce this in our
> local environment?
>
> Thanks,
> Deepak
>
>

Re: Cassandra crashes when using offheap_objects for memtable_allocation_type

[Aaron Ploetz]

primary key ((partition_key, clustering_key))

Also, this primary key definition does not define a partitioning key and a
clustering key.  It defines a *composite* partition key.

If you want it to instantiate both a partition and clustering key, get rid
of one set of parens.

primary key (partition_key, clustering_key)


On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester
 wrote:

> I just changed these properties to increase flushed file size (decrease
> number of compactions):
>
>- memtable_allocation_type from heap_buffers to offheap_objects
>- memtable_offheap_space_in_mb: from default (2048) to 8192
>
> Using default value for other memtable/compaction/commitlog configurations
> .
>
> After a few hours some of nodes stopped to do any mutations (dropped
> mutaion increased) and also pending flushes increased, they were just up
> and running and there was only a single CPU core with 100% usage(other
> cores was 0%). other nodes on the cluster determines the node as DN. Could
> not access 7199 and also could not create thread dump even with jstack -F.
>
> Restarting Cassandra service fixes the problem but after a while some
> other node would be DN.
>
> Am i missing some configurations?  What should i change in cassandra
> default configuration to maximize write throughput in single node/cluster
> in write-heavy scenario for the data model:
> Data mode is a single table:
>   create table test(
>   text partition_key,
>   text clustering_key,
>   set rows,
>   primary key ((partition_key, clustering_key))
>
>
> vCPU: 12
> Memory: 32GB
> Node data size: 2TB
> Apache cassandra 3.11.2
> JVM heap size: 16GB, CMS, 1GB newgen
>
> Sent using Zoho Mail 
>
>
>
>

Re: Cassandra crashes when using offheap_objects for memtable_allocation_type

[Aaron Ploetz]

I would try running it with memtable_offheap_space_in_mb at the default for
sure, but definitely lower than 8GB.  With 32GB of RAM, you're already
allocating half of that for your heap, and then halving the remainder for
off heap memtables.  What's left may not be enough for the OS, etc.  Giving
some of that back, will allow more to be used for page cache, which always
helps.

"JVM heap size: 16GB, CMS, 1GB newgen"

For CMS GC with a 16GB heap, 1GB is way too small for new gen.  You're
going to want that to be at least 40% of the max heap size.  Some folks
here even advocate for setting Xmn as high as 50% of Xmx/s.

If you want to stick with CMS GC, take a look at
https://issues.apache.org/jira/browse/CASSANDRA-8150.  There's plenty of
good info in there on CMS GC tuning.  Make sure to read through the whole
ticket, so that you understand what each setting does.  You can't just
pick-and-choose.

Regards,

Aaron


On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester
 wrote:

> I just changed these properties to increase flushed file size (decrease
> number of compactions):
>
>- memtable_allocation_type from heap_buffers to offheap_objects
>- memtable_offheap_space_in_mb: from default (2048) to 8192
>
> Using default value for other memtable/compaction/commitlog configurations
> .
>
> After a few hours some of nodes stopped to do any mutations (dropped
> mutaion increased) and also pending flushes increased, they were just up
> and running and there was only a single CPU core with 100% usage(other
> cores was 0%). other nodes on the cluster determines the node as DN. Could
> not access 7199 and also could not create thread dump even with jstack -F.
>
> Restarting Cassandra service fixes the problem but after a while some
> other node would be DN.
>
> Am i missing some configurations?  What should i change in cassandra
> default configuration to maximize write throughput in single node/cluster
> in write-heavy scenario for the data model:
> Data mode is a single table:
>   create table test(
>   text partition_key,
>   text clustering_key,
>   set rows,
>   primary key ((partition_key, clustering_key))
>
>
> vCPU: 12
> Memory: 32GB
> Node data size: 2TB
> Apache cassandra 3.11.2
> JVM heap size: 16GB, CMS, 1GB newgen
>
> Sent using Zoho Mail 
>
>
>
>

Re: Cassandra didn't order data according to clustering order

[Aaron Ploetz]

It also helps to think about it with the token values of the partition key
in place.  Assume I have a table "users_by_dept" keyed like this: PRIMARY
KEY ((department),username).

Querying that table with the token function on the partition key looks like
this:

SELECT token(department),department,username,email FROM users_by_dept ;

 system.token(department) | department  | username | email
--+-+--+--
 -8838453544589358145 | Engineering |   Dinesh |
din...@piedpiper.com
 -8838453544589358145 | Engineering | Gilfoyle |
thedark...@piedpiper.com
 -8838453544589358145 | Engineering |  Richard |
rich...@piedpiper.com
 -4463195796437695053 |   Marketing |   Erlich |
erl...@piedpiper.com
 -3086103490616810985 |  Finance/HR |Jared |
don...@piedpiper.com

(5 rows)

As you can see, department doesn't appear to be in any discernable order
until you apply the token function to it.

Aaron


On Sun, Jul 15, 2018 at 8:05 AM, Soheil Pourbafrani 
wrote:

> To the point, Thanks!
>
> On Sun, Jul 15, 2018 at 4:31 PM, shalom sagges 
> wrote:
>
>> The clustering column is ordered per partition key.
>>
>> So if for example I create the following table:
>> create table desc_test (
>>id text,
>>name text,
>>PRIMARY KEY (id,name)
>> ) WITH CLUSTERING ORDER BY (name DESC );
>>
>>
>> I insert a few rows:
>>
>> insert into desc_test (id , name ) VALUES ( 'abc', 'abc');
>> insert into desc_test (id , name ) VALUES ( 'abc', 'bcd');
>> insert into desc_test (id , name ) VALUES ( 'abc', 'aaa');
>> insert into desc_test (id , name ) VALUES ( 'fgh', 'aaa');
>> insert into desc_test (id , name ) VALUES ( 'fgh', 'bcd');
>> insert into desc_test (id , name ) VALUES ( 'fgh', 'abc');
>>
>>
>> And then read:
>> select * from desc_test;
>>
>>  id  | name
>> -+--
>>   fgh |  bcd
>>   fgh |  abc
>>   fgh |  aaa
>>  abc |  bcd
>>  abc |  abc
>>  abc |  aaa
>>
>> (6 rows)
>>
>>
>> You can see that the data is properly ordered in descending mode, BUT
>> *for each partition key. *
>> So in order to achieve what you want, you will have to add the relevant
>> partition key for each select query.
>>
>> Hope this helps
>>
>>
>> On Sun, Jul 15, 2018 at 2:16 PM, Soheil Pourbafrani <
>> soheil.i...@gmail.com> wrote:
>>
>>> I created table using the command:
>>> CREATE TABLE correlated_data (
>>> processing_timestamp bigint,
>>> generating_timestamp bigint,
>>> data text,
>>> PRIMARY KEY (processing_timestamp, generating_timestamp)
>>> ) WITH CLUSTERING ORDER BY (generating_timestamp DESC);
>>>
>>>
>>> When I get data using the command :
>>> SELECT * FROM correlated_data LIMIT 1 ;
>>>
>>> I expect it return the row with the biggest field "generating_timestamp",
>>> but I got the same row every time I run the query, while row with bigger "
>>> generating_timestamp" exists. What's the problem?
>>>
>>
>>
>

Re: Openstack and Cassandra

[Aaron Ploetz]

Shalom,

We (Target) have been challenged by our management team to leverage
OpenStack whenever possible, and that includes Cassandra.  I was against it
at first, but we have done some stress testing with it and had application
teams try it out.  So far, there haven't been any issues.

A good use case for Cassandra on OpenStack, is to support an
internal-facing application that needs to scale for disk footprint, or to
spin-up a quick dev environment.  When building clusters to support those
solutions, we haven't had any problems due to simply deploying on
OpenStack.  Our largest Cassandra cluster on OpenStack is currently around
30 nodes.  OpenStack is a good solution for that particular use case as we
can easily add/remove nodes to accommodate the dynamic disk usage
requirements.

However, when query latency is a primary concern, I do still recommend that
we use one of our external cloud providers.

Hope that helps,

Aaron

On Thu, Dec 22, 2016 at 9:51 AM, Shalom Sagges 
wrote:

> Thanks Vladimir!
>
> I guess I'll just have to deploy and continue from there.
>
>
>
>
> Shalom Sagges
> DBA
> T: +972-74-700-4035 <+972%2074-700-4035>
>  
>  We Create Meaningful Connections
> 
>
>
> On Thu, Dec 22, 2016 at 5:20 PM, Vladimir Yudovin 
> wrote:
>
>> Hi Shalom,
>>
>> I don't see any reason why it wouldn't work,  but obviously, any resource
>> sharing affects performance. You can expect less degradation with SSD
>> disks, I guess.
>>
>>
>> Best regards, Vladimir Yudovin,
>> *Winguzone  - Cloud Cassandra Hosting*
>>
>>
>>  On Wed, 21 Dec 2016 13:31:22 -0500 *Shalom Sagges
>> >* wrote 
>>
>> Hi Everyone,
>>
>> I am looking into the option of deploying a Cassandra cluster on
>> Openstack nodes instead of physical nodes due to resource management
>> considerations.
>>
>> Does anyone has any insights regarding this?
>> Can this combination work properly?
>> Since the disks (HDDs) are part of one physical machine that divide their
>> capacity to various instances (not only Cassandra), will this affect
>> performance, especially when the commitlog directory will probably reside
>> with the data directory?
>>
>> I'm at a loss here and don't have any answers for that matter.
>>
>> Can anyone assist please?
>>
>> Thanks!
>>
>>
>>
>>
>> Shalom Sagges
>> DBA
>> T: +972-74-700-4035 <+972%2074-700-4035>
>> 
>> 
>> 
>> We Create Meaningful Connections
>>
>>
>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>

Re: Question about hector api documentation

[Aaron Ploetz]

I used to be surprised that people still ask about Hector here; and that
questions here on Hector always seem to mirror new Hector questions on
Stack Overflow.  The problem (I think), is that places like Edureka! are
still charging people $300 for a Cassandra training class, where they still
actively teach people to use Hector:

http://www.edureka.co/cassandra-course-curriculum

Lesson to be learned, not all online training resources are created equally.

New development with a framework that's been dead for 2 years isn't the
best way to ensure success.  Stick with the DataStax Java driver.

Aaron


On Fri, Jun 24, 2016 at 11:09 AM, Jonathan Haddad  wrote:

> +1, do not use Hector. It hasn't had a commit in years and uses the thrift
> protocol which is now marked deprecated. The DataStax Java driver is
> recommended, possibly with Achilles to make things a bit nicer.
> On Thu, Jun 23, 2016 at 9:20 PM Noorul Islam K M 
> wrote:
>
>>
>> The very first line README tells the story
>>
>>
>> THIS PROJECT IS NO LONGER ACTIVE
>>
>>
>> But you should be able to generate doc from source code.
>>
>> Regards,
>> Noorul
>>
>>
>> Sungju Hong  writes:
>>
>> > Hello,
>> >
>> > I'm finding hector java api doc.
>> >
>> > I searched though google but couldn't find hector api doc.
>> >
>> > This link is broken also.
>> > https://hector-client.github.io/hector/build/html/content/api.html#
>> >
>> > Can I know the way to get the doc?
>> >
>> > Thanks.
>> >
>> > Sungju.
>>
>

Re: Detect Repair is finish

[Aaron Ploetz]

You can do this with two nodetool commands:
nodetool compactionstats will check on active Merkle Tree calculations.

nodetool netstats will list out any active repair streams.

Aaron Morton posted a Bash script/command a while back that you can just
paste into the (linux) command line to monitor active streams for you:

while true; do date; diff (nodetool -h localhost netstats) (sleep 5
 nodetool -h localhost netstats); done

Hope this helps,

Aaron Ploetz


On Mon, Jun 22, 2015 at 8:40 AM, Jean Tremblay 
jean.tremb...@zen-innovations.com wrote:

 Hi,

 What is the best way to see if a repair is finished? Is there a JMX object
 or is there a command to see if a repair is finished?
 What happens if by mistake an operator starts a repair before the previous
 is not yet finished? Will they execute both one after the other or at the
 same time?

 Thanks for your help.

 Jean