It's actually correct to do it how it is today.
Insertion date does not matter, what matters is the time after tombstones
are supposed to be deleted.
If the delete got to all nodes, sure, no problem, but if any of the nodes
didn't get the delete, and you would get rid of the tombstones before
That's not how gc_grace_seconds work.
gc_grace_seconds controls how much time *after* a tombstone can be deleted,
it can actually be deleted, in order to give you enough time to run repairs.
Say you have data that is about to expire on March 16 8am, and
gc_grace_seconds is 10 days.
After Mar 16
the old table into a single row in the new table, the
> read speed should be much faster. Keep in mind that this may not work if
> the partition size of the original table is too large (approximately
> >16MB), as the mutation size is limited to up to half of the commitlog
>
f you
> want it to be faster, you can store the number of rows elsewhere if that's
> the only thing you need.
> On 11/04/2023 07:13, Gil Ganz wrote:
>
> Hey
> I have a 4.0.4 cluster, with reads of partitions that are a bit on the
> bigger side, taking longer than I would expect.
> Read
Hey
I have a 4.0.4 cluster, with reads of partitions that are a bit on the
bigger side, taking longer than I would expect.
Reading entire partition that has ~7 rows, total partition size of 4mb,
takes 120ms, I would expect it to take less.
This is after major compaction, so there is only one
> - Index build/rebuild
> - Streaming (repair, bootstrap, move, decom)
>
> If you have repairs running, you can try pausing/cancelling them and/or
> stopping validation/index_build compactions.
>
>
>
> On Tue, Apr 4, 2023 at 2:29 PM Gil Ganz wrote:
>
>> If i
.
>
>
>
> On Tue, Apr 4, 2023 at 1:55 PM Gil Ganz wrote:
>
>> More information - from another node in the cluster
>>
>> I can see many txn files although I only have two compactions running.
>> [user@server808 new_table-44263b406bf111ed8bd9
80ace3677a/nb-10334-big-Data.db
(deleted)
I will add that this cluster contains a single table with twcs.
gil
On Tue, Apr 4, 2023 at 10:14 AM Gil Ganz wrote:
> Hey
> I have a 4.0.7 cluster in which I see weird behavior.
> I expect that once compaction finishes, the old ssta
Hey
I have a 4.0.7 cluster in which I see weird behavior.
I expect that once compaction finishes, the old sstables that were part of
the compaction set will be deleted, but it appears they are deleted much
later, thus causing space issues.
For example this is what I have in the log, only one
Thanks Erick, indeed with curl with the redirect flag I can see the file
there.
On Mon, Oct 24, 2022 at 8:21 AM Erick Ramirez
wrote:
> redhat.cassandra.apache.org/40x/ redirects to
> apache.jfrog.io/artifactory/cassandra-rpm/40x/. When I curl it on the
> command line, I can see that the
Hey
Seems like the following release of 4.0.7 a few hours ago ago, repo
settings were a bit changed.
Where can one download the 4.0.6 cassandra-tools rpm from?
It's not in https://apache.jfrog.io/ui/native/cassandra-rpm/40x/ and
https://redhat.cassandra.apache.org/40x/ points to a login page with
Hey
Do reads that come from a read repair are somehow counted as part of the
local read metric?
i.e
org.apache.cassandra.metrics.Table... :
ReadLatency.1m_rate
Version is 4.0.4
Gil
hey do.
>
>
>
> Similarly, when the cluster was created, I added the seeds nodes in
> numerically ascending order and then the other nodes in a similar fashion.
> So why doesn’t nodetool display the status in that same order?
>
>
>
> *From:* Gil Ganz
> *Sent:* Monday,
I can understand why the number of nodes sending at once might be
interesting somehow, but why would the order of the nodes matter?
On Fri, Sep 9, 2022 at 10:27 AM Marc Hoppins wrote:
> Curiosity as to which data/node starts first, what determines the delivery
> sequence, how many nodes send
except stopping
>> big/unnecessary compactions manually (using nodetool stop) whenever they
>> appears by some shell scrips (using crontab)
>>
>> Sent using Zoho Mail <https://www.zoho.com/mail/>
>>
>>
>>
>> On Fri, 02 Sep 2022 10:59:22 +04
Hey
When deciding which sstables to compact together, how is the priority
determined between tasks, and can I do something about it?
In some cases (mostly after removing a node), it takes a while for
compactions to keep up with the new data the came from removed nodes, and I
see it is busy on
Reason I would like to suppress it is I think this is due to network
disconnects we know are happening, and looks like it's not going to change.
Since it doesn't happen that often, and not causing a real issue , I would
like to have cleaner logs if possible.
On Thu, Sep 1, 2022 at 9:55 AM Erick
Hey
We have an issue in few of our 4.0.4 clusters, these are on-prem, multiple
datacenters around the world clusters, and our logs have many errors like
this :
ERROR [Messaging-EventLoop-3-26] 2022-09-01 05:57:28,142
InboundMessageHandler.java:300 -
as a regression).
>
>
>
> On Tue, Jun 7, 2022 at 7:52 AM Gil Ganz wrote:
>
>> Yes, I know the issue with the peers table, we had it in different
>> clusters, in this case it appears the cause of the problem was indeed a bad
>> ip in the seed list.
>> After removing it
ete from
> system.peers_v2 where peer=..." to fix it, as on our client side, the
> Python cassandra-driver, reads the token ring information from this table
> and uses it for routing requests.
> On 07/06/2022 05:22, Gil Ganz wrote:
>
> Only errors I see in the logs prior to goss
however I'm not aware of any open issues in 4.0.4
> that would result in this.
>
> Would be eager to investigate immediately if so.
>
> – Scott
>
> On Jun 6, 2022, at 11:04 AM, Gil Ganz wrote:
>
>
> Hey
> We have a big cluster (>500 nodes, onprem, multi
Hey
We have a big cluster (>500 nodes, onprem, multiple datacenters, most with
vnodes=32, but some with 128), that was recently upgraded from 3.11.9 to
4.0.4. Servers are all centos 7.
We have been dealing with a few issues related to gossip since :
1 - The moment the last node in the cluster was
Feb 6, 2022, at 12:52 AM, Gil Ganz wrote:
>
>
> Hey
> I'm trying to enable full query log on cassandra 4.01 node and it's
> causing cassandra to shutdown
>
> nodetool enablefullquerylog --path /mnt/fql_data
>
> Cassandra has shutdown.
> error: null
Hey
I'm trying to enable full query log on cassandra 4.01 node and it's causing
cassandra to shutdown
nodetool enablefullquerylog --path /mnt/fql_data
Cassandra has shutdown.
error: null
-- StackTrace --
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
Looks like it's not happening, seen a couple of cases like this, in all
cases the files belonged to nodes that died and had to be removed, and I
can see this behaviour on all the nodes in the relevant data center.
All these nodes had to be removed with force (not by choice..), I wonder if
that
Hey
What should happen to hints that were generated for a node that has been
removed from the cluster, before hints were sent? I'm seeing old hint files
that are not handled, they are not sent anywhere and not being deleted.
Version is 3.11.9
gil
gc_grace_seconds set is the default 10 days (these tables do not have data
with ttl, or anything else that might create tombstones), and
max_hint_window_in_ms parameter is set to 18h.
I'm checking it by looking at both the hints directory and hints metrics,
and as far as check time, I took a node
Hey
I have a cluster, 3.11.9, in which I am enabling hints, but it appears that
no hints are created when other nodes are down.
I can see that hints are enabled by running "nodetool statushandoff", and
gc_grace_seconds is high enough on all tables, so I'm expecting to see
hints being created,
re you determining it's counters that are the
> problem? Is it stalling on the Initializing counters log line or something?
>
> raft.so - Cassandra consulting, support, and managed services
>
>
> On Mon, Apr 26, 2021 at 3:25 AM Gil Ganz wrote:
>
>> Hey
>> I have
Hey
I have a cluster, 3.11.6, startup is very slow, i3en.xlarge server with
about 1tb of data, takes 45 minutes to startup, almost 40 minutes of that
is loading the saved counter cache from disk (200mb), and I can see that in
these 40 minutes the amount of data read from disk is very high, up to
n their case.
>
> Is that your case too? Bigger RAM, more cores and higher CPU frequency to
> help "fix" the performance issue? I really hope not.
>
>
> On 11/03/2021 09:57, Gil Ganz wrote:
>
> Yes. 192gb.
>
> On Thu, Mar 11, 2021 at 10:29 AM Kane Wilson
>
Yes. 192gb.
On Thu, Mar 11, 2021 at 10:29 AM Kane Wilson wrote:
> That is a very large heap. I presume you are using G1GC? How much memory
> do your servers have?
>
> raft.so - Cassandra consulting, support, managed services
>
> On Thu., 11 Mar. 2021, 18:29 Gil Ganz, wr
re seeing is the increased NTR
> requests?
>
> raft.so - Cassandra consulting, support, and managed services
>
>
> On Mon, Mar 8, 2021 at 10:47 PM Gil Ganz wrote:
>
>>
>> Hey,
>> We have a 3.11.9 cluster (recently upgraded from 2.1.14), and after the
>
Hey,
We have a 3.11.9 cluster (recently upgraded from 2.1.14), and after the
upgrade we have an issue when we remove a node.
The moment I run the removenode command, 3 servers in the same dc start to
have a high amount of pending native-transport-requests (getting to around
1M) and clients are
I think the value of the r6gd (assuming cpu is good compared to intel) is
more cpu, not disk.
I'm not running on spark the cassandra servers, and having more cpu cores
in my cluster will sure help. It all depends on the workloads, some
workloads need more io, some cpu.
i3 servers are great
it's not the same, notice I wrote r6gd, these are the ones with nvme, i'm
looking just at those.
I do not need all the space that i3en gives me (and probably won't be able
to use it all due to memory usage, or have other issues just like you
mention), so the plan is use the big enough r6gd nodes,
Hey
Does anyone have experience with running cassandra in aws on arm servers?
Currently running on i3 servers and thinking about upgrading, looking at
i3en vs r6gd.
Obviously a lot of considerations here, but trying to understand what's the
verdict on these arm processors running cassandra.
Gil
Eric - I understand, thing is repairs are causing issues and I would like
to have the option to only run them on the tables that really need that.
Kane - Thanks, good idea, I will check that metric.
On Fri, Feb 26, 2021 at 12:07 AM Kane Wilson wrote:
> You should be able to use the Table
Hey
I'm running cassandra 3.11.9 and I have a lot of messages like this:
DEBUG [ReadRepairStage:2] 2021-02-25 16:41:11,464 ReadCallback.java:244 -
Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(4059620144736691554,
an opportunity to brush up my java
skills ;)
On Thu, Jul 9, 2020 at 7:59 PM Jeff Jirsa wrote:
>
>
> On Thu, Jul 9, 2020 at 9:02 AM Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
>> On Thu, Jul 9, 2020 at 5:54 PM Gil Ganz wrote:
>>
>> Another questio
Great, thank you very much!
On Thu, Jul 9, 2020 at 7:02 PM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:
> On Thu, Jul 9, 2020 at 5:54 PM Gil Ganz wrote:
>
>> That sounds very interesting Alex, so just to be sure I understand, it
>> was like this
>> 1
the built in one?
Another question, did changing the compaction strategy from one twcs to the
other trigger merging of old sstables?
gil
On Thu, Jul 9, 2020 at 4:51 PM Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:
> On Thu, Jul 9, 2020 at 3:38 PM Gil Ganz wrote:
>
>
Hey
I have a 2.11.4 cluster with tables that are defined with twcs, using
jeff's jar
https://github.com/jeffjirsa/twcs
All working great, but now I want to upgrade to 3.11, and I have a problem,
cassandra won't start, it fails on the following error
ERROR [main] 2020-07-09 13:30:29,823
ing authentication on
> performance
>
> DSR> Set the Auth cache to a long validity
>
> DSR> Don’t go crazy with RF of system auth
>
> DSR> Drop bcrypt rounds if you see massive cpu spikes on reconnect storms
>
>
> >> On Jun 1, 2020, at 11:26 PM, G
Hi
I have a production 3.11.6 cluster which I'm might want to enable
authentication in, I'm trying to understand what will be the performance
impact, if any.
I understand each use case might be different, trying to understand if
there is a common % people usually see their performance hit, or if
isable the debug.log one, or change
> org.apache.cassandra.service to log at INFO instead
>
> Nobody needs to see every digest mismatch and that someone thought this
> was a good idea is amazing to me. Someone should jira that to be a trace.
>
>
> On Mar 8, 2020, at 3:25 AM, Gil
r on your cluster. But if these messages come back
> again, you need to check what's causing these data inconsistencies.
>
>
> On Sun, Mar 8, 2020 at 10:11 AM Gil Ganz wrote:
>
>> Hey all
>> I have a lot of debug message about read repairs in my debug log :
>>
>&
Hey all
I have a lot of debug message about read repairs in my debug log :
DEBUG [ReadRepairStage:346] 2020-03-08 08:09:12,959 ReadCallback.java:242 -
Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(-28476014476640,
d".
>
>
>
> On Wed, Dec 19, 2018 at 3:40 AM Gil Ganz wrote:
>
>> sounds like the foreground read repair can cause issues to twcs (mix old
>> and new data in same sstable), is there a way to disable the foreground
>> read repair? is that indeed the case that i
this (again, this is 2.1, with your jar, might be something specifically
there). thanks
gil
On Wed, Dec 19, 2018 at 1:39 PM Gil Ganz wrote:
> sounds like the foreground read repair can cause issues to twcs (mix old
> and new data in same sstable), is there a way to disable the foreground
> re
sounds like the foreground read repair can cause issues to twcs (mix old
and new data in same sstable), is there a way to disable the foreground
read repair? is that indeed the case that it's problematic?
On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz wrote:
> hey jeff, attaching more informat
51 matches
Mail list logo
