We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
So we had to drop 7 MVs in production, as soon as we dropped the first
Materialized View, our cluster became unstable and app started giving 100%
error, what we noticed:
1. As soon as MV was dropped , cluster became unstable and
I believe we had something similar happen on 3.0.15 a while back. We had a
cluster that created mass havoc by creating MVs on a large existing
dataset. We thought we had stabilized the cluster, but saw similar issues
as you when we dropped the MVs.
We interpreted our errors to mean that we should
Jonathan,
As per https://issues.apache.org/jira/browse/CASSANDRA-13696 the issue,
Digest mismatch Exception if hints file has UnknownColumnFamily, is fixed
for 3.0.15 , did you still faced this issue on 3.0.15 ?
Thanks
Surbhi
On Tue, 18 Feb 2020 at 17:40, Jonathan Koppenhofer
wrote:
> I
Thanks Eric, Let me go back to the app team
On Tue, Feb 18, 2020 at 6:49 PM Erick Ramirez
wrote:
> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>>
>
> Which exact version of C* is it again?
>
>> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
>>
Reading from a non-existent table shouldn’t crash the database
Clearly the hint error invoked the fs error handler - probably incorrectly -
which shut down the db. That’s not ok and deserves a JIRA.
If dropping a table causes a hint checksum error then it needs to be fixed.
>> On Feb 18,
>
> We are Cassandra 3.11.0 unfortunately :(
>
Oh, right. That's why the hint read failure is causing the node to
shutdown. We've at least identified that. I was worried that there was
another bug we didn't know about. Cheers!
>
Just to add to my above point because here we are dropping MV not a regular
table.
And MV does read before write , Is this the reason we are seeing the below
message? Trying to understand
WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932
HintsReader.java:237 - Failed to read a hint for
Thanks Eric...
On Tue, 18 Feb 2020 at 22:06, Erick Ramirez
wrote:
> Just to add to my above point because here we are dropping MV not a
>> regular table.
>> And MV does read before write , Is this the reason we are seeing the
>> below message? Trying to understand
>>
>> WARN
>
> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>
Which exact version of C* is it again?
> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
> socket; closing
>
This is
We are Cassandra 3.11.0 unfortunately :(
On Tue, 18 Feb 2020 at 19:41, Erick Ramirez
wrote:
> Clearly the hint error invoked the fs error handler - probably incorrectly
>> - which shut down the db. That’s not ok and deserves a JIRA.
>>
>
> It's supposed to have been fixed by CASSANDRA-13696 in
Hello.
It is a question.
The operation of the write operation and the read operation at the consistency
level QUORUM was confirmed in the following
environment.
4-node configuration
・ 10.17.38.204 (seed node)
・ 10.17.38.210
・ 10.17.38.215
・ 10.17.38.218
First, the following command was
A quick eyeball seems correct to me. Do you have concerns with the TRACE
output?
As a side note, the QUORUM consistency is not really relevant because you
only have 1 replica in the dc1 DC. Cheers!
Hi Eric,
As per https://issues.apache.org/jira/browse/CASSANDRA-13696 , this issue
happens even with write traffic
"I did more investigation today. Seems it's more serious than I thought.
Even there's no down node, "drop table" + write traffic, will trigger the
problem."
Thanks
Surbhi
On Tue,
I'm not sure I understand your last response. Was there a question in there
somewhere? Cheers!
>
Forensics are gone at this point, so I can't verify exact errors, but
wanted to mention we had seen something similar to corroborate your
experience and warn others.
The version would have been 3.0.15 or 3.11.3 as that is what we were
deploying on our clusters at the time. I think it was more
>
> Clearly the hint error invoked the fs error handler - probably incorrectly
> - which shut down the db. That’s not ok and deserves a JIRA.
>
It's supposed to have been fixed by CASSANDRA-13696 in 3.0.15/3.11.1 but
I'm waiting for Surbhi to confirm the exact C* version. Cheers!
So should upgrading to 3.11.1 will solve this issue?
On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta wrote:
> Thanks Eric...
>
> On Tue, 18 Feb 2020 at 22:06, Erick Ramirez
> wrote:
>
>> Just to add to my above point because here we are dropping MV not a
>>> regular table.
>>> And MV does read
Application team confirmed that they are * not* referencing the dropped MVs
anywhere for reading or writing
On Tue, 18 Feb 2020 at 22:25, Surbhi Gupta wrote:
> So should upgrading to 3.11.1 will solve this issue?
>
> On Tue, 18 Feb 2020 at 22:18, Surbhi Gupta
> wrote:
>
>> Thanks Eric...
>>
Thanks Jonathan, it still helps...
Anyone knows the solution?
On Tue, 18 Feb 2020 at 18:08, Jonathan Koppenhofer
wrote:
> Forensics are gone at this point, so I can't verify exact errors, but
> wanted to mention we had seen something similar to corroborate your
> experience and warn others.
>
We tested this is dev and test before dropping in production but did not
see this issue in dev/test .
I am yet to hear from application team.
However if it was application read was happening from the dropped MV then
we would have caught this error in dev/test itself.
And we are running Cassandra
>
> Just to add to my above point because here we are dropping MV not a
> regular table.
> And MV does read before write , Is this the reason we are seeing the below
> message? Trying to understand
>
> WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 -
> Failed to read a
>
> So should upgrading to 3.11.1 will solve this issue?
>
Upgrading off 3.11.0 will prevent nodes going down as a result of the hint
replay bug in CASSANDRA-13696, yes. But I'd recommend upgrading to the
latest C* 3.11.6 unless you have a very specific reason for upgrading to
3.11.1 which was
22 matches
Mail list logo
