RE: [EXTERNAL] Re: Even after the drop table, the data actually was not erased.

[Durity, Sean R]

We have found it very useful to set up an infrastructure where we can execute a 
nodetool command (or any other arbitrary command) from a single (non-Cassandra) 
host that will get executed on each node across the cluster (or a list of 
nodes).


Sean Durity

From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Monday, January 15, 2018 1:19 PM
To: user cassandra.apache.org 
Subject: [EXTERNAL] Re: Even after the drop table, the data actually was not 
erased.

As you said, the auto_bootstrap setting was turned on.

Well I was talking about the 'auto_snapshot' ;-). I understand that's what you 
meant to say.

This command seems to apply only to one node. Can it be applied cluster-wide? 
Or should I run this command on each node?

Indeed, 'nodetool clearsnapshot' is only for the node where you run the 
command, like most of the nodetool commands (repair is a bit specific).

C*heers,
---
Alain Rodriguez - @arodream - 
al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-01-15 1:56 GMT+00:00 Eunsu Kim 
>:
Thank you for your response.  As you said, the auto_bootstrap setting was 
turned on.
The actual data was deleted with the 'nodetool clearsnapshot' command.
This command seems to apply only to one node. Can it be applied cluster-wide? 
Or should I run this command on each node?




On 12 Jan 2018, at 8:10 PM, Alain RODRIGUEZ 
> wrote:

Hello,

However, the actual size of the data directory did not decrease at all. Disk 
Load monitored by JMX has been decreased.

This sounds like 'auto_snapshot' is enabled. This option will trigger a 
snapshot before any table drop / truncate to prevent user mistakes mostly. Then 
the data is removed but as it is still referenced by the snapshot (hard link), 
space cannot be freed.

Running 'nodetool clearsnapshot' should help reducing the dataset size in this 
situation.


The client fails to establish a connection and I see the following exceptions 
in the Cassandra logs.
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
cfId…

This does not look like a failed connection to me but rather a try to query 
some inexistent data. If that's the data you just deleted (keyspace / table), 
this is expected. If not there is an other issue, I hope not related to the 
delete in this case...

C*heers,
---
Alain Rodriguez - @arodream - 
al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2018-01-12 7:14 GMT+00:00 Eunsu Kim 
>:
hi everyone

On the development server, I dropped all the tables and even keyspace dropped 
to change the table schema.
Then I created the keyspace and the table.

However, the actual size of the data directory did not decrease at all. Disk 
Load monitored by JMX has been decreased.




After that, Cassandra does not work normally.

The client fails to establish a connection and I see the following exceptions 
in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
cfId…….org.apache.cassandra.io.FSReadError:
 java.io.IOException: Digest mismatch exception……


After the data is forcibly deleted, Cassandra is restarted in a clean state and 
works well.

Can anyone guess why this is happening?

Thank you in advance.






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability 

Re: Even after the drop table, the data actually was not erased.

[Alain RODRIGUEZ]

>
> As you said, the auto_bootstrap setting was turned on.


Well I was talking about the 'auto_snapshot' ;-). I understand that's what
you meant to say.

This command seems to apply only to one node. Can it be applied
> cluster-wide? Or should I run this command on each node?


Indeed, 'nodetool clearsnapshot' is only for the node where you run the
command, like most of the nodetool commands (repair is a bit specific).

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-01-15 1:56 GMT+00:00 Eunsu Kim :

> Thank you for your response.  As you said, the auto_bootstrap setting was
> turned on.
> The actual data was deleted with the 'nodetool clearsnapshot' command.
> This command seems to apply only to one node. Can it be applied
> cluster-wide? Or should I run this command on each node?
>
>
>
> On 12 Jan 2018, at 8:10 PM, Alain RODRIGUEZ  wrote:
>
> Hello,
>
> However, the actual size of the data directory did not decrease at all.
>> Disk Load monitored by JMX has been decreased.
>
>
> This sounds like 'auto_snapshot' is enabled. This option will trigger a
> snapshot before any table drop / truncate to prevent user mistakes mostly.
> Then the data is removed but as it is still referenced by the snapshot
> (hard link), space cannot be freed.
>
> Running 'nodetool clearsnapshot' should help reducing the dataset size in
> this situation.
>
>
> The client fails to establish a connection and I see the following
>> exceptions in the Cassandra logs.
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
>> for cfId…
>
>
> This does not look like a failed connection to me but rather a try to
> query some inexistent data. If that's the data you just deleted (keyspace /
> table), this is expected. If not there is an other issue, I hope not
> related to the delete in this case...
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
> 2018-01-12 7:14 GMT+00:00 Eunsu Kim :
>
>> hi everyone
>>
>> On the development server, I dropped all the tables and even keyspace
>> dropped to change the table schema.
>> Then I created the keyspace and the table.
>>
>> However, the actual size of the data directory did not decrease at all.
>> Disk Load monitored by JMX has been decreased.
>>
>> 
>>
>>
>> After that, Cassandra does not work normally.
>>
>> The client fails to establish a connection and I see the following
>> exceptions in the Cassandra logs.
>>
>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
>> table for cfId…….org.apache.cassandra.io.FSReadError:
>> java.io.IOException: Digest mismatch exception……
>>
>>
>> After the data is forcibly deleted, Cassandra is restarted in a clean
>> state and works well.
>>
>> Can anyone guess why this is happening?
>>
>> Thank you in advance.
>>
>
>
>

Re: Even after the drop table, the data actually was not erased.

[Eunsu Kim]

Thank you for your response.  As you said, the auto_bootstrap setting was 
turned on. 
The actual data was deleted with the 'nodetool clearsnapshot' command.
This command seems to apply only to one node. Can it be applied cluster-wide? 
Or should I run this command on each node?



> On 12 Jan 2018, at 8:10 PM, Alain RODRIGUEZ  wrote:
> 
> Hello,
> 
> However, the actual size of the data directory did not decrease at all. Disk 
> Load monitored by JMX has been decreased.
> 
> This sounds like 'auto_snapshot' is enabled. This option will trigger a 
> snapshot before any table drop / truncate to prevent user mistakes mostly. 
> Then the data is removed but as it is still referenced by the snapshot (hard 
> link), space cannot be freed.
> 
> Running 'nodetool clearsnapshot' should help reducing the dataset size in 
> this situation.
> 
> 
> The client fails to establish a connection and I see the following exceptions 
> in the Cassandra logs. 
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
> cfId…
> 
> This does not look like a failed connection to me but rather a try to query 
> some inexistent data. If that's the data you just deleted (keyspace / table), 
> this is expected. If not there is an other issue, I hope not related to the 
> delete in this case...
> 
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com 
> 
> France / Spain
> 
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com 
> 
> 
> 
> 2018-01-12 7:14 GMT+00:00 Eunsu Kim  >:
> hi everyone
> 
> On the development server, I dropped all the tables and even keyspace dropped 
> to change the table schema.
> Then I created the keyspace and the table.
> 
> However, the actual size of the data directory did not decrease at all. Disk 
> Load monitored by JMX has been decreased.
> 
> 
> 
> 
> After that, Cassandra does not work normally.
> 
> The client fails to establish a connection and I see the following exceptions 
> in the Cassandra logs.
> 
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
> cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException: Digest 
> mismatch exception……
> 
> 
> After the data is forcibly deleted, Cassandra is restarted in a clean state 
> and works well.
> 
> Can anyone guess why this is happening?
> 
> Thank you in advance.
> 

Re: Even after the drop table, the data actually was not erased.

[Alain RODRIGUEZ]

Hello,

However, the actual size of the data directory did not decrease at all.
> Disk Load monitored by JMX has been decreased.


This sounds like 'auto_snapshot' is enabled. This option will trigger a
snapshot before any table drop / truncate to prevent user mistakes mostly.
Then the data is removed but as it is still referenced by the snapshot
(hard link), space cannot be freed.

Running 'nodetool clearsnapshot' should help reducing the dataset size in
this situation.


The client fails to establish a connection and I see the following
> exceptions in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
> for cfId…


This does not look like a failed connection to me but rather a try to query
some inexistent data. If that's the data you just deleted (keyspace /
table), this is expected. If not there is an other issue, I hope not
related to the delete in this case...

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2018-01-12 7:14 GMT+00:00 Eunsu Kim :

> hi everyone
>
> On the development server, I dropped all the tables and even keyspace
> dropped to change the table schema.
> Then I created the keyspace and the table.
>
> However, the actual size of the data directory did not decrease at all.
> Disk Load monitored by JMX has been decreased.
>
>
>
> After that, Cassandra does not work normally.
>
> The client fails to establish a connection and I see the following
> exceptions in the Cassandra logs.
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
> for cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException:
> Digest mismatch exception……
>
>
> After the data is forcibly deleted, Cassandra is restarted in a clean
> state and works well.
>
> Can anyone guess why this is happening?
>
> Thank you in advance.
>

Even after the drop table, the data actually was not erased.

[Eunsu Kim]

hi everyone

On the development server, I dropped all the tables and even keyspace dropped 
to change the table schema.
Then I created the keyspace and the table.

However, the actual size of the data directory did not decrease at all. Disk 
Load monitored by JMX has been decreased.




After that, Cassandra does not work normally.

The client fails to establish a connection and I see the following exceptions 
in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for 
cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException: Digest 
mismatch exception……


After the data is forcibly deleted, Cassandra is restarted in a clean state and 
works well.

Can anyone guess why this is happening?

Thank you in advance.