Re: Hints folder missing in Cassandra

2018-02-09 Thread Nicolas Guyomar 
Hi,

There are no piece of code in Cassandra that would remove this folder. You
should start looking elsewhere, like other people mentioned (chef, ansible
and so on), good luck


On 8 February 2018 at 22:54, test user  wrote:

> Does anyone have more inputs on the missing hints folder, rather why it
> gets deleted.
>
> Has anyone run into this scenario before?
>
> Regards,
> Cassandra User
>
> On Wed, Feb 7, 2018 at 9:21 PM, test user  wrote:
>
>> The problem is even though, I get that O_RDONLY WARN message, if I try to
>> navigate to the path where the hints folder is stored, the folder is not
>> present.
>> I cannot check the permissions on that folder, its already missing, got
>> deleted somehow.
>>
>> I believe everything runs as the root user.
>>
>> I do see a lot of sstable activity performed by MemTableFlushWriter
>> (ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable)
>> before and after this WARN message.
>>
>> It is not a space issue, I checked that already.
>>
>>
>>
>> On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall 
>> wrote:
>>
>>>
>>> The environment is built using established images for Cassandra 3.10.
 Unfortunately the debug log does not indicate any errors before I start
 seeing the WARN for missing hints folder. I understand that hints file will
 be deleted after replay is complete, but not sure of the root cause of why
 the hints folder is getting deleted.
 When I look at the nodetool status or nodetool ring - it indicates that
 all nodes are up and running in normal state, no node went down. Also, I do
 not see anything the debug logs indicating that a node went down. In such a
 scenario, I am not sure why would HintsWriterExecutor would get triggered.


>>> That error code (O_RDONLY) in the log message indicates that the hints
>>> folder has had its permission bits set to read only.
>>>
>>> We've had several issues with some of the tools doing this type of thing
>>> when they are run as the root user. Is this specific node one on which you
>>> use any of the tools like sstableloader or similar? If so, are you running
>>> them as root?
>>>
>>> Another thought - if it is on a different partition than the data
>>> directory, is there free space left on the underlying device holding:
>>> /var/lib/cassandra/hints?
>>>
>>>
>>> --
>>> -
>>> Nate McCall
>>> Wellington, NZ
>>> @zznate
>>>
>>> CTO
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>
>>
>

Re: Hints folder missing in Cassandra

2018-02-08 Thread test user 
Does anyone have more inputs on the missing hints folder, rather why it
gets deleted.

Has anyone run into this scenario before?

Regards,
Cassandra User

On Wed, Feb 7, 2018 at 9:21 PM, test user  wrote:

> The problem is even though, I get that O_RDONLY WARN message, if I try to
> navigate to the path where the hints folder is stored, the folder is not
> present.
> I cannot check the permissions on that folder, its already missing, got
> deleted somehow.
>
> I believe everything runs as the root user.
>
> I do see a lot of sstable activity performed by MemTableFlushWriter
> (ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable)
> before and after this WARN message.
>
> It is not a space issue, I checked that already.
>
>
>
> On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall 
> wrote:
>
>>
>> The environment is built using established images for Cassandra 3.10.
>>> Unfortunately the debug log does not indicate any errors before I start
>>> seeing the WARN for missing hints folder. I understand that hints file will
>>> be deleted after replay is complete, but not sure of the root cause of why
>>> the hints folder is getting deleted.
>>> When I look at the nodetool status or nodetool ring - it indicates that
>>> all nodes are up and running in normal state, no node went down. Also, I do
>>> not see anything the debug logs indicating that a node went down. In such a
>>> scenario, I am not sure why would HintsWriterExecutor would get triggered.
>>>
>>>
>> That error code (O_RDONLY) in the log message indicates that the hints
>> folder has had its permission bits set to read only.
>>
>> We've had several issues with some of the tools doing this type of thing
>> when they are run as the root user. Is this specific node one on which you
>> use any of the tools like sstableloader or similar? If so, are you running
>> them as root?
>>
>> Another thought - if it is on a different partition than the data
>> directory, is there free space left on the underlying device holding:
>> /var/lib/cassandra/hints?
>>
>>
>> --
>> -
>> Nate McCall
>> Wellington, NZ
>> @zznate
>>
>> CTO
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>

Re: Hints folder missing in Cassandra

2018-02-07 Thread test user 
The problem is even though, I get that O_RDONLY WARN message, if I try to
navigate to the path where the hints folder is stored, the folder is not
present.
I cannot check the permissions on that folder, its already missing, got
deleted somehow.

I believe everything runs as the root user.

I do see a lot of sstable activity performed by MemTableFlushWriter
(ColumnFamilyStore), CompactionExecutor, PerDiskFlushWriter (MemTable)
before and after this WARN message.

It is not a space issue, I checked that already.



On Wed, Feb 7, 2018 at 3:49 PM, Nate McCall  wrote:

>
> The environment is built using established images for Cassandra 3.10.
>> Unfortunately the debug log does not indicate any errors before I start
>> seeing the WARN for missing hints folder. I understand that hints file will
>> be deleted after replay is complete, but not sure of the root cause of why
>> the hints folder is getting deleted.
>> When I look at the nodetool status or nodetool ring - it indicates that
>> all nodes are up and running in normal state, no node went down. Also, I do
>> not see anything the debug logs indicating that a node went down. In such a
>> scenario, I am not sure why would HintsWriterExecutor would get triggered.
>>
>>
> That error code (O_RDONLY) in the log message indicates that the hints
> folder has had its permission bits set to read only.
>
> We've had several issues with some of the tools doing this type of thing
> when they are run as the root user. Is this specific node one on which you
> use any of the tools like sstableloader or similar? If so, are you running
> them as root?
>
> Another thought - if it is on a different partition than the data
> directory, is there free space left on the underlying device holding:
> /var/lib/cassandra/hints?
>
>
> --
> -
> Nate McCall
> Wellington, NZ
> @zznate
>
> CTO
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Hints folder missing in Cassandra

2018-02-07 Thread Nate McCall 
> The environment is built using established images for Cassandra 3.10.
> Unfortunately the debug log does not indicate any errors before I start
> seeing the WARN for missing hints folder. I understand that hints file will
> be deleted after replay is complete, but not sure of the root cause of why
> the hints folder is getting deleted.
> When I look at the nodetool status or nodetool ring - it indicates that
> all nodes are up and running in normal state, no node went down. Also, I do
> not see anything the debug logs indicating that a node went down. In such a
> scenario, I am not sure why would HintsWriterExecutor would get triggered.
>
>
That error code (O_RDONLY) in the log message indicates that the hints
folder has had its permission bits set to read only.

We've had several issues with some of the tools doing this type of thing
when they are run as the root user. Is this specific node one on which you
use any of the tools like sstableloader or similar? If so, are you running
them as root?

Another thought - if it is on a different partition than the data
directory, is there free space left on the underlying device holding:
/var/lib/cassandra/hints?


-- 
-
Nate McCall
Wellington, NZ
@zznate

CTO
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Hints folder missing in Cassandra

2018-02-07 Thread test user 
The environment is built using established images for Cassandra 3.10.
Unfortunately the debug log does not indicate any errors before I start
seeing the WARN for missing hints folder. I understand that hints file will
be deleted after replay is complete, but not sure of the root cause of why
the hints folder is getting deleted.
When I look at the nodetool status or nodetool ring - it indicates that all
nodes are up and running in normal state, no node went down. Also, I do not
see anything the debug logs indicating that a node went down. In such a
scenario, I am not sure why would HintsWriterExecutor would get triggered.

Regards,
Cassandra User

On Wed, Feb 7, 2018 at 11:25 AM, Rahul Singh <rahul.xavier.si...@gmail.com>
wrote:

> Do you have any automation being applied via chef or likewise? Was the
> environment built using automation or using established images ?
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Feb 7, 2018, 10:37 AM -0600, test user <casst...@gmail.com>, wrote:
>
> The directory */var/lib/cassandra/hints* was created, when I brought up
> the cluster. I am running into this issue until after the cluster is up and
> running for about 2 weeks.
>
> Regards,
> Cassandra User
>
> On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev <kyrylo_lebed...@epam.com>
> wrote:
>
>> Hi,
>>
>> Not sure what could be the reason, but the issue is obvious: directory
>> */var/lib/cassandra/hints* is missing or cassandra user doesn't have
>> enough permissions on it.
>>
>>
>> Regards,
>>
>> Kyrill
>> --
>> *From:* test user <casst...@gmail.com>
>> *Sent:* Wednesday, February 7, 2018 8:14:59 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Fwd: Hints folder missing in Cassandra
>>
>>
>> Has anyone run into the issue mentioned below?
>>
>>
>> -- Forwarded message --
>> From: *test user* <casst...@gmail.com>
>> Date: Tue, Feb 6, 2018 at 3:28 PM
>> Subject: Hints folder missing in Cassandra
>> To: user-subscr...@cassandra.apache.org
>>
>>
>> Hello All,
>>
>>
>>
>> I am using Cassandra 3.10. I have 3-node cluster (say node0, node1,
>> node2).
>>
>> I have been using nodetool utility to monitor the cluster.
>>
>> nodetool status – indicates all nodes are up and in normal state
>>
>> nodetool ring – indicates all nodes are up
>>
>>
>>
>> Running into following issue:
>>
>> *WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 -
>> open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).*
>>
>> For some reason, the hints folder on node1 is missing. Not sure how it
>> got deleted. One node in the cluster always seems to run into this issue.
>>
>> The other two nodes node0 and node2 do not have this issue, and the hints
>> folder is present.
>>
>>
>>
>> The debug logs do not seem to indicate the root cause.
>>
>>
>>
>> Please let me know how to debug this further to understand the root cause.
>>
>>
>>
>> Regards,
>>
>> Cassandra User
>>
>>
>>
>>
>

Re: Hints folder missing in Cassandra

2018-02-07 Thread Rahul Singh 
Do you have any automation being applied via chef or likewise? Was the 
environment built using automation or using established images ?

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Feb 7, 2018, 10:37 AM -0600, test user <casst...@gmail.com>, wrote:
> The directory /var/lib/cassandra/hints was created, when I brought up the 
> cluster. I am running into this issue until after the cluster is up and 
> running for about 2 weeks.
>
> Regards,
> Cassandra User
>
> > On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev <kyrylo_lebed...@epam.com> 
> > wrote:
> > > Hi,
> > > Not sure what could be the reason, but the issue is obvious: directory 
> > > /var/lib/cassandra/hints is missing or cassandra user doesn't have enough 
> > > permissions on it.
> > >
> > > Regards,
> > > Kyrill
> > > From: test user <casst...@gmail.com>
> > > Sent: Wednesday, February 7, 2018 8:14:59 AM
> > > To: user@cassandra.apache.org
> > > Subject: Fwd: Hints folder missing in Cassandra
> > >
> > >
> > > Has anyone run into the issue mentioned below?
> > >
> > >
> > > -- Forwarded message --
> > > From: test user <casst...@gmail.com>
> > > Date: Tue, Feb 6, 2018 at 3:28 PM
> > > Subject: Hints folder missing in Cassandra
> > > To: user-subscr...@cassandra.apache.org
> > >
> > >
> > > Hello All,
> > >
> > > I am using Cassandra 3.10. I have 3-node cluster (say node0, node1, 
> > > node2).
> > > I have been using nodetool utility to monitor the cluster.
> > > nodetool status – indicates all nodes are up and in normal state
> > > nodetool ring – indicates all nodes are up
> > >
> > > Running into following issue:
> > > WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 - 
> > > open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).
> > > For some reason, the hints folder on node1 is missing. Not sure how it 
> > > got deleted. One node in the cluster always seems to run into this issue.
> > > The other two nodes node0 and node2 do not have this issue, and the hints 
> > > folder is present.
> > >
> > > The debug logs do not seem to indicate the root cause.
> > >
> > > Please let me know how to debug this further to understand the root cause.
> > >
> > > Regards,
> > > Cassandra User
> > >
> > >
> > >
>

Re: Hints folder missing in Cassandra

2018-02-07 Thread test user 
The directory */var/lib/cassandra/hints *was created, when I brought up the
cluster. I am running into this issue until after the cluster is up and
running for about 2 weeks.

Regards,
Cassandra User

On Wed, Feb 7, 2018 at 1:28 AM, Kyrylo Lebediev <kyrylo_lebed...@epam.com>
wrote:

> Hi,
>
> Not sure what could be the reason, but the issue is obvious: directory
> */var/lib/cassandra/hints* is missing or cassandra user doesn't have
> enough permissions on it.
>
>
> Regards,
>
> Kyrill
> --
> *From:* test user <casst...@gmail.com>
> *Sent:* Wednesday, February 7, 2018 8:14:59 AM
> *To:* user@cassandra.apache.org
> *Subject:* Fwd: Hints folder missing in Cassandra
>
>
> Has anyone run into the issue mentioned below?
>
>
> -- Forwarded message --
> From: *test user* <casst...@gmail.com>
> Date: Tue, Feb 6, 2018 at 3:28 PM
> Subject: Hints folder missing in Cassandra
> To: user-subscr...@cassandra.apache.org
>
>
> Hello All,
>
>
>
> I am using Cassandra 3.10. I have 3-node cluster (say node0, node1,
> node2).
>
> I have been using nodetool utility to monitor the cluster.
>
> nodetool status – indicates all nodes are up and in normal state
>
> nodetool ring – indicates all nodes are up
>
>
>
> Running into following issue:
>
> *WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 -
> open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).*
>
> For some reason, the hints folder on node1 is missing. Not sure how it got
> deleted. One node in the cluster always seems to run into this issue.
>
> The other two nodes node0 and node2 do not have this issue, and the hints
> folder is present.
>
>
>
> The debug logs do not seem to indicate the root cause.
>
>
>
> Please let me know how to debug this further to understand the root cause.
>
>
>
> Regards,
>
> Cassandra User
>
>
>
>

Re: Hints folder missing in Cassandra

2018-02-07 Thread Kyrylo Lebediev 
Hi,

Not sure what could be the reason, but the issue is obvious: directory 
/var/lib/cassandra/hints is missing or cassandra user doesn't have enough 
permissions on it.


Regards,

Kyrill


From: test user <casst...@gmail.com>
Sent: Wednesday, February 7, 2018 8:14:59 AM
To: user@cassandra.apache.org
Subject: Fwd: Hints folder missing in Cassandra


Has anyone run into the issue mentioned below?


-- Forwarded message --
From: test user <casst...@gmail.com<mailto:casst...@gmail.com>>
Date: Tue, Feb 6, 2018 at 3:28 PM
Subject: Hints folder missing in Cassandra
To: 
user-subscr...@cassandra.apache.org<mailto:user-subscr...@cassandra.apache.org>



Hello All,



I am using Cassandra 3.10. I have 3-node cluster (say node0, node1, node2).

I have been using nodetool utility to monitor the cluster.

nodetool status – indicates all nodes are up and in normal state

nodetool ring – indicates all nodes are up



Running into following issue:

WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 - 
open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).

For some reason, the hints folder on node1 is missing. Not sure how it got 
deleted. One node in the cluster always seems to run into this issue.

The other two nodes node0 and node2 do not have this issue, and the hints 
folder is present.



The debug logs do not seem to indicate the root cause.



Please let me know how to debug this further to understand the root cause.



Regards,

Cassandra User

Fwd: Hints folder missing in Cassandra

2018-02-06 Thread test user 
Has anyone run into the issue mentioned below?


-- Forwarded message --
From: test user <casst...@gmail.com>
Date: Tue, Feb 6, 2018 at 3:28 PM
Subject: Hints folder missing in Cassandra
To: user-subscr...@cassandra.apache.org


Hello All,



I am using Cassandra 3.10. I have 3-node cluster (say node0, node1, node2).

I have been using nodetool utility to monitor the cluster.

nodetool status – indicates all nodes are up and in normal state

nodetool ring – indicates all nodes are up



Running into following issue:

*WARN  [HintsWriteExecutor:1] 2018-02-05 06:57:09,607 CLibrary.java:295 -
open(/var/lib/cassandra/hints, O_RDONLY) failed, errno (2).*

For some reason, the hints folder on node1 is missing. Not sure how it got
deleted. One node in the cluster always seems to run into this issue.

The other two nodes node0 and node2 do not have this issue, and the hints
folder is present.



The debug logs do not seem to indicate the root cause.



Please let me know how to debug this further to understand the root cause.



Regards,

Cassandra User