Re: large system hint partition

[Romain Hardouin]

Hi,
> More recent (I think 2.2) don't have this problem since they write hints to 
>the file system as per the commit log
Flat files hints were implemented starting from 3.0  
https://issues.apache.org/jira/browse/CASSANDRA-6230
Best,
Romain

Re: large system hint partition

[Graham Sanderson]

The reason for large partitions is that the partition key is just the uuid of 
the target node

More recent (I think 2.2) don't have this problem since they write hints to the 
file system as per the commit log

Sadly the large partitions make things worse when you are hinting hence 
presumably under stress

Sent from my iPhone

> On Sep 16, 2016, at 6:13 PM, Nicolas Douillet  
> wrote:
> 
> Hi Erza, 
> 
> Have you a dead node in your cluster?
> Because the coordinator stores a hint about dead replicas in the local 
> system.hints when a node is dead or didn't respond to a write request.
> 
> --
> Nicolas
> 
> 
> 
>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel  a 
>> écrit :
>> What would be the likely causes of large system hint partitions? Normally 
>> large partition warnings are for user defined tables which they are writing 
>> large partitions to. In this case, it appears C* is writing large partitions 
>> to the system.hints table. Gossip is not backed up.
>> 
>> version: C* 2.2.7
>> WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220 
>> BigTableWriter.java:184 - Writing large partition 
>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>> 
>> 
>> 
>> Thanks,
>> 
>> Ezra

Re: large system hint partition

[Ezra Stuetzel]

Yeah I tried that, but oddly the table had nothing in it.

I changed the compaction strategy from leveled to sizetierd and ran a major
compaction on each node. I haven't seen the message logged on any node in a
few days which makes me think that fixed it because it is normally logged
multiple times per day.

On Sun, Sep 18, 2016 at 4:29 AM, Carlos Alonso  wrote:

> By inspecting the contents on your system.hints table, specifically the
> host_id column, you can see which is the destination host of those hints
> and check if it is one of the alive or dead ones.
>
> Carlos Alonso | Software Engineer | @calonso 
>
> On 18 September 2016 at 04:35, Ezra Stuetzel 
> wrote:
>
>> Hey Nicolas,
>>
>> There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
>> both show 4 healthy nodes. In the past we have had some nodes we eliminated
>> by using 'nodetool assassinate'. However, I checked system.peers table on
>> all 4 of our nodes and they each show 3 peers as expected. So it doesn't
>> appear that any nodes have any awareness of an unreachable node which could
>> be causing hints to back up. Any ideas for further troubleshooting what the
>> hints are?
>>
>> Thanks,
>> Ezra
>>
>> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
>> nicolas.douil...@gmail.com> wrote:
>>
>>> Hi Erza,
>>>
>>> Have you a dead node in your cluster?
>>> Because the coordinator stores a hint about dead replicas in the local
>>> system.hints when a node is dead or didn't respond to a write request.
>>>
>>> --
>>> Nicolas
>>>
>>>
>>>
>>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel 
>>> a écrit :
>>>
 What would be the likely causes of large system hint partitions?
 Normally large partition warnings are for user defined tables which they
 are writing large partitions to. In this case, it appears C* is writing
 large partitions to the system.hints table. Gossip is not backed up.

 version: C* 2.2.7

 WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
 BigTableWriter.java:184 - Writing large partition
 system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)


 Thanks,

 Ezra

>>>
>>
>

Re: large system hint partition

[Carlos Alonso]

By inspecting the contents on your system.hints table, specifically the
host_id column, you can see which is the destination host of those hints
and check if it is one of the alive or dead ones.

Carlos Alonso | Software Engineer | @calonso 

On 18 September 2016 at 04:35, Ezra Stuetzel 
wrote:

> Hey Nicolas,
>
> There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
> both show 4 healthy nodes. In the past we have had some nodes we eliminated
> by using 'nodetool assassinate'. However, I checked system.peers table on
> all 4 of our nodes and they each show 3 peers as expected. So it doesn't
> appear that any nodes have any awareness of an unreachable node which could
> be causing hints to back up. Any ideas for further troubleshooting what the
> hints are?
>
> Thanks,
> Ezra
>
> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
> nicolas.douil...@gmail.com> wrote:
>
>> Hi Erza,
>>
>> Have you a dead node in your cluster?
>> Because the coordinator stores a hint about dead replicas in the local
>> system.hints when a node is dead or didn't respond to a write request.
>>
>> --
>> Nicolas
>>
>>
>>
>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel 
>> a écrit :
>>
>>> What would be the likely causes of large system hint partitions?
>>> Normally large partition warnings are for user defined tables which they
>>> are writing large partitions to. In this case, it appears C* is writing
>>> large partitions to the system.hints table. Gossip is not backed up.
>>>
>>> version: C* 2.2.7
>>>
>>> WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>>> BigTableWriter.java:184 - Writing large partition
>>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>>
>>>
>>> Thanks,
>>>
>>> Ezra
>>>
>>
>

Re: large system hint partition

[Ezra Stuetzel]

Hey Nicolas,

There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
both show 4 healthy nodes. In the past we have had some nodes we eliminated
by using 'nodetool assassinate'. However, I checked system.peers table on
all 4 of our nodes and they each show 3 peers as expected. So it doesn't
appear that any nodes have any awareness of an unreachable node which could
be causing hints to back up. Any ideas for further troubleshooting what the
hints are?

Thanks,
Ezra

On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
nicolas.douil...@gmail.com> wrote:

> Hi Erza,
>
> Have you a dead node in your cluster?
> Because the coordinator stores a hint about dead replicas in the local
> system.hints when a node is dead or didn't respond to a write request.
>
> --
> Nicolas
>
>
>
> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel  a
> écrit :
>
>> What would be the likely causes of large system hint partitions? Normally
>> large partition warnings are for user defined tables which they are writing
>> large partitions to. In this case, it appears C* is writing large
>> partitions to the system.hints table. Gossip is not backed up.
>>
>> version: C* 2.2.7
>>
>> WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>> BigTableWriter.java:184 - Writing large partition
>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>
>>
>> Thanks,
>>
>> Ezra
>>
>

Re: large system hint partition

[Nicolas Douillet]

Hi Erza,

Have you a dead node in your cluster?
Because the coordinator stores a hint about dead replicas in the local
system.hints when a node is dead or didn't respond to a write request.

--
Nicolas



Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel  a
écrit :

> What would be the likely causes of large system hint partitions? Normally
> large partition warnings are for user defined tables which they are writing
> large partitions to. In this case, it appears C* is writing large
> partitions to the system.hints table. Gossip is not backed up.
>
> version: C* 2.2.7
>
> WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
> BigTableWriter.java:184 - Writing large partition
> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>
>
> Thanks,
>
> Ezra
>