By inspecting the contents on your system.hints table, specifically the
host_id column, you can see which is the destination host of those hints
and check if it is one of the alive or dead ones.

Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>

On 18 September 2016 at 04:35, Ezra Stuetzel <ezra.stuet...@riskiq.net>
wrote:

> Hey Nicolas,
>
> There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
> both show 4 healthy nodes. In the past we have had some nodes we eliminated
> by using 'nodetool assassinate'. However, I checked system.peers table on
> all 4 of our nodes and they each show 3 peers as expected. So it doesn't
> appear that any nodes have any awareness of an unreachable node which could
> be causing hints to back up. Any ideas for further troubleshooting what the
> hints are?
>
> Thanks,
> Ezra
>
> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
> nicolas.douil...@gmail.com> wrote:
>
>> Hi Erza,
>>
>> Have you a dead node in your cluster?
>> Because the coordinator stores a hint about dead replicas in the local
>> system.hints when a node is dead or didn't respond to a write request.
>>
>> --
>> Nicolas
>>
>>
>>
>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ezra.stuet...@riskiq.net>
>> a écrit :
>>
>>> What would be the likely causes of large system hint partitions?
>>> Normally large partition warnings are for user defined tables which they
>>> are writing large partitions to. In this case, it appears C* is writing
>>> large partitions to the system.hints table. Gossip is not backed up.
>>>
>>> version: C* 2.2.7
>>>
>>> WARN  [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>>> BigTableWriter.java:184 - Writing large partition
>>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>>
>>>
>>> Thanks,
>>>
>>> Ezra
>>>
>>
>

Reply via email to