Yeah I tried that, but oddly the table had nothing in it. I changed the compaction strategy from leveled to sizetierd and ran a major compaction on each node. I haven't seen the message logged on any node in a few days which makes me think that fixed it because it is normally logged multiple times per day.
On Sun, Sep 18, 2016 at 4:29 AM, Carlos Alonso <i...@mrcalonso.com> wrote: > By inspecting the contents on your system.hints table, specifically the > host_id column, you can see which is the destination host of those hints > and check if it is one of the alive or dead ones. > > Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> > > On 18 September 2016 at 04:35, Ezra Stuetzel <ezra.stuet...@riskiq.net> > wrote: > >> Hey Nicolas, >> >> There are no dead nodes. 'nodetool status' and 'nodetool describecluster' >> both show 4 healthy nodes. In the past we have had some nodes we eliminated >> by using 'nodetool assassinate'. However, I checked system.peers table on >> all 4 of our nodes and they each show 3 peers as expected. So it doesn't >> appear that any nodes have any awareness of an unreachable node which could >> be causing hints to back up. Any ideas for further troubleshooting what the >> hints are? >> >> Thanks, >> Ezra >> >> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet < >> nicolas.douil...@gmail.com> wrote: >> >>> Hi Erza, >>> >>> Have you a dead node in your cluster? >>> Because the coordinator stores a hint about dead replicas in the local >>> system.hints when a node is dead or didn't respond to a write request. >>> >>> -- >>> Nicolas >>> >>> >>> >>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ezra.stuet...@riskiq.net> >>> a écrit : >>> >>>> What would be the likely causes of large system hint partitions? >>>> Normally large partition warnings are for user defined tables which they >>>> are writing large partitions to. In this case, it appears C* is writing >>>> large partitions to the system.hints table. Gossip is not backed up. >>>> >>>> version: C* 2.2.7 >>>> >>>> WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220 >>>> BigTableWriter.java:184 - Writing large partition >>>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes) >>>> >>>> >>>> Thanks, >>>> >>>> Ezra >>>> >>> >> >
