Just to add , we are using 24GB heap size. On Mon, 10 Feb 2020 at 09:08, Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
> Hi Jon, > > We are on multi datacenter(On Prim) setup. > We also noticed too many messages like below: > > DEBUG [GossipStage:1] 2020-02-10 09:38:52,953 FailureDetector.java:457 - > Ignoring interval time of 3258125997 for /10.x.x.x > > DEBUG [GossipStage:1] 2020-02-10 09:38:52,954 FailureDetector.java:457 - > Ignoring interval time of 2045630029 for /10.y.y.y > > DEBUG [GossipStage:1] 2020-02-10 09:38:52,954 FailureDetector.java:457 - > Ignoring interval time of 2045416737 for /10.z.z.z > > > > Currently the value of phi_convict_threshold is not set which makes it to > 8 (default) . > Can this also cause hints buildup even when we can see that all nodes are > UP ? > Recommended value of phi_convict_threshold is 12 in AWS multi datacenter > environment. > > Thanks > Surbhi > > On Sun, 9 Feb 2020 at 21:42, Surbhi Gupta <surbhi.gupt...@gmail.com> > wrote: > >> Thanks a lot Jon.. >> Will try the recommendations and let you know the results.... >> >> On Fri, Feb 7, 2020 at 10:52 AM Jon Haddad <j...@jonhaddad.com> wrote: >> >>> There's a few things you can do here that might help. >>> >>> First off, if you're using the default heap settings, that's a serious >>> problem. If you've got the head room, my recommendation is to use 16GB >>> heap with 12 GB new gen and pin your memtable heap space to 2GB. Set your >>> max tenuring threshold to 6 and your survivor ratio to 6. You don't need a >>> lot of old gen space with cassandra, almost everything that will show up >>> there is memtable related, and we allocate a *lot* whenever we read data >>> off disk. >>> >>> Most folks use the default disk read ahead setting of 128KB. You can >>> check this setting using blockdev --report, under the RA column. You'll >>> see 256 there, that's in 512 byte sectors. MVs rely on a read before a >>> write, so for every read off disk you do, you'll pull additional 128KB into >>> your page cache. This is usually a waste and puts WAY too much pressure on >>> your disk. On SSD, I always change this to 4KB. >>> >>> Next, be sure you're setting your compression rate accordingly. I wrote >>> a long post on the topic here: >>> https://thelastpickle.com/blog/2018/08/08/compression_performance.html. >>> Our default compression is very unfriendly for read heavy workloads if >>> you're reading small rows. If your records are small, 4KB compression >>> chunk length is your friend. >>> >>> I have some slides showing pretty good performance improvements from the >>> above 2 changes. Specifically, I went from 16K reads a second at 180ms p99 >>> latency up to 63K reads / second at 21ms p99. Disk usage dropped by a >>> factor of 10. Throw in those JVM changes I recommended and things should >>> improve even further. >>> >>> Generally speaking, I recommend avoiding MVs, as they can be a giant >>> mine if you aren't careful. They're not doing any magic behind the scenes >>> that makes scaling easier, and in a lot of cases they're a hinderance. You >>> still need to understand the underlying data and how it's laid out to use >>> them properly, which is 99% of the work. >>> >>> Jon >>> >>> On Fri, Feb 7, 2020 at 10:32 AM Michael Shuler <mich...@pbandjelly.org> >>> wrote: >>> >>>> That JIRA still says Open, so no, it has not been fixed (unless there's >>>> a fixed duplicate in JIRA somewhere). >>>> >>>> For clarification, you could update that ticket with a comment >>>> including >>>> your environmental details, usage of MV, etc. I'll bump the priority up >>>> and include some possible branchX fixvers. >>>> >>>> Michael >>>> >>>> On 2/7/20 10:53 AM, Surbhi Gupta wrote: >>>> > Hi, >>>> > >>>> > We are getting hit by the below bug. >>>> > Other than lowering hinted_handoff_throttle_in_kb to 100 any other >>>> work >>>> > around ? >>>> > >>>> > https://issues.apache.org/jira/browse/CASSANDRA-13810 >>>> > >>>> > Any idea if it got fixed in later version. >>>> > We are on Open source Cassandra 3.11.1 . >>>> > >>>> > Thanks >>>> > Surbhi >>>> > >>>> > >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>>> For additional commands, e-mail: user-h...@cassandra.apache.org >>>> >>>>
