Hi I've addressed the issues with Git. I believe this is what Stefan asking for: https://github.com/sedulam/cassandra/tree/12201 I've also added more tests for BHCS, including more for wide rows following Jeff's suggestion.
Thanks for the directions so far! If there's something else you would like to see tested or some metrics, please let me know what would be relevant. All the best Pedro Gordo On 13 June 2017 at 15:43, Pedro Gordo <pedro.gordo1...@gmail.com> wrote: > Hi all > > Although a couple of people engaged with me directly to talk about BHCS, I > would also like to get the community opinion on this, so I thought I could > get the discussion started by saying what the advantages would be and in > which type of tables BHCS would do a good job. Please keep in mind that all > my assumptions are without any real world experience on Cassandra, so this > is where I expect to see some input of the C* veterans to help me steer > BHCS implementation in the right direction if needed. This is a long email, > so there's a TLDR if you don't want to read everything. This is intended > for high-level discussion. For code level discussion, please refer to the > document in JIRA. > I'm aware that some might not like that no compaction occurs outside of > the burst hour, but I thought of solutions for that, so please read the > planned improvements below. > > *TL;DR* > BHCS tries to address these issues with the current compaction strategies: > - Necessity of allocating large storage during big compactions in STCS -> > Through the sstable_max_size property of BHCS, we can keep SSTables below a > certain size, so we wouldn't have issues with size during compaction > - We might get to a point where to return the results of a query, we need > to read from a large number of SSTables -> BHCS addresses this by making > sure that the number of SSTables where a key exists will be consistently > maintained at a low level after every compaction. The number of SSTables > where a key exists is configurable, so in the limit, you could set it to 1 > for optimal read performance. > - Continuous high I/O of LCS -> addressed by the scheduling feature of > BHCS. > > *Longer explanation:* > > *Where would it be advantageous using BHCS?* > - Read-bound tables: due to BHCS maintaining the number of key copies at a > low level, the read speed would be consistently fast. Since there's not a > lot of writes in this type of table, even if there are new SSTables > produced containing that key, the number SSTables containing that key would > be set again to 1 after burst hour (BH). > - Write-bound tables: in this scenario, there's a lot of SSTables created > outside of BH, but few reads, so the issue with existing strategies would > be a continuous high I/O dedicated to compaction. With BHCS during these > active hours, we would have an increase in disk size, but I assume that > this disk increase outside the BH would be tolerable since a lot of space > would be released during the burst. Still, if that's a big issue, I plan to > address this with the improvement (1). > > *Where is BHCS NOT recommended and what improvements can be done to make > it viable?* > - Read and write-heavy tables: because outside BH, SSTables would increase > until the burst kicks in, there can be an increase in the read speed and > disk used space. This could also be solved with improvement (1), (3) or (5). > > *Planned Improvements:* > (1) - The user could indicate that he wants continuous compaction. This > would change the strategy in such a way that outside of the Burst Hour, > STCS would be used to maintain an acceptable read speed and disk used > space. And then when BH would kick in, it would set key copies and disk > size again to optimal levels. > (2) - During table creation the user, might not be aware of the compaction > configurable details, so a user-friendly configuration would be provided. > If the user sets the table as a Write-and-Read heavy table, then > improvement (1) would be activated. Otherwise, the strategy would default > to its current config to save resources during the outside the BH. > (3) - Instead, of just one burst hour, we could set several periods for > BHCS to run during the day (for instance, every 3 hours or another > schedule). > > *Ideas:* > (4) - Continuously evaluate how many pending compactions we have and I/O > status, and then based on that, we start (or not) the compaction. > (5) - If outside the BH, the size for all the SSTables in a family set > reaches a certain threshold, then background compaction can occur anyway. > This threshold should be elevated due to the high CPU usage of BHCS. > > Please let me know your thoughts on this. Thanks! > > Best regards > > Pedro Gordo > > On 10 June 2017 at 22:22, J. D. Jordan <jeremiah.jor...@gmail.com> wrote: > >> GitHub has some good guides on how to use git and make a pull request for >> a project. >> >> https://guides.github.com/introduction/flow/ >> https://guides.github.com/activities/forking/ >> >> On Jun 10, 2017, at 3:17 PM, Pedro Gordo <pedro.gordo1...@gmail.com> >> wrote: >> >> Hi all >> >> I've added to JIRA, a document explaining how BHCS works with code >> snippets, and the motivation behind it. Because I'm not sure we can send >> attachments to the mailing list, please get the document from JIRA: >> https://issues.apache.org/jira/browse/CASSANDRA-12201 >> >> I'll check how to address the Git history in the next days. Can you please >> point me to a repo that you merged into C*, with a good history, so I can >> check it out and replicate the format in mine? >> >> Best regards >> Pedro Gordo >> >> >
