Re: [VOTE] Release Apache Cassandra 2.2.0-rc1
Two tickets that warrant attention: https://issues.apache.org/jira/browse/CASSANDRA-9557 https://issues.apache.org/jira/browse/CASSANDRA-9558 On Fri, Jun 5, 2015 at 2:15 PM, Jason Brown jasedbr...@gmail.com wrote: +1 On Friday, June 5, 2015, Sylvain Lebresne sylv...@datastax.com wrote: +1 On Fri, Jun 5, 2015 at 5:41 PM, Brandon Williams dri...@gmail.com javascript:; wrote: +1 On Fri, Jun 5, 2015 at 10:27 AM, Jake Luciani j...@apache.org javascript:; wrote: I propose the following artifacts for release as 2.2.0-rc1. sha1: b0ae285bdc7377a64ed92f01c67ff46b40ecaac0 Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.2.0-rc1-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-1059/org/apache/cassandra/apache-cassandra/2.2.0-rc1/ Staging repository: https://repository.apache.org/content/repositories/orgapachecassandra-1059/ The artifacts as well as the debian package are also available here: http://people.apache.org/~jake The vote will be open for 72 hours (longer if needed). [1]: http://goo.gl/B6LLNm (CHANGES.txt) [2]: http://goo.gl/mr638q (NEWS.txt)
Re: Requiring Java 8 for C* 3.0
+1, from a testing perspective we run dtest and unit tests on hotspot 8 and openjdk 8 and have seen no problems. On Thu, May 7, 2015 at 12:09 PM, Jonathan Ellis jbel...@gmail.com wrote: We discussed requiring Java 8 previously and decided to remain Java 7-compatible, but at the time we were planning to release 3.0 before Java 7 EOL. Now that 8099 and increased emphasis on QA have delayed us past Java 7 EOL, I think it's worth reopening this discussion. If we require 8, then we can use lambdas, LongAdder, StampedLock, Streaming collections, default methods, etc. Not just in 3.0 but over 3.x for the next year. If we don't, then people can choose whether to deploy on 7 or 8 -- but the vast majority will deploy on 8 simply because 7 is no longer supported without a premium contract with Oracle. 8 also has a more advanced G1GC implementation (see CASSANDRA-7486). I think that gaining access to the new features in 8 as we develop 3.x is worth losing the ability to run on a platform that will have been EOL for a couple months by the time we release. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder, http://www.datastax.com @spyced
Re: Staging Branches
In the meantime can we agree on having cassci to validate personal merged branches before pushing either, in case of non-trunk patches? +100 from me, but why the exception for trunk? Wouldn't it be easier to wait for the dev branch tests to pass and then do all the merging at once (2.0, 2.1,3.0, trunk)? On Thu, May 7, 2015 at 1:21 PM, Aleksey Yeschenko alek...@apache.org wrote: Agreed. I would like to wait and see how we do without extra branches for a release or two. That will give us a better idea of how much pain the extra steps will protect us from. In the meantime can we agree on having cassci to validate personal merged branches before pushing either, in case of non-trunk patches? That doesn’t require any more infra setup, and with the delta between 2.0 and 2.1, and 2.1 and 3.0, and, sometimes, 2.0 and 3.0, doing so is crucial. I know that at least Sam does that already, but I want it to be an agreed upon procedure. Oh, ans as Benedict said, it’d be nice for all those commits to start including CHANGES.txt and the commit messages, both. With the developers total/committers ratio that we have now, it helps the process scale better. -- AY On May 7, 2015 at 19:02:36, Benedict Elliott Smith ( belliottsm...@datastax.com) wrote: I would argue that we must *at least* do the following for now. If we get this right, the extra staging branches can certainly wait to be assessed until later. IMO, any patch should have a branch in CI for each affected mainline branch, and should have the commit completely wired up (CHANGES.txt, commit message, the works), so that it can be merged straight in. If it conflicts significantly, it can be bumped back to the author/reviewer to refresh. On Thu, May 7, 2015 at 4:16 PM, Aleksey Yeschenko alek...@apache.org wrote: I would argue that we must *at least* do the following for now. If your patch is 2.1-based, you need to create a private git branch for that and a merged trunk branch ( and -trunk). And you don’t push anything until cassci validates all of those three branches, first. An issue without a link to cassci for both of those branches passing doesn’t qualify as done to me. That alone will be enough to catch most merge-related regressions. Going with staging branches would also prevent any issues from concurrent pushes, but given the opposition, I’m fine with dropping that requirement, for now. -- AY On May 7, 2015 at 18:04:20, Josh McKenzie (josh.mcken...@datastax.com) wrote: Merging is *hard*. Especially 2.1 - 3.0, with many breaking API changes (this is before 8099, which is going to make a *world* of hurt, and will stick around for a year). It is *very* easy to break things, with even the utmost care. While I agree re:merging, I'm not convinced the proportion of commits that will benefit from a staging branch testing pipeline is high enough to justify the time and complexity overhead to (what I expect are) the vast majority of commits that are smaller, incremental changes that won't benefit from this. On Thu, May 7, 2015 at 9:56 AM, Ariel Weisberg ariel.weisb...@datastax.com wrote: Hi, Sorry didn't mean to blame or come off snarky. I just it is important not to #include our release process from somewhere else. We don't have to do anything unless it is necessary to meet some requirement of what we are trying to do. So the phrase Trunk is always releasable definitely has some wiggle room because you have to define what your release process is. If your requirement is that at any time you be able to tag trunk and ship it within minutes then yes staging branches help solve that problem. The reality is that the release process always takes low single digit days because you branch trunk, then wait for longer running automated tests to run against that branch. If there happens to be a failure you may have to update the branch, but you have bounded how much brokeness sits between you and release already. We also don't have a requirement to be able to ship nigh immediately. We can balance the cost of extra steps and process against the cost of having to delay some releases some of the time by a few days and pick whichever is more important. We are stilly reducing the amount of time it takes to get a working release. Reduced enough that we should be able to ship every month without difficulty. I have been on a team roughly our size that shipped every three weeks without having staging branches. Trunk broke infrequently enough it wasn't an issue and when it did break it wasn't hard to address. The real pain point was flapping tests and the diffusion of responsibility that prevented them from getting fixed. If I were trying to sell staging branches I would work the angle that I want to be able to bisect trunk without coming
A proposal for how we use JIRA in the tick-tock release process
In the interests of making the tick tock release process as smooth and efficient as possible, I’d like to propose a few procedural JIRA rules: * Let’s use the In Progress status to indicate when development is actually in progress. This can be a very useful indicator to testers that it’s the right time to engage the developer to discuss testing plans and agree on the Definition of Done for that ticket. * Let’s use the Testing status after a patch has been reviewed, and before the patch gets merged, to be an opportunity for people to chime in about whether or not the proposed change has adequate testing and meets the Definition of Done. It’s not my intention to add needless formalities to the process or to slow things down for the developers - test planning and test implementation should always be done concurrently while a test is In Progress, so the Testing status for a ticket should be short lived. What it gives us is a more solid way of knowing that what gets merged into trunk is in as best shape as it can be, and is always deliverable. I would also note that the Testing phase should not be regarded as only for the DataStax test engineering team. It really should be a collaborative phase where we all can discuss the tests that everyone has contributed. If a developer is confident that all the testing is in place (unit tests, dtests, etc.) then they should feel free to skip the testing status. -- [image: datastax_logo.png] http://www.datastax.com/ Ryan McGuire Software Engineering Manager in Test | r...@datastax.com [image: linkedin.png] https://www.linkedin.com/in/enigmacurry [image: twitter.png] http://twitter.com/enigmacurry http://github.com/enigmacurry
Re: 3.0 and the Cassandra release process
I'm taking notes from the infrastructure doc and wrote down some action items for my team: https://gist.github.com/EnigmaCurry/d53eccb55f5d0986c976 -- [image: datastax_logo.png] http://www.datastax.com/ Ryan McGuire Software Engineering Manager in Test | r...@datastax.com [image: linkedin.png] https://www.linkedin.com/in/enigmacurry [image: twitter.png] http://twitter.com/enigmacurry http://github.com/enigmacurry On Thu, Mar 19, 2015 at 1:08 PM, Ariel Weisberg ariel.weisb...@datastax.com wrote: Hi, I realized one of the documents we didn't send out was the infrastructure side changes I am looking for. This one is maybe a little rougher as it was the first one I wrote on the subject. https://docs.google.com/document/d/1Seku0vPwChbnH3uYYxon0UO-b6LDtSqluZiH--sWWi0/edit?usp=sharing The goal is to have infrastructure that gives developers as close to immediate feedback as possible on their code before they merge. Feedback that is delayed to after merging to trunk should come in a day or two and there is a product owner (Michael Shuler) responsible for making sure that issues are addressed quickly. QA is going to help by providing developers with a better tools for writing higher level functional tests that explore all of the functions together along with the configuration space without developers having to do any work other then plugging in functionality to exercise and then validate something specific. This kind of harness is hard to get right and make reliable and expressive so they have their work cut out for them. It's going to be an iterative process where the tests improve as new work introduces missing coverage and as bugs/regressions drive the introduction of new tests. The monthly retrospective (planning on doing that first of the month) is also going to help us refine the testing and development process. Ariel On Thu, Mar 19, 2015 at 7:23 AM, Jason Brown jasedbr...@gmail.com wrote: +1 to this general proposal. I think the time has finally come for us to try something new, and this sounds legit. Thanks! On Thu, Mar 19, 2015 at 12:49 AM, Phil Yang ud1...@gmail.com wrote: Can I regard the odd version as the development preview and the even version as the production ready? IMO, as a database infrastructure project, stable is more important than other kinds of projects. LTS is a good idea, but if we don't support non-LTS releases for enough time to fix their bugs, users on non-LTS release may have to upgrade a new major release to fix the bugs and may have to handle some new bugs by the new features. I'm afraid that eventually people would only think about the LTS one. 2015-03-19 8:48 GMT+08:00 Pavel Yaskevich pove...@gmail.com: +1 On Wed, Mar 18, 2015 at 3:50 PM, Michael Kjellman mkjell...@internalcircle.com wrote: For most of my life I’ve lived on the software bleeding edge both personally and professionally. Maybe it’s a personal weakness, but I guess I get a thrill out of the problem solving aspect? Recently I came to a bit of an epiphany — the closer I keep to the daily build — generally the happier I am on a daily basis. Bugs happen, but for the most part (aside from show stopper bugs), pain points for myself in a given daily build can generally can be debugged to 1 or maybe 2 root causes, fixed in ~24 hours, and then life is better the next day again. In comparison, the old waterfall model generally means taking an “official” release at some point and waiting for some poor soul (or developer) to actually run the thing. No matter how good the QA team is, until it’s actually used in the real world, most bugs aren’t found. If you and your organization can wait 24 hours * number of bugs discovered after people actually started using the thing, you end up with a “usable build” around the holy-grail minor X.X.5 release of Cassandra. I love the idea of the LTS model Jonathan describes because it means more code can get real testing and “bake” for longer instead of sitting largely unused on some git repository in a datacenter far far away. A lot of code has changed between 2.0 and trunk today. The code has diverged to the point that if you write something for 2.0 (as the most stable major branch currently available), merging it forward to 3.0 or after generally means rewriting it. If the only thing that comes out of this is a smaller delta of LOC between the deployable version/branch and what we can develop against and what QA is focused on I think that’s a massive win. Something like CASSANDRA-8099 will need 2x the baking time of even many of the more risky changes the project has made. While I wouldn’t want to run a build with CASSANDRA-8099 in it anytime soon
Re: [VOTE] Release Apache Cassandra 2.0.12
dtests looks good except for CASSANDRA-7281 https://issues.apache.org/jira/browse/CASSANDRA-7281, but looks like that got pushed up to 2.0.13. On Fri, Jan 9, 2015 at 5:34 PM, Jake Luciani j...@apache.org wrote: I propose the following artifacts for release as 2.0.12. sha1: df1f5ead0950d4d3058cf6fe0fcae9ef528014fa Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.0.12-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-1041/org/apache/cassandra/apache-cassandra/2.0.12/ Staging repository: https://repository.apache.org/content/repositories/orgapachecassandra-1041/ The artifacts as well as the debian package are also available here: http://people.apache.org/~jake The vote will be open for 72 hours (longer if needed). [1]: http://goo.gl/7JmQWg (CHANGES.txt) [2]: http://goo.gl/bqxiFR (NEWS.txt)
Re: [VOTE] Release Apache Cassandra 2.1.0-rc5
+1 On Sun, Aug 3, 2014 at 10:44 PM, Jake Luciani jak...@gmail.com wrote: +1 On Saturday, August 2, 2014, Sylvain Lebresne sylv...@datastax.com wrote: I propose the following artifacts for release as 2.1.0-rc5. Unless someone strongly object, we'll keep the vote period for this one to 24h. sha1: cfb335e39b080c031faf76671c67cc003af22635 Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.1.0-rc5-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-1022/org/apache/cassandra/apache-cassandra/2.1.0-rc5/ Staging repository: https://repository.apache.org/content/repositories/orgapachecassandra-1022/ The artifacts as well as the debian package are also available here: http://people.apache.org/~slebresne/ The vote will be open for 24 hours (longer if needed). [1]: http://goo.gl/AC2QUA (CHANGES.txt) [2]: http://goo.gl/zUtXcO (NEWS.txt) -- http://twitter.com/tjake
Re: CQL unit tests vs dtests
We actually have some jython tests for a few test suites we wanted to use the java driver with: https://github.com/riptano/cassandra-dtest-jython On Thu, May 22, 2014 at 2:36 PM, Jake Luciani jak...@gmail.com wrote: Jython! :D On Thu, May 22, 2014 at 12:09 PM, Benedict Elliott Smith belliottsm...@datastax.com wrote: I would for defining the cql tests in a way that permits them being run as both dtests and unit tests. But since we're on python for dtests that could be troublesome. On 22 May 2014 17:03, Jeremiah D Jordan jerem...@datastax.com wrote: The only thing I worry about here is that the unit tests don't come into the system the same way user queries will. So we still need the system level dtests. So I don't think all CQL tests should be unit tests, but I am all for there being unit level CQL tests. On May 22, 2014, at 10:58 AM, Sylvain Lebresne sylv...@datastax.com wrote: On Wed, May 21, 2014 at 10:46 PM, Jonathan Ellis jbel...@gmail.com wrote: I do think that CQL tests in general make more sense as unit tests, but I'm not so anal that I'm going to insist on rewriting existing ones. But in theory, if I had an infinite army of interns, sure. I'd have one of them do that. :) But in the real world, compared to saying we don't have any cql unit tests, so we should always write them as dtests to be consistent I think having mixed unit + dtests is the lesser of evils. Fair enough. I'll try to make CQL tests as unit tests from now on as much as possible (I can't promise a few misfire in the short term). Let's hope you find your infinite intern army someday. -- Sylvain -- http://twitter.com/tjake
Re: [VOTE] Release Apache Cassandra 2.0.7
I'd like to see some dev response to my notes on CASSANDRA-6525https://issues.apache.org/jira/browse/CASSANDRA-6525, but other than that dtests for 2.0 look good. On Mon, Apr 14, 2014 at 2:32 PM, Pavel Yaskevich pove...@gmail.com wrote: Can I push new release of the thrift-server before we roll 2.0.7? On Mon, Apr 14, 2014 at 10:57 AM, Jonathan Ellis jbel...@gmail.com wrote: +1 On Apr 14, 2014 10:39 AM, Sylvain Lebresne sylv...@datastax.com wrote: sha1: 7dbbe9233ce83c2a473ba2510c827a661de99400 Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.0.7-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-1009/org/apache/cassandra/apache-cassandra/2.0.7/ Staging repository: https://repository.apache.org/content/repositories/orgapachecassandra-1009/ The artifacts as well as the debian package are also available here: http://people.apache.org/~slebresne/ The vote will be open for 72 hours (longer if needed). [1]: http://goo.gl/6yg6Xh (CHANGES.txt) [2]: http://goo.gl/GxmBC9 (NEWS.txt)
Re: cassandra-2.1 testing
Thanks Mikhail, I remember I fixed this in ccmhttps://github.com/pcmanus/ccm/commit/2c19025dc51578f074b889ad4b78ac9e0625e715, but then I forgot that cqlsh_tests wasn't using the ccm api for invoking that. On Tue, Apr 8, 2014 at 5:54 PM, Mikhail Stepura mikhail.step...@outlook.com wrote: I suspect cqlsh-related tests are failing because cqlsh was trying to connect to 9160 (thrift) Could someone verify if switching to 'binary' solves this problem? Here is the patch: https://github.com/Mishail/cassandra-dtest/commit/33d648066df3f34a42035397fb366540784b6139 https://issues.apache.org/jira/browse/CASSANDRA-7003 https://issues.apache.org/jira/browse/CASSANDRA-7004 On Apr 8, 2014, at 13:47, Michael Shuler mich...@pbandjelly.org wrote: Here is the current state of unit and dtest results for the cassandra-2.1 branch, from the commit I started with this morning. origin/cassandra-2.1 commit 0fa5cba35b9c97ad34dedbbd89a933f5b9d156e9 Date: Tue Apr 8 12:26:09 2014 -0500 Unit Tests Failing: org.apache.cassandra.db.CommitLogTest.testExceedSegmentSizeWithOverhead Error Message: java.lang.IllegalArgumentException: Mutation of 33554430 bytes is too large for the maxiumum size of 16777216 at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:205) at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:192) at org.apache.cassandra.db.CommitLogTest.testExceedSegmentSizeWithOverhead(CommitLogTest.java:177) Jira: https://issues.apache.org/jira/browse/CASSANDRA-6764(Unresolved; 2.1 beta2): Using Batch commitlog_sync is slow and doesn't actually batch writes Dtests Failing: bootstrap_test Should be fixed with a patch to ccm: https://github.com/pcmanus/ccm/pull/109 concurrent_schema_changes_test.TestConcurrentSchemaChanges.snapshot_test Jira: https://issues.apache.org/jira/browse/CASSANDRA-7002 concurrent_schema_changes_test snapshot_test dtest needs to account for hashed data dirs in 2.1 cqlsh_tests.TestCqlsh.test_eat_glass Jira: https://issues.apache.org/jira/browse/CASSANDRA-7003 cqlsh_tests test_eat_glass dtest fails on 2.1 cqlsh_tests.TestCqlsh.test_simple_insert Jira: https://issues.apache.org/jira/browse/CASSANDRA-7004 cqlsh_tests test_simple_insert dtest fails on 2.1 paxos_tests Jira: ToDo, once I have a little better understanding of the errors repair_test Jira: https://issues.apache.org/jira/browse/CASSANDRA-7005 repair_test dtest fails on 2.1 secondary_indexes_test.TestSecondaryIndexes.test_6924 Jira: https://issues.apache.org/jira/browse/CASSANDRA-7006 secondary_indexes_test test_6924 dtest fails on 2.1 topology_test Jira: https://issues.apache.org/jira/browse/CASSANDRA-7009 topology_test dtest fails in 2.1 upgrade_supercolumns_test Jira: https://issues.apache.org/jira/browse/CASSANDRA-7008 upgrade_supercolumns_test dtest failing in 2.1 upgrade_through_versions_test Actively being improved by Russ Hatch - will hold on Jira for the moment -- Kind regards, Michael
Re: Compaction Progress Greater than 100%
Looks like the same bug to me, whether it's caused by the same thing as it was originally doesn't really matter. Can you add your steps to reproduce it there? I'll reopen the bug for you. Thanks for the bug report! On Wed, Apr 2, 2014 at 11:43 PM, Ji Cheng memol...@gmail.com wrote: Hi, I found the compaction progress is greater than 100%. I'm running 2.0.5. Not sure if it is a regression of CASSANDRA-4807https://issues.apache.org/jira/browse/CASSANDRA-4807 since we are performing size-tiered compaction in level 0 in 2.0. jicheng@S407:~$ /opt/cassandra/bin/nodetool -h 192.168.0.190 compactionstats pending tasks: 244 compaction typekeyspace table completed total unit progress Validation user address 3118858200 2725845729 bytes 114.42% related configuration in cf definition: not CQL3 table column_type = 'Standard' min_compaction_threshold = 4 max_compaction_threshold = 32 compaction_strategy = 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy' compaction_strategy_options = {'sstable_size_in_mb' : '160'} compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'} related configuration in cassandra.yaml: concurrent_compactors: 4 multithreaded_compaction: false Thanks. Cheng
Re: [jira] [Commented] (CASSANDRA-6746) Reads have a slow ramp up in speed
32G nodes. I'm not mucking with heap settings so it's using -Xms8023M -Xmx8023M -Xmn1600M
Re: [VOTE] Release Apache Cassandra 2.0.1 (strike 2)
I retract my -1 (for now, I can't reproduce CASSANDRA-6074.) Sorry for the noise. On Fri, Sep 20, 2013 at 8:15 PM, Ryan McGuire r...@datastax.com wrote: Unfortunately, -1 from me. See CASSANDRA-6074 https://issues.apache.org/jira/browse/CASSANDRA-6074 On Fri, Sep 20, 2013 at 12:16 PM, Pavel Yaskevich pove...@gmail.comwrote: +1 On Sep 20, 2013, at 5:55 AM, Gary Dusbabek gdusba...@gmail.com wrote: +1 On Fri, Sep 20, 2013 at 4:39 AM, Sylvain Lebresne sylv...@datastax.com wrote: Same player, try again. I propose the following artifacts for release as 2.0.1. sha1: eb96db6c19515e6d1215230f29d25b46fcd005ef Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.0.1-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-085/org/apache/cassandra/apache-cassandra/2.0.1/ Staging repository: https://repository.apache.org/content/repositories/orgapachecassandra-085/ The artifacts as well as the debian package are also available here: http://people.apache.org/~slebresne/ The vote will be open for 72 hours (longer if needed). [1]: http://goo.gl/LNAdjU (CHANGES.txt) [2]: http://goo.gl/KuHgLs (NEWS.txt)
Re: [VOTE] Release Apache Cassandra 2.0.1 (strike 2)
Unfortunately, -1 from me. See CASSANDRA-6074 https://issues.apache.org/jira/browse/CASSANDRA-6074 On Fri, Sep 20, 2013 at 12:16 PM, Pavel Yaskevich pove...@gmail.com wrote: +1 On Sep 20, 2013, at 5:55 AM, Gary Dusbabek gdusba...@gmail.com wrote: +1 On Fri, Sep 20, 2013 at 4:39 AM, Sylvain Lebresne sylv...@datastax.com wrote: Same player, try again. I propose the following artifacts for release as 2.0.1. sha1: eb96db6c19515e6d1215230f29d25b46fcd005ef Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.0.1-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-085/org/apache/cassandra/apache-cassandra/2.0.1/ Staging repository: https://repository.apache.org/content/repositories/orgapachecassandra-085/ The artifacts as well as the debian package are also available here: http://people.apache.org/~slebresne/ The vote will be open for 72 hours (longer if needed). [1]: http://goo.gl/LNAdjU (CHANGES.txt) [2]: http://goo.gl/KuHgLs (NEWS.txt)
Re: Apache Cassandra 2.0.0 rc1
I've seen similar slowdowns in 2.0 - roughly the same 15%. For instance, http://i.imgur.com/Qi6oo62.png That's a comparison of 1.2.6 to 2.0beta1. Ignore the lines that say speculative_retry=95percentile and ALWAYS. Default 1.2 beats Default 2.0 in overall operation time. Bisecting the cause of the problem is on my TODO list. On Mon, Aug 12, 2013 at 2:27 PM, Jonathan Ellis jbel...@gmail.com wrote: We haven't seen that in our tests. Suggest bisecting to figure out what changed it for your workload. On Mon, Aug 12, 2013 at 2:47 AM, Radim Kolar h...@filez.com wrote: thrift, sync -- Jonathan Ellis Project Chair, Apache Cassandra co-founder, http://www.datastax.com @spyced