For those who missed it, my talk discussing this CEP at ApacheCon is now available to view: https://www.youtube.com/watch?v=YAE7E-QEAvk
From: Oleksandr Petrov <oleksandr.pet...@gmail.com> Date: Monday, 11 October 2021 at 10:11 To: dev <dev@cassandra.apache.org> Subject: Re: [DISCUSS] CEP-15: General Purpose Transactions > I support this proposal. From what I can understand, this proposal moves us towards having the building blocks we need to correctly deliver some of the most often requested features in Cassandra. Same here. I also support this proposal and believe it opens up many new opportunities (while not limiting us / not narrowing our future options), can help us implement features we've all wanted to have implemented for years, and make significant improvements in the subsystems that were a source of issues for a long time. I think it's also good to start with CAS batches: it's a great way to make the feature available and work incrementally. After this lands, people will be able to use Accord/MPT in different subsystems and get busy implementing all sorts of other features and improvements on top of it. On Sat, Oct 9, 2021 at 4:18 PM Joseph Lynch <joe.e.ly...@gmail.com> wrote: > > With the proposal hitting the one-month mark, the contributors are > interested in gauging the developer community's response to the proposal. > > I support this proposal. From what I can understand, this proposal > moves us towards having the building blocks we need to correctly > deliver some of the most often requested features in Cassandra. For > example it seems to unlock: batches that actually work, registers that > offer fast compare and swap, global secondary indices that can be > correctly maintained, and more. Therefore, given the benefit to the > community, I support working towards that foundation that will allow > us to build solutions in Cassandra that pay consensus closer to > mutation instead of lazily at read/repair time. > > I think the feedback in this thread around interface (what statements > will this facilitate and how will the library integrate with Cassandra > itself), performance (how fast will these transactions be, will we > offer bounded stale reads, etc ...), and implementation (how does this > compare/contrast with other consensus approaches) has been > informative, but at this point I think it makes sense to start trying > to make incremental progress towards a functional integration to > discover any remaining areas for improvement. > > Cheers and thank you! > -Joey > > > > On Thu, Oct 7, 2021 at 10:51 AM C. Scott Andreas <sc...@paradoxica.net> > wrote: > > > > Hi Jonathan, > > > > Following up on my message yesterday as it looks like our replies may > have crossed en route. > > > > Thanks for bumping your message from earlier in our discussion. I > believe we have addressed most of these questions on the thread, in > addition to offering a presentation on this and related work at ApacheCon, > a discussion hosted following that presentation at ApacheCon, and in ASF > Slack. Contributors have further offered an opportuntity to discuss > specific questions via videoconference if it helps to speak live. I'd be > happy to do so as well. > > > > Since your original message, discussion has covered a lot of ground on > the related databases you've mentioned: > > – Henrik has shared expertise related to MongoDB and its implementation. > > – You've shared an overview of Calvin. > > – Alex Miller has helped us review the work relative to other Paxos > algorithms and identified a few great enhancements to incorporate. > > – The paper discusses related approaches in FoundationDB, CockroachDB, > and Yugabyte. > > – Subsequent discussion has contrasted the implementation to DynamoDB, > Google Cloud BigTable, and Google Cloud Spanner (noting specifically that > the protocol achieves Spanner's 1x round-trip without requiring specialized > hardware). > > > > In my reply yesterday, I've attempted to crystallize what becomes > possible via CQL: one-shot multi-partition transactions in the first > implementation and a 4x latency reduction on writes / 2x latency reduction > on reads relative to today; along with the ability to build upon this work > to enable interactive transactions in the future. > > > > I believe we've exercised the questions you've raised and am grateful > for the ground we've covered. If you have further questions that are > difficult to exercise via email, please let me know if you'd like to > arrange a call (open-invite); we'd be happy to discuss live as well. > > > > With the proposal hitting the one-month mark, the contributors are > interested in gauging the developer community's response to the proposal. > We warrant our ability to focus durably on the project; execute this > development on ASF JIRA in collaboration with other contributors; engage > with members of the developer and user community on feedback, enhancements, > and bugs; and intend deliver it to completion at a standard of readiness > suitable for production transactional systems of record. > > > > Thanks, > > > > – Scott > > > > On Oct 6, 2021, at 8:25 AM, C. Scott Andreas <sc...@paradoxica.net> > wrote: > > > > > > > > Hi folks, > > > > Thanks for discussion on this proposal, and also to Benedict who’s been > fielding questions on the list! > > > > I’d like to restate the goals and problem statement captured by this > proposal and frame context. > > > > Today, lightweight transactions limit users to transacting over a single > partition. This unit of atomicity has a very low upper limit in terms of > the amount of data that can be CAS’d over; and doing so leads many to > design contorted data models to cram different types of data into one > partition for the purposes of being able to CAS over it. We propose that > Cassandra can and should be extended to remove this limit, enabling users > to issue one-shot transactions that CAS over multiple keys – including CAS > batches, which may modify multiple keys. > > > > To enable this, the CEP authors have designed a novel, leaderless > paxos-based protocol unique to Cassandra, offered a proof of its > correctness, a whitepaper outlining it in detail, along with a prototype > implementation to incubate development, and integrated it with Maelstrom > from jepsen.io to validate linearizability as more specific test > infrastructure is developed. This rigor is remarkable, and I’m thrilled to > see such a degree of investment in the area. > > > > Even users who do not require the capability to transact across > partition boundaries will benefit. The protocol reduces message/WAN > round-trips by 4x on writes (4 → 1) and 2x on reads (2 → 1) in the common > case against today’s baseline. These latency improvements coupled with the > enhanced flexibility of what can be transacted over in Cassandra enable new > classes of applications to use the database. > > > > In particular, 1xRTT read/write transactions across partitions enable > Cassandra to be thought of not just as a strongly consistent database, but > even a transactional database - a mode many may even prefer to use by > default. Given this capability, Apache Cassandra has an opportunity to > become one of – or perhaps the only – database in the industry that can > store multiple petabytes of data in a single database; replicate it across > many regions; and allow users to transact over any subset of it. These are > capabilities that can be met by no other system I’m aware of on the market. > Dynamo’s transactions are single-DC. Google Cloud BigTable does not support > transactions. Spanner, Aurora, CloudSQL, and RDS have far lower scalability > limits or require specialized hardware, etc. > > > > This is an incredible opportunity for Apache Cassandra - to surpass the > scalability and transactional capability of some of the most advanced > systems in our industry - and to do so in open source, where anyone can > download and deploy the software to achieve this without cost; and for > students and researchers to learn from and build upon as well (a team from > UT-Austin has already reached out to this effect). > > > > As Benedict and Blake noted, the scope of what’s captured in this > proposal is also not terminal. While the first implementation may extend > today’s CAS semantics to multiple partitions with lower latency, the > foundation is suitable to build interactive transactions as well — which > would be remarkable and is something that I hadn’t considered myself at the > onset of this project. > > > > To that end, the CEP proposes the protocol, offers a validated > implementation, and the initial capability of extending today’s > single-partition transactions to multi-partition; while providing the > flexibility to build upon this work further. > > > > A simple example of what becomes possible when this work lands and is > integrated might be: > > > > ––– > > BEGIN BATCH > > UPDATE tbl1 SET value1 = newValue1 WHERE partitionKey = k1 > > UPDATE tbl2 SET value2 = newValue2 WHERE partitionKey = k2 AND > conditionValue = someCondition > > APPLY BATCH > > ––– > > > > I understand that this query is present in the CEP and my intent isn’t > to recommend that folks reread it if they’ve given a careful reading > already. But I do think it’s important to elaborate upon what becomes > possible when this query can be issued. > > > > Users of Cassandra who have designed data models that cram many types of > data into a single partition for the purposes of atomicity no longer need > to. They can design their applications with appropriate schemas that > wouldn’t leave Codd holding his nose. They’re no longer pushed into > antipatterns that result in these partitions becoming huge and potentially > unreadable. Cassandra doesn’t become fully relational in this CEP - but it > becomes possible and even easy to design applications that transact across > tables that mimic a large amount of relational functionality. And for users > who are content to transact over a single table, they’ll find those > transactions become up to 4x faster today due to the protocol’s reduction > in round-trips. The library’s loose coupling to Apache Cassandra and > ability to be incubated out-of-tree also enables other applications to take > advantage of the protocol and is a nice step toward bringing modularity to > the project. There are a lot of good things happening here. > > > > I know I’m listed as an author - but figured I should go on record to > say “I support this CEP.” :) > > > > Thanks, > > > > – Scott > > > > On Oct 6, 2021, at 8:05 AM, Jonathan Ellis <jbel...@gmail.com> wrote: > > > > > > The problem that I keep pointing out is that you've created this CEP for > > Accord without first getting consensus that the goals and the tradeoffs > it > > makes to achieve those goals (and that it will impose on future work > around > > transactions) are the right ones for Cassandra long term. > > > > At this point I'm done repeating myself. For the convenience of anyone > > following this thread intermittently, I'll quote my first reply on this > > thread to illustrate the kind of discussion I'd like to have. > > > > ----- > > > > The whitepaper here is a good description of the consensus algorithm > itself > > as well as its robustness and stability characteristics, and its > comparison > > with other state-of-the-art consensus algorithms is very useful. In the > > context of Cassandra, where a consensus algorithm is only part of what > will > > be implemented, I'd like to see a more complete evaluation of the > > transactional side of things as well, including performance > characteristics > > as well as the types of transactions that can be supported and at least a > > general idea of what it would look like applied to Cassandra. This will > > allow the PMC to make a more informed decision about what tradeoffs are > > best for the entire long-term project of first supplementing and > ultimately > > replacing LWT. > > > > (Allowing users to mix LWT and AP Cassandra operations against the same > > rows was probably a mistake, so in contrast with LWT we’re not looking > for > > something fast enough for occasional use but rather something within a > > reasonable factor of AP operations, appropriate to being the only way to > > interact with tables declared as such.) > > > > Besides Accord, this should cover > > > > - Calvin and FaunaDB > > - A Spanner derivative (no opinion on whether that should be Cockroach or > > Yugabyte, I don’t think it’s necessary to cover both) > > - A 2PC implementation (the Accord paper mentions DynamoDB but I suspect > > there is more public information about MongoDB) > > - RAMP > > > > Here’s an example of what I mean: > > > > =Calvin= > > > > Approach: global consensus (Paxos in Calvin, Raft in FaunaDB) to order > > transactions, then replicas execute the transactions independently with > no > > further coordination. No SPOF. Transactions are batched by each sequencer > > to keep this from becoming a bottleneck. > > > > Performance: Calvin paper (published 2012) reports linear scaling of > TPC-C > > New Order up to 500,000 transactions/s on 100 machines (EC2 XL machines > > with 7GB ram and 8 virtual cores). Note that TPC-C New Order is composed > > of four reads and four writes, so this is effectively 2M reads and 2M > > writes as we normally measure them in C*. > > > > Calvin supports mixed read/write transactions, but because the > transaction > > execution logic requires knowing all partition keys in advance to ensure > > that all replicas can reproduce the same results with no coordination, > > reads against non-PK predicates must be done ahead of time > (transparently, > > by the server) to determine the set of keys, and this must be retried if > > the set of rows affected is updated before the actual transaction > executes. > > > > Batching and global consensus adds latency -- 100ms in the Calvin paper > and > > apparently about 50ms in FaunaDB. Glass half full: all transactions > > (including multi-partition updates) are equally performant in Calvin > since > > the coordination is handled up front in the sequencing step. Glass half > > empty: even single-row reads and writes have to pay the full coordination > > cost. Fauna has optimized this away for reads but I am not aware of a > > description of how they changed the design to allow this. > > > > Functionality and limitations: since the entire transaction must be known > > in advance to allow coordination-less execution at the replicas, Calvin > > cannot support interactive transactions at all. FaunaDB mitigates this by > > allowing server-side logic to be included, but a Calvin approach will > never > > be able to offer SQL compatibility. > > > > Guarantees: Calvin transactions are strictly serializable. There is no > > additional complexity or performance hit to generalizing to multiple > > regions, apart from the speed of light. And since Calvin is already > paying > > a batching latency penalty, this is less painful than for other systems. > > > > Application to Cassandra: B-. Distributed transactions are handled by the > > sequencing and scheduling layers, which are leaderless, and Calvin’s > > requirements for the storage layer are easily met by C*. But Calvin also > > requires a global consensus protocol and LWT is almost certainly not > > sufficiently performant, so this would require ZK or etcd (reasonable > for a > > library approach but not for replacing LWT in C* itself), or an > > implementation of Accord. I don’t believe Calvin would require additional > > table-level metadata in Cassandra. > > > > On Wed, Oct 6, 2021 at 9:53 AM bened...@apache.org <bened...@apache.org> > > wrote: > > > > The problem with dropping a patch on Jira is that there is no opportunity > > to point out problems, either with the fundamental approach or with the > > specific implementation. So please point out some problems I can engage > > with! > > > > > > From: Jonathan Ellis <jbel...@gmail.com> > > Date: Wednesday, 6 October 2021 at 15:48 > > To: dev <dev@cassandra.apache.org> > > Subject: Re: [DISCUSS] CEP-15: General Purpose Transactions > > On Wed, Oct 6, 2021 at 9:21 AM bened...@apache.org <bened...@apache.org> > > wrote: > > > > > The goals of the CEP are stated clearly, and these were the goals we > had > > > going into the (multi-month) research project we undertook before > > proposing > > > this CEP. These goals are necessarily value judgements, so we cannot > > expect > > > that everyone will agree that they are optimal. > > > > > > > Right, so I'm saying that this is exactly the most important thing to get > > consensus on, and creating a CEP for a protocol to achieve goals that you > > have not discussed with the community is the CEP equivalent of dropping a > > patch on Jira without discussing its goals either. > > > > That's why our conversations haven't gone anywhere, because I keep saying > > "we need discuss the goals and tradeoffs", and I'll give an example of > what > > I mean, and you keep addressing the examples (sometimes very shallowly, > "it > > would be possible to X" or "Y could be done as an optimization") while > > ignoring the request to open a discussion around the big picture. > > > > > > > > -- > > Jonathan Ellis > > co-founder, http://www.datastax.com > > @spyced > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > -- alex p
