> > Transactional acknowledgement also needs to be taken into account > > I don't think we have to treat `transactional acknowledgement` as a special > case. currently `acknowledgment` are actually "append" operations into > cursor ledgers. > So the problem set can be reduced as `atomic append` to both data ledgers > and cursor ledgers. in that way, we can use one solution for handling > appending data and updating cursors.
Acknowledges are different though as they have to take conflicts into account. For publishing messages, transactions are only really a case of making sure if one message from the transaction is visible, then all messages from the transaction are visible, i.e. atomicity. There's no need for isolation or consistency in ACID terms because all these messages are independent. However, in the case of acknowledgements, we want to ensure that only one transaction can acknowledges a specific message (leaving cumulative ack aside for now). This means that before committing the transaction we need to check if the transaction can be committed. If any message acknowledged by the transaction has already been acknowledged at this point, then we need to abort the transaction as a conflict has occurred. > Kafka's implementation is interleaving committed messages with uncommitted > messages at storage. Personally I think it is a very ugly design and > implementation. I don't think it's particularly ugly. Databases using MVCC do something similar. This has a nice property for the clients. Our transactional system should be optimistic, as the only cause for aborts will be client crashes and conflicts on acks, both of which should be rare events. Transactions will likely live for a very short time. So if the data is interleaved, it can be cached at the client, and the client can then surface it to the user when then commit arrives shortly afterwards, without any modification to the dispatch path. It also gives clients the ability to read uncommitted data, though i don't know why they would want to. > 1) for any transaction, write the messages to a separate ledger (or multiple separate ledger). When you say a separate ledger, are you talking about a ledger per transaction? Or is there one log for the system? -Ivan
