I still can not edit the wiki. Can any of the pmc members grant me the permissions?
- Xi On Sat, Sep 17, 2016 at 10:35 PM, Xi Liu <xi.liu....@gmail.com> wrote: > Sijie, > > I attempted to create a wiki page under that space. I found that I am not > authorized with edit permission. > > Can any of the committers grant me the wiki edit permission? My account is > "xi.liu.ant". > > - Xi > > > On Tue, Sep 13, 2016 at 9:26 AM, Sijie Guo <si...@apache.org> wrote: > >> This sounds interesting ... I will take a closer look and give my comments >> later. >> >> At the same time, do you mind creating a wiki page to put your idea there? >> You can add your wiki page under >> https://cwiki.apache.org/confluence/display/DL/Project+Proposals >> >> You might need to ask in the dev list to grant the wiki edit permissions >> to >> you once you have a wiki account. >> >> - Sijie >> >> >> On Mon, Sep 12, 2016 at 2:20 AM, Xi Liu <xi.liu....@gmail.com> wrote: >> >> > Hello, >> > >> > I asked the transaction support in distributedlog user group two months >> > ago. I want to raise this up again, as we are looking for using >> > distributedlog for building a transactional data service. It is a major >> > feature that is missing in distributedlog. We have some ideas to add >> this >> > to distributedlog and want to know if they make sense or not. If they >> are >> > good, we'd like to contribute and develop with the community. >> > >> > Here are the thoughts: >> > >> > ------------------------------------------------- >> > >> > From our understanding, DL can provide "at-least-once" delivery semantic >> > (if not, please correct me) but not "exactly-once" delivery semantic. >> That >> > means that a message can be delivered one or more times if the reader >> > doesn't handle duplicates. >> > >> > The duplicates come from two places, one is at writer side (this assumes >> > using write proxy not the core library), while the other one is at >> reader >> > side. >> > >> > - writer side: if the client attempts to write a record to the write >> > proxies and gets a network error (e.g timeouts) then retries, the >> retrying >> > will potentially result in duplicates. >> > - reader side:if the reader reads a message from a stream and then >> crashes, >> > when the reader restarts it would restart from last known position >> (DLSN). >> > If the reader fails after processing a record and before recording the >> > position, the processed record will be delivered again. >> > >> > The reader problem can be properly addressed by making use of the >> sequence >> > numbers of records and doing proper checkpointing. For example, in >> > database, it can checkpoint the indexed data with the sequence number of >> > records; in flink, it can checkpoint the state with the sequence >> numbers. >> > >> > The writer problem can be addressed by implementing an idempotent >> writer. >> > However, an alternative and more powerful approach is to support >> > transactions. >> > >> > *What does transaction mean?* >> > >> > A transaction means a collection of records can be written >> transactionally >> > within a stream or across multiple streams. They will be consumed by the >> > reader together when a transaction is committed, or will never be >> consumed >> > by the reader when the transaction is aborted. >> > >> > The transaction will expose following guarantees: >> > >> > - The reader should not be exposed to records written from uncommitted >> > transactions (mandatory) >> > - The reader should consume the records in the transaction commit order >> > rather than the record written order (mandatory) >> > - No duplicated records within a transaction (mandatory) >> > - Allow interleaving transactional writes and non-transactional writes >> > (optional) >> > >> > *Stream Transaction & Namespace Transaction* >> > >> > There will be two types of transaction, one is Stream level transaction >> > (local transaction), while the other one is Namespace level transaction >> > (global transaction). >> > >> > The stream level transaction is a transactional operation on writing >> > records to one stream; the namespace level transaction is a >> transactional >> > operation on writing records to multiple streams. >> > >> > *Implementation Thoughts* >> > >> > - A transaction is consist of begin control record, a series of data >> > records and commit/abort control record. >> > - The begin/commit/abort control record is written to a `commit` log >> > stream, while the data records will be written to normal data log >> streams. >> > - The `commit` log stream will be the same log stream for stream-level >> > transaction, while it will be a *system* stream (or multiple system >> > streams) for namespace-level transactions. >> > - The transaction code looks like as below: >> > >> > <code> >> > >> > Transaction txn = client.transaction(); >> > Future<DLSN> result1 = txn.write(stream-0, record); >> > Future<DLSN> result2 = txn.write(stream-1, record); >> > Future<DLSN> result3 = txn.write(stream-2, record); >> > Future<Pair<DLSN, DLSN>> result = txn.commit(); >> > >> > </code> >> > >> > if the txn is committed, all the write futures will be satisfied with >> their >> > written DLSNs. if the txn is aborted, all the write futures will be >> failed >> > together. there is no partial failure state. >> > >> > - The actually data flow will be: >> > >> > 1. writer get a transaction id from the owner of the `commit' log stream >> > 1. write the begin control record (synchronously) with the transaction >> id >> > 2. for each write within the same txn, it will be assigned a local >> sequence >> > number starting from 0. the combination of transaction id and local >> > sequence number will be used later on by the readers to de-duplicate >> > records. >> > 3. the commit/abort control record will be written based on the results >> > from 2. >> > >> > - Application can supply a timeout for the transaction when #begin() a >> > transaction. The owner of the `commit` log stream can abort transactions >> > that never be committed/aborted within their timeout. >> > >> > - Failures: >> > >> > * all the log records can be simply retried as they will be >> de-duplicated >> > probably at the reader side. >> > >> > - Reader: >> > >> > * Reader can be configured to read uncommitted records or committed >> records >> > only (by default read uncommitted records) >> > * If reader is configured to read committed records only, the read ahead >> > cache will be changed to maintain one additional pending committed >> records. >> > the pending committed records map is bounded and records will be dropped >> > when read ahead is moving. >> > * when the reader hits a commit record, it will rewind to the begin >> record >> > and start reading from there. leveraging the proper read ahead cache and >> > pending commit records cache, it would be good for both short >> transactions >> > and long transactions. >> > >> > - DLSN, SequenceId: >> > >> > * We will add a fourth field to DLSN. It is `local sequence number` >> within >> > a transaction session. So the new DLSN of records in a transaction will >> be >> > the DLSN of commit control record plus its local sequence number. >> > * The sequence id will be still the position of the commit record plus >> its >> > local sequence number. The position will be advanced with total number >> of >> > written records on writing the commit control record. >> > >> > - Transaction Group & Namespace Transaction >> > >> > using one single log stream for namespace transaction can cause the >> > bottleneck problem since all the begin/commit/end control records will >> have >> > to go through one log stream. >> > >> > the idea of 'transaction group' is to allow partitioning the writers >> into >> > different transaction groups. >> > >> > clients can specify the `group-name` when starting the transaction. if >> > there is no `group-name` specified, it will use the default `commit` >> log in >> > the namespace for creating transactions. >> > >> > ------------------------------------------------- >> > >> > I'd like to collect feedbacks on this idea. Appreciate any comments and >> if >> > anyone is also interested in this idea, we'd like to collaborate with >> the >> > community. >> > >> > >> > - Xi >> > >> > >
