Main question I have then is what happens in the case of retry due to network failure, where the client doesn't get a success response, but it succeeds / will succeed proxy server side? What do I have to ensure that no duplicates are written to distributedlog in such a case?
-Cameron On Wed, Oct 5, 2016 at 11:33 AM, Leigh Stewart <lstew...@twitter.com.invalid > wrote: > > > > So, my basic question is if this is currently possible in the proxy? I > > don't believe it gives these guarantees as it stands today, but I am not > > 100% of how all of the futures in the code handle failures. > > > > As long as you use this method > <http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-4> > to write this is possible. > > The writeBulk is not atomic and we will probably deprecated it at some > point. > > If not, where in the code would be the relevant places to add the ability > > to do this, and would the project be interested in a pull request? > > > Does the example linked above meet your requirements? > > Thx > > On Tue, Oct 4, 2016 at 12:39 PM, Cameron Hatfield <kin...@gmail.com> > wrote: > > > I have a question about the Proxy Client. Basically, for our use cases, > we > > want to guarantee ordering at the key level, irrespective of the ordering > > of the partition it may be assigned to as a whole. Due to the source of > the > > data (HBase Replication), we cannot guarantee that a single partition > will > > be owned for writes by the same client. This means the proxy client works > > well (since we don't care which proxy owns the partition we are writing > > to). > > > > > > However, the guarantees we need when writing a batch consists of: > > Definition of a Batch: The set of records sent to the writeBatch endpoint > > on the proxy > > > > 1. Batch success: If the client receives a success from the proxy, then > > that batch is successfully written > > > > 2. Inter-Batch ordering : Once a batch has been written successfully by > the > > client, when another batch is written, it will be guaranteed to be > ordered > > after the last batch (if it is the same stream). > > > > 3. Intra-Batch ordering: Within a batch of writes, the records will be > > committed in order > > > > 4. Intra-Batch failure ordering: If an individual record fails to write > > within a batch, all records after that record will not be written. > > > > 5. Batch Commit: Guarantee that if a batch returns a success, it will be > > written > > > > 6. Read-after-write: Once a batch is committed, within a limited > time-frame > > it will be able to be read. This is required in the case of failure, so > > that the client can see what actually got committed. I believe the > > time-frame part could be removed if the client can send in the same > > sequence number that was written previously, since it would then fail and > > we would know that a read needs to occur. > > > > > > So, my basic question is if this is currently possible in the proxy? I > > don't believe it gives these guarantees as it stands today, but I am not > > 100% of how all of the futures in the code handle failures. > > If not, where in the code would be the relevant places to add the ability > > to do this, and would the project be interested in a pull request? > > > > > > Thanks, > > Cameron > > >
