Re: Distributed Log as Kafka's backend

Agree with Sijie, I think this is exciting work and I didn't mean to cut off your options. My objection was just about code organization. A contribs project seems like a good compromise for now, until we can think of a better place to put the code. Sijie's right though, if we want to fully produc

Re: Duplicate transaction ids

It simply must be non-decreasing- dup ids are ok. Since txid is used for positioning, if you had a long series of identical txids you might have to scan through a lot of records to find the record you were really looking for if you were doing a lookup. Other than that there should be no impact. T

Re: Duplicate transaction ids

matching" > Perhaps they are not equivalent or perhaps I have the wrong idea? > Any idea which way it is? > Thanks > ---- Original message From: Leigh Stewart > Date: 20/09/2016 18:48 (GMT+00:00) To: > dev@distributedlog.incubator.apache.org Subject: Re: Duplic

Re: Deploy DL bookkeeper branch to maven central

Hey John - I think we're pretty close to merging the needed changes to upstream bk. Can you tolerate this for another few weeks? Thanks. On Tue, Sep 20, 2016 at 10:25 AM, john.lonergan wrote: > for the moment wound or be possible to build the BK fork as an artefact in > the twitter group in cen

Re: user mail list

There's an active pull request from Sijie which contains instructions for how to build the website: https://github.com/apache/incubator-distributedlog/pull/30 On Thu, Sep 29, 2016 at 9:50 PM, Khurrum Nasim wrote: > I confirmed that I can subscribe to the user@d.i.a.o list. Thanks Chris. > > I t

Re: tutorial about setting up a global replicated log

I am currently working on this (busy production week) - follow here: https://issues.apache.org/jira/browse/DL-35 On Thu, Sep 29, 2016 at 9:58 PM, Jay Juma wrote: > Can anyone help on this? > > - Jay > > On Sun, Sep 18, 2016 at 11:17 AM, Jay Juma wrote: > > > Hello, > > > > Do you have a tutoria

Re: Question about replication repair

Streams are segmented across sets of bookies called ensembles. http://distributedlog.incubator.apache.org/docs/latest/basics/introduction#log-segments So loss of 3 bookies (assuming ensemble size 3) would never result in loss of an entire stream. Regardless, losing all replicas is non-recoverabl

Re: Proxy Client - Batch Ordering / Commit

> > So, my basic question is if this is currently possible in the proxy? I > don't believe it gives these guarantees as it stands today, but I am not > 100% of how all of the futures in the code handle failures. > As long as you use this method

Re: Proxy Client - Batch Ordering / Commit

to > network failure, where the client doesn't get a success response, but it > succeeds / will succeed proxy server side? > What do I have to ensure that no duplicates are written to distributedlog > in such a case? > > -Cameron > > On Wed, Oct 5, 2016 at 11

Re: Proxy Client - Batch Ordering / Commit

ed, Oct 5, 2016 at 2:58 PM, Cameron Hatfield wrote: > Basically, its point 6 above. What ends up being the timeout for me to > ensure that a result is committed or not, so that I guarantee a read > actually gives me the last value?? > > > -Cameron > > On Wed, Oct 5, 2016 at 2:

Re: October 2016 Podling Report Draft Feedback

Will do. On Sat, Oct 8, 2016 at 7:11 AM, Sijie Guo wrote: > Ah, Chris, thank you so much. > > Leigh was helping with the report. Leigh, can you incorporate Chris's > comments? > > - Sijie > > On Fri, Oct 7, 2016 at 6:00 PM, Chris Nauroth > wrote: > >> Thank you, Khurrum. >> >> I suspect it will

Re: Proxy Client - Batch Ordering / Commit

Cameron: Another thing we've discussed but haven't really thought through - We might be able to support some kind of epoch write request, where the epoch is guaranteed to have changed if the writer has changed or the ledger was ever fenced off. Writes include an epoch and are rejected if the epoch

Re: October 2016 Podling Report Draft Feedback

Done On Sat, Oct 8, 2016 at 7:14 AM, Leigh Stewart wrote: > Will do. > > On Sat, Oct 8, 2016 at 7:11 AM, Sijie Guo wrote: > >> Ah, Chris, thank you so much. >> >> Leigh was helping with the report. Leigh, can you incorporate Chris's >> comments? >&g

Re: Projections

Eventstore looks very cool. At Twitter, filtering/projections are currently implemented in a separate service. I'm obviously not familiar with Eventstore architecture, but is it possible DL could still be used as a backend and projections could be implemented overtop? On Tue, Oct 18, 2016 at 5:

Re: hundreds of millions of streams?

DL is not able to handle 100s of millions of streams. 10^5-106 is probably ok. ZK is probably the biggest challenge (we are looking at ways to eliminate this as we would like to scale to 10^6-10^7 in the not too distant future), but 100s of millions is so far beyond what we've worked with there wou

Re: hundreds of millions of streams?

Poule- would you mind sharing some information on Event Sourcing? Are you referring to something like http://martinfowler.com/eaaDev/EventSourcing.html ? On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart wrote: > DL is not able to handle 100s of millions of streams. 10^5-106 is probably > ok

Re: [Discuss] Transaction Support

Interesting proposal. A couple quick notes while you continue to flesh this out. a. just to be sure - does this eliminate the need to save seqno with checkpoint? b. i.e. another way to describe this kind of improvement is "support records (atomic writes) larger than 1MB", iiuc. the advantage bein

Re: hundreds of millions of streams?

by replaying that stream, I can rebuild a user's cart state > > > > Le 28 oct. 2016 à 10:13, Leigh Stewart a > écrit : > > > > Poule- would you mind sharing some information on Event Sourcing? Are you > > referring to something like > > http://martinfowl

Re: Distributedlog Podling Report Draft - November 2016

LGTM On Tue, Nov 1, 2016 at 9:31 PM, Sijie Guo wrote: > +1 LGTM. > > - Sijie > > On Tue, Nov 1, 2016 at 7:18 PM, Jia Zhai wrote: > > > Thanks so much for your suggestions. Here is a new draft: > > > > === > > DistributedLog > > > > DistributedLog is a high-performance replicated log ser

Re: Use DL stream to store offsets?

We have in fact built something like this. No plans as yet to release, but I think we would like to eventually. On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Nasim wrote: > As part of implementing the kafka subscriber interface, I am wondering is > there anyone use DL stream for storing the offsets? >

Re: Use DL stream to store offsets?

Whats your timeline Khurrum? Maybe we can work something out. On Wed, Nov 2, 2016 at 8:15 AM, Leigh Stewart wrote: > We have in fact built something like this. No plans as yet to release, but > I think we would like to eventually. > > On Wed, Nov 2, 2016 at 2:43 AM, Khurrum Na

Re: FileSystem API over distributedlog logs

Sure we could do it. We skipped last time because dl was not OSS. Need to find some time though - lets discuss quickly next week. On Fri, Nov 11, 2016 at 12:10 PM, Sijie Guo wrote: > /cc Leigh > > I don't think we pushed the DL related code to kestrel. As I think kestrel > has been in the depre

Re: [DISCUSSION] First Release

+1 On Tue, Nov 15, 2016 at 7:01 PM, Franck Cuny wrote: > +1 and I agree to not make DL-2 a blocker. > > On Tue, Nov 15, 2016 at 2:02 AM, Xi Liu wrote: > > > +1 especially on DL-23. > > > > - Xi > > > > On Sat, Nov 12, 2016 at 2:22 AM, Khurrum Nasim > > wrote: > > > > > +1 (non-binding) > > > >

Re: Use DL stream to store offsets?

> On Wed, Nov 2, 2016 at 9:19 AM, Leigh Stewart > > wrote: > > > Whats your timeline Khurrum? Maybe we can work something out. > > > > On Wed, Nov 2, 2016 at 8:15 AM, Leigh Stewart > > wrote: > > > > > We have in fact built something like

Re: Proxy Client - Batch Ordering / Commit

gt; > ordering > > > > > > > > > > > consistence > > > > > > > > > > > >> > with > > > > > > > > > > > >> > > the original request, when exp

Re: DistributedLog Podling Report Draft - December 2016

#shipit On Wed, Dec 7, 2016 at 7:28 PM, Sijie Guo wrote: > If no objections, I will submit the proposal tonight. > > - Sijie > > On Tue, Dec 6, 2016 at 7:04 PM, Khurrum Nasim > wrote: > > > +1 also excited about the first release. > > > > - KN > > > > On Tue, Dec 6, 2016 at 5:51 PM, Jia Zhai w

Re: Is there a REST API for Write and Read proxies?

The challenge is the protocol uses a redirection mechanism so theres a protocol beyond thrift/transport. As Jay says we plan to make improvements in this area. On Wed, Dec 21, 2016 at 8:53 AM, Jay Juma wrote: > Hi Asko, > > I don't think there is a REST api available in the proxy service. The A

Re: [DISCUSSION] First Release

> > > own branch and switch to 4.5. >>>> > > >>>> > > Hope this make sense. >>>> > > >>>> > > Sijie >>>> > > >>>> > > On Nov 16, 2016 8:04 PM, "Flavio Junqueira" wrote: &g

Re: Intro

Hey Wenson - welcome! On Fri, Dec 30, 2016 at 1:39 PM, Wenson Li wrote: > Hi all, > > I wanted to introduce myself and say Hi to the community. I am currently > working with Xi Liu. We are using DistributedLog to build a transactional > replicated data store. Xi has made a few proposals that we

Re: [VOTE] Release 0.4.0, release candidate #2

+1 On Mon, Jan 16, 2017 at 8:13 AM, Jon Derrick wrote: > +1 > > LGTM. compiled the source packages and ran dbench. the license files look > good. > > - jd > > On Tue, Jan 10, 2017 at 11:56 PM, Sijie Guo wrote: > > > Hi all, > > > > Please review and vote on the release candidate #2 for the vers

Re: [VOTE] Release 0.4.0, release candidate #3

+1 On Mar 31, 2017 11:03 AM, "Yiming Zang" wrote: > +1 > > Same as Jia Zhai, build succeed after adding > "src/main/resources/DISCLAIMER.bin.txt" > at line 232 of pom.xml > > MD5 and SHA looks good. > > dlog tool can run without any issue from the binary. > > > On Thu, Mar 30, 2017 at 5:27 PM, J

Re: [DISCUSS] DP-8: Symlinked Log

+ Dan On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo wrote: > Hi all, > > I created a proposal for supporting symlinks in Dlog, in order to migrate > a namespace from flat namespace to a hierarchical namespace. > > https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log > > Please take a

Re: [DISCUSS] DP-8: Symlinked Log

gt;> https://issues.apache.org/jira/browse/DL-77. I think we should try to >> solve the general reconfiguration problem correctly first, and then that >> should enable us to solve both this problem and those two more naturally. I >> expect that a correct reconfiguration solution wou

Re: DistributedLog Podling Report Draft - May 2017

gt; How has the community developed since the last report? > > 1. community > > - Sijie gave a talk about DistributedLog at Strata+Hadoop San Jose. > - The DistributedLog paper is accepted at ICDE 2017. > - Leigh Stewart gave a presentation about DistributedLog at ICDE. > > 2.

Re: [VOTE] Merge DistributedLog as the subproject of Apache BookKeeper

+1 On Fri, Jun 9, 2017 at 2:48 PM, Uma gangumalla wrote: > +1 (binding) > > Regards, > Uma > > On Thu, Jun 8, 2017 at 5:21 PM, Sijie Guo wrote: > > > ( /cc bookkeeper dev@ and incubator general@ for awareness ) > > > > Hi all, > > > > There was a joint discussion between BookKeeper PMC and Dist

Re: [DISCUSS] Slack Channel for BookKeeper

🛳it On Mon, Jul 24, 2017 at 6:50 PM, Jia Zhai wrote: > 👍 It is great to have a slack channel. It make things more effective and > smooth. > > On Tue, Jul 25, 2017 at 8:11 AM, Sijie Guo wrote: > > > Hi all, > > > > What do you guys all think about having a dedicated slack channel for > > informa

Re: [VOTE] Slack Channel for BookKeeper

+1 On Thu, Jul 27, 2017 at 11:02 AM, Sijie Guo wrote: > Start a vote thread for transferring the DL slack channel to BK. > > The propose is: > > - transfer the dl slack channel apachedistributedlog.slack.com to > apachebookkeeper.slack.com > - the owner will be BookKeeper PMC (priv...@bookkeeper

[jira] [Closed] (DL-35) Document on how to setup a global DL cluster

[ https://issues.apache.org/jira/browse/DL-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leigh Stewart closed DL-35. --- > Document on how to setup a global DL cluster > > >

[jira] [Reopened] (DL-35) Document on how to setup a global DL cluster

[ https://issues.apache.org/jira/browse/DL-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leigh Stewart reopened DL-35: - > Document on how to setup a global DL clus

[jira] [Commented] (DL-35) Document on how to setup a global DL cluster

[ https://issues.apache.org/jira/browse/DL-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664154#comment-15664154 ] Leigh Stewart commented on DL-35: - Checked in, still need to deploy the latest

[jira] [Resolved] (DL-35) Document on how to setup a global DL cluster

[ https://issues.apache.org/jira/browse/DL-35?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leigh Stewart resolved DL-35. - Resolution: Fixed > Document on how to setup a global DL clus

[jira] [Commented] (DL-10) Please document how to run distributedlog-benchmark #26

[ https://issues.apache.org/jira/browse/DL-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664357#comment-15664357 ] Leigh Stewart commented on DL-10: - Pull request: https://github.com/apache/incub

[jira] [Commented] (DL-11) Would you please help explain more for the IDs of a record? #28

[ https://issues.apache.org/jira/browse/DL-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15676896#comment-15676896 ] Leigh Stewart commented on DL-11: - It looks like this is already answered in the

[jira] [Resolved] (DL-11) Would you please help explain more for the IDs of a record? #28

[ https://issues.apache.org/jira/browse/DL-11?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leigh Stewart resolved DL-11. - Resolution: Won't Fix > Would you please help explain more for the IDs of a rec

[jira] [Commented] (DL-84) Reduce HashedWheelTimer instance number

[ https://issues.apache.org/jira/browse/DL-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722632#comment-15722632 ] Leigh Stewart commented on DL-84: - First two are client, second two are server. Client

[jira] [Updated] (DL-97) Remove unused methods in BKLogHandler

[ https://issues.apache.org/jira/browse/DL-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leigh Stewart updated DL-97: Assignee: Sijie Guo (was: Leigh Stewart) > Remove unused methods in BKLogHand

[jira] [Commented] (DL-97) Remove unused methods in BKLogHandler

[ https://issues.apache.org/jira/browse/DL-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767329#comment-15767329 ] Leigh Stewart commented on DL-97: - Assigning to [~si...@apache.org] since he merged

[jira] [Commented] (DL-102) Add routing service to write proxy server side

[ https://issues.apache.org/jira/browse/DL-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767493#comment-15767493 ] Leigh Stewart commented on DL-102: -- https://github.com/apache/incubator-distributedlog/

[jira] [Commented] (DL-102) Add routing service to write proxy server side

[ https://issues.apache.org/jira/browse/DL-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767568#comment-15767568 ] Leigh Stewart commented on DL-102: -- Serverside depends on TwitterRegionResolver So I g