[jira] [Resolved] (DL-59) Travis CI build timeout after findbugs:check

2016-11-01 Thread Sijie Guo (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DL-59?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo resolved DL-59.
-
Resolution: Fixed

Merged the change. Thanks [~khurrumnasimm]

> Travis CI build timeout after findbugs:check
> 
>
> Key: DL-59
> URL: https://issues.apache.org/jira/browse/DL-59
> Project: DistributedLog
>  Issue Type: Test
>  Components: builds
>Reporter: Khurrum Nasim
>Assignee: Khurrum Nasim
> Fix For: 0.4.0
>
>
> I found that after adding findbugs in the travis ci build, the builds kept 
> timeout due to reaching the hard limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DL-59) Travis CI build timeout after findbugs:check

2016-11-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DL-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627844#comment-15627844
 ] 

ASF GitHub Bot commented on DL-59:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/38


> Travis CI build timeout after findbugs:check
> 
>
> Key: DL-59
> URL: https://issues.apache.org/jira/browse/DL-59
> Project: DistributedLog
>  Issue Type: Test
>  Components: builds
>Reporter: Khurrum Nasim
>Assignee: Khurrum Nasim
> Fix For: 0.4.0
>
>
> I found that after adding findbugs in the travis ci build, the builds kept 
> timeout due to reaching the hard limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog pull request #38: DL-59: Check findbugs warning bef...

2016-11-01 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/38


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-distributedlog git commit: DL-59: Check findbugs warning before packaging

2016-11-01 Thread sijie 
Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master f88c1e6e4 -> 298abf86b


DL-59: Check findbugs warning before packaging

check findbugs errors before running package

Author: Khurrum Nasim 

Reviewers: Sijie Guo 

Closes #38 from khurrumnasimm/kn/DL-59


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/298abf86
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/298abf86
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/298abf86

Branch: refs/heads/master
Commit: 298abf86b7f8b9b62b20d5d26946db3dea5b8b8d
Parents: f88c1e6
Author: Khurrum Nasim 
Authored: Tue Nov 1 22:51:35 2016 -0700
Committer: Sijie Guo 
Committed: Tue Nov 1 22:51:35 2016 -0700

--
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/298abf86/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index fc464b3..9953fed 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -24,11 +24,11 @@ before_install:
   - echo "MAVEN_OPTS='-Xmx3072m -XX:MaxPermSize=512m'" > ~/.mavenrc
 
 install:
-  - travis_retry mvn -B install clean -U -DskipTests=true
+  - travis_retry mvn -B install clean findbugs:check -U -DskipTests=true
 
 script:
   - travis_retry mvn clean apache-rat:check
-  - travis_wait 60 mvn package findbugs:check
+  - travis_wait 60 mvn package 
 
 cache:
   directories:

[GitHub] incubator-distributedlog pull request #39: DL-60: Add user mail list to the ...

2016-11-01 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/39


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-60) [documentation] Add user mail list to community page

2016-11-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DL-60?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15627839#comment-15627839
 ] 

ASF GitHub Bot commented on DL-60:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/39


> [documentation] Add user mail list to community page
> 
>
> Key: DL-60
> URL: https://issues.apache.org/jira/browse/DL-60
> Project: DistributedLog
>  Issue Type: Task
>  Components: documentation
>Reporter: Khurrum Nasim
>Assignee: Khurrum Nasim
> Fix For: 0.4.0
>
>
> {quote}
> it is worth separating the user and dev discussion into two mail lists. As 
> from subscribers' perspective, I can easily setup filtering rules on two 
> different mail lists and get all the information that I really care about. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-distributedlog git commit: DL-60: Add user mail list to the community page

2016-11-01 Thread sijie 
Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master c2546a9c1 -> f88c1e6e4


DL-60: Add user mail list to the community page

Author: Khurrum Nasim 

Reviewers: Sijie Guo 

Closes #39 from khurrumnasimm/kn/add_user_list


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/f88c1e6e
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/f88c1e6e
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/f88c1e6e

Branch: refs/heads/master
Commit: f88c1e6e42a5936332f7a07edfc981d6cfba702c
Parents: c2546a9
Author: Khurrum Nasim 
Authored: Tue Nov 1 22:49:12 2016 -0700
Committer: Sijie Guo 
Committed: Tue Nov 1 22:49:12 2016 -0700

--
 website/community/community.md | 15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/f88c1e6e/website/community/community.md
--
diff --git a/website/community/community.md b/website/community/community.md
index 3837d2a..cf63407 100644
--- a/website/community/community.md
+++ b/website/community/community.md
@@ -24,8 +24,21 @@ There are many ways to get help from the Apache 
DistributedLog community. The [m
   
   
 
+  user@@distributedlog.incubator.apache.org
+  User related discussions
+
+ mailto:user-subscr...@distributedlog.incubator.apache.org";>Subscribe
+ mailto:user-digest-subscr...@distributedlog.incubator.apache.org";>Subscribe
+ mailto:user-unsubscr...@distributedlog.incubator.apache.org";>Unsubscribe
+ mailto:u...@distributedlog.incubator.apache.org";>Post
+
+  http://mail-archives.apache.org/mod_mbox/incubator-distributedlog-user/";>Archives
 
+
+  
+  
+
   dev@@distributedlog.incubator.apache.org
-  Users and Development related discussions
+  Development related discussions
 
  mailto:dev-subscr...@distributedlog.incubator.apache.org";>Subscribe
  mailto:dev-digest-subscr...@distributedlog.incubator.apache.org";>Subscribe

Fwd: BookKeeper Meetup at Yahoo - Tuesday, Nov 15, 2016

2016-11-01 Thread Sijie Guo 
The next bookkeeper meetup at Yahoo campus on Nov 15. Three talks with
focus on distributed database/storage and replication. We will cover
reliable data replication using DistributedLog. If you are interested in
this topic, please join us. Community participation is appreciated.

- Sijie


-- Forwarded message --
From: Sijie Guo 
Date: Tue, Nov 1, 2016 at 10:12 PM
Subject: BookKeeper Meetup at Yahoo - Tuesday, Nov 15, 2016
To: d...@bookkeeper.apache.org, u...@bookkeeper.apache.org


http://www.meetup.com/Apache-BookKeeper/events/235098921/

Community participation is greatly appreciated. Please RSVP.

Agenda

• 6:00 - 6-30 - Socialize over food and beers

• 6:30 - 7:00 - Reliable Data Replication using Apache
DistributedLog/BookKeeper (Twitter)

• 7:00 - 7:30 - Sherpa - Distributed NoSQL database (Yahoo)

• 7:30 - 8:00 - Apache BookKeeper as a longterm distributed store
(Salesforce)

- Sijie

Re: Distributedlog Podling Report Draft - November 2016

2016-11-01 Thread Leigh Stewart 
LGTM

On Tue, Nov 1, 2016 at 9:31 PM, Sijie Guo  wrote:

> +1 LGTM.
>
> - Sijie
>
> On Tue, Nov 1, 2016 at 7:18 PM, Jia Zhai  wrote:
>
> > Thanks so much for your suggestions. Here is a new draft:
> >
> > ===
> > DistributedLog
> >
> > DistributedLog is a high-performance replicated log service. It offers
> > durability, replication and strong consistency, which provides a
> > fundamental building block for building reliable distributed systems.
> > DistributedLog has been incubating since 2016-06-24.
> >
> > Three most important issues to address in the move towards graduation:
> > 1.Continue to grow the community, and increase diversity of community.
> > 2.Improve documentation, including documentation of project and
> processes.
> > 3.Successful releases.
> >
> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware
> > of?
> > None
> >
> > How has the community developed since the last report?
> > 1. Increase in contributions from community.
> >- 8 created and 4 resolved issues in community JIRA in October.
> > 2. Lots of engagement on documentation.
> >- Enhance existing documents,
> >- Setup guides for developers and committers.
> > 3. Increased traffic on the mailing list, in particular, due to
> committers
> > engaging more actively with contributors.
> >- we have 35 people subscribed mail list.
> >- 125 messages to distributedlog mail list in October.
> >
> > How has the project developed since the last report?
> > 1. Documentation has improved for project build and project deployment.
> > Added more information on  community page.
> >  New added pages:
> >   https://cwiki.apache.org/confluence/display/DL/Developer+Guide
> >   https://cwiki.apache.org/confluence/display/DL/Committer+Guide
> > 2. Involved more discussion of new ideas and bring in new features.
> Include
> > major discussions like 'transaction support', 'batch commit',
> "EventStore"
> > . etc.
> > 3. First release expected on November, and repackaging of the project
> under
> > apache namespace are being discussed.
> > - Pull requests for repackaging.
> > - The major blocker is https://issues.apache.org/jira/browse/DL-2. We
> were
> > expecting to let DL depends on an official bk version.
> >
> > 
> >
> > On Tue, Nov 1, 2016 at 5:54 PM, Sijie Guo  wrote:
> >
> > > Thank you Jia and Flavio. Comments inline.
> > >
> > > On Tue, Nov 1, 2016 at 1:54 AM, Jia Zhai  wrote:
> > >
> > > > Hi Sijie,
> > > > Would you please help address some of Flavio's comments first, You
> may
> > > have
> > > > more information for story behind the result. I would like to handle
> > this
> > > > after your comments.
> > > >
> > > > Thanks a lot.
> > > > -Jia
> > > >
> > > > On Tue, Nov 1, 2016 at 10:42 AM, Jia Zhai 
> wrote:
> > > >
> > > > > Thanks a lot for your comments, will address them and post a new
> > draft.
> > > > >
> > > > > On Mon, Oct 31, 2016 at 6:02 PM, Flavio Junqueira <
> > > > > fpjunque...@yahoo.com.invalid> wrote:
> > > > >
> > > > >> Thanks, Jia. It would be good to have concrete metrics where
> > possible.
> > > > >> See my comments below:
> > > > >>
> > > > >> > On 30 Oct 2016, at 10:53, Jia Zhai  wrote:
> > > > >> >
> > > > >> > Thanks a lot for your suggestions Flavio.
> > > > >> > And Guys, Here is a draft. Please help comments on it.
> > > > >> > Regards.
> > > > >> > -Jia
> > > > >> >
> > > > >> > =
> > > > >> >
> > > > >> > DistributedLog
> > > > >> >
> > > > >> > DistributedLog is a high-performance replicated log service. It
> > > offers
> > > > >> >
> > > > >> > durability, replication and strong consistency, which provides a
> > > > >> fundamental
> > > > >> >
> > > > >> > building block for building reliable distributed systems.
> > > > >> >
> > > > >> > DistributedLog has been incubating since 2016-06-24.
> > > > >> >
> > > > >> > Three most important issues to address in the move towards
> > > graduation:
> > > > >> > 1.   Continue to grow the community, especially people from
> > > > outside
> > > > >> > Twitter.
> > > > >> > [Ask the PMC for helping check the number of people subscribed
> to
> > > dev@
> > > > >> mail
> > > > >> > list]
> > > > >>
> > > > >> I think you're referring to the project PMC checking the
> > subscription
> > > > >> list. I checked and we have 35 people subscribed. A lot of people
> > have
> > > > >> gmail and apache.org  addresses so it is hard
> > to
> > > > >> tell affiliation. One typical sign of community diversity is to
> have
> > > > active
> > > > >> committers from multiple organizations, which is just a natural
> > > > extension
> > > > >> of a diverse community. As folks contribute to the project, I'd
> > expect
> > > > the
> > > > >> PPMC to offer committership to the contributors.
> > > > >>
> > > > >> Please do not refer to Twitter specifically, just say that we want
> > to
> > > > >> increase diversity.
> > > >
> > >
> > > Agreed. We should just say t

Re: Distributedlog Podling Report Draft - November 2016

2016-11-01 Thread Sijie Guo 
+1 LGTM.

- Sijie

On Tue, Nov 1, 2016 at 7:18 PM, Jia Zhai  wrote:

> Thanks so much for your suggestions. Here is a new draft:
>
> ===
> DistributedLog
>
> DistributedLog is a high-performance replicated log service. It offers
> durability, replication and strong consistency, which provides a
> fundamental building block for building reliable distributed systems.
> DistributedLog has been incubating since 2016-06-24.
>
> Three most important issues to address in the move towards graduation:
> 1.Continue to grow the community, and increase diversity of community.
> 2.Improve documentation, including documentation of project and processes.
> 3.Successful releases.
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
> of?
> None
>
> How has the community developed since the last report?
> 1. Increase in contributions from community.
>- 8 created and 4 resolved issues in community JIRA in October.
> 2. Lots of engagement on documentation.
>- Enhance existing documents,
>- Setup guides for developers and committers.
> 3. Increased traffic on the mailing list, in particular, due to committers
> engaging more actively with contributors.
>- we have 35 people subscribed mail list.
>- 125 messages to distributedlog mail list in October.
>
> How has the project developed since the last report?
> 1. Documentation has improved for project build and project deployment.
> Added more information on  community page.
>  New added pages:
>   https://cwiki.apache.org/confluence/display/DL/Developer+Guide
>   https://cwiki.apache.org/confluence/display/DL/Committer+Guide
> 2. Involved more discussion of new ideas and bring in new features. Include
> major discussions like 'transaction support', 'batch commit', "EventStore"
> . etc.
> 3. First release expected on November, and repackaging of the project under
> apache namespace are being discussed.
> - Pull requests for repackaging.
> - The major blocker is https://issues.apache.org/jira/browse/DL-2. We were
> expecting to let DL depends on an official bk version.
>
> 
>
> On Tue, Nov 1, 2016 at 5:54 PM, Sijie Guo  wrote:
>
> > Thank you Jia and Flavio. Comments inline.
> >
> > On Tue, Nov 1, 2016 at 1:54 AM, Jia Zhai  wrote:
> >
> > > Hi Sijie,
> > > Would you please help address some of Flavio's comments first, You may
> > have
> > > more information for story behind the result. I would like to handle
> this
> > > after your comments.
> > >
> > > Thanks a lot.
> > > -Jia
> > >
> > > On Tue, Nov 1, 2016 at 10:42 AM, Jia Zhai  wrote:
> > >
> > > > Thanks a lot for your comments, will address them and post a new
> draft.
> > > >
> > > > On Mon, Oct 31, 2016 at 6:02 PM, Flavio Junqueira <
> > > > fpjunque...@yahoo.com.invalid> wrote:
> > > >
> > > >> Thanks, Jia. It would be good to have concrete metrics where
> possible.
> > > >> See my comments below:
> > > >>
> > > >> > On 30 Oct 2016, at 10:53, Jia Zhai  wrote:
> > > >> >
> > > >> > Thanks a lot for your suggestions Flavio.
> > > >> > And Guys, Here is a draft. Please help comments on it.
> > > >> > Regards.
> > > >> > -Jia
> > > >> >
> > > >> > =
> > > >> >
> > > >> > DistributedLog
> > > >> >
> > > >> > DistributedLog is a high-performance replicated log service. It
> > offers
> > > >> >
> > > >> > durability, replication and strong consistency, which provides a
> > > >> fundamental
> > > >> >
> > > >> > building block for building reliable distributed systems.
> > > >> >
> > > >> > DistributedLog has been incubating since 2016-06-24.
> > > >> >
> > > >> > Three most important issues to address in the move towards
> > graduation:
> > > >> > 1.   Continue to grow the community, especially people from
> > > outside
> > > >> > Twitter.
> > > >> > [Ask the PMC for helping check the number of people subscribed to
> > dev@
> > > >> mail
> > > >> > list]
> > > >>
> > > >> I think you're referring to the project PMC checking the
> subscription
> > > >> list. I checked and we have 35 people subscribed. A lot of people
> have
> > > >> gmail and apache.org  addresses so it is hard
> to
> > > >> tell affiliation. One typical sign of community diversity is to have
> > > active
> > > >> committers from multiple organizations, which is just a natural
> > > extension
> > > >> of a diverse community. As folks contribute to the project, I'd
> expect
> > > the
> > > >> PPMC to offer committership to the contributors.
> > > >>
> > > >> Please do not refer to Twitter specifically, just say that we want
> to
> > > >> increase diversity.
> > >
> >
> > Agreed. We should just say that we want to increase diversity.
> >
> >
> > > >>
> > > >> > 2.   Improve documentation.
> > > >>
> > > >> Project documentation is important, but from a community
> perspective,
> > it
> > > >> is more important to have all processes documented well: how to
> > > contribute,
> > > >> how to merge, notes for first time committers, how

Re: Distributedlog Podling Report Draft - November 2016

2016-11-01 Thread Jia Zhai 
Thanks so much for your suggestions. Here is a new draft:

===
DistributedLog

DistributedLog is a high-performance replicated log service. It offers
durability, replication and strong consistency, which provides a
fundamental building block for building reliable distributed systems.
DistributedLog has been incubating since 2016-06-24.

Three most important issues to address in the move towards graduation:
1.Continue to grow the community, and increase diversity of community.
2.Improve documentation, including documentation of project and processes.
3.Successful releases.

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
of?
None

How has the community developed since the last report?
1. Increase in contributions from community.
   - 8 created and 4 resolved issues in community JIRA in October.
2. Lots of engagement on documentation.
   - Enhance existing documents,
   - Setup guides for developers and committers.
3. Increased traffic on the mailing list, in particular, due to committers
engaging more actively with contributors.
   - we have 35 people subscribed mail list.
   - 125 messages to distributedlog mail list in October.

How has the project developed since the last report?
1. Documentation has improved for project build and project deployment.
Added more information on  community page.
 New added pages:
  https://cwiki.apache.org/confluence/display/DL/Developer+Guide
  https://cwiki.apache.org/confluence/display/DL/Committer+Guide
2. Involved more discussion of new ideas and bring in new features. Include
major discussions like 'transaction support', 'batch commit', "EventStore"
. etc.
3. First release expected on November, and repackaging of the project under
apache namespace are being discussed.
- Pull requests for repackaging.
- The major blocker is https://issues.apache.org/jira/browse/DL-2. We were
expecting to let DL depends on an official bk version.



On Tue, Nov 1, 2016 at 5:54 PM, Sijie Guo  wrote:

> Thank you Jia and Flavio. Comments inline.
>
> On Tue, Nov 1, 2016 at 1:54 AM, Jia Zhai  wrote:
>
> > Hi Sijie,
> > Would you please help address some of Flavio's comments first, You may
> have
> > more information for story behind the result. I would like to handle this
> > after your comments.
> >
> > Thanks a lot.
> > -Jia
> >
> > On Tue, Nov 1, 2016 at 10:42 AM, Jia Zhai  wrote:
> >
> > > Thanks a lot for your comments, will address them and post a new draft.
> > >
> > > On Mon, Oct 31, 2016 at 6:02 PM, Flavio Junqueira <
> > > fpjunque...@yahoo.com.invalid> wrote:
> > >
> > >> Thanks, Jia. It would be good to have concrete metrics where possible.
> > >> See my comments below:
> > >>
> > >> > On 30 Oct 2016, at 10:53, Jia Zhai  wrote:
> > >> >
> > >> > Thanks a lot for your suggestions Flavio.
> > >> > And Guys, Here is a draft. Please help comments on it.
> > >> > Regards.
> > >> > -Jia
> > >> >
> > >> > =
> > >> >
> > >> > DistributedLog
> > >> >
> > >> > DistributedLog is a high-performance replicated log service. It
> offers
> > >> >
> > >> > durability, replication and strong consistency, which provides a
> > >> fundamental
> > >> >
> > >> > building block for building reliable distributed systems.
> > >> >
> > >> > DistributedLog has been incubating since 2016-06-24.
> > >> >
> > >> > Three most important issues to address in the move towards
> graduation:
> > >> > 1.   Continue to grow the community, especially people from
> > outside
> > >> > Twitter.
> > >> > [Ask the PMC for helping check the number of people subscribed to
> dev@
> > >> mail
> > >> > list]
> > >>
> > >> I think you're referring to the project PMC checking the subscription
> > >> list. I checked and we have 35 people subscribed. A lot of people have
> > >> gmail and apache.org  addresses so it is hard to
> > >> tell affiliation. One typical sign of community diversity is to have
> > active
> > >> committers from multiple organizations, which is just a natural
> > extension
> > >> of a diverse community. As folks contribute to the project, I'd expect
> > the
> > >> PPMC to offer committership to the contributors.
> > >>
> > >> Please do not refer to Twitter specifically, just say that we want to
> > >> increase diversity.
> >
>
> Agreed. We should just say that we want to increase diversity.
>
>
> > >>
> > >> > 2.   Improve documentation.
> > >>
> > >> Project documentation is important, but from a community perspective,
> it
> > >> is more important to have all processes documented well: how to
> > contribute,
> > >> how to merge, notes for first time committers, how to cut a release,
> > check
> > >> licenses.
> >
>
> Yup. We have setup a guide for developers and committers:
>
> https://cwiki.apache.org/confluence/display/DL/Developer+Guide
> https://cwiki.apache.org/confluence/display/DL/Committer+Guide
>
> They are linked under the community tab in
> http://distributedlog.incubator.apache.org/

Podling Report Reminder - November 2016

2016-11-01 Thread johndament 
Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, November 02).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.

This should be appended to the Incubator Wiki page at:

http://wiki.apache.org/incubator/November2016

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

[jira] [Commented] (DL-45) DL should allow ByteBuffer based API and should avoid copying of arrays

2016-11-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DL-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625755#comment-15625755
 ] 

ASF GitHub Bot commented on DL-45:
--

Github user arvindkandhare commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/21
  
@sijie The BK changes are final and ready to be checked in once you are 
satisfied with them. This particular pull request still needs a couple of 
changes.


> DL should allow ByteBuffer based API and should avoid copying of arrays
> ---
>
> Key: DL-45
> URL: https://issues.apache.org/jira/browse/DL-45
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-core, distributedlog-protocol
>Reporter: Arvind Kandhare
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog issue #21: DL-45: DL should allow ByteBuffer based ...

2016-11-01 Thread arvindkandhare 
Github user arvindkandhare commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/21
  
@sijie The BK changes are final and ready to be checked in once you are 
satisfied with them. This particular pull request still needs a couple of 
changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: hundreds of millions of streams?

2016-11-01 Thread Poule Dodue 
as example geteventstore.com is limited to 2 billion events per stream
imho anything 1+ billion events per stream is ok!



> Le 1 nov. 2016 à 06:17, Sijie Guo  a écrit :
> 
> Poule,
> 
> how long do you expect to keep the events for your 1 cart or 1 user? or in
> the words, how large is your event stream per cart or per user. it seems
> that you want some very lightweight stream/queue solutions - like support
> very large amount of streams and per stream throughput is not very high.
> 
> - Sijie
> 
> On Sun, Oct 30, 2016 at 8:00 AM, Poule Dodue  wrote:
> 
>> i'm not expert but ref. my previous cart example, imho i need a way to
>> fetch events of 1 cart or 1 user.
>> the actual physical implementation on DL I don't know the technicalities,
>> as long I can get 1 cart with the
>> less cpu cycles possible.
>> 
>> 
>>> Le 30 oct. 2016 à 09:59, john.lonergan  a
>> écrit :
>>> 
>>> As an FYI.
>>> It's a misrepresentation to suggest that ES/CQRS requires each
>> actor/aggregate root history to be materialised as a physically distinct
>> stream. There may be specific implementations that require this, and I know
>> there are others that do not.So regarding those references given, it's is
>> not a requirement of this tech that we generate one stream in DL per actor
>> for ES.It might for instance be necessary for Greg Young's event store to
>> have one phys stream per aggregate sure to implementation choices.
>>>  Original message From: Poule Dodue <
>> pouledo...@hotmail.com> Date: 28/10/2016  16:24  (GMT+00:00) To:
>> dev@distributedlog.incubator.apache.org Subject: Re: hundreds of millions
>> of streams?
>>> yes aka ES/CQRS
>>> 
>>> some links:
>>> 
>>> https://msdn.microsoft.com/en-us/library/jj554200.aspx
>>> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
>>> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>>> 
>>> it needs lot of streams to basically replay events for any entity on a
>> system.
>>> 
>>> example: i could replay events for all changes that happened in 1 Cart
>> of 1 User:
>>> 
>>> 
>>> (read events from stream "cart-of-user-233293111" ):
>>> 
>>> 1- added item X
>>> 2- deleted item X
>>> 3- added item Y
>>> 
>>> 
>>> by replaying that stream, I can rebuild a user's cart state
>>> 
>>> 
 Le 28 oct. 2016 à 10:13, Leigh Stewart 
>> a écrit :
 
 Poule- would you mind sharing some information on Event Sourcing? Are
>> you
 referring to something like
 http://martinfowler.com/eaaDev/EventSourcing.html ?
 
 On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart 
>> wrote:
 
> DL is not able to handle 100s of millions of streams. 10^5-106 is
>> probably
> ok.
> ZK is probably the biggest challenge (we are looking at ways to
>> eliminate
> this as we would like to scale to 10^6-10^7 in the not too distant
>> future),
> but 100s of millions is so far beyond what we've worked with there
>> would
> likely be other scaling challenges on the way to that point.
> 
> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue 
> wrote:
> 
>> In Event Sourcing, we need to have 1 stream per entity/aggregate so
>> for
>> a typical prod system it means we need hundreds of millions of
>> streams.
>> 
>> Is DL able to handle that or it is limited to, say, few hundreds
>> thousands of streams?
>> 
>> 
>> 
> 
>>> 
>> 
>>

[GitHub] incubator-distributedlog issue #21: DL-45: DL should allow ByteBuffer based ...

2016-11-01 Thread sijie 
Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/21
  
@arvindkandhare is the 
[twitter/bookkeeper#4](https://github.com/twitter/bookkeeper/pull/4) change 
finalized?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-45) DL should allow ByteBuffer based API and should avoid copying of arrays

2016-11-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DL-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625059#comment-15625059
 ] 

ASF GitHub Bot commented on DL-45:
--

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/21
  
@arvindkandhare is the 
[twitter/bookkeeper#4](https://github.com/twitter/bookkeeper/pull/4) change 
finalized?


> DL should allow ByteBuffer based API and should avoid copying of arrays
> ---
>
> Key: DL-45
> URL: https://issues.apache.org/jira/browse/DL-45
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-core, distributedlog-protocol
>Reporter: Arvind Kandhare
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog issue #28: Fix the typo in merge script on formatti...

2016-11-01 Thread sijie 
Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/28
  
ping @franckcuny 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-23) Move distributedlog build to depend on central maven repo

2016-11-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DL-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625052#comment-15625052
 ] 

ASF GitHub Bot commented on DL-23:
--

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/27
  
@jiazhai it would good that you can send out the second part of this 
change. I took a look at this change initially. the layout seems to be fine. I 
would like to see how will you change the server side and decide later on. /cc 
@leighst 

also, this change is touching quite bunch of files. it will definitely 
conflict with the repackaging change made by @khurrumnasimm . so we'd 
coordinate on merging them.


> Move distributedlog build to depend on central maven repo
> -
>
> Key: DL-23
> URL: https://issues.apache.org/jira/browse/DL-23
> Project: DistributedLog
>  Issue Type: Improvement
>  Components: distributedlog-client, distributedlog-core
>Reporter: Sijie Guo
>
> Reported https://github.com/twitter/distributedlog/issues/37.
> Currently they are a few dependencies only exists at maven.twttr. There are 
> two major pieces, one is the finagle-serverset and while the other one is 
> thrift library. The finagle-serverset is majorly used for service-discovery 
> by distributedlog-client and thrift is used by scrooge and the rpc wire 
> protocol between client and write proxy.
> There are two options:
> - one is to look for a solution to move those dependencies to central maven 
> repo.
> - the other one is to see if we can make thrift and its service discovery 
> pluginnable. so we can support both finagle-thrift and gRPC-protobuf. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog issue #27: DL-23: update libthrift version in distr...

2016-11-01 Thread sijie 
Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/27
  
@jiazhai it would good that you can send out the second part of this 
change. I took a look at this change initially. the layout seems to be fine. I 
would like to see how will you change the server side and decide later on. /cc 
@leighst 

also, this change is touching quite bunch of files. it will definitely 
conflict with the repackaging change made by @khurrumnasimm . so we'd 
coordinate on merging them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (DL-60) [documentation] Add user mail list to community page

2016-11-01 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/DL-60?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625028#comment-15625028
 ] 

ASF GitHub Bot commented on DL-60:
--

Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/39
  
LGTM +1


> [documentation] Add user mail list to community page
> 
>
> Key: DL-60
> URL: https://issues.apache.org/jira/browse/DL-60
> Project: DistributedLog
>  Issue Type: Task
>  Components: documentation
>Reporter: Khurrum Nasim
>Assignee: Khurrum Nasim
> Fix For: 0.4.0
>
>
> {quote}
> it is worth separating the user and dev discussion into two mail lists. As 
> from subscribers' perspective, I can easily setup filtering rules on two 
> different mail lists and get all the information that I really care about. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-distributedlog issue #39: DL-60: Add user mail list to the communi...

2016-11-01 Thread sijie 
Github user sijie commented on the issue:

https://github.com/apache/incubator-distributedlog/pull/39
  
LGTM +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Repackaging the package namespace

2016-11-01 Thread Sijie Guo 
Khurrum,

thank your so much for making this change. I will try to take a look and
get back to you soon.

- Sijie

On Fri, Oct 21, 2016 at 2:41 AM, Khurrum Nasim 
wrote:

> Forgot the pull request -
> https://github.com/apache/incubator-distributedlog/pull/41
>
> - KN
>
> On Fri, Oct 21, 2016 at 2:41 AM, Khurrum Nasim 
> wrote:
>
> > I just sent a pull request - repacked the package from
> > com.twitter.distributedlog to org.apache.distributedlog.
> >
> > (basically just renamed files and sed/replace the package name)
> >
> > Let me know if it is okay.
> >
> > - KN
> >
> > On Sat, Oct 8, 2016 at 6:55 AM, Sijie Guo  wrote:
> >
> >> Khurrum, I don't think anyone is working on this now. Feel free to pick
> it
> >> up.
> >>
> >> - Sijie
> >>
> >> On Fri, Oct 7, 2016 at 10:37 AM, Khurrum Nasim  >
> >> wrote:
> >>
> >> > Is anyone working on repackaging? If no, can I pick this up?
> >> >
> >> > - KN
> >> >
> >> > On Sat, Sep 17, 2016 at 10:27 PM, Khurrum Nasim <
> >> khurrumnas...@gmail.com>
> >> > wrote:
> >> >
> >> > > +1
> >> > >
> >> > > On Mon, Sep 12, 2016 at 6:16 PM, Stevo Slavić 
> >> wrote:
> >> > >
> >> > >> + 1 (non-binding) for doing it asap
> >> > >>
> >> > >> On Mon, Sep 12, 2016 at 11:40 AM, Flavio Junqueira  >
> >> > >> wrote:
> >> > >>
> >> > >> > It isn't strictly necessary, here is what the documentation says:
> >> > >> >
> >> > >> > On Repackaging
> >> > >> >
> >> > >> > It is recommended - but not mandated - that source is repackaged
> >> under
> >> > >> the
> >> > >> > Apache namespace. There is no need to use the incubator
> namespace.
> >> For
> >> > >> > example, Java source might be repackaged to org.apache.foo.Bar
> or a
> >> > DTD
> >> > >> to
> >> > >> > http://dtd.apache.org/foo/bar.
> >> > >> >
> >> > >> > Existing open source projects moving to Apache may well need to
> >> > consider
> >> > >> > carefully how they will approach this transition.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > My suggestion is to repackage before the first release, though.
> >> This
> >> > way
> >> > >> > applications written against the early release will have the
> right
> >> > >> > namespace already.
> >> > >> >
> >> > >> > -Flavio
> >> > >> >
> >> > >> > > On 10 Sep 2016, at 20:32, Sijie Guo  wrote:
> >> > >> > >
> >> > >> > > Any suggestions from mentors for the first release?
> >> > >> > >
> >> > >> > > - Sijie
> >> > >> > >
> >> > >> > > On Tue, Aug 30, 2016 at 11:18 PM, Sijie Guo 
> >> > wrote:
> >> > >> > >
> >> > >> > >> Does anyone know if it is required to repackage the namespace
> to
> >> > >> under
> >> > >> > >> org.apache for the first release? Any suggestions from
> mentors?
> >> > >> > >>
> >> > >> > >> - Sijie
> >> > >> > >>
> >> > >> >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: hundreds of millions of streams?

2016-11-01 Thread Sijie Guo 
Poule,

how long do you expect to keep the events for your 1 cart or 1 user? or in
the words, how large is your event stream per cart or per user. it seems
that you want some very lightweight stream/queue solutions - like support
very large amount of streams and per stream throughput is not very high.

- Sijie

On Sun, Oct 30, 2016 at 8:00 AM, Poule Dodue  wrote:

> i'm not expert but ref. my previous cart example, imho i need a way to
> fetch events of 1 cart or 1 user.
> the actual physical implementation on DL I don't know the technicalities,
> as long I can get 1 cart with the
> less cpu cycles possible.
>
>
> > Le 30 oct. 2016 à 09:59, john.lonergan  a
> écrit :
> >
> > As an FYI.
> >  It's a misrepresentation to suggest that ES/CQRS requires each
> actor/aggregate root history to be materialised as a physically distinct
> stream. There may be specific implementations that require this, and I know
> there are others that do not.So regarding those references given, it's is
> not a requirement of this tech that we generate one stream in DL per actor
> for ES.It might for instance be necessary for Greg Young's event store to
> have one phys stream per aggregate sure to implementation choices.
> >  Original message From: Poule Dodue <
> pouledo...@hotmail.com> Date: 28/10/2016  16:24  (GMT+00:00) To:
> dev@distributedlog.incubator.apache.org Subject: Re: hundreds of millions
> of streams?
> > yes aka ES/CQRS
> >
> > some links:
> >
> > https://msdn.microsoft.com/en-us/library/jj554200.aspx
> > http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
> > http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
> >
> > it needs lot of streams to basically replay events for any entity on a
> system.
> >
> > example: i could replay events for all changes that happened in 1 Cart
> of 1 User:
> >
> >
> > (read events from stream "cart-of-user-233293111" ):
> >
> > 1- added item X
> > 2- deleted item X
> > 3- added item Y
> > 
> >
> > by replaying that stream, I can rebuild a user's cart state
> >
> >
> >> Le 28 oct. 2016 à 10:13, Leigh Stewart 
> a écrit :
> >>
> >> Poule- would you mind sharing some information on Event Sourcing? Are
> you
> >> referring to something like
> >> http://martinfowler.com/eaaDev/EventSourcing.html ?
> >>
> >> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart 
> wrote:
> >>
> >>> DL is not able to handle 100s of millions of streams. 10^5-106 is
> probably
> >>> ok.
> >>> ZK is probably the biggest challenge (we are looking at ways to
> eliminate
> >>> this as we would like to scale to 10^6-10^7 in the not too distant
> future),
> >>> but 100s of millions is so far beyond what we've worked with there
> would
> >>> likely be other scaling challenges on the way to that point.
> >>>
> >>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue 
> >>> wrote:
> >>>
>  In Event Sourcing, we need to have 1 stream per entity/aggregate so
> for
>  a typical prod system it means we need hundreds of millions of
> streams.
> 
>  Is DL able to handle that or it is limited to, say, few hundreds
>  thousands of streams?
> 
> 
> 
> >>>
> >
>
>

Re: Proxy Client - Batch Ordering / Commit

2016-11-01 Thread Sijie Guo 
I created https://issues.apache.org/jira/browse/DL-63 for tracking the
proposed idea here.



On Wed, Oct 26, 2016 at 4:53 PM, Sijie Guo 
wrote:

> On Tue, Oct 25, 2016 at 11:30 AM, Cameron Hatfield 
> wrote:
>
> > Yes, we are reading the HBase WAL (from their replication plugin
> support),
> > and writing that into DL.
> >
>
> Gotcha.
>
>
> >
> > From the sounds of it, yes, it would. Only thing I would say is make the
> > epoch requirement optional, so that if I client doesn't care about dupes
> > they don't have to deal with the process of getting a new epoch.
> >
>
> Yup. This should be optional. I can start a wiki page on how we want to
> implement this. Are you interested in contributing to this?
>
>
> >
> > -Cameron
> >
> > On Wed, Oct 19, 2016 at 7:43 PM, Sijie Guo 
> > wrote:
> >
> > > On Wed, Oct 19, 2016 at 7:17 PM, Sijie Guo  wrote:
> > >
> > > >
> > > >
> > > > On Monday, October 17, 2016, Cameron Hatfield 
> > wrote:
> > > >
> > > >> Answer inline:
> > > >>
> > > >> On Mon, Oct 17, 2016 at 11:46 AM, Sijie Guo 
> wrote:
> > > >>
> > > >> > Cameron,
> > > >> >
> > > >> > Thank you for your summary. I liked the discussion here. I also
> > liked
> > > >> the
> > > >> > summary of your requirement - 'single-writer-per-key,
> > > >> > multiple-writers-per-log'. If I understand correctly, the core
> > concern
> > > >> here
> > > >> > is almost 'exact-once' write (or a way to explicit tell if a write
> > can
> > > >> be
> > > >> > retried or not).
> > > >> >
> > > >> > Comments inline.
> > > >> >
> > > >> > On Fri, Oct 14, 2016 at 11:17 AM, Cameron Hatfield <
> > kin...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > > Ah- yes good point (to be clear we're not using the proxy this
> > way
> > > >> > > today).
> > > >> > >
> > > >> > > > > Due to the source of the
> > > >> > > > > data (HBase Replication), we cannot guarantee that a single
> > > >> partition
> > > >> > > will
> > > >> > > > > be owned for writes by the same client.
> > > >> > >
> > > >> > > > Do you mean you *need* to support multiple writers issuing
> > > >> interleaved
> > > >> > > > writes or is it just that they might sometimes interleave
> writes
> > > and
> > > >> > you
> > > >> > > >don't care?
> > > >> > > How HBase partitions the keys being written wouldn't have a
> > one->one
> > > >> > > mapping with the partitions we would have in HBase. Even if we
> did
> > > >> have
> > > >> > > that alignment when the cluster first started, HBase will
> > rebalance
> > > >> what
> > > >> > > servers own what partitions, as well as split and merge
> partitions
> > > >> that
> > > >> > > already exist, causing eventual drift from one log per
> partition.
> > > >> > > Because we want ordering guarantees per key (row in hbase), we
> > > >> partition
> > > >> > > the logs by the key. Since multiple writers are possible per
> range
> > > of
> > > >> > keys
> > > >> > > (due to the aforementioned rebalancing / splitting / etc of
> > hbase),
> > > we
> > > >> > > cannot use the core library due to requiring a single writer for
> > > >> > ordering.
> > > >> > >
> > > >> > > But, for a single log, we don't really care about ordering aside
> > > from
> > > >> at
> > > >> > > the per-key level. So all we really need to be able to handle is
> > > >> > preventing
> > > >> > > duplicates when a failure occurs, and ordering consistency
> across
> > > >> > requests
> > > >> > > from a single client.
> > > >> > >
> > > >> > > So our general requirements are:
> > > >> > > Write A, Write B
> > > >> > > Timeline: A -> B
> > > >> > > Request B is only made after A has successfully returned
> (possibly
> > > >> after
> > > >> > > retries)
> > > >> > >
> > > >> > > 1) If the write succeeds, it will be durably exposed to clients
> > > within
> > > >> > some
> > > >> > > bounded time frame
> > > >> > >
> > > >> >
> > > >> > Guaranteed.
> > > >> >
> > > >>
> > > >> > > 2) If A succeeds and B succeeds, the ordering for the log will
> be
> > A
> > > >> and
> > > >> > > then B
> > > >> > >
> > > >> >
> > > >> > If I understand correctly here, B is only sent after A is
> returned,
> > > >> right?
> > > >> > If that's the case, It is guaranteed.
> > > >>
> > > >>
> > > >>
> > > >> >
> > > >> >
> > > >> >
> > > >> > > 3) If A fails due to an error that can be relied on to *not* be
> a
> > > lost
> > > >> > ack
> > > >> > > problem, it will never be exposed to the client, so it may
> > > (depending
> > > >> on
> > > >> > > the error) be retried immediately
> > > >> > >
> > > >> >
> > > >> > If it is not a lost-ack problem, the entry will be exposed. it is
> > > >> > guaranteed.
> > > >>
> > > >> Let me try rephrasing the questions, to make sure I'm understanding
> > > >> correctly:
> > > >> If A fails, with an error such as "Unable to create connection to
> > > >> bookkeeper server", that would be the type of error we would expect
> to
> > > be
> > > >> able to retry immediately, as that result means no action was taken
> on
> > > any
> > > >> log / etc, so no entry

[jira] [Created] (DL-63) Epoch write support

2016-11-01 Thread Sijie Guo (JIRA) 
Sijie Guo created DL-63:
---

 Summary: Epoch write support 
 Key: DL-63
 URL: https://issues.apache.org/jira/browse/DL-63
 Project: DistributedLog
  Issue Type: Bug
Reporter: Sijie Guo


Discussion mail thread: 
http://mail-archives.apache.org/mod_mbox/incubator-distributedlog-dev/201610.mbox/browser

{quote}
- when a log stream is owned,  the proxy use the last transaction id as the
epoch
- when a client connects (handshake with the proxy), it will get the epoch
for the stream.
- the writes issued by this client will carry the epoch to the proxy.
- add a new rpc - fenceThenGetLastDLSN - it would force the proxy to bump
the epoch.
- if fenceThenGetLastDLSN happened, all the outstanding writes with old
epoch will be rejected with exceptions (e.g. EpochFenced).
- The DLSN returned from fenceThenGetLastDLSN can be used as the bound for
deduplications on failures.
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributedlog Podling Report Draft - November 2016

2016-11-01 Thread Sijie Guo 
Thank you Jia and Flavio. Comments inline.

On Tue, Nov 1, 2016 at 1:54 AM, Jia Zhai  wrote:

> Hi Sijie,
> Would you please help address some of Flavio's comments first, You may have
> more information for story behind the result. I would like to handle this
> after your comments.
>
> Thanks a lot.
> -Jia
>
> On Tue, Nov 1, 2016 at 10:42 AM, Jia Zhai  wrote:
>
> > Thanks a lot for your comments, will address them and post a new draft.
> >
> > On Mon, Oct 31, 2016 at 6:02 PM, Flavio Junqueira <
> > fpjunque...@yahoo.com.invalid> wrote:
> >
> >> Thanks, Jia. It would be good to have concrete metrics where possible.
> >> See my comments below:
> >>
> >> > On 30 Oct 2016, at 10:53, Jia Zhai  wrote:
> >> >
> >> > Thanks a lot for your suggestions Flavio.
> >> > And Guys, Here is a draft. Please help comments on it.
> >> > Regards.
> >> > -Jia
> >> >
> >> > =
> >> >
> >> > DistributedLog
> >> >
> >> > DistributedLog is a high-performance replicated log service. It offers
> >> >
> >> > durability, replication and strong consistency, which provides a
> >> fundamental
> >> >
> >> > building block for building reliable distributed systems.
> >> >
> >> > DistributedLog has been incubating since 2016-06-24.
> >> >
> >> > Three most important issues to address in the move towards graduation:
> >> > 1.   Continue to grow the community, especially people from
> outside
> >> > Twitter.
> >> > [Ask the PMC for helping check the number of people subscribed to dev@
> >> mail
> >> > list]
> >>
> >> I think you're referring to the project PMC checking the subscription
> >> list. I checked and we have 35 people subscribed. A lot of people have
> >> gmail and apache.org  addresses so it is hard to
> >> tell affiliation. One typical sign of community diversity is to have
> active
> >> committers from multiple organizations, which is just a natural
> extension
> >> of a diverse community. As folks contribute to the project, I'd expect
> the
> >> PPMC to offer committership to the contributors.
> >>
> >> Please do not refer to Twitter specifically, just say that we want to
> >> increase diversity.
>

Agreed. We should just say that we want to increase diversity.


> >>
> >> > 2.   Improve documentation.
> >>
> >> Project documentation is important, but from a community perspective, it
> >> is more important to have all processes documented well: how to
> contribute,
> >> how to merge, notes for first time committers, how to cut a release,
> check
> >> licenses.
>

Yup. We have setup a guide for developers and committers:

https://cwiki.apache.org/confluence/display/DL/Developer+Guide
https://cwiki.apache.org/confluence/display/DL/Committer+Guide

They are linked under the community tab in
http://distributedlog.incubator.apache.org/


> We need to make sure all this is in place once we suggest
> >> graduation.
> >>
> >>
> >> > 3.   First Apache release
> >>
> >> When is the first release due? it would be good for this community to
> >> release sooner rather later. I'd expect multiple releases upon
> graduation.
> >> I think I'd just say "Release successfully".
>

Or "Multiple successful releases"


> >>
> >> >
> >> >
> >> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> >> aware
> >> >
> >> > of?
> >> >
> >> > None
> >> >
> >> > How has the community developed since the last report?
> >> >
> >> > 1. Increase in contributions from community.
> >> >
> >>
> >> How do we measure this? Here are a few suggestions:
> >>
> >> - Number of messages to the list
> >> - Number of jiras created/resolved
> >> - Number of individual contributors
>

Jia,

I think you can collect

- number of messages by checking
http://mail-archives.apache.org/mod_mbox/incubator-distributedlog-dev/
- number of jiras by filtering jira queue
- number of individual contributors : probably by looking at the number of
contributors at github?


> >>
> >> > 2. Lots of engagement on documentation.
> >> >
> >>
> >> What are you using to determine this? Mailing list discussion or jira?
> >>
> >> There are other interesting threads going on too, like transactions and
> >> event sourcing.
>

I think pull requests can be used for counting this.


> >>
> >> > 3. Improved responsiveness of committers to community members
> >> >
> >>
> >> I'd phrase this differently. I'd say something like:
> >>
> >> Increased traffic on the mailing list, in particular, due to committers
> >> engaging more actively with contributors.
> >>
> >> >
> >> > How has the project developed since the last report?
> >> >
> >> > 1. Documentation has improved for project build and project
> deployment.
> >> > Added more information on  community page.
> >> >
> >>
> >> Please try to be a bit more specific.
> >>
> >> > 2. Involved more discussion of new ideas and bring in new features.
> >> Include
> >> > major discussions like 'transaction support', 'batch commit' .etc.
> >>
> >> Ok, I see you talked about other thread

[jira] [Updated] (DL-2) DistributedLog should work with the official apache bookkeeper

2016-11-01 Thread Sijie Guo (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DL-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-2:
---
Priority: Blocker  (was: Major)

> DistributedLog should work with the official apache bookkeeper
> --
>
> Key: DL-2
> URL: https://issues.apache.org/jira/browse/DL-2
> Project: DistributedLog
>  Issue Type: Task
>  Components: distributedlog-core
>Reporter: Sijie Guo
>Priority: Blocker
> Fix For: 0.4.0
>
>
> Currently the version of bookkeeper that distributedlog depends on is 
> Twitter's version. http://distributedlog.io/operations/bookkeeper.html 
> It is twitter's production version 4.3.4-TWTTR, which is available in 
> https://github.com/twitter/bookkeeper
> Once the major Twitter's bookkeeper changes (BOOKKEEPER-670, BOOKKEEPER-759 
> and BOOKKEEPER-757) merged into main master, we should change pom files to 
> use the official bookkeeper versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DL-2) DistributedLog should work with the official apache bookkeeper

2016-11-01 Thread Sijie Guo (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DL-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sijie Guo updated DL-2:
---
Priority: Major  (was: Blocker)

> DistributedLog should work with the official apache bookkeeper
> --
>
> Key: DL-2
> URL: https://issues.apache.org/jira/browse/DL-2
> Project: DistributedLog
>  Issue Type: Task
>  Components: distributedlog-core
>Reporter: Sijie Guo
> Fix For: 0.4.0
>
>
> Currently the version of bookkeeper that distributedlog depends on is 
> Twitter's version. http://distributedlog.io/operations/bookkeeper.html 
> It is twitter's production version 4.3.4-TWTTR, which is available in 
> https://github.com/twitter/bookkeeper
> Once the major Twitter's bookkeeper changes (BOOKKEEPER-670, BOOKKEEPER-759 
> and BOOKKEEPER-757) merged into main master, we should change pom files to 
> use the official bookkeeper versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Distributedlog Podling Report Draft - November 2016

2016-11-01 Thread Jia Zhai 
Hi Sijie,
Would you please help address some of Flavio's comments first, You may have
more information for story behind the result. I would like to handle this
after your comments.

Thanks a lot.
-Jia

On Tue, Nov 1, 2016 at 10:42 AM, Jia Zhai  wrote:

> Thanks a lot for your comments, will address them and post a new draft.
>
> On Mon, Oct 31, 2016 at 6:02 PM, Flavio Junqueira <
> fpjunque...@yahoo.com.invalid> wrote:
>
>> Thanks, Jia. It would be good to have concrete metrics where possible.
>> See my comments below:
>>
>> > On 30 Oct 2016, at 10:53, Jia Zhai  wrote:
>> >
>> > Thanks a lot for your suggestions Flavio.
>> > And Guys, Here is a draft. Please help comments on it.
>> > Regards.
>> > -Jia
>> >
>> > =
>> >
>> > DistributedLog
>> >
>> > DistributedLog is a high-performance replicated log service. It offers
>> >
>> > durability, replication and strong consistency, which provides a
>> fundamental
>> >
>> > building block for building reliable distributed systems.
>> >
>> > DistributedLog has been incubating since 2016-06-24.
>> >
>> > Three most important issues to address in the move towards graduation:
>> > 1.   Continue to grow the community, especially people from outside
>> > Twitter.
>> > [Ask the PMC for helping check the number of people subscribed to dev@
>> mail
>> > list]
>>
>> I think you're referring to the project PMC checking the subscription
>> list. I checked and we have 35 people subscribed. A lot of people have
>> gmail and apache.org  addresses so it is hard to
>> tell affiliation. One typical sign of community diversity is to have active
>> committers from multiple organizations, which is just a natural extension
>> of a diverse community. As folks contribute to the project, I'd expect the
>> PPMC to offer committership to the contributors.
>>
>> Please do not refer to Twitter specifically, just say that we want to
>> increase diversity.
>>
>> > 2.   Improve documentation.
>>
>> Project documentation is important, but from a community perspective, it
>> is more important to have all processes documented well: how to contribute,
>> how to merge, notes for first time committers, how to cut a release, check
>> licenses. We need to make sure all this is in place once we suggest
>> graduation.
>>
>>
>> > 3.   First Apache release
>>
>> When is the first release due? it would be good for this community to
>> release sooner rather later. I'd expect multiple releases upon graduation.
>> I think I'd just say "Release successfully".
>>
>> >
>> >
>> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
>> aware
>> >
>> > of?
>> >
>> > None
>> >
>> > How has the community developed since the last report?
>> >
>> > 1. Increase in contributions from community.
>> >
>>
>> How do we measure this? Here are a few suggestions:
>>
>> - Number of messages to the list
>> - Number of jiras created/resolved
>> - Number of individual contributors
>>
>> > 2. Lots of engagement on documentation.
>> >
>>
>> What are you using to determine this? Mailing list discussion or jira?
>>
>> There are other interesting threads going on too, like transactions and
>> event sourcing.
>>
>> > 3. Improved responsiveness of committers to community members
>> >
>>
>> I'd phrase this differently. I'd say something like:
>>
>> Increased traffic on the mailing list, in particular, due to committers
>> engaging more actively with contributors.
>>
>> >
>> > How has the project developed since the last report?
>> >
>> > 1. Documentation has improved for project build and project deployment.
>> > Added more information on  community page.
>> >
>>
>> Please try to be a bit more specific.
>>
>> > 2. Involved more discussion of new ideas and bring in new features.
>> Include
>> > major discussions like 'transaction support', 'batch commit' .etc.
>>
>> Ok, I see you talked about other threads here.
>>
>> >
>> > 3. First release and repackaging of the project under apache namespace
>> are
>> >
>> >   being discussed, such as:
>> >
>> > - Pull requests for repackaging.
>> >
>> > - Twitter folks are working on porting twitter's bk version back to
>> > community
>>
>> It is a bit unclear why this is important, and I'd again avoid referring
>> to a specific company if possible.
>>
>> If there is a major change happening and the community is all involved,
>> then I'd mention it, but as stated it sounds like this is relevant to the
>> BookKeeper community, although it affects this community because of the
>> dependency.
>>
>> >
>> > - Expect to cut a release on Nov.
>>
>> We should mention this above when we refer to the first release. I'd also
>> be interested in knowing why the delay, what is holding it, and who is the
>> release manager. We don't necessarily need to add it to the report,
>> although more detail isn't bad.
>>
>> -Flavio
>>
>> >
>> >
>> >
>> >
>> > On Fri, Oct 28, 2016 at 11:10 PM, Flavio Junqueira 
>> wrote:
>> >
>> >> Hi Jia,
>> >>