[VOTE] DistributedLog Release 0.5.0, release candidate #2

[Jia Zhai]

Hi everyone,

Please review and vote on the release candidate #2 for the version 0.5.0,
as follows:

[ ] +1, Approve the release

[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:

* JIRA release notes [1],

* the official Apache source release to be deployed to dist.apache.org [2],

* all artifacts to be deployed to the Maven Central Repository [3],

* source code tag "v0.5.0" [4],


The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PPMC affirmative votes.


[1] https://github.com/apache/distributedlog/pull/180

[2] https://dist.apache.org/repos/dist/dev/bookkeeper/distribute
dlog/0.5.0-rc2/

[3] https://repository.apache.org/content/repositories/orgapache
bookkeeper-1014/

[4] https://github.com/apache/distributedlog/tree/release-0.5.0


Thanks,

-Jia

[VOTE] DistributedLog Release 0.5.0, release candidate #2

[Jia Zhai]

Hi everyone,

Please review and vote on the release candidate #2 for the version 0.5.0,
as follows:

[ ] +1, Approve the release

[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:

* JIRA release notes [1],

* the official Apache source release to be deployed to dist.apache.org [2],

* all artifacts to be deployed to the Maven Central Repository [3],

* source code tag "v0.5.0" [4],


The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PPMC affirmative votes.


[1] https://github.com/apache/distributedlog/pull/180

[2] https://dist.apache.org/repos/dist/dev/bookkeeper/
distributedlog/0.5.0-rc2/


[3] https://repository.apache.org/content/repositories/
orgapachebookkeeper-1014/


[4] https://github.com/apache/distributedlog/tree/release-0.5.0


Thanks,

-Jia

Re: [VOTE] DistributedLog Release 0.5.0, release candidate #1

[Jia Zhai]

Thanks for the verify. and Thanks for the fix @Sijie. Let's re-create one.

On Tue, Sep 12, 2017 at 12:46 AM, Sijie Guo <guosi...@gmail.com> wrote:

> Ah, I think the problem was introduced by the bookkeeper upgrade. we are
> using `ByteBuf` for better memory management. However, it seems that thrift
> only uses heap byte buffer. so when thrift tries to serialize a non-heap
> byte buffer, it would fail with "UnsupportedOperationException".
>
> Sent a pull request for fixing this -
> https://github.com/apache/distributedlog/pull/187
>
> Jia, we should cancel this rc, and recreate one once 187 is fixed.
>
> - Sijie
>
> On Mon, Sep 11, 2017 at 8:01 AM, Xi Liu <xi.liu@gmail.com> wrote:
>
> > I hit the same issue here.
> >
> > On Mon, Sep 11, 2017 at 7:59 AM, Khurrum Nasim <khurrumnas...@gmail.com>
> > wrote:
> >
> > > -1
> > >
> > > - verified source package, binary package. all md5, sha1 and asc look
> > good.
> > > - NOTICE, LICENSE look good
> > > - source package can compile.
> > > - both source package and binary package: verified tutorial, can run
> > > successful with core library.
> > >
> > > However, it failed to run tutorial with write proxy. I suspected this
> is
> > > related to ByteBuf changes recently.
> > >
> > > Sep 11, 2017 7:55:54 AM com.twitter.util.RootMonitor$ handle
> > > SEVERE: Exception propagated to the root monitor!
> > > java.lang.UnsupportedOperationException
> > > at java.nio.ByteBuffer.array(ByteBuffer.java:994)
> > > at
> > > org.apache.thrift.protocol.TBinaryProtocol.writeBinary(
> > > TBinaryProtocol.java:211)
> > > at
> > > org.apache.distributedlog.thrift.service.DistributedLogService$
> > > writeWithContext_args.write(DistributedLogService.java:6327)
> > > at
> > > org.apache.distributedlog.thrift.service.DistributedLogService$
> > > ServiceToClient.writeWithContext(DistributedLogService.java:1225)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl$
> > > WriteOp.sendWriteRequest(DistributedLogClientImpl.java:428)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl$
> > > AbstractWriteOp.sendRequest(DistributedLogClientImpl.java:365)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl.
> > > sendWriteRequest(DistributedLogClientImpl.java:957)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl.access$300(
> > > DistributedLogClientImpl.java:106)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl$
> > StreamOp.doSend(
> > > DistributedLogClientImpl.java:201)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl$
> StreamOp.send(
> > > DistributedLogClientImpl.java:185)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl$4.onSuccess(
> > > DistributedLogClientImpl.java:887)
> > > at
> > > org.apache.distributedlog.client.DistributedLogClientImpl$4.onSuccess(
> > > DistributedLogClientImpl.java:879)
> > >
> > > - KN
> > >
> > > On Thu, Sep 7, 2017 at 5:04 AM, Jia Zhai <zhaiji...@gmail.com> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > > > 0.5.0, as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> dist.apache.org
> > > > [2],
> > > > * all artifacts to be deployed to the Maven Central Repository [3],
> > > > * source code tag "v0.5.0" [4],
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PPMC affirmative votes.
> > > >
> > > > [1] https://github.com/apache/distributedlog/pull/180
> > > > [2] https://dist.apache.org/repos/dist/dev/bookkeeper/
> > > > distributedlog/0.5.0-rc1/
> > > > [3] https://repository.apache.org/content/repositories/
> > > > orgapachebookkeeper-1013/
> > > > [4] https://github.com/apache/distributedlog/tree/release-0.5.0
> > > >
> > > > Thanks,
> > > > -Jia
> > > >
> > >
> >
>

[VOTE] DistributedLog Release 0.5.0, release candidate #1

[Jia Zhai]

Hi everyone,


Please review and vote on the release candidate #1 for the version
0.5.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
* all artifacts to be deployed to the Maven Central Repository [3],
* source code tag "v0.5.0" [4],

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PPMC affirmative votes.

[1] https://github.com/apache/distributedlog/pull/180
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/distributedlog/0.5.0-rc1/
[3] https://repository.apache.org/content/repositories/orgapachebookkeeper-1013/
[4] https://github.com/apache/distributedlog/tree/release-0.5.0

Thanks,
-Jia

Re: Relax durability

[Jia Zhai]

I second that!  Thanks Enrico for starting and holding this productive
discussion.  Thanks Enrico, JV, Sijie and other guys to make this
consensus.
Looking forward for the design.

On Fri, Sep 1, 2017 at 12:53 AM, Venkateswara Rao Jujjuri  wrote:

> Hi all,
>
> It has been a great and lively discussion. I can say this is one of the
> highly trended topics in the recent BK community discussion.
> Kudos to Enrico for starting this.
>
> Enrico, Sijie and I met and discussed this further and came up with the
> following consensus on how to move forward.
>
> * Introduce LedgerType/LedgerProperties which goes into ZK metadata.
> * No changes to AddEntry API (application view); but AddEntry RPC will add
> a flag to bookies to inform about the type/durability.
> * Introduce a sync() RPC which needs to be called explicitly on RD ledgers.
> * No changes to LAC and how we update it.
> * No changes to the behavior of readEntries() API, which reads only until
> LAC.
> * Applications can use readUnConcirmed API to read until last add pushed.
> * Segregate stats based on the ledgertype.
>
>
> Enrico is going to merge two docs and publish a detailed design. Thanks a
> lot Enrico
>
>
> On Mon, Aug 21, 2017 at 10:01 PM, Sijie Guo  wrote:
>
> > On Aug 21, 2017 5:44 AM, "Enrico Olivelli"  wrote:
> >
> > As the issue is really huge, I need to narrow the design and
> implementation
> > efforts to a specific case at the moment: I am interested in having a
> > per-ledger flag to not require fsynch on entries on journal.
> >
> >
> > It is good to narrow down the implementation. However because there are
> > different requirements from different people. It would be good to discuss
> > and cover all thoughts.
> >
> >
> > If the "no-synch" flag is applied per ledger than we have to decide what
> to
> > do on the LAC protocol, I see two opposite ways:
> > 1) the LAC will never advanced (no fsynch is guaranteed on journal)
> > 2) the LAC is advanced as usual but it will be possible to have missing
> > entries
> >
> >
> > Personally I am -1 to approach 2) as for the reasons I stated in previous
> > emails.
> >
> >
> > There is a "gray" situation:
> > 3) as entries will be interleaved on the journal with entries of other
> > "synch" ledgers it will be possible to detect some kind of "synched"
> > entries and return the info to the writing client which in turn will be
> > able to advance the LAC:
> > this option is not useful as the behavior is unpredictable
> >
> > For my "urgent" usecase I would prefer 2), but 1) is possible too,
> because
> > I am using LedgerHandlerAdv (I have manual allocation of entry ids) +
> > readUnconfirmedEntries (which allows to read entries even if LAC did not
> > advance)
> >
> >
> > As JV suggested, please start the design doc and let's iterate over it
> > before the implementation.
> >
> >
> > -- Enrico
> >
> >
> > 2017-08-19 14:09 GMT+02:00 Enrico Olivelli :
> >
> > >
> > >
> > > On ven 18 ago 2017, 20:12 Sijie Guo  wrote:
> > >
> > >> /cc (distributedlog-dev@)
> > >>
> > >> I know JV has similar use cases. This might require a broad
> discussion.
> > >> The
> > >> most tricky part would be LAC protocol - when can the client advance
> the
> > >> LAC. I think a BP, initially with a google doc shared to the community
> > >> would be good to start the discussion. because I would expect a lot
> > points
> > >> to discuss for this topic. Once we finalize the details, we can copy
> the
> > >> google doc content back to the wiki page.
> > >>
> > >
> > > Thank you Sijie and JV for pointing me to the right direction.
> > > I had underestimated the problems related to the ensemble changes, and
> > > also effectively in my projects  it can happen that a single
> > 'transaction'
> > > can span more then one ledger so the ordering issues are nore complex
> > than
> > > I expected. If somehow it would be possible to keep ordering inside the
> > > scope of a single ledger it is very hard to get it using multiple
> > ledgers.
> > >
> > > Next week I will write the doc, but I think I am going to split the
> > > problem into multiple parts.
> > > I see that the LAC must be advanced only when an fsynch is done. This
> > will
> > > preserve correctness as Sijie told.
> > >
> > > I think that the problems related to the ordering of events must be
> > > addressed at application level and it would be the best thing to have
> > such
> > > support in DL.
> > >
> > > For instance at first glance I omage that we should add in BK some
> > support
> > > in order to let the application receive notifications of changes to LAC
> > to
> > > the writer more easily.
> > >
> > > The first step would be to add a new flag to addEntry to receive
> > > acknowledge on fwrite and flush (with the needed changes to the
> journal),
> > > and in the addresponse a flag wjich tells that the entry has been
> synched
> > > or only flushed, and handle 

Re: Starting DistributedLog release process for 0.5.0

[Jia Zhai]

Thanks a lot.  :)
I created an issue here
<https://github.com/apache/distributedlog/issues/167> for tracking 0.5.0
release related work.

Regards.
-Jia

On Thu, Aug 31, 2017 at 2:34 AM, Sijie Guo <guosi...@gmail.com> wrote:

> +1 for cutting 0.5.0
>
> I think the main change I'd like to include is using the ByteBuf class. I
> will also mark other issues that need to be included in 0.5.0
>
> - Sijie
>
> On Tue, Aug 29, 2017 at 9:05 PM, Jia Zhai <zhaiji...@gmail.com> wrote:
>
> > Hi all, :)
> >
> > We have already bumped the BookKeeper version to official
> > apache-bookkeeper-4.5.0 in DistributedLog, and most of DistributedLog
> > graduation items are fixed,  so It would be a good time to cut a release
> > for DistributedLog 0.5.0(maybe at the end of this week or early next
> > week?).
> >
> > Please help check and mark 0.5.0 for the omitted issues, which we want to
> > put in release 0.5.0.
> >
> > Any thoughts?
> >
> > Thanks a lot.
> > -Jia
> >
>

Starting DistributedLog release process for 0.5.0

[Jia Zhai]

Hi all, :)

We have already bumped the BookKeeper version to official
apache-bookkeeper-4.5.0 in DistributedLog, and most of DistributedLog
graduation items are fixed,  so It would be a good time to cut a release
for DistributedLog 0.5.0(maybe at the end of this week or early next week?).

Please help check and mark 0.5.0 for the omitted issues, which we want to
put in release 0.5.0.

Any thoughts?

Thanks a lot.
-Jia

Re: [DistributedLog Proposal] Use Github for issue management

[Jia Zhai]

Thanks.
will creating the basic labels, and update contribute guide for this.

On Tue, Aug 15, 2017 at 2:26 PM, Sijie Guo <guosi...@gmail.com> wrote:

> Jia,
>
> it seems that we all agreed to do this and no complaints. let's close this
> and move forward.
>
> - Sijie
>
> On Fri, Aug 11, 2017 at 12:35 PM, Enrico Olivelli <eolive...@gmail.com>
> wrote:
>
> > For bookkeeper we are still in a work in progress phase, maybe we can do
> > the switch for DL when we will be more organized in BK.
> > My feeling is that we are on the good way for BK, we did engaged some new
> > contributors I think because github is open to everyone nowadays
> >
> > My two cents
> > Enrico
> >
> > Il ven 11 ago 2017, 21:32 Sijie Guo <guosi...@gmail.com> ha scritto:
> >
> > > On Fri, Aug 11, 2017 at 10:19 AM, Henry Saputra <
> henry.sapu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Should use same issue management as BookKeeper.
> > > >
> > > > One thing that I miss from JIRA is ability to distinguish Epic,
> Story,
> > > and
> > > > Tasks for issues.
> > > >
> > >
> > > Same feeling here :)
> > >
> > >
> > > >
> > > > On Thu, Aug 10, 2017 at 12:30 AM Sijie Guo <guosi...@gmail.com>
> wrote:
> > > >
> > > > > +1 personally I had a very good experience on using github issues
> as
> > > the
> > > > > issue tracking.
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Tue, Aug 8, 2017 at 5:28 PM, Jia Zhai <zhaiji...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > As part of DistributedLog graduation, DistributedLog git repo is
> > > > renamed
> > > > > to
> > > > > > 'distributedlog <https://github.com/apache/distributedlog>' and
> > > moved
> > > > to
> > > > > > gitbox, as what BookKeeper did a month ago.
> > > > > >
> > > > > > While in BookKeeper user feels more effective and smooth to use
> > > Github
> > > > > > issue for issue management,   I am proposing to also try
> > > > DistributedLog's
> > > > > > issue management in Github issue as BP-9
> > > > > > <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> > > > > > BP-9+-+Github+issues+for+Issue+Tracking>
> > > > > >
> > > > > > Best Regards.
> > > > > > -Jia
> > > > > >
> > > > >
> > > >
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>

Re: Updates on DL graduation items

[Jia Zhai]

Hi Sijie,
Is this 2 links also in the plan?
https://travis-ci.org/apache/incubator-distributedlog/
https://coveralls.io/repos/github/apache/incubator-distributedlog/badge.svg?branch=master

On Thu, Aug 10, 2017 at 1:35 AM, Sijie Guo  wrote:

> Updates:
>
> INFRA fixed the website issue. Now the content is serving under both
> http://bookkeeper.apache.org/distributedlog/ and
> http://distributedlog.incubator.apache.org/. Once we confirmed the
> graduation to incubator, the distributedlog incubator dns will be removed.
>
> - Sijie
>
> On Tue, Aug 8, 2017 at 11:48 AM, Sijie Guo  wrote:
>
> > Hi all,
> >
> > INFRA already completed the tasks for moving the resources to bookkeeper.
> >
> > https://issues.apache.org/jira/browse/INFRA-14671
> >
> > A summary for this:
> >
> > - Git repo : https://github.com/distributedlog
> > - mailing lists: {user,dev,commits}@distributedlog.incubator.apache.org
> =>
> > distributedlog-{user,dev,commits}@bookkeeper.apache.org
> > - dist directory are moved to under bookkeeper/distributedlog
> > - website will be under https://bookkeeper.apache.org/distributedlog/
> >
> > I also created a JIRA for changes documentation and website at our side:
> > https://issues.apache.org/jira/browse/DL-207
> >
> > Any thoughts?
> >
> > - Sijie
> >
>

Re: BP-12: Improve documentation

[Jia Zhai]

I would like to help on the docker deployment part.
Thanks.

On Wed, Aug 9, 2017 at 7:39 PM, Enrico Olivelli  wrote:

> Il mer 9 ago 2017, 07:47 Sijie Guo  ha scritto:
>
> > (/cc dev@distributedlog for calling for contributions)
> >
> > Hi all,
> >
> > Prior to 4.5.0, bookkeeper doesn't have good documentation. In 4.5.0
> > release, the community did a lot of work on improving the current
> > documentation along with new bookkeeper website. However, the
> documentation
> > itself is stilling lagging behind of production usages and
> implementations.
> > A while ago, the community was trying to improve documentation but it
> > didn't actually happen due to lack of efforts.
> >
> > I am starting a BP for calling for contributions to improve the
> > documentation. Here is the BP:
> >
> >
> > https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> BP-12+-+Improve+documentation
> >
> > Any thoughts? If you are interested in participating in this, please
> > comment here as well.
> >
>
> +1
>
> >
> I did start to write about the client API some months ago. I wuold like to
> continue on this topic.
> It is very important and interesting.while writing docs I found several
> problems and so this activity will make us grow in a good direction and
> become more 'owners' of own project.
>
> Apart from writing docs in the next months I would like to help in cleaning
> up the client API anf the various plugin APIs
>
> Enrico
>
>
> > - Sijie
> >
> --
>
>
> -- Enrico Olivelli
>

Re: Updates on DL graduation items

[Jia Zhai]

Con~.
:) Seems the git repo should be: https://github.com/apache/distributedlog

On Wed, Aug 9, 2017 at 5:07 AM, Enrico Olivelli  wrote:

> Congrats Sijie for DL graduation !
>
> Il mar 8 ago 2017, 20:48 Sijie Guo  ha scritto:
>
> > Hi all,
> >
> > INFRA already completed the tasks for moving the resources to bookkeeper.
> >
> > https://issues.apache.org/jira/browse/INFRA-14671
> >
> > A summary for this:
> >
> > - Git repo : https://github.com/distributedlog
> > - mailing lists: {user,dev,commits}@distributedlog.incubator.apache.org
> =>
> > distributedlog-{user,dev,commits}@bookkeeper.apache.org
> > - dist directory are moved to under bookkeeper/distributedlog
> > - website will be under https://bookkeeper.apache.org/distributedlog/
> >
> > I also created a JIRA for changes documentation and website at our side:
> > https://issues.apache.org/jira/browse/DL-207
> >
> > Any thoughts?
> >
> > - Sijie
> >
> --
>
>
> -- Enrico Olivelli
>