Pulsar Summit San Francisco 2022 CFP is open!

[Sijie Guo]

Hi, all the Pulsar community members,

Pulsar Summit San Francisco 2022 CFP is open now! It will be our *first-ever
in-person Pulsar Summit *in San Francisco! We're glad to invite you to
share your experience and add up to the wonderfulness.

As a speaker, you will receive the chance to demonstrate your experience
and deep knowledge, and your bio and session featured on the Pulsar Summit
website. We would also like you to help us make Pulsar Summit San Francisco
2022 a big success by spreading the word on Twitter, LinkedIn, and other
social media platforms.

Please submit your talks [1] by May 20th (Pacific Time). We
welcome submissions from around the globe. Also, first-time speakers are
welcomed. For more details, please check the submission link, and do not
hesitate to contact us at organiz...@pulsar-summit.org if you have any
questions.

We look forward to your participation and hope to see you in person in San
Francisco!

For any more updates about Pulsar Summit San Francisco, you can follow
@PulsarSummit  on Twitter or add yourself
to the waitlist on the home page .

[1] Submission link: https://sessionize.com/pulsar-summit-san-francisco-2022
[2] Pulsar Summit Homepage: https://pulsar-summit.org/

Re: [ANNOUNCE] New committer Prashant Kumar

[Sijie Guo]

Congratulations, Prashant!

- Sijie

On Mon, Jan 3, 2022 at 10:06 AM Enrico Olivelli  wrote:

> Welcome!
>
> Very well deserved
>
> Enrico
>
> Il Lun 3 Gen 2022, 18:17 Jonathan Ellis  ha scritto:
>
> > Congratulations, Prashant!
> >
> > On Mon, Jan 3, 2022 at 11:15 AM Henry Saputra 
> > wrote:
> >
> > > Hi All, Happy New Year 2022!
> > >
> > > The Project Management Committee (PMC) for Apache BookKeeper has
> invited
> > > Prashant Kumar to become a committer and we are pleased to announce
> that
> > he
> > > has accepted.
> > >
> > >
> > > Being a committer enables easier contribution to the poject since there
> > is
> > > no need to go via the patch submission process.
> > >
> > > Please join me to congratulate and welcome Prashant.
> > >
> > > Thanks,
> > >
> > > Henry
> > >
> >
> >
> > --
> > Jonathan Ellis
> > co-founder, http://www.datastax.com
> > @spyced
> >
>

Re: BP-46 running without journal PR ready for review

[Sijie Guo]

Jack,

Awesome! Glad to know the PR is finally out for review.

Sijie

On Fri, Dec 17, 2021 at 10:48 AM Jack Vanlightly 
wrote:

> Hi all,
>
> The BP-46 running without the journal changes are ready for review.
>
> https://github.com/apache/bookkeeper/pull/2936
>
> You can view the BP PR here:
> https://github.com/apache/bookkeeper/pull/2706
>
> Also Ivan and I did a talk about it which might help describe some of the
> context around the journal and our changes.
> https://www.youtube.com/watch?v=Cwcbffybfac
>
> Thanks
> Jack
>

Re: [DISCUSS] Bookkeeper 4.14.3 release

[Sijie Guo]

+1

On Tue, Dec 14, 2021 at 6:35 AM Matteo Merli  wrote:

> There are few fixes in the 4.14 that would be good to release soon.
>
> Given that the timeline for 4.15 can be quite long, I'd propose to
> release one more patch release right now.
>
> Let me know if there are other fixes to include.
>
> Matteo
>
>
> --
> Matteo Merli
> 
>

Re: BP-44: direct io entry logging

[Sijie Guo]

+1 Looks very promising

On Wed, Dec 15, 2021 at 9:36 AM Maurice Barnum 
wrote:

> In order to improve IO utilization, Splunk developed support for logging
> entries bypassing the kernel's buffer cache via O_DIRECT.  The code has
> been in production for several months, running on Linux, with an
> implementation for MacOS also included.
>
> BP-44 proposes to merge the code into the mainstream code base.
>
> https://github.com/apache/bookkeeper/pull/2932 is mostly complete merge of
> the code base to the Bookkeeper master branch.  Some unit tests fail, most
> likely due to merge errors that need to be resolved.
>
> Comments appreciated.
>

Re: [ANNOUNCE] new committer: Jack Vanlightly

[Sijie Guo]

Congrats, Jack!

- Sijie

On Mon, Nov 15, 2021 at 2:37 AM Enrico Olivelli  wrote:

> The Project Management Committee (PMC) for Apache BookKeeper
> has invited Jack Vanlightly to become a committer and we are pleased
> to announce that he has accepted.
>
>
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
>
> Congrats Jack !
>
> Enrico
>

Re: [VOTE] Release 4.14.3 release candidate #2

[Sijie Guo]

+1

- signatures are good
- source package is good
- binary package is good

On Tue, Nov 9, 2021 at 5:17 PM Yong Zhang 
wrote:

> Hi guys,
>
> Any PMC give this release one more +1 to close this vote?
>
> Thanks!
>
> On Mon, 8 Nov 2021 at 17:40, Nicolò Boschi  wrote:
>
> > +1 (non binding)
> >
> > Verified the followings:
> > - checksum and signatures
> > - build from sources
> > - ledgers write/read with "localbookie" command
> >
>

Re: [DISCUSS] Release 4.14.2

[Sijie Guo]

+1

On Thu, Aug 12, 2021 at 11:59 PM Yong Zhang  wrote:
>
> Hi,
>
> We have changed the BouncyCastle at this PR
> https://github.com/apache/bookkeeper/pull/2631,
> which introduces an Incompatible issue. Detail:
> https://github.com/apache/pulsar/issues/10937.
>
> This also blocks the user upgrade their charts to pulsar 2.8.0
> https://github.com/apache/pulsar-helm-chart/pull/130
>
> We have fixed it by https://github.com/apache/bookkeeper/pull/2740,
> so I want to start a new release of bookkeeper for unblocking the users.
>
> If there are no objections, I'll move forward with the patch release.
>
> Thanks,
> Yong

[ANNOUNCE] New Committer: Yong Zhang

[Sijie Guo]

The Project Management Committee (PMC) for Apache BookKeeper has
invited Yong Zhang
to become a committer and we are pleased to announce that he has accepted!

Congratulations and welcome onboard Yong Zhang! Looking forward to
seeing more contributions from Yong!

Please join us to welcome Yong Zhang!

Sijie Guo
on behalf of Apache BookKeeper PMC

Re: [VOTE] BookKeeper release 4.14.1, release candidate #0

[Sijie Guo]

+1

Verified the followings:

- checksum, signatures
- source package can build
- binary package can run

- Sijie

On Fri, May 28, 2021 at 10:34 PM Matteo Merli  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #0 for the version
> 4.14.1, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "release-4.14.1-rc0" [4] with git sha [5]
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/2724
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.1-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1053/
> [4] https://github.com/apache/bookkeeper/releases/tag/release-4.14.1-rc0
> [5] f7a94422e44a7da6a5a8667853ad468bb0194642
>
>
> Thank you,
> Matteo
>
> --
> Matteo Merli
> 
>

Re: [VOTE] Release 4.14.0, release candidate #0

[Sijie Guo]

+1 (binding)

All things below are good.

  * Signatures
  * Source package structure, and compile
  * Binary package
  * Running standalone bookie and client

On Mon, May 17, 2021 at 9:43 AM Matteo Merli  wrote:

> +1 (binding)
>
> Checked:
>   * Signatures
>   * Source package structure, and compile
>   * Binary package
>   * Running standalone bookie and client
>
>
> --
> Matteo Merli
> 
>
> On Fri, May 14, 2021 at 2:23 PM Andrey Yegorov
>  wrote:
> >
> > I do not hear voices against it so I assume everyone is ok with this.
> > (I started building RC1 and docker hanged mid-process; happy to not deal
> > with retry).
> >
> > The vote has been out for about 5 days already.
> > I need votes from 3 PMCs to proceed.
> >
> >
> > On Fri, May 14, 2021 at 12:10 PM Henry Saputra 
> > wrote:
> >
> > > I am +1 to keep Gradle out from this release since the files are ok
> not to
> > > be included.
> > >
> > > I am ok to keep RC0 for 4.14.0 for VOTE as release candidate.
> > >
> > > - Henry
> > >
> > > On Fri, May 14, 2021 at 12:14 AM Enrico Olivelli 
> > > wrote:
> > >
> > > > Andrey,
> > > > I have merged the patch.
> > > >
> > > > In my opinion the source tarball should reflect the git repo without
> > > > the '.git' database.
> > > > That said, the most important thing is that the source release
> > > > contains the files to build the project and produce working binaries.
> > > >
> > > > This is in the spirit of Open Source (of the ASF at least), you can
> > > > pick the sources, modify them, build and run the new binaries.
> > > >
> > > > AFAIK The Gradle build is not fully implemented so it is not so
> > > > important to see those files at the moment
> > > > I agree that we can strip away the website, it is useless to the
> > > > purpose of building the project.
> > > >
> > > > Enrico
> > > >
> > > >
> > > > Il giorno gio 13 mag 2021 alle ore 23:43 Andrey Yegorov
> > > >  ha scritto:
> > > > >
> > > > > I looked at the situation with src release artifact, and:
> > > > > - it was definitely built from the right changelist (I spot checked
> > > some
> > > > > files affected by the last commit)
> > > > > - site/ is explicitly excluded
> > > > >
> > > >
> > >
> https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L61
> > > > > - site2 is not excluded (should be fixed)
> > > > > - gradle files are not explicitly included, pom are
> > > > >
> > > >
> > >
> https://github.com/apache/bookkeeper/blob/732b6cf2a6576c844a4b43176125e82d500a6ea8/bookkeeper-dist/src/assemble/src.xml#L33
> > > > >
> > > > > that src.xml is used to package the release artifacts
> > > > >
> > > >
> > >
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-dist/pom.xml#L55
> > > > >
> > > > > It looks like the plugin resolves the situation with the file not
> > > > matching
> > > > > includes and excludes by excluding it.
> > > > >
> > > > > PR that adds gradle into the includes patterns and excludes site2:
> > > > > https://github.com/apache/bookkeeper/pull/2714
> > > > >
> > > > > I can either rebuild the RC or we can decide to proceed.
> > > > > Gradle was not used to build/release 4.14.0
> > > > >
> > > > >
> > > > > On Thu, May 13, 2021 at 8:34 AM Andrey Yegorov <
> > > > andrey.yego...@datastax.com>
> > > > > wrote:
> > > > >
> > > > > > I ran the release script, as described in the release procedure.
> > > > > > It all ran from the docker container.
> > > > > >
> > > > > > On Wed, May 12, 2021 at 11:30 PM Henry Saputra <
> > > > henry.sapu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Andrey,
> > > > > >>
> > > > > >> How did you generate the source release artifact? Looks like it
> is
> > > > missing
> > > > > >> some files like the site directory and the gradle build files.
> > > > > >>
> > > > > >> - Henry
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Sun, May 9, 2021 at 11:16 PM Andrey Yegorov <
> > > > > >> andrey.yego...@datastax.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi everyone,
> > > > > >> > Please review and vote on the release candidate #0 for the
> version
> > > > > >> 4.14.0,
> > > > > >> > as follows:
> > > > > >> > [ ] +1, Approve the release
> > > > > >> > [ ] -1, Do not approve the release (please provide specific
> > > > comments)
> > > > > >> >
> > > > > >> > The complete staging area is available for your review, which
> > > > includes:
> > > > > >> > * Release notes [1]
> > > > > >> > * The official Apache source and binary distributions to be
> > > > deployed to
> > > > > >> > dist.apache.org [2]
> > > > > >> > * All artifacts to be deployed to the Maven Central
> Repository [3]
> > > > > >> > * Source code tag "release-4.14.0" [4] with git sha
> > > > > >> > 4729682f00f05f23a1821211cbce064e653edb83
> > > > > >> >
> > > > > >> > BookKeeper's KEYS file contains PGP keys we used to sign this
> > > > release:
> > > > > >> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > > > >> >
> > > > > 

Re: Cutting BookKeeper 4.13.1 or 4.14.0 release?

[Sijie Guo]

+1

On Tue, May 4, 2021 at 2:22 AM Yunze Xu 
wrote:

> Hello,
> About 10 days ago I found a heap memory copy problem in Apache Pulsar, see
> [1].
> It’s a problem of BK side because when `LedgerHandle#asyncAddEntry`
> accepts a `CompositeByteBuf` or a wrapper, it will finally call
> `ByteBuf#nioBuffer()`, which would make a heap copy from direct memory.
> [2] fixed this problem and has been merged for a week.
>
> Since it has a significant impact on Pulsar, Pulsar side needs a new BK
> release with [2] merged to fix it. Is there any plan to cut a 4.13.1
> release or 4.14.0 release so that we can upgrade the dependency in Pulsar?
>
> Thanks,
> Yunze
>
> [1] https://github.com/apache/pulsar/pull/10330 <
> https://github.com/apache/pulsar/pull/10330>
>
> [2] https://github.com/apache/bookkeeper/pull/2701 <
> https://github.com/apache/bookkeeper/pull/2701>
>
>

[ANNOUNCE] Introducing Apache Pulsar Hackathon 2021!!!

[Sijie Guo]

Hi, Pulsar and BookKeeper communities,

The adoption of Apache Pulsar and BookKeeper is accelerating as
organizations around the world pursue cloud-native technologies. To build
on this momentum, we are excited to announce the first-ever Apache Pulsar
Hackathon 2021, *taking place on May 6-7th*. This event will give you an
opportunity to:

- Engage and grow with the Pulsar/BookKeeper community
- Generate ideas to enhance Pulsar/BookKeeper and its ecosystem
- Drive contributions as part of a diverse group of developers and data
architects

*Hackathon Categories:*

The Hackathon is all things about Pulsar, BookKeeper, and the ecosystem
around them. To help inspire you, we’ve created five categories for this
year’s hackathon:

- *Pulsar Enhancement*: adding new features, improving performance, etc.
- *BookKeeper Enhancement*: adding new features, improving performance, etc.
- *Pulsar + Big Data Ecosystem Integration*: integrating Pulsar with other
influential data systems for easy usage
- *Pulsar + Flink Solution*: developing end-to-end general data processing
solutions based on Pulsar and Apache Flink
- *Pulsar + Cloud*: enabling Pulsar to run on cloud environments easily and
seamlessly

*Hackathon Timeline* (All times listed in PST):

- April 28, 2021 - Registration Deadline
- May 6-7, 2021 - Hackathon
- May 6, 9:00 AM - Virtual Live Kick-Off
- May 6, 9:15 AM - 10 PM - Hacking Time
- May 7, 9:00 AM - 10 PM - Hacking Time
- May 7, 11:59 PM - Video Submission Deadline
- June 16-17, 2021 - Winners Announced at the Pulsar Summit North America
2021

*Hackathon Prizes*:

- *First-place: $5,000*
- Second-place: $2,500
- Third-place: $1,000
- All participants: Pulsar and StreamNative swag

*How to Participate:*

- Sign up! We encourage teams up to four people. If you already have the
team you will be working with and your idea, you can sign up here

.
- If you are looking for teammates, ideas, or just to connect with the
community, we encourage you to join Apache Pulsar on Slack (designated
hackathon channel: #pulsar-hackathon-2021
).

More details can be found here
.
Join us and showcase your creativity and skills!

- Sijie

[ANNOUNCE] Pulsar Summit North America 2021 Call-for-speakers and Registration is now open

[Sijie Guo]

Dear Apache Pulsar community,

We are excited to announce that Pulsar Summit North America 2021 will be
hosted virtually on June 16th - 17th. Call-for-speakers and attendee
registration are now open for the conference!

Pulsar Summit is the conference dedicated to Apache Pulsar, and the
messaging and event streaming community. The conference gathers an
international audience of CTOs/CIOs, developers, data architects, data
scientists, Apache Pulsar committers/contributors, and the messaging and
streaming community. Together they share experiences, exchange ideas, and
knowledge, and receive hands-on training sessions led by Pulsar experts.

*Join Us and Speak at Pulsar Summit!*

Do you have a Pulsar story to share? Join us and speak at the summit! You
will be on stage with all the top Pulsar thought-leaders. It is a great way
to participate and raise your profile in the rapidly growing Apache Pulsar
community.

We are looking for Pulsar stories that are innovative, informative, or
thought-provoking. Here are some suggestions for what to talk about:

   - Use cases, operations, tools, techniques, or the Pulsar ecosystem
   - A deep dive into technologies
   - Best practices and lessons learned
   - A Pulsar success story
   - Anything that inspires the audience!

*To speak at the summit, please submit an abstract
 about your
presentation.* Remember to keep your proposal short, relevant and engaging.

*Important Dates:*

   - CFP opens: Feb 18th, 2021
   - CFP closes: Mar 26th, 2021
   - Speaker notifications sent: April 2nd, 2021
   - Schedule announcement: April 9th, 2021

If you want some advice or feedback on your proposal, or have any questions
about the summit, please do not hesitate to contact us at
organiz...@pulsar-summit.org. We are happy to help!

*Register for the Summit*

If you are interested in attending Pulsar Summit North America 2021,
please sign
up in the Apache Foundation's Hopin instance
. Once you are
registered, we will keep you updated on the summit.

Help us make *#PulsarSummit NA 2021* a big success by spreading the word
and submitting your proposal! Follow us on Twitter
 to receive the latest updates of the
conference!

See you at Pulsar Summit North America 2021!

- Sijie

Re: Unbounded memory usage for WQ > AQ ?

[Sijie Guo]

On Mon, Jan 18, 2021 at 10:18 AM Flavio Junqueira  wrote:

> >>> Regarding recovery reads, recovery read doesn't need to be
> deterministic.
> >>> For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
> >>> either including it or excluding it in the sealed ledger is correct
> >>> behavior. The bookkeeper client guarantees that once a ledger is
> sealed,
> >>> the entries in the sealed ledger can always be read and can be read
> >>> consistently.
> >>
> >>> I am not sure it is a problem unless I misunderstand it.
> >>
> >> It is true that it doesn't violate any safety property, but it is a
> strange
> >> check to me. It looks like an implementation artefact rather than an
> >> explicit protocol design choice. But not a huge deal.
> >>
> >
> > It was discussed in the earlier days as a design choice for this
> protocol.
> >
> > If we want to make it deterministic, we might need to consider what is
> the
> > performance penalty.
>
>
> I don't quite follow the observation about a deterministic check. The
> example that Sijie provides makes sense to me if I understand it right as
> the entry does not have enough replicas, so it can go either way when the
> ledger is close. But, that assumes that no later entry has been
> acknowledged, otherwise we have a data loss if we skip the entry and
> consequently have a problem with the protocol. If anyone cares to explain
> the deterministic check referred to, I'd appreciate.
>

Based on my understanding, Jack wants the behavior on recovering an entry
does not have enough replicas to be deterministic. i.e. If the entry does
not have enough replicas, we can always exclude the entry. Jack, did I get
you right?

- Sijie


>
> -Flavio
>
> > On 18 Jan 2021, at 18:51, Sijie Guo  wrote:
> >
> > Jack,
> >
> > Thank you for your replies! That's good as there are not violations of
> > bookkeeper protocol.
> >
> > Comments inline.
> >
> > On Mon, Jan 18, 2021 at 3:20 AM Jack Vanlightly
> > mailto:jvanligh...@splunk.com.invalid>>
> wrote:
> >
> >>> Did you guys see any issues with the ledger auditor?
> >>
> >>> The active writer can't guarantee it writing entries to WQ because it
> can
> >>> crash during retrying adding entries to (WQ - AQ) bookies.
> >>
> >> The need to repair AQ replicated entries is clear and the auditor is one
> >> such strategy. Ivan has also worked on a self-healing bookie strategy
> where
> >> each bookie itself is able to detect these holes and is able to obtain
> the
> >> missing entries itself. The detection of these holes using this
> strategy is
> >> more efficient as it only requires network calls for the ledger metadata
> >> scanning (to zk) and the missing entry reads (to other bookies). The
> >> auditor as I understand it, reads all entries of all ledgers from all
> >> bookies (of an entries ensemble) meaning these entries cross the
> network.
> >> Using the auditor approach is likely to be run less frequently due to
> the
> >> network cost.
> >>
> >
> > Agreed on the efficiency part. I think the Salesforce team introduced the
> > Disk Scrubber to solve that problem already unless I confused something
> > there.
> >
> > +JV Jujjuri mailto:vjujj...@salesforce.com>>
> can chime in on this part.
> >
> >
> >>
> >> I do also wonder if the writer, on performing an ensemble change, should
> >> replay "AQ but not WQ" entries, this would just leave writer failures
> >> causing these AQ replicated entries.
> >>
> >
> > The writer can do that. But there are no guarantees there. You still
> need a
> > mechanism to repair the under-replicated entries.
> > It will also make the writer become much complicated to maintain.
> >
> >
> >>
> >>> Regarding recovery reads, recovery read doesn't need to be
> deterministic.
> >>> For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
> >>> either including it or excluding it in the sealed ledger is correct
> >>> behavior. The bookkeeper client guarantees that once a ledger is
> sealed,
> >>> the entries in the sealed ledger can always be read and can be read
> >>> consistently.
> >>
> >>> I am not sure it is a problem unless I misunderstand it.
> >>
> >> It is true that it doesn't violate any safety property, but it is a
> strang

Re: Unbounded memory usage for WQ > AQ ?

[Sijie Guo]

> One concern for me in this thread is case (3). I'd expect a client that
doesn't crash to not give up, and eventually replace the bookie if it is
unresponsive.

The current implementation doesn't retry replacing a bookie if an entry is
already acknowledged (receiving AQ responses). It relies on inspection to
repair the hole.

So the memory pressure is not coming from retrying. It is straight that the
bookkeeper client references the sendBuffers until it receives any
responses from the slow bookie. The bookkeeper client allows enqueuing
addEntry operations because the operations meet the AQ requirements. Pulsar
does add `maxPendingPublishdRequestsPerConnection` mechanism to throttle
the add operations. But this won't work as bookkeeper will notify the
callbacks once the operations meet the AQ requirements. But there is a huge
amount of memory (throughput * timeout period) referenced by a slow bookie.
Hence we have to add a memory-based throttling mechanism as Matteo
suggested.

If we want to add the retry logic to replace a bookie, this will add more
pressure to the memory. But it can still be solved by a memory-based
back-pressure mechansim.

Thanks,
Sijie

On Mon, Jan 18, 2021 at 8:10 AM Flavio Junqueira  wrote:

> In the scenario that WQ > AQ, a client acknowledges the add of an entry e
> to the application once it receives AQ bookie acks. Say now that the client
> is not able to write a copy of e to at least one bookie b, it could be
> because:
>
> 1- The client crashed before it is able to do it
> 2- Bookie b crashed
> 3- The client gave up trying
>
> In case (1), the client crashed and the ledger will be recovered by some
> reader. For all entries that have been acknowledged, including e, I'd
> expect them to be readable from the closed ledger. Each one of these
> entries that haven't been written to bookie b should be written there as
> part of the recovery process.
>
> In case (2), the client is not able to write entry e to the crashed bookie
> b, so it will replace the bookie and write e to the new bookie. I see in
> this discussion that there is an option to disable bookie replacement, I'm
> ignoring that for this discussion.
>
> In case (3), the client say discards the entry after adding successfully
> to AQ bookies, and gives up at some point because it can't reach the
> bookie. The client maybe replaces bookie b or bookie b eventually comes
> back and the client proceeds with the adds. In either case, there is a hole
> that can only be fixed by inspecting the ledger.
>
> One concern for me in this thread is case (3). I'd expect a client that
> doesn't crash to not give up, and eventually replace the bookie if it is
> unresponsive. But, that certainly leads to the memory pressure problem that
> was also mentioned in the thread, for which one potential direction also
> mentioned is to apply back pressure.
>
> Thanks,
> -Flavio
>
> > On 18 Jan 2021, at 12:20, Jack Vanlightly 
> wrote:
> >
> >> Did you guys see any issues with the ledger auditor?
> >
> >> The active writer can't guarantee it writing entries to WQ because it
> can
> >> crash during retrying adding entries to (WQ - AQ) bookies.
> >
> > The need to repair AQ replicated entries is clear and the auditor is one
> > such strategy. Ivan has also worked on a self-healing bookie strategy
> where
> > each bookie itself is able to detect these holes and is able to obtain
> the
> > missing entries itself. The detection of these holes using this strategy
> is
> > more efficient as it only requires network calls for the ledger metadata
> > scanning (to zk) and the missing entry reads (to other bookies). The
> > auditor as I understand it, reads all entries of all ledgers from all
> > bookies (of an entries ensemble) meaning these entries cross the network.
> > Using the auditor approach is likely to be run less frequently due to the
> > network cost.
> >
> > I do also wonder if the writer, on performing an ensemble change, should
> > replay "AQ but not WQ" entries, this would just leave writer failures
> > causing these AQ replicated entries.
> >
> >> Regarding recovery reads, recovery read doesn't need to be
> deterministic.
> >> For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
> >> either including it or excluding it in the sealed ledger is correct
> >> behavior. The bookkeeper client guarantees that once a ledger is sealed,
> >> the entries in the sealed ledger can always be read and can be read
> >> consistently.
> >
> >> I am not sure it is a problem unless I misunderstand it.
> >
> > It is true that it doesn't violate any safety property, but it is

Re: Unbounded memory usage for WQ > AQ ?

[Sijie Guo]

Jack,

Thank you for your replies! That's good as there are not violations of
bookkeeper protocol.

Comments inline.

On Mon, Jan 18, 2021 at 3:20 AM Jack Vanlightly
 wrote:

> > Did you guys see any issues with the ledger auditor?
>
> > The active writer can't guarantee it writing entries to WQ because it can
> > crash during retrying adding entries to (WQ - AQ) bookies.
>
> The need to repair AQ replicated entries is clear and the auditor is one
> such strategy. Ivan has also worked on a self-healing bookie strategy where
> each bookie itself is able to detect these holes and is able to obtain the
> missing entries itself. The detection of these holes using this strategy is
> more efficient as it only requires network calls for the ledger metadata
> scanning (to zk) and the missing entry reads (to other bookies). The
> auditor as I understand it, reads all entries of all ledgers from all
> bookies (of an entries ensemble) meaning these entries cross the network.
> Using the auditor approach is likely to be run less frequently due to the
> network cost.
>

Agreed on the efficiency part. I think the Salesforce team introduced the
Disk Scrubber to solve that problem already unless I confused something
there.

+JV Jujjuri  can chime in on this part.


>
> I do also wonder if the writer, on performing an ensemble change, should
> replay "AQ but not WQ" entries, this would just leave writer failures
> causing these AQ replicated entries.
>

The writer can do that. But there are no guarantees there. You still need a
mechanism to repair the under-replicated entries.
It will also make the writer become much complicated to maintain.


>
> > Regarding recovery reads, recovery read doesn't need to be deterministic.
> > For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
> > either including it or excluding it in the sealed ledger is correct
> > behavior. The bookkeeper client guarantees that once a ledger is sealed,
> > the entries in the sealed ledger can always be read and can be read
> > consistently.
>
> > I am not sure it is a problem unless I misunderstand it.
>
> It is true that it doesn't violate any safety property, but it is a strange
> check to me. It looks like an implementation artefact rather than an
> explicit protocol design choice. But not a huge deal.
>

It was discussed in the earlier days as a design choice for this protocol.

If we want to make it deterministic, we might need to consider what is the
performance penalty.


>
> Jack
>
>
> On Mon, Jan 18, 2021 at 7:07 AM Sijie Guo  wrote:
>
> > [ External sender. Exercise caution. ]
> >
> > Sorry for being late in this thread.
> >
> > If I understand this correctly, the main topic is about the "hole" when
> WQ
> > > AQ.
> >
> > > This leaves a "hole" as the entry is now replicated only to 2 bookies,
> >
> > We do have one hole when ensemble change is enabled and WQ > AQ. That
> was a
> > known behavior. But the hole will be repaired by the ledger auditor as JV
> > said. Did you guys see any issues with the ledger auditor?
> >
> > > I'd think that we guarantee that an entry that is acknowledged is
> > eventually written WQ ways and that it is observable by readers when the
> > ledger is closed.
> >
> > To Flavio's question, we don't guarantee (and can't guarantee) that the
> > active writer will eventually write the entries to WQ. For the active
> > writers, we only guarantee entries are written to AQ. The ledger auditor
> is
> > to ensure all the entries are written to WQ.
> >
> > The active writer can't guarantee it writing entries to WQ because it can
> > crash during retrying adding entries to (WQ - AQ) bookies.
> >
> > >  A single successful read is enough. However
> > there is an odd behavior in that as soon as (WQ-AQ)+1 reads fail with
> > explicit NoSuchEntry/Ledger, the read is considered failed and the ledger
> > recovery process ends there. This means that given the responses
> > b1->success, b2->NoSuchLedger, b3->NoSuchLedger, whether the read is
> > considered successful is non-deterministic.
> >
> > Regarding recovery reads, recovery read doesn't need to be deterministic.
> > For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
> > either including it or excluding it in the sealed ledger is correct
> > behavior. The bookkeeper client guarantees that once a ledger is sealed,
> > the entries in the sealed ledger can always be read and can be read
> > consistently.
> >
> > I am not sure it is a problem unless I misunderstand it.
>

Re: Unbounded memory usage for WQ > AQ ?

[Sijie Guo]

Sorry for being late in this thread.

If I understand this correctly, the main topic is about the "hole" when WQ
> AQ.

> This leaves a "hole" as the entry is now replicated only to 2 bookies,

We do have one hole when ensemble change is enabled and WQ > AQ. That was a
known behavior. But the hole will be repaired by the ledger auditor as JV
said. Did you guys see any issues with the ledger auditor?

> I'd think that we guarantee that an entry that is acknowledged is
eventually written WQ ways and that it is observable by readers when the
ledger is closed.

To Flavio's question, we don't guarantee (and can't guarantee) that the
active writer will eventually write the entries to WQ. For the active
writers, we only guarantee entries are written to AQ. The ledger auditor is
to ensure all the entries are written to WQ.

The active writer can't guarantee it writing entries to WQ because it can
crash during retrying adding entries to (WQ - AQ) bookies.

>  A single successful read is enough. However
there is an odd behavior in that as soon as (WQ-AQ)+1 reads fail with
explicit NoSuchEntry/Ledger, the read is considered failed and the ledger
recovery process ends there. This means that given the responses
b1->success, b2->NoSuchLedger, b3->NoSuchLedger, whether the read is
considered successful is non-deterministic.

Regarding recovery reads, recovery read doesn't need to be deterministic.
For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
either including it or excluding it in the sealed ledger is correct
behavior. The bookkeeper client guarantees that once a ledger is sealed,
the entries in the sealed ledger can always be read and can be read
consistently.

I am not sure it is a problem unless I misunderstand it.

- Sijie

On Fri, Jan 15, 2021 at 7:43 AM Jack Vanlightly
 wrote:

> Let's set up a call and create any issues from that. I have already created
> the patches in our (Splunk) fork and it might be easiest or not to wait
> until we re-sync up with the open source repo. We can include the fixes in
> the discussion.
>
> Jack
>
> On Fri, Jan 15, 2021 at 4:33 PM Flavio Junqueira  wrote:
>
> > [ External sender. Exercise caution. ]
> >
> > Hi Jack,
> >
> > Thanks for getting back.
> >
> > > What's the best way to share the TLA+ findings?
> >
> > Would you be able to share the spec? I'm ok with reading TLA+.
> >
> > As for sharing your specific findings, I'd suggest one of the following:
> >
> > 1- Create an email thread describing the scenarios that trigger a bug.
> > 2- Create issues, one for each problem you found.
> > 3- Create a discussion on the project Slack, perhaps a channel specific
> > for it.
> > 4- Set up a zoom call to present and discuss with the community.
> >
> > Option 2 is ideal from a community perspective, but we can also set up a
> > call inviting everyone and create issues out of that discussion. We can
> in
> > fact set up a call even if we create the issues ahead of time.
> >
> > Does it make sense?
> >
> > -Flavio
> >
> > > On 15 Jan 2021, at 16:14, Jack Vanlightly  .INVALID>
> > wrote:
> > >
> > > Hi Flavio,
> > >
> > >>> This is an example of a scenario corresponding to what we suspect is
> a
> > > bug introduced earlier, but Enrico is arguing that this is not the
> > intended
> > > behavior, and at this point, I agree.
> > >
> > >>> By the time a successful callback is received, the client might only
> > > have replicated AQ ways, so the guarantee can only be at that point of
> > > being able to tolerate AQ - 1 crashes. The ledger configuration states
> > that
> > > the application wants to have WQ copies >> of each entry, though. I'd
> > > expect a ledger to have WQ copies of each entry up to the final entry
> > > number when it is closed. Do you see it differently?
> > >
> > > I also agree and was pretty surprised when I discovered the behaviour.
> It
> > > is not something that users expect and I think we need to correct it.
> So
> > > I'm with you.
> > >
> > > What's the best way to share the TLA+ findings?
> > >
> > > Jack
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Jan 15, 2021 at 3:59 PM Flavio Junqueira 
> wrote:
> > >
> > >> [ External sender. Exercise caution. ]
> > >>
> > >>> Let's say we have WQ 3 and AQ 2. An add (e100) has reached AQ and the
> > >>> confirm callback to the client is called and the LAC is set to
> 100.Now
> > >> the
> > >>> 3rd bookie times out. Ensemble change is executed and all pending
> adds
> > >> that
> > >>> are above the LAC of 100 are replayed to another bookie, meaning that
> > the
> > >>> entry e100 is not replayed to another bookie, causing this entry to
> > meet
> > >>> the rep factor of only AQ.
> > >>
> > >> This is an example of a scenario corresponding to what we suspect is a
> > bug
> > >> introduced earlier, but Enrico is arguing that this is not the
> intended
> > >> behavior, and at this point, I agree.
> > >>
> > >>> This is alluded to in the docs as they state
> > >>> that AQ is also the minimum guaranteed 

Re: [VOTE] Apache BookKeeper Release 4.12.1, release candidate #0

[Sijie Guo]

+1 (binding)

- Source package looks good
- Binary package looks good
- The release tag is good

- Sijie

On Thu, Jan 7, 2021 at 11:43 PM Enrico Olivelli  wrote:

> Thank you Amit, Matteo and Henry,
> we need at least one more +1 from a PMC in order to validate the release
> (assuming my own +1)
>
> Enrico
>
> Il giorno ven 8 gen 2021 alle ore 01:47 Henry Saputra <
> henry.sapu...@gmail.com> ha scritto:
>
> > +1 (binding)
> > LICENSE file looks good
> > NOTICE file looks good - need to be updated to 2021 but should not be
> > blocker
> > Signature file looks good
> > Checksum file matched
> > No binary in the source artifact distribution
> > Source compiled and tests passed (OSX, Java 1.8)
> >
> >
> > - Henry
> >
> > On Mon, Dec 28, 2020 at 5:34 AM Enrico Olivelli 
> > wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #0 for the version
> > 4.12.1,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > The complete staging area is available for your review, which includes:
> > > * Release notes [1]
> > > * The official Apache source and binary distributions to be deployed to
> > > dist.apache.org [2]
> > > * All artifacts to be deployed to the Maven Central Repository [3]
> > > * Source code tag "v4.12.1-rc0" [4] with git sha
> > > 8756a7e36b8d1ba014b8117b4661cddb41328425
> > >
> > > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > >
> > > Please download these packages and review this release candidate:
> > >
> > > - Review release notes
> > > - Download the source package (verify shasum, and asc) and follow the
> > > instructions to build and run the bookkeeper service.
> > > - Download the binary package (verify shasum, and asc) and follow the
> > > instructions to run the bookkeeper service.
> > > - Review maven repo, release tag, licenses, and any other things you
> > think
> > > it is important to a release.
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Enrico Olivelli
> > >
> > > [1]
> > >
> > >
> >
> https://raw.githubusercontent.com/apache/bookkeeper/f0a1fc99a034725690d872a41abe6306c70c890f/site/docs/4.12.1/overview/releaseNotes.md
> > > [2]
> > >
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.12.1-rc0/
> > > [3]
> > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1050/
> > > [4] https://github.com/apache/bookkeeper/tree/v4.12.1-rc0
> > >
> >
>

Re: [VOTE] Apache BookKeeper 4.12.0 release candidate 0

[Sijie Guo]

+1 (binding)

- Both source package and binaries are good.
- Test compiling the source package.
- Tags and license are good

- Sijie

On Thu, Nov 12, 2020 at 2:17 AM Enrico Olivelli  wrote:

> +1 (binding)
> - Build sources on Mac, with JDK14
> - Tested the built sources, run a few smoke tests with JDK14
> - tested bookieId on a small cluster
>
> I noticed a small issue on "bookkeper standalone" mode, that is not
> implementing BP-38 correctly and thus you cannot use BookieId, but
> "bookkeeper standalone" is mostly a dev mode utility, so not a big deal,
> I have already raised a PR to fix it
>
> Enrico
>
> Il giorno mer 11 nov 2020 alle ore 16:47 Jia Zhai  ha
> scritto:
>
> > Hi everyone,
> > Please review and vote on release candidate #0 for version 4.12.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * Release notes [1]
> > * The official Apache source and binary distributions to be deployed to
> > dist.apache.org [2]
> > * All artifacts to be deployed to the Maven Central Repository [3]
> > * Source code tag "v4.12.0-rc0" [4] with git sha
> > dcf6bea03610fb69663cd8468a034b8a3823214e
> >
> > BookKeeper's KEYS file contains PGP keys we used to sign this release:
> > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> >
> > Please download these packages and review this release candidate:
> >
> > - Review release notes
> > - Download the source package (verify shasum, and asc) and follow the
> > instructions to build and run the bookkeeper service.
> > - Download the binary package (verify shasum, and asc) and follow the
> > instructions to run the bookkeeper service.
> > - Review maven repo, release tag, licenses, and any other things you
> think
> > it is important to a release.
> >
> > The vote will be open for at least 72 hours. It is adopted by the
> majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Jia
> >
> > [1] https://github.com/apache/bookkeeper/pull/2478
> > [2]
> > https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.12.0-rc0/
> > [3]
> >
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1049
> > [4] https://github.com/apache/bookkeeper/tree/v4.12.0-rc0
> >
>

Re: Contributing Splunk changes back to OSS

[Sijie Guo]

On Tue, Nov 10, 2020 at 3:09 AM Ivan Kelly  wrote:

> Hi folks,
>
> It's been about a year since Streamlio joined Splunk and since then
> we've had a bit of forking with our BK branch.
> It has gotten to a stage where it's starting to be a problem for us,
> so we'd like to start to get things back in sync.
>
> There are a couple of big chunks of work to come back.
> We've added a data integrity checker that replaces a lot of the
> functionality of autorecovery and allows us to run without a journal.
> We refactored the bookie to allow dependency injection.
> We've rewritten the entry logger to use direct I/O (allowing 2GBps
> writes per bookie).
>

+1 Looking forward to the changes.


>
> One other thing we've done is to change the build system to use gradle.
> The major driver for this was that maven is just slow, even before you
> start running tests.
> "mvn clean package -DskipTests" takes 4m30 on my laptop. "./gradlew
> clean jar" takes 40s.
> Subsequent builds on gradle are much much faster, as it does
> incremental building.
> Incremental building exists in maven, but it doesn't work.
> Gradle also handle multimodule projects better. If I make a change in
> bookkeeper-common,
> "./gradlew :bookkeeper-server:test" will pick up the change without
> having to explicitly
> "mvn install" the bookkeeper-common. In my opinion it's just a much
> nicer build system
> to work with. Even the poms it generates are better as they avoid
> dependency pollution.
>
> What are peoples opinions on moving BookKeeper to gradle (assuming
> I/splunk do the legwork)?
> If people are open to it, I'll submit a BP.
>

+1. My only question is how do you do an Apache release. I'd like to see BP
covering that question.


>
> Another thing that BK (and the whole ecosystem) is missing is
> structured logging.
> We also plan to add structured logging to BK in soon. This is a major
> motivator for converging the branches,
> as it touches a lot of places.
>
> Anyhow, any feedback appreciated.
>
> -Ivan
>

Re: Cutting Apache BookKeeper 4.12.0

[Sijie Guo]

Jia - Thank you for taking care of this!

- Sijie

On Tue, Nov 10, 2020 at 1:05 AM Jia Zhai  wrote:

> We will start the 4.12 release today.
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Thu, Nov 5, 2020 at 5:18 AM Enrico Olivelli 
> wrote:
>
> > Il giorno mer 4 nov 2020 alle ore 22:17 Sijie Guo 
> ha
> > scritto:
> >
> > > Okay. I will do the release.
> > >
> >
> >
> > Thanks.
> > I will report any showstopper as soon as possible. btw at the moment I
> > don't see any problem with a release
> >
> > please also check the list of pending PRs if there is something appealing
> > for the release
> >
> > Enrico
> >
> > >
> > > - Sijie
> > >
> > > On Wed, Nov 4, 2020 at 11:06 AM Enrico Olivelli 
> > > wrote:
> > >
> > > > Sijie
> > > > Thanks for your proposal.
> > > > I just started to prepare the website and the release notes.
> > > >
> > > > I am testing locally current master and also I am trying to test it
> on
> > > > current Pulsar master.
> > > >
> > > > I am happy that you do the release
> > > >
> > > > Please confirm that you are going to do it
> > > >
> > > > Enrico
> > > >
> > > > Il Mer 4 Nov 2020, 17:37 Sijie Guo  ha scritto:
> > > >
> > > > > +1
> > > > >
> > > > > If you need any help with the release, I can do the release this
> > time.
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Tue, Nov 3, 2020 at 3:05 AM Enrico Olivelli <
> eolive...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello BookKeepers,
> > > > > > I would like to cut 4.12.0
> > > > > >
> > > > > > There are still a few interesting pull requests pending review
> > > > > > https://github.com/apache/bookkeeper/pulls
> > > > > >
> > > > > > Please note that we don't have a working public CI environment,
> due
> > > to
> > > > > the
> > > > > > migration of ASF Jenkins, but every pull request is validated
> with
> > > > GitHub
> > > > > > Actions so I am pretty confident that the codebase is in great
> > shape.
> > > > > >
> > > > > > I am also going to talk about BP-41 at PulsarSummit Asia 2020, it
> > > would
> > > > > be
> > > > > > good to have the release done within the middle of November.
> > > > > > Probably they will also release Apache Pulsar 2.7.0 with BK
> 4.12.0,
> > > > > before
> > > > > > PulsarSummit
> > > > > >
> > > > > > If you are aware of showstoppers or about issues that we should
> > > > consider
> > > > > > before cutting the release please chime in or ping me.
> > > > > >
> > > > > > I will start the release procedure within the end of this week,
> > > > hopefully
> > > > > > tomorrow or the day after
> > > > > >
> > > > > > Best regards
> > > > > > Enrico
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Cutting Apache BookKeeper 4.12.0

[Sijie Guo]

Okay. I will do the release.

- Sijie

On Wed, Nov 4, 2020 at 11:06 AM Enrico Olivelli  wrote:

> Sijie
> Thanks for your proposal.
> I just started to prepare the website and the release notes.
>
> I am testing locally current master and also I am trying to test it on
> current Pulsar master.
>
> I am happy that you do the release
>
> Please confirm that you are going to do it
>
> Enrico
>
> Il Mer 4 Nov 2020, 17:37 Sijie Guo  ha scritto:
>
> > +1
> >
> > If you need any help with the release, I can do the release this time.
> >
> > - Sijie
> >
> > On Tue, Nov 3, 2020 at 3:05 AM Enrico Olivelli 
> > wrote:
> >
> > > Hello BookKeepers,
> > > I would like to cut 4.12.0
> > >
> > > There are still a few interesting pull requests pending review
> > > https://github.com/apache/bookkeeper/pulls
> > >
> > > Please note that we don't have a working public CI environment, due to
> > the
> > > migration of ASF Jenkins, but every pull request is validated with
> GitHub
> > > Actions so I am pretty confident that the codebase is in great shape.
> > >
> > > I am also going to talk about BP-41 at PulsarSummit Asia 2020, it would
> > be
> > > good to have the release done within the middle of November.
> > > Probably they will also release Apache Pulsar 2.7.0 with BK 4.12.0,
> > before
> > > PulsarSummit
> > >
> > > If you are aware of showstoppers or about issues that we should
> consider
> > > before cutting the release please chime in or ping me.
> > >
> > > I will start the release procedure within the end of this week,
> hopefully
> > > tomorrow or the day after
> > >
> > > Best regards
> > > Enrico
> > >
> >
>

Re: Cutting Apache BookKeeper 4.12.0

[Sijie Guo]

+1

If you need any help with the release, I can do the release this time.

- Sijie

On Tue, Nov 3, 2020 at 3:05 AM Enrico Olivelli  wrote:

> Hello BookKeepers,
> I would like to cut 4.12.0
>
> There are still a few interesting pull requests pending review
> https://github.com/apache/bookkeeper/pulls
>
> Please note that we don't have a working public CI environment, due to the
> migration of ASF Jenkins, but every pull request is validated with GitHub
> Actions so I am pretty confident that the codebase is in great shape.
>
> I am also going to talk about BP-41 at PulsarSummit Asia 2020, it would be
> good to have the release done within the middle of November.
> Probably they will also release Apache Pulsar 2.7.0 with BK 4.12.0, before
> PulsarSummit
>
> If you are aware of showstoppers or about issues that we should consider
> before cutting the release please chime in or ping me.
>
> I will start the release procedure within the end of this week, hopefully
> tomorrow or the day after
>
> Best regards
> Enrico
>

Re: BookKeeper Board Report due by Wed Aug 12

[Sijie Guo]

Thank you, Enrico!

I have just submitted the report.

- Sijie

On Tue, Aug 11, 2020 at 1:44 PM Enrico Olivelli  wrote:

> Sijie,
> this is my proposal of report to the board (sharing it on dev@ this way
> everyone can know about this important activity on the lifecycle of the
> project)
>
> ## Description:
> The mission of BookKeeper is the creation and maintenance of software
> related
> to Replicated Log Services which can be used to build replicated state
> machines. During the past years it became a more general basic building
> block
> for Distributed Storage Systems, as Apache Pulsar and other OpenSource
> projects like Pravega.io and HerdDB.org.
>
> ## Issues:
> No relevant issues for the board at the moment.
>
> ## Membership Data:
> Apache BookKeeper was founded 2014-11-19 (6 years ago) There are currently
> 22
> committers and 16 PMC members in this project. The Committer-to-PMC ratio
> is
> roughly 3:2.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Charan Reddy G on 2019-07-24.
> - No new committers. Last addition was Rajan Dhabalia on 2020-03-23.
>
> ## Project Activity:
> The project is quite active even if we are not cutting releases very often.
> 4.11.0 was release on 2020-07-09
> 4.10.0 was released on 2019-11-06.
>
> The project is moving forward with new features and bugfixes.
>
>
> ## Community Health:
> New users are often using the Slack channel instead of the ML in order to
> get
> in touch with the project, this has the effect to a reduced activity on the
> user@ mailing list.
> The activity on the ML is stable on dev@, and we are also following a
> formal process to get major changes in (BookKeeper Proposals design
> documents).
>
> We had a BookKeeper dedicated track at PulsarSummit.com, a conference
> devoted to Apache Pulsar.
> We also have a few talks about BookKeeper at ApacheCon@Home.
>
> We recently moved to GitHub Actions the CI jobs relevant to testing Pull
> Requests.
>
>
>
>
>
>
>
>
> Il giorno mar 11 ago 2020 alle ore 00:05 Roy T. Fielding <
> field...@apache.org> ha scritto:
>
>> Hello again,
>>
>> According to our records, you are listed as the chair of BookKeeper,
>> a committee that is due to submit a report by Wed Aug 12
>> for the next ASF board meeting. Please submit your report as
>> described below.
>>
>> Thank you for being a responsible project chair and helping us maintain
>> oversight over the Apache Software Foundation. If, for whatever reason,
>> a full report is not possible by the deadline, please report just that.
>> It's okay to postpone a report by a month.
>>
>> Reports received after Wed Aug 12 will be postponed to the
>> next regular meeting.
>>
>>
>> Submitting your report
>> --
>>
>> Full details about reporting to the board are at
>>
>>   https://www.apache.org/foundation/board/reporting
>>
>> Please be aware that the board is looking for your personal observations,
>> assessment, and ideas, not just raw statistics.
>>
>> Chairs may use one of several mechanisms to submit or edit their report:
>>
>>  a) the Apache Reporter Service
>> https://reporter.apache.org/
>>
>>  b) the Whimsy online agenda tool
>> https://whimsy.apache.org/board/agenda/2020-08-19/BookKeeper
>>
>>  c) carefully editing and committing changes to the dated agenda in
>> https://svn.apache.org/repos/private/foundation/board
>>
>>  d) or, if none of the above work, send an email to bo...@apache.org with
>> Subject: [REPORT] BookKeeper
>>
>> If you believe it won't be possible to prepare a report before the
>> deadline,
>> or if the PMC is aware that the Chair is unavailable, please report that
>> and we can reschedule or have someone else report on your behalf.
>>
>>
>> Attending the Board Meeting
>> ---
>>
>> The formal board meeting (usually an online videoconference) will be held
>> at
>>
>>   Wed, 19 Aug 2020 at 12:30 UTC
>>
>> which in other time zones is
>>
>>   https://timeanddate.com/s/429w
>>
>> As always, chairs and ASF members are welcome to attend the board meeting.
>> However, in most cases, we will not be using meeting time to discuss
>> reports
>> unless you specifically request time to speak in person.
>>
>> During the week prior to the meeting, the directors will read the received
>> reports, make comments (if any) within the agenda tool, discuss those
>> comments on the board and/or private committee lists, and vote to approve.
>>
>> If we have comments on a report, we will forward them during the
>> review and attempt to complete any associated action items as well.
>> This will allow us to be more responsive to project needs and give you
>> an opportunity to expand on your report if additional details are
>> requested prior to the meeting.
>>
>> Regular board meetings are held monthly, as scheduled at
>>
>>   https://svn.apache.org/repos/private/committers/board/calendar.txt
>>
>>
>> Requesting a Board Action
>> -
>>
>> If you want the board 

Re: Slack channel for HerdDB hosted on BK slack

[Sijie Guo]

+1

On Wed, Jul 8, 2020 at 1:34 AM Enrico Olivelli  wrote:

> Hi,
> Can I create an #herddb channel on our BookKeeper slack space ?
>
> After my talk at PulsarSummit we started having a few new contacts on
> herddb-dev mailing list, but sometimes slack is more efficient.
>
> HerdDB community is still too little to have a dedicated space, it would be
> awkward for users to subscribe to another workspace (I don't know your
> experience but I am subscribed to lots of Slack workspaces and I don't like
> it very much).
>
> There won't be much traffic.
>
> Regards
> Enrico
>

Re: Announcing ApacheCon @Home 2020

[Sijie Guo]

Hi Enrico,

I have reached out to Rich to create a track for pulsar/bookkeeper. If you
have any talk ideas, please submit the talks and let me know your
submissions. I will include that on the track.

- Sijie

On Mon, Jun 29, 2020 at 11:04 AM Enrico Olivelli 
wrote:

> Hey Bookkeepers
> This is a great chance to add visibility for our project
>
> If you have any kind of project or any cool thing that can be done with BK
> please try to share it.
>
> Unfortunately BK is a very low level building block. It is not easy to see
> its value while looking at it standalone.
>
> Best regards
> Enrico
>
> Il Lun 29 Giu 2020, 14:54 Rich Bowen  ha scritto:
>
> > Hi, Apache enthusiast!
> >
> > (You’re receiving this because you’re subscribed to one or more dev or
> > user mailing lists for an Apache Software Foundation project.)
> >
> > The ApacheCon Planners and the Apache Software Foundation are pleased to
> > announce that ApacheCon @Home will be held online, September 29th
> > through October 1st, 2020. We’ll be featuring content from dozens of our
> > projects, as well as content about community, how Apache works, business
> > models around Apache software, the legal aspects of open source, and
> > many other topics.
> >
> > Full details about the event, and registration, is available at
> > https://apachecon.com/acah2020
> >
> > Due to the confusion around how and where this event was going to be
> > held, and in order to open up to presenters from around the world who
> > may previously have been unable or unwilling to travel, we’ve reopened
> > the Call For Presentations until July 13th. Submit your talks today at
> > https://acna2020.jamhosted.net/
> >
> > We hope to see you at the event!
> > Rich Bowen, VP Conferences, The Apache Software Foundation
> >
>

Re: [VOTE] Release 4.11.0, release candidate #0

[Sijie Guo]

+1 (binding)

On Tue, Jun 30, 2020 at 9:18 PM Jia Zhai  wrote:

> +1 (binding)
>
> Environment: macOS 10.15.5
>
> - verified packages checksum ( asc and sha good)
>
> - the source package build and test all run successfully.
>
> - in both binary package(server & all), 'bin/bookkeeper standalone' and
> 'bin/bookkeeper shell bookiesanity' runs well.
>
>
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Tue, Jun 30, 2020 at 11:14 PM Matteo Minardi - Diennea <
> minardi.mat...@hotmail.it> wrote:
>
> > +1 (binding)
> > - Built source package with JDK13 on Mac, tests ok
> > - Ran base bk-shell tests with 4.11 built binaries
> > - Built Bookkeeper Visual Manager with the new version.
> > - Connected both to 4.10 and 4.11 servers with BKVM and showing bookies
> > data fine!
> >
> > Matteo.
> >
> > Il giorno 30/06/20, 14:26 "Enrico Olivelli"  ha
> > scritto:
> >
> > +1 (binding)
> > - verified checksums and signatures
> > - built source package with JDK8 on Fedora, all tests passed
> > - performed a few smoke tests with built binaries, on JDK8, only one
> > comment (see below)
> > - built and run tests of HerdDB, all passed, only one comment (see
> > below)
> > - build and run Java tests of https://pravega.io all tests passed
> >
> > Thank you very much Rajan for putting this all together !
> >
> > Best regards
> > Enrico
> >
> > Problem 1, bkctl, non blocker:
> > "bkctl" does not work on JDK14, this is due to deprecated and removed
> > command line options. The fix is trivial, just fix the bash scripts
> >
> > Problem 2, HerdDB, non blocker
> > HerdDB is able to boot a Bookie, using a Java unofficial API
> > (BookieServer).
> > The signature of the constructor changed due to BP-38, the fix on
> > HerdDB
> > EmbeddedBookie is straightforward, just add BookieServiceInfo.NO_INFO
> > as
> > third parameter.
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> > (default-compile) on project herddb-core: Compilation failure
> > [ERROR]
> >
> >
> /data/dev/herddb/herddb-core/src/main/java/herddb/cluster/EmbeddedBookie.java:[146,24]
> > no suitable constructor found for
> >
> >
> BookieServer(org.apache.bookkeeper.conf.ServerConfiguration,org.apache.bookkeeper.stats.StatsLogger)
> > [ERROR] constructor
> >
> >
> org.apache.bookkeeper.proto.BookieServer.BookieServer(org.apache.bookkeeper.conf.ServerConfiguration)
> > is not applicable
> > [ERROR]   (actual and formal argument lists differ in length)
> > [ERROR] constructor
> >
> >
> org.apache.bookkeeper.proto.BookieServer.BookieServer(org.apache.bookkeeper.conf.ServerConfiguration,org.apache.bookkeeper.stats.StatsLogger,java.util.function.Supplier)
> > is not applicable
> > [ERROR]   (actual and formal argument lists differ in length)
> >
> >
> >
> >
> >
> >
> > Il giorno mar 30 giu 2020 alle ore 06:17 Anup Ghatage <
> > ghat...@gmail.com>
> > ha scritto:
> >
> > > LGTM!
> > >
> > > non-binding +1
> > >
> > > On Mon, Jun 29, 2020 at 8:56 PM Rajan Dhabalia <
> rdhaba...@apache.org
> > >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #0 for the
> version
> > > 4.11.0,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > The complete staging area is available for your review, which
> > includes:
> > > > * Release notes [1]
> > > > * The official Apache source and binary distributions to be
> > deployed to
> > > > dist.apache.org [2]
> > > > * All artifacts to be deployed to the Maven Central Repository
> [3]
> > > > * Source code tag "release-4.11.0" [4] with git sha
> > > > ceb140ba0bcab72ac3d22e7ace7712cea30bdfe9
> > > > BookKeeper's KEYS file contains PGP keys we used to sign this
> > release:
> > > > https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> > > >
> > > > Please download these packages and review this release candidate:
> > > > - Review release notes
> > > > - Download the source package (verify shasum, and asc) and follow
> > the
> > > > instructions to build and run the bookkeeper service.
> > > > - Download the binary package (verify shasum, and asc) and follow
> > the
> > > > instructions to run the bookkeeper service.
> > > > - Review maven repo, release tag, licenses, and any other things
> > you
> > > think
> > > > it is important to a release.
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > Rajan
> > > >
> > > > [1] Release note:
> > > >
> > > >
> > >
> >
> 

Re: Re-working the BookKeeper site

[Sijie Guo]

+1

@Anup Ghatage  thank you for driving this!

- Sijie

On Mon, Jun 15, 2020 at 9:07 AM Anup Ghatage  wrote:

> Thanks for your comments Enrico.
>
> This is just the prototype, we shall ask for consent from all the
> companies before using their logos.
> I checked and the website is not bad on Mobile.
> I see the release procedure being similar to that of the pulsar website.
> From what I see, all we have to do is create markdown files and push them
> to a specific folder and then build, similar to now.
> They have a script which does the 'build' and deploy. I'll reach out to
> the folks involved to see if there is more to it.
>
> I also reached out to Matteo Minardi (In slack) and Lamberken (on github)
> and they're both enthusiastic about getting this rolling.
> We'll come up with a plan and reply on this thread when it's ready.
>
> Regards,
> Anup
>
>
> ..
>
> On Sun, Jun 14, 2020 at 12:00 AM Enrico Olivelli 
> wrote:
>
>> Anup
>> Great idea.
>> I am totally +1 to your idea
>>
>> Just a couple of items:
>> - we must ask for consent before using logos
>> - we should make the website look well on mobile
>> - think about the release procedure and how we will have to update the
>> website after the release (now we have a script but it is awkward)
>>
>> Thank you very much
>> Enrico
>>
>> Il Dom 14 Giu 2020, 02:26 Anup Ghatage  ha scritto:
>>
>>> Hi Bookies,
>>>
>>> We've had quite a few issues reported recently about the site.
>>> Some are about the fact that the site has dead links and some are about
>>> the content.
>>> I tried to build the site recently and it didn't even build.
>>>
>>> @Sijie Guo  suggested here
>>> <https://github.com/apache/bookkeeper/pull/2344#issuecomment-636364183>that
>>> we should try and use docusaurus for the BookKeeper website as it's much
>>> easier to maintain. And I must agree! I played around with it a little and
>>> after using the Pulsar website as a template, I think this is fantastic.
>>>
>>> So I propose, we go ahead and start development of the new website under
>>> 'site2' in the root directory. Once the development is complete, we can
>>> decommission the old website and have the domain point to our new website.
>>>
>>> I spent most of today trying to re-imagine the website and have charted
>>> out the tasks and workflow, so if you're interested, please reply to this
>>> email and we can get started!
>>>
>>> I have put together a test website at:
>>> http://ghatage.com/bookkeeper-site/
>>> It's inspired by the Pulsar website and has A LOT of dead links and is
>>> yet to be populated.
>>> However, generation and maintenance of the website has been super easy.
>>> Please check it out and let me know what you guys think!
>>>
>>> Regards,
>>> Anup
>>>
>>
>
> --
> Anup Ghatage
> www.ghatage.com
>

Re: Question on disableEnsembleChange

[Sijie Guo]

Hi JV,

It unset the failed bookies and resend the write requests, no?

- Sijie

On Sat, May 30, 2020 at 11:05 PM Venkateswara Rao Jujjuri 
wrote:

> I am looking at a feature to disableEnsembleChanges
> If the disableEnsembleChangeFeature
> 
> is
> available  it is calling unsetSuccessAndSendWriteRequest
>
> 
> without any ensemble changes, but the why are we calling
> unsetSuccessAndSendWriteRequest, which is actually removing
> 
> the Ack
> and resending the write.
> If there is an ensemble changes this makes sense but in this case ensemble
> hasn't changed. Wondering if this is a bug/oversight.
>
>
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
>
>

Announcing Pulsar Summit Virtual Conference 2020 | June 17-18

[Sijie Guo]

Hi all,

Over the past few weeks, after having to reschedule Pulsar Summit San
Francisco 2020, we have been working on delivering the event in an online
format instead. Today, we are excited to announce the Pulsar Summit Virtual
Conference, happening on June 17-18!

Join the speakers from Splunk, Verizon Media, Iterable, Yahoo! JAPAN,
TIBCO, OVHcloud, Clever Cloud, and many more live with real-time Q
sessions. The conference will be running for 2 days across several time
zones, to ensure that the global Pulsar community can participate.

Register today to save your spot! https://pulsar-summit.org/registration

Follow @PulsarSummit for more updates!

See you online!

Best,
Your Pulsar Summit Team

Re: Board report draft for May 2020

[Sijie Guo]

Anup, thank you!

I am going to send the report.

- Sijie

On Tue, May 12, 2020 at 10:27 PM Enrico Olivelli 
wrote:

> Good
>
> Sijie have you sent it?
>
> Enrico
>
> Il Mer 13 Mag 2020, 00:34 Anup Ghatage  ha scritto:
>
>> Thank you for your comments Enrico, I've made the changes as you
>> suggested.
>> I have tried to tell the story behind the metrics, tried to include the
>> work being done and the work that was done in the past quarter.
>> Please find below the updated report, let me know what you think.
>>
>> -
>>
>> ## Description:
>> BookKeeper is a scalable, fault-tolerant, and low-latency storage service
>> optimized for append-only workloads. It has been used as
>> a fundamental service to build high available and replicated services
>> in companies like Twitter, Yahoo and Salesforce. It is also the log
>> segment store for Apache DistributedLog and message store for Apache
>> Pulsar.
>>
>> Apache DistributedLog is a high-level API and service layer for
>> Apache BookKeeper, providing easier access to the BookKeeper
>> primitives. It is a subproject of Apache BookKeeper.
>>
>> ## Issues:
>> There are no issues requiring board attention at this time.
>>
>> ## Membership Data:
>> Apache BookKeeper was founded 2014-11-19 (5 years ago)
>> There are currently 23 committers and 16 PMC members in this project.
>> The Committer-to-PMC ratio is roughly 3:2.
>>
>> Community changes, past quarter:
>> - No new PMC members.
>> - Rajan Dhabalia was added as committer on 2020-03-23
>>
>> ## Project Activity:
>> Recently, we've seen lots of activity from upstream consumers such as
>> Apache Pulsar.
>> Companies such as Salesforce and Diennea contributing back to the
>> community.
>> We are also seeing a new interest from users at Dell EMC contributing back
>> and starting conversations.
>> Several bugs (issues) were opened and fixed from these upstream consumers
>> which will be going into our next release.
>> Some of the important features being worked on/recently merged are:
>> - Bookkeeper Proposal - 38: Publish Bookie Service Info on Metadata
>> Service
>> - Bookkeeper Proposal - 40: Audit Logging for Apache Bookkeeper
>> - Migrated Project CI from Jenkins to Github Actions on 2020-01-22.
>>
>> Releases
>> - 4.10.0 was released on 2019-11-06.
>> - 4.9.2 was released on 2019-05-16.
>> - Supporting project Bookkeeper Visual Manager released v1.0.0 on
>> 2020-04-12.
>>
>> ## Community Health:
>> Community has been active in making infrastructural changes to Apache
>> Bookkeeper this past quarter.
>> We have had initiatives which moved us to Github Actions from Jenkins and
>> also cut down the build and test time by more than 50%.
>> Other development activity has seemed to go from mailing lists to our
>> slack
>> channel (apachebookkeeper.slack.com) where we had around 38 new users of
>> Apache Bookkeeper.
>> In the past quarter, the above activity has resulted in the following
>> GitHub activity:
>> - 38 PRs opened (8% increase)
>> - 40 PRs closed (48% increase)
>> - 50 commits (51% increase)
>> - 20 code contributors (66% increase)
>>
>> Mailing List activity:
>> - 1272 emails in iss...@bookkeeper.apache.org (4% increase)
>> - 30 emails in u...@bookkeeper.apache.org
>> - 59 emails in the dev@bookkeeper.apache.org
>> - We'll be pushing to bring the discussions back to the mailing lists.
>>
>> Meetups and Conferences:
>> - Several talks are scheduled for the Bookkeeper Track in the Apache
>> Pulsar
>> Summit 2020  to be held in 17 and
>> 18th
>> June 2020.
>>
>

Re: Rollback of Apache Bookeeper

[Sijie Guo]

This is the one of the approaches that you can downgrade.

But a general approach I mentioned in my previous is doing rollback via
re-replicating the data. Basically you can start a brand new bookie server
running with the old version that is configured using old disk format. Then
you can tear down the bookie server that runs a newer version that is
configured to use a new disk format. The auto-recovery process will
re-replicate the data to the bookies run old version. This is the way you
achieve "rollback".

If you are managing a very large bookkeeper cluster, you typically
configure a rack-aware placement policy. When you want to roll out a newer
version of bookkeeper, you typically deploy the newer version of bookkeeper
to a set of bookies within one single rack. So you make sure an ensemble
has both old version and newer version. You canary the newer version for a
while to ensure this version is running well. Then you roll out the newer
version to the whole cluster. If there is any problem observed during the
canary period, you can stop the bookies that already rolled out the newer
version. The auto-recovery process will re-replicate the data to the
bookies running an old version. In this way, you "rollback" the whole
cluster to an old version. This is a generally safe approach that applies
to all versions. So you don't need to take special care for individual
versions.

- Sijie

On Mon, Apr 20, 2020 at 3:13 AM Subash K  wrote:

> Okay, now I got. If we upgrade and we are using the latest disk format
> version, before rollback we can downgrade the disk format version
> (bookkeeper conf) as per the Bookkeeper version we are rolling back to and
> once all the files are written in expected format we can rollback the
> binaries to the older version.
>
>
>
> Is there any API or tool through which we can check the disk format
> version of all data before rollback?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Monday, April 20, 2020 3:24 PM
> *To:* us...@pulsar.apache.org
> *Cc:* Bookkeeper-Dev 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
>
>
> Il giorno lun 20 apr 2020 alle ore 11:49 Subash K 
> ha scritto:
>
> Thanks Enrico and Sijie, so I see the only way to rollback is not to use
> the new data format after upgrade. And once we change the data format, then
> we lose the option of rollback.
>
>
>
> And regarding “For the second question, it depends on the changes in a
> given release. In certain releases, the disk format change supports
> rollback. In other releases, it doesn't. But you usually can still do
> rollback via re-replicating the data. So in general, you can assume you can
> do rollback..” -- I believe you are pointing out that, at times the new
> data format version is compatible with the older version. But if it is not
> compatible, then I hope we can’t re-replicate the data (to older version)
> as the entire Bookkeeper cluster data will be in new version of data
> format. I believe in that case, we might need a dedicated tool to rollback
> the data to previous version and then only we can rollback our binaries.
> Please let me know if my understanding is wrong.
>
>
>
> Also, after upgrade if we step up the data format version, does compaction
> (minor/major) of Bookkeeper re-write the existing data according to the new
> data format version?
>
>
>
> The bookie uses the configured disk format version.
>
> It is always able to read the old format, and it will write during regular
> writes and during compaction only the format configured in bk_server.conf
>
>
>
> Enrico
>
>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo 
> *Sent:* Monday, April 20, 2020 12:47 PM
> *To:* Bookkeeper-Dev ; us...@pulsar.apache.org
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
> + users@ back to the discussion.
>
>
>
> Sorry for being late to the discussion here.
>
>
>
> When talking about "rollback" behavior, it usually involves two questions:
>
>
>
> 1) does wire protocol support rollback?
>
> 2) does data(metadata) format support rollback?
>
>
>
> For the first question, the bookkeeper provides very good wire protocol
> compatibility between versions. So if there are no data format changes, it
> is generally safe to assume that you can rollback bookies to an old version.
>
>
>
> For the second question, it depends on the changes in a given release. In
> certain releases, the disk format change supports rollback. In other
> releases, it doesn't. But you usually can still do rollback via
> re-replicating the data. So in general, you can assume you can do rollback..
&

Re: Rollback of Apache Bookeeper

[Sijie Guo]

+ users@ back to the discussion.

Sorry for being late to the discussion here.

When talking about "rollback" behavior, it usually involves two questions:

1) does wire protocol support rollback?
2) does data(metadata) format support rollback?

For the first question, the bookkeeper provides very good wire protocol
compatibility between versions. So if there are no data format changes, it
is generally safe to assume that you can rollback bookies to an old version.

For the second question, it depends on the changes in a given release. In
certain releases, the disk format change supports rollback. In other
releases, it doesn't. But you usually can still do rollback via
re-replicating the data. So in general, you can assume you can do rollback..

Hope this clarifies things.

- Sijie

On Fri, Apr 17, 2020 at 11:48 AM Enrico Olivelli 
wrote:

> I am forwarding this request from user@
>
> We should state somehow how we support rollbacks.
>
> In general we do not have much support for it.
> We only try to not enable new format of data on disk by default. This way
> you can update the binaries and then use the new format only if you need to
> use a new feature that requires such format.
> If you enable the new format there is no way to rollback.
> We could have some tool that tries to convert to the previous format.
>
> We should also talk about this topic on the website, even if we do not
> still provide any tool for the rollback.
>
> Thoughts?
>
>
> -- Forwarded message -
> Da: Subash K 
> Date: Ven 17 Apr 2020, 16:32
> Subject: RE: Rollback of Apache Bookeeper
> To: u...@bookkeeper.apache.org 
>
>
> Thanks a lot Enrico 
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, April 17, 2020 7:10 PM
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
> Subash
>
> I will discuss about this topic on dev@bookkeeper.apache.org list
>
> best regards
>
> Enrico
>
>
>
> Il giorno ven 17 apr 2020 alle ore 12:47 Subash K 
> ha scritto:
>
> If rollback is technically supported by Bookkeeper, may be updating the
> website will help us better for now.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, April 17, 2020 3:44 PM
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
> Il Ven 17 Apr 2020, 11:59 Subash K  ha scritto:
>
> Sure. But as I’m just evaluating Apache Pulsar along with Bookkeeper, we
> are trying to understand various factors like Deployment, Life Cycle
> Management, etc.. Because these factors will be critical as it would create
> an impact on our existing product strategy.
>
>
>
> For now, we will take in that Rollback is not supported out of box. I feel
> that, rollback support should be brought in at the earliest to support
> organizations like us.
>
> In my opinion current releases of Bookkeeper support rollback.
>
> We do not have automated testing but we have this principle of being
> backward compatible with the previous version.
>
>
>
> Maybe we just lack an official statement on the website about this rule.
>
>
>
> Enrico
>
>
>
>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* Friday, April 17, 2020 3:16 PM
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
> Il Ven 17 Apr 2020, 10:59 Subash K  ha scritto:
>
> Thanks for your answers Enrico.
>
>
>
> On a high level, I understand that rollback is not supported and there is
> no plan to test and support it. But technically it is possible to do a
> rollback if new feature is not enabled after upgrade which is likely to
> change the file format.
>
>
>
> From the document, I see that there is option to rollback the file format
> after upgrade. Will this help us if we have to perform the rollback of data
> format as well?
>
>
>
> bookkeeper-server/bin/bookkeeper upgrade --rollback
>
> Actually that command was needed for very old versions of BK.
>
> I have been using it since 4.4 and never needed that command.
>
> In my opinion it is because since 4.4 BK payed more attention in making
> changes more safely
>
>
>
> A good option to you may be to check the version you are using and the one
> you want to upgrade to and if you have questions ask on this ML.
>
>
>
> We will be happy to support
>
>
>
> Enrico
>
>
>
>
>
>
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Enrico Olivelli 
> *Sent:* 16 April 2020 19:47
> *To:* user 
> *Subject:* Re: Rollback of Apache Bookeeper
>
>
>
>
>
>
>
> Il giorno gio 16 apr 2020 alle ore 15:01 Subash K 
> ha scritto:
>
> Hi Enrico,
>
>
>
> Thanks for your response.
>
>
>
> I’ve following queries on top of your response:
>
>- In general is community planning to test and support issues in
>rollback of Bookkeeper down the lane?
>
> Unfortunately no.
>
> We are only testing regular upgrades, not rollback.
>
> It should be doable with current docker based integration testing tools.
>
>
>
>
>
>
>-
>- We might not frequently upgrade 

Re: Release 4.11.0 and 4.10.0 "stable"

[Sijie Guo]

I think we can make 4.10.1 first?

- Sijie

On Sun, Apr 19, 2020 at 11:05 PM Enrico Olivelli 
wrote:

> Alessandro,
> Thanks for pointing this out.
>
> From my point of view these items are to be checked:
> - We have problems on Integration Tests (on GitHub Actions,
>
> https://github.com/apache/bookkeeper/pull/2315/checks?check_run_id=600090302
> ,
> but I can reproduce the issue locally)
> - we have a few PR pending https://github.com/apache/bookkeeper/pulls some
> of them are trivial but very useful new features or bug fixes
> - we have this blocker issue from Ivan
> https://github.com/apache/bookkeeper/pull/2303 that is related to a patch
> from Rajan already committed to master, we must reach consensus here
>
> I hope that we can address all of this topic soon, then we can cut 4.11.
>
> +1 about making 4.10 marked as stable as soon as we release 4.11
>
> Enrico
>
>
> Il giorno lun 20 apr 2020 alle ore 07:22 Alessandro Luccaroni - Diennea
>  ha scritto:
>
> > Hi BookKeepers,
> > today is theoretically code freeze day for 4.11, see
> > https://bookkeeper.apache.org/community/releases/
> >
> > Also, in the last 4 month we have released no minor release for 4.10 and
> > 4.9 (and 4.9.2 is still referenced as the "stable" release).
> >
> > Are there any plan to proceed with a 4.11 release? Do we wait for 4.10.1
> > to declare 4.10 stable or we simply have to cast a vote?
> >
> > Regards,
> > Alessandro Luccaroni
> > Platform Manager @ Diennea - MagNews
> > Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519
> > Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy
> >
> >
> > 
> >
> > CONFIDENTIALITY & PRIVACY NOTICE
> > This e-mail (including any attachments) is strictly confidential and may
> > also contain privileged information. If you are not the intended
> recipient
> > you are not authorised to read, print, save, process or disclose this
> > message. If you have received this message by mistake, please inform the
> > sender immediately and destroy this e-mail, its attachments and any
> copies.
> > Any use, distribution, reproduction or disclosure by any person other
> than
> > the intended recipient is strictly prohibited and the person responsible
> > may incur in penalties.
> > The use of this e-mail is only for professional purposes; there is no
> > guarantee that the correspondence towards this e-mail will be read only
> by
> > the recipient, because, under certain circumstances, there may be a need
> to
> > access this email by third subjects belonging to the Company.
> >
>

Re: [ANNOUNCE] New BookKeeper Committer: Rajan Dhabalia

[Sijie Guo]

Congratulations Rajan! Thank you for your contribution!

- Sijie

On Mon, Mar 23, 2020 at 12:24 PM Rajan Dhabalia 
wrote:

> Thank you Enrico and other members for sharing the opportunity.
>
> Thanks,
> Rajan
>
> On Mon, Mar 23, 2020 at 11:19 AM Enrico Olivelli 
> wrote:
>
> > The Apache BookKeeper PMC recently extended committer karma to Rajan
> > Dhabalia and he has accepted.
> >
> > Rajan Dhabalia has been making great contributions to bookkeeper
> > community and he also a committer in Apache Pulsar project.
> >
> > It is great to have him onboard as bookkeeper committer.
> >
> > We are looking forward to more contributions from him.
> >
> > Congratulations and welcome onboard, Rajan !
> >
> > Enrico on behave of the BookKeeper PMC
> >
>

Re: Still problems with CI

[Sijie Guo]

We can port the pulsarbot approach to bookkeeper. So non-committers can use
a similar bot command to trigger re-running tests.

- Sijie

On Mon, Mar 2, 2020 at 12:58 PM Enrico Olivelli  wrote:

> Hi,
> We are still having troubles with CI.
> Even trivial patches fail, due to GitHub Actions or due to flakyness.
>
> Does anyone have time to help fixing the problem?
>
> The fact that GtiHub actions cannot be restarted by non committers is
> quite unfortunate.
>
> Enrico
>

Re: GitHub actions checkout is failing

[Sijie Guo]

I think the checkout action is changed to v2.

Added Yijie here. Since he helped fixing the issue in Pulsar, he can help
fixing the issue in bookkeeper as well.

Sijie

On Sun, Feb 23, 2020 at 2:29 AM Enrico Olivelli  wrote:

> Hi guys,
> it looks like GitHub ACtions sometimes fails to checkout Pull requests
>
> It is happening ofter and ofter
>
> https://github.com/apache/bookkeeper/pull/2268/checks?check_run_id=462945652
>
>
> Enrico
>

Re: Troubles with GitHub Actions and Python client - integration tests

[Sijie Guo]

Yong,

Can you help look into this?

Thanks,
Sijie

On Wed, Feb 5, 2020 at 4:34 AM Enrico Olivelli  wrote:

> Hi,
> integration tests started failing on GitHub Actions.
>
> see:
>
> https://pipelines.actions.githubusercontent.com/EMeCU0s3nV8mEzpqEGqbbDjtOUKCLJV4IPXg8mh4qtgdpzCGW6/_apis/pipelines/1/runs/367/signedlogcontent/3?urlExpires=2020-02-05T12%3A31%3A02.1144046Z=HMACV1=YN%2Bwk9c3Z0ZOIuI8mfhtj0L8hmaG%2B94ieufVJITSD8w%3D
>
> Please any Python Client and/or GitHub actions user help
>
> Enrico
>
>
> Copying the log...
>
> 020-02-05T11:55:42.3185364Z adding 'bookkeeper/proto/kv_pb2.py'
> 2020-02-05T11:55:42.3195826Z adding 'bookkeeper/proto/kv_rpc_pb2.py'
> 2020-02-05T11:55:42.3200493Z adding 'bookkeeper/proto/kv_rpc_pb2_grpc.py'
> 2020-02-05T11:55:42.3204309Z adding 'bookkeeper/proto/kv_store_pb2.py'
> 2020-02-05T11:55:42.3214399Z adding 'bookkeeper/proto/storage_pb2.py'
> 2020-02-05T11:55:42.3219234Z adding 'bookkeeper/proto/storage_pb2_grpc.py'
> 2020-02-05T11:55:42.3229668Z adding 'bookkeeper/proto/stream_pb2.py'
> 2020-02-05T11:55:42.3241984Z adding
> 'apache_bookkeeper_client-4.11.0_SNAPSHOT.dist-info/METADATA'
> 2020-02-05T11:55:42.3242447Z adding
> 'apache_bookkeeper_client-4.11.0_SNAPSHOT.dist-info/WHEEL'
> 2020-02-05T11:55:42.3242784Z adding
> 'apache_bookkeeper_client-4.11.0_SNAPSHOT.dist-info/namespace_packages.txt'
> 2020-02-05T11:55:42.3243119Z adding
> 'apache_bookkeeper_client-4.11.0_SNAPSHOT.dist-info/top_level.txt'
> 2020-02-05T11:55:42.3243478Z adding
> 'apache_bookkeeper_client-4.11.0_SNAPSHOT.dist-info/RECORD'
> 2020-02-05T11:55:42.3245125Z removing build/bdist.linux-x86_64/wheel
> 2020-02-05T11:57:28.9689829Z [ERROR] The command '/bin/sh -c
> /opt/bookkeeper/scripts/install-python-client.sh' returned a non-zero
> code: 1
> 2020-02-05T11:59:09.7211864Z [ERROR] The command '/bin/sh -c
> /opt/bookkeeper/scripts/install-python-client.sh' returned a non-zero
> code: 1
> 2020-02-05T11:59:09.7357291Z [ERROR] Failed to execute goal
> com.spotify:dockerfile-maven-plugin:1.3.7:build (default) on project
> current-version-image: Could not build image: The command '/bin/sh -c
> /opt/bookkeeper/scripts/install-python-client.sh' returned a non-zero
> code: 1 -> [Help 1]
> 2020-02-05T11:59:09.7357906Z [ERROR]
>

[ANNOUNCE] Pulsar Summit San Francisco 2020 Call for Presentation extended!

[Sijie Guo]

Hi everyone,

We know the CFP of Pulsar Summit SF 2020 runs over holidays.
To give you more time to submit a talk, we decided to extend the Call for
Presentations for Pulsar Summit San Francisco 2020 until Friday February
14th.

The conference takes place on April 28th with one day of talks.
If you are working on an interesting Pulsar project or use case that you'd
like to share with many enthusiastic Pulsar users and committer, you should
submit a talk proposal.

We are looking for talks on the following topics:
* Use Cases
* Operations
* Technology Deep Dive
* Ecosystem
* BookKeeper

You can find more detailed track descriptions and the form to submit a
proposal on the Pulsar Summit website at

--> https://pulsar-summit.org/call-for-presentations/
--> https://sessionize.com/pulsar-summit-san-francisco-2020/

As usual, accepted speakers get a free conference pass.

Best regards,
Sijie Guo

Re: Too many pending PRs

[Sijie Guo]

I thought we are waiting for migrating to Github Actions. Yong's vote is
waiting for anther binding vote. Can you approve that?

Thanks,
Sijie

On Tue, Jan 21, 2020 at 1:23 PM Enrico Olivelli  wrote:

> Hi Bookkeepers,
> We have plenty of pending pull requests on github.
>
> We need to help contributors move forward.
> For each patch we need at least 2 binding approvals, fortunately there are
> many small patches.
> We are also receiving contributions about docs, I would like to encourage
> them.
> We have a blocker problem on Travis for website changes.
>
> It's up to us
>
> Best regards
> Enrico
>

BookKeeper Track at Pulsar Summit

[Sijie Guo]

Hi,

We have the Call-For-Presentations (CFP) and pre-registration open for
Pulsar Summit. Since Pulsar uses BookKeeper as the event storage, we have
seen an increasing interest in BookKeeper and have created a dedicated
track for BookKeeper. If you have any talk ideas about your BookKeeper
usage or BookKeeper internals, you are welcome to submit a talk about
BookKeeper:  https://sessionize.com/pulsar-summit-san-francisco-2020/

Thanks,
Sijie

Re: [VOTE] Migrate the Jenkins CI to Github Actions

[Sijie Guo]

+1 binding

On Mon, Jan 20, 2020 at 6:50 PM Jia Zhai  wrote:

> +1
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Sat, Jan 18, 2020 at 12:48 AM Alessandro Luccaroni - Diennea <
> alessandro.luccar...@diennea.com> wrote:
>
> > +1 (non-binding)
> >
> > Alessandro
> >
> > > -Messaggio originale-
> > > Da: Yong Zhang 
> > > Inviato: venerdì 17 gennaio 2020 15:21
> > > A: dev@bookkeeper.apache.org
> > > Oggetto: [VOTE] Migrate the Jenkins CI to Github Actions
> > >
> > > Hi, BookKeepers
> > >
> > > Since the Jenkins CI is not anymore working when a new pull request
> > > created. This is a vote to migrate the Jenkins to Github Actions.
> > >
> > > Please vote with your opinions. The vote will be open for at least 72
> > hours.
> > >
> > > Thanks,
> > > Yong
> >
> > 
> >
> > CONFIDENTIALITY & PRIVACY NOTICE
> > This e-mail (including any attachments) is strictly confidential and may
> > also contain privileged information. If you are not the intended
> recipient
> > you are not authorised to read, print, save, process or disclose this
> > message. If you have received this message by mistake, please inform the
> > sender immediately and destroy this e-mail, its attachments and any
> copies.
> > Any use, distribution, reproduction or disclosure by any person other
> than
> > the intended recipient is strictly prohibited and the person responsible
> > may incur in penalties.
> > The use of this e-mail is only for professional purposes; there is no
> > guarantee that the correspondence towards this e-mail will be read only
> by
> > the recipient, because, under certain circumstances, there may be a need
> to
> > access this email by third subjects belonging to the Company.
> >
>

[ANNOUNCE] Apache Pulsar 2.5.0 released

[Sijie Guo]

The Apache Pulsar team is proud to announce Apache Pulsar version 2.5.0.

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management
for
subscribers, and cross-datacenter replication.

For Pulsar release details and downloads, visit:

https://pulsar.apache.org/download

Release Notes are at:
http://pulsar.apache.org/en/release-notes/#250-mdash-2019-12-06-a-id250a

We would like to thank the contributors that made the release possible.

Regards,

The Pulsar Team

Re: Are we really using libthrift and httpclient ?

[Sijie Guo]

SGTM

On Fri, Jan 17, 2020 at 9:16 AM Enrico Olivelli  wrote:

> Any insight over this problem ?
>
> If no one objects I will update my patch in order to clean up that 'shade
> plugin' stuff
> https://github.com/apache/bookkeeper/pull/2233
>
> Please note that bookkeeper and distributed log 'shaded' artifacts won't
> be changed, only the 'pure' DL Log Jar
>
> Enrico
>
> Il giorno gio 9 gen 2020 alle ore 09:48 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
>> Hi,
>> During the last weeks I have been "fighting" with Travis and the license
>> checker due to the presence/absence of
>> org.apache.httpcomponents-httpclient-4.4.1.jar
>> org.apache.httpcomponents-httpcore-4.4.1.jar
>> org.apache.thrift-libthrift-0.9.3.jar
>>
>> If I understand correctly libthrift is used by DistributedLog, and this
>> is turn is used by StreamStorage.
>> httpclient is a dependency imported by libthrift.
>>
>> Currently DistributedLog shades (and relocates package names) libthrift.
>> The result is that we are hiddenly bundling libthrift inside
>> distributelog jar and thus we have it in our server distribution.
>> It looks like we are now bundling httpclient (current master).
>>
>> Now that I am dropping the -Dstream Maven profile such jars disappeared
>> again.
>> The reason is that Maven is doing his best to deal with
>> dependency-reduced-poms but it is not so good and depending on the
>> activation of profiles (and on the Maven version) those jars may happen to
>> finish inside the "lib" directory of the server.
>>
>> Please also note that if we bundled a shaded version of libthrift we
>> should deal with the license, and we have to enhance the license checker
>>
>> Let's come to the questions:
>> 1) do we need to shade libthrift ? (perhaps the answer is no, as we
>> already have a distributed-log-shaded package)
>> 2) do we need really httpclient in the classpath ? (we also have a
>> pending PR that wants to add it explicitly as a dependency)
>>
>> In order to make things simpler I prefer to drop that shade stuff and
>> have the jars in the original form on "lib" and for downstream clients
>>
>> I am tagging you @siije directly because as far as I know you are the one
>> that most knows about DL and StreamStorage
>>
>> This is my patch that is currently stuck with this problem
>> https://github.com/apache/bookkeeper/pull/2233
>>
>>
>> Enrico
>>
>>
>>
>>
>>
>>

Re: [VOTE] EOL BookKeeper release versions up to 4.7.x

[Sijie Guo]

+1

On Fri, Jan 17, 2020 at 12:08 AM Alessandro Luccaroni - Diennea <
alessandro.luccar...@diennea.com> wrote:

> Hi BookKeepers,
> BookKeeper 4.6.2 was released in April 2018
> BookKeeper 4.7.3 was released in December 2018
>
> Every system in the wild that I know off is using releases newer than
> 4.6.2.
> I think Pravega is the last OSS project I know still on 4.7.3, but there
> is a pull currently in the works to upgrade to 4.9.x
>
> This is a vote to officially mark as EOL all the BK release up to 4.7.3
> (included).
> Please vote with your opinions. The vote will be open for at least 72
> hours.
>
> Alessandro
>
> 
>
> CONFIDENTIALITY & PRIVACY NOTICE
> This e-mail (including any attachments) is strictly confidential and may
> also contain privileged information. If you are not the intended recipient
> you are not authorised to read, print, save, process or disclose this
> message. If you have received this message by mistake, please inform the
> sender immediately and destroy this e-mail, its attachments and any copies.
> Any use, distribution, reproduction or disclosure by any person other than
> the intended recipient is strictly prohibited and the person responsible
> may incur in penalties.
> The use of this e-mail is only for professional purposes; there is no
> guarantee that the correspondence towards this e-mail will be read only by
> the recipient, because, under certain circumstances, there may be a need to
> access this email by third subjects belonging to the Company.
>

Re: [VOTE] Relax Time Based Release Plan to 6 months

[Sijie Guo]

+1

On Fri, Jan 17, 2020 at 12:08 AM Alessandro Luccaroni - Diennea <
alessandro.luccar...@diennea.com> wrote:

> Hi BookKeepers,
> since early 2019 the rate of contribution have slowed down significantly,
> this is a vote to relax the Time Based Release Plan from the actual "every
> 3 months" to a new release "every 6 month".
>
> Please vote with your opinions. The vote will be open for at least 72
> hours.
>
> Alessandro
>
> 
>
> CONFIDENTIALITY & PRIVACY NOTICE
> This e-mail (including any attachments) is strictly confidential and may
> also contain privileged information. If you are not the intended recipient
> you are not authorised to read, print, save, process or disclose this
> message. If you have received this message by mistake, please inform the
> sender immediately and destroy this e-mail, its attachments and any copies.
> Any use, distribution, reproduction or disclosure by any person other than
> the intended recipient is strictly prohibited and the person responsible
> may incur in penalties.
> The use of this e-mail is only for professional purposes; there is no
> guarantee that the correspondence towards this e-mail will be read only by
> the recipient, because, under certain circumstances, there may be a need to
> access this email by third subjects belonging to the Company.
>

Re: Website update after 4.10, Time Based Release plan and EOL

[Sijie Guo]

Yes. Let's start a vote?

Thanks,
Sijie

On Mon, Jan 13, 2020 at 3:41 AM Enrico Olivelli  wrote:

> I would like to EOL up to 4.7.x
>
> We also have a few CI jobs for 4.7
> It a total waste of resources
>
> Enrico
>
> Il mar 3 dic 2019, 18:29 Sijie Guo  ha scritto:
>
> > Yes we need votes for both proposals.
> >
> > We can wait to see if the new release window work before declaring an
> > official policy.
> >
> > It might be worthing waiting for a few more days before starting a vote.
> So
> > other people can chime in if they have different ideas.
> >
> > Thanks,
> > Sijie
> >
> > On Tue, Dec 3, 2019 at 12:05 AM Alessandro Luccaroni - Diennea <
> > alessandro.luccar...@diennea.com> wrote:
> >
> > > Do we need a vote for both proposal?
> > >
> > > Regarding the EOL Policy: do we want to wait and see if the new release
> > > window works before declaring an official policy?
> > >
> > > Regards,
> > > Alessandro
> > >
> > > > -Messaggio originale-
> > > > Da: Sijie Guo 
> > > > Inviato: martedì 3 dicembre 2019 08:19
> > > > A: Bookkeeper-Dev 
> > > > Oggetto: Re: Website update after 4.10, Time Based Release plan and
> EOL
> > > >
> > > > Yes. We can probably relax the release window to be every 6 months.
> If
> > > the
> > > > growth of development increases again, we can revert it back to
> every 3
> > > > months.
> > > >
> > > > Regarding EOL policy, we haven't really discussed how to proceed it.
> > > Most of
> > > > the systems I know are already using newer versions than 4.7.0. I
> think
> > > we
> > > > can consider deprecating the versions older than 4.7.0.
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Wed, Nov 20, 2019 at 5:49 AM Alessandro Luccaroni - Diennea <
> > > > alessandro.luccar...@diennea.com> wrote:
> > > >
> > > > > Hi BookKeepers,
> > > > > after the 4.10 release I see that the
> > > > > https://bookkeeper.apache.org/community/releases/ page needs some
> > > > care.
> > > > > We are still listing 4.10 as the next feature release and we have
> > > > > skipped
> > > > > 3 release windows.
> > > > >
> > > > > I can see on Github that since the early 2019 the rate of
> > contribution
> > > > > have slowed down a bit, so I think we should relax the Time Based
> > > > > Release Plan (new release every 6 month?).
> > > > >
> > > > > Apart from that, do we have an official EOL Policy for BookKeeper?
> On
> > > > > the website I see that we list all the version of BK back to v4.0.0
> > > > > (2011) Many of these older version had very little usage, but since
> > we
> > > > > are starting to see many project using BK downstream maybe we
> should
> > > > > clarify the full lifecycle of BK
> > > > >
> > > > > Alessandro Luccaroni
> > > > > Platform Manager @ Diennea - MagNews
> > > > > Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519 Viale
> > > > > G.Marconi 30/14 - 48018 Faenza (RA) - Italy
> > > > >
> > > > >
> > > > > 
> > > > >
> > > > > CONFIDENTIALITY & PRIVACY NOTICE
> > > > > This e-mail (including any attachments) is strictly confidential
> and
> > > > > may also contain privileged information. If you are not the
> intended
> > > > > recipient you are not authorised to read, print, save, process or
> > > > > disclose this message. If you have received this message by
> mistake,
> > > > > please inform the sender immediately and destroy this e-mail, its
> > > > attachments and any copies.
> > > > > Any use, distribution, reproduction or disclosure by any person
> other
> > > > > than the intended recipient is strictly prohibited and the person
> > > > > responsible may incur in penalties.
> > > > > The use of this e-mail is only for professional purposes; there is
> no
> > > > > guarantee that the correspondence towards this e-mail will be read
> > > > > only by the recipient, because, under certain circumstances, there
> > may
> > > > > be a need to access this email by third subjects belonging to the
> > > Company.
> > > > >
> > >
> > > 
> > >
> > > CONFIDENTIALITY & PRIVACY NOTICE
> > > This e-mail (including any attachments) is strictly confidential and
> may
> > > also contain privileged information. If you are not the intended
> > recipient
> > > you are not authorised to read, print, save, process or disclose this
> > > message. If you have received this message by mistake, please inform
> the
> > > sender immediately and destroy this e-mail, its attachments and any
> > copies.
> > > Any use, distribution, reproduction or disclosure by any person other
> > than
> > > the intended recipient is strictly prohibited and the person
> responsible
> > > may incur in penalties.
> > > The use of this e-mail is only for professional purposes; there is no
> > > guarantee that the correspondence towards this e-mail will be read only
> > by
> > > the recipient, because, under certain circumstances, there may be a
> need
> > to
> > > access this email by third subjects belonging to the Company.
> > >
> >
>

Re: Problems with Jenkins

[Sijie Guo]

I think it is worth considering moving CI to Github Actions. Jenkins has
been very terrible :(

- Sijie

On Sun, Jan 5, 2020 at 9:01 PM Enrico Olivelli  wrote:

> Hi,
> We have problems on node H30 on ASF Jenkins
>
> I have excluded that node manually from some of our precommit jobs thant
> seems to always prefer that node.
>
> This is the ticket on INFRA
> https://issues.apache.org/jira/browse/INFRA-19660
>
> Enrico
>

Re: BP-38 Publish Bookie Service Info on Metadata Service

[Sijie Guo]

Hi Enrico,

Thank you for putting all these together. We are interested in this feature.

I have made some comments in the proposal. PTAL.

Thanks,
Sijie

On Tue, Dec 17, 2019 at 5:25 AM Enrico Olivelli  wrote:

> Hi,
> any interest in this proposal ?
>
> It will open up the way to having more information about the services
> exposed by bookies available to the clients and even to the PlacementPolicy
> (in the future)
>
> I would like to see this kind of support in 4.11
>
> Enrico
>
> Il giorno ven 13 dic 2019 alle ore 00:30 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
> > Hi all,
> > I have created a new BookKeeper Proposal "BP-38 Publish Bookie Service
> > Info on Metadata Service"
> >
> > Proposal:
> > https://github.com/apache/bookkeeper/pull/2214
> >
> > Implementation (draft):
> > https://github.com/apache/bookkeeper/pull/2213
> >
> > Master ticket:
> > https://github.com/apache/bookkeeper/issues/2215
> >
> > Any feedback is really appreciated, for small comments you can comment
> > directly on  PR #2214 (proposal page) other wise we can just follow this
> > email thread
> >
> > Best regards
> > Enrico
> >
> >
>

Re: [DISCUSS] Dropping Twitter modules

[Sijie Guo]

LGTM. +1

On Fri, Jan 3, 2020 at 10:25 PM Enrico Olivelli  wrote:

> Thank you Dave
>
> can you please "Approve" the pull request ?
> Unfortunately Travis has problems and the build fails, I have started
> another thread to fix the problem
>
> Enrico
>
> Il giorno ven 3 gen 2020 alle ore 15:24 David Rusek 
> ha scritto:
>
> > Confirming what I said on slack, it’s not worth us maintaining these
> > modules.
> >
> > On Fri, Jan 3, 2020 at 5:54 AM Jia Zhai  wrote:
> >
> > > +1 to remove it.
> > >
> > > Best Regards.
> > >
> > >
> > > Jia Zhai
> > >
> > > Beijing, China
> > >
> > > Mobile: +86 15810491983
> > >
> > >
> > >
> > >
> > > On Fri, Jan 3, 2020 at 6:37 PM Enrico Olivelli 
> > > wrote:
> > >
> > >> This is the patch
> > >> https://github.com/apache/bookkeeper/pull/2232
> > >>
> > >> @Sijie @Dave please take a look
> > >>
> > >> Enrico
> > >>
> > >> Il giorno ven 3 gen 2020 alle ore 11:00 Enrico Olivelli <
> > >> eolive...@gmail.com>
> > >> ha scritto:
> > >>
> > >> > Hi,
> > >> > on slack Dave Rusek told me it is useless to keep this stuff.
> > >> > It seems that he did not receive this message, I am posting this
> > message
> > >> > just to let him confirm
> > >> >
> > >> > btw I am preparing a patch that drops all of this stuff
> > >> >
> > >> > Enrico
> > >> >
> > >> > Enrico
> > >> >
> > >> > Il giorno sab 28 dic 2019 alle ore 09:52 Enrico Olivelli <
> > >> > eolive...@gmail.com> ha scritto:
> > >> >
> > >> >> Hi folks,
> > >> >> I propose to drop the four modules that contain Twitter specific
> > stuff.
> > >> >> Those modules did not receive changes for long time, nor
> > >> issues/questions
> > >> >> from users.
> > >> >>
> > >> >> We recently added a new Maven profile (-Dtwitter) to skip them from
> > >> build.
> > >> >>
> > >> >> The modules are:
> > >> >> - Http Server:
> > >> >>
> > >>
> >
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-http/twitter-http-server/pom.xml
> > >> >> - Finagle Stats Provider
> > >> >>
> > >>
> >
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-stats-providers/twitter-finagle-provider/pom.xml
> > >> >> - Science Stats Provider
> > >> >>
> > >>
> >
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-stats-providers/twitter-science-provider/pom.xml
> > >> >> - Ostrich Stats Provider
> > >> >>
> > >>
> >
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-stats-providers/twitter-ostrich-provider/pom.xml
> > >> >>
> > >> >> Best regards
> > >> >>
> > >> >> Enrico
> > >> >>
> > >> >
> > >>
> > > --
> >
> > Dave Rusek
> > dave.ru...@gmail.com
> > @davidjrusek
> >
>

Re: [DISCUSS] drop -Dstream - Always build stream storage service

[Sijie Guo]

Enrico,

LGTM. +1

- Sijie

On Fri, Jan 3, 2020 at 10:49 PM David Rusek  wrote:

> Another option could be to create a Github bookkeeper org and host these as
> separate projects under the same umbrella. I think this might be an option
> for the future but I wanted to throw it out there.
>
> On Fri, Jan 3, 2020 at 7:43 AM David Rusek  wrote:
>
> > +1
> >
> > On Fri, Jan 3, 2020 at 5:57 AM Jia Zhai  wrote:
> >
> >> +1 for me
> >>
> >> Best Regards.
> >>
> >>
> >> Jia Zhai
> >>
> >> Beijing, China
> >>
> >> Mobile: +86 15810491983
> >>
> >>
> >>
> >>
> >> On Fri, Jan 3, 2020 at 6:46 PM Enrico Olivelli 
> >> wrote:
> >>
> >> > ping
> >> >
> >> > Il giorno sab 28 dic 2019 alle ore 17:15 Enrico Olivelli <
> >> > eolive...@gmail.com> ha scritto:
> >> >
> >> > > Hi Bookkeepers,
> >> > > I think that having the -Dstream profile is a nuisance.
> >> > > It enables streamstorage and distributedlog modules.
> >> > >
> >> > > The release procedure enables it forcibly.
> >> > >
> >> > > On CI we have lots of tricks and we have a very complex structure of
> >> > Maven
> >> > > modules.
> >> > >
> >> > > Currently we also have a problem with LICENSE checker on Travis
> >> > > https://github.com/apache/bookkeeper/pull/2228
> >> > >
> >> > > When I work on bookkeeper-server module (main code) even if I enable
> >> > > -Dstream I don't see troubles of any kind.
> >> > >
> >> > > Thoughts?
> >> > >
> >> > > Enrico
> >> > >
> >> >
> >>
> > --
> >
> > Dave Rusek
> > dave.ru...@gmail.com
> > @davidjrusek
> >
> --
>
> Dave Rusek
> dave.ru...@gmail.com
> @davidjrusek
>

Re: Audit Logging for BookieShell

[Sijie Guo]

+1 the proposal looks good to me.

If there is no objections about this, you can submit the proposal to bk
website, following the process here:
http://bookkeeper.apache.org/community/bookkeeper_proposals/

Thanks,
Sijie

On Tue, Dec 3, 2019 at 3:11 PM Anup Ghatage  wrote:

> Hi Everyone!,
>
> I have written up a proposal for audit logging in BookieShell.
> <
> https://docs.google.com/document/d/1bVZfvEnzxmsS2Ldmir6p0PigrNXfg5v3NMtC_b1zrEg/edit?usp=sharing
> >
> The proposal is simple but has a large impact on compliance certifications.
> I'll start an official BP based on reception of the feature.
> Kindly take a look when you have time and feel free to add comments on the
> doc!
>
> Regards
> --
> Anup Ghatage
> www.ghatage.com
>

Re: Missing BookKeeper tags on dockerhub

[Sijie Guo]

Cool. Thank you Enrico!

- Sijie

On Tue, Dec 3, 2019 at 4:06 AM Enrico Olivelli  wrote:

> It is all done now
> https://hub.docker.com/r/apache/bookkeeper/tags
>
> We should update the release guide
>
> Enrico
>
> Il giorno mar 3 dic 2019 alle ore 09:03 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
>> I have compared our git tags for 4.9.1 and 4.9.2 and 4.10.0
>>
>> it seems that we missed these special "tags":
>>
>> https://github.com/apache/bookkeeper/tree/release-4.9.2-docker
>> https://github.com/apache/bookkeeper/tree/release-4.10.0-docker
>>
>> It seems that we changed something in 4.8.2 with this commit from @Ivan
>> Kelly 
>>
>> https://github.com/apache/bookkeeper/commit/5e9828b795d7ef82c17f72a6c9efb6d0e0b4c8d7
>>
>> Enrico
>>
>> Il giorno mar 3 dic 2019 alle ore 07:56 Sijie Guo 
>> ha scritto:
>>
>>> Enrico,
>>>
>>> It is docker autobuild. We don't manage this part. I would suggest
>>> filling an INFRA ticket to see what is the issue.
>>>
>>> - Sijie
>>>
>>> On Mon, Dec 2, 2019 at 3:59 AM Enrico Olivelli 
>>> wrote:
>>>
>>>> @Sijie Guo   @Ivan Kelly 
>>>> Do you have any idea ?
>>>>
>>>> I can't remember if we have ever switched to a new way of pushing such
>>>> tags.
>>>>
>>>> Enrico
>>>>
>>>> Il giorno ven 29 nov 2019 alle ore 11:02 Enrico Olivelli <
>>>> eolive...@gmail.com> ha scritto:
>>>>
>>>>> Hi,
>>>>> It looks like dockerhub lost our latest tags: 4.9.2 and 4.10.0.
>>>>>
>>>>> https://hub.docker.com/r/apache/bookkeeper/tags
>>>>>
>>>>> IIRC when I released 4.10.0 the image was there, but I may be wrong.
>>>>>
>>>>> Shall we open a ticket to INFRA ?
>>>>>
>>>>> Enrico
>>>>>
>>>>

Re: Website update after 4.10, Time Based Release plan and EOL

[Sijie Guo]

Yes we need votes for both proposals.

We can wait to see if the new release window work before declaring an
official policy.

It might be worthing waiting for a few more days before starting a vote. So
other people can chime in if they have different ideas.

Thanks,
Sijie

On Tue, Dec 3, 2019 at 12:05 AM Alessandro Luccaroni - Diennea <
alessandro.luccar...@diennea.com> wrote:

> Do we need a vote for both proposal?
>
> Regarding the EOL Policy: do we want to wait and see if the new release
> window works before declaring an official policy?
>
> Regards,
> Alessandro
>
> > -----Messaggio originale-
> > Da: Sijie Guo 
> > Inviato: martedì 3 dicembre 2019 08:19
> > A: Bookkeeper-Dev 
> > Oggetto: Re: Website update after 4.10, Time Based Release plan and EOL
> >
> > Yes. We can probably relax the release window to be every 6 months. If
> the
> > growth of development increases again, we can revert it back to every 3
> > months.
> >
> > Regarding EOL policy, we haven't really discussed how to proceed it.
> Most of
> > the systems I know are already using newer versions than 4.7.0. I think
> we
> > can consider deprecating the versions older than 4.7.0.
> >
> > Thanks,
> > Sijie
> >
> > On Wed, Nov 20, 2019 at 5:49 AM Alessandro Luccaroni - Diennea <
> > alessandro.luccar...@diennea.com> wrote:
> >
> > > Hi BookKeepers,
> > > after the 4.10 release I see that the
> > > https://bookkeeper.apache.org/community/releases/ page needs some
> > care.
> > > We are still listing 4.10 as the next feature release and we have
> > > skipped
> > > 3 release windows.
> > >
> > > I can see on Github that since the early 2019 the rate of contribution
> > > have slowed down a bit, so I think we should relax the Time Based
> > > Release Plan (new release every 6 month?).
> > >
> > > Apart from that, do we have an official EOL Policy for BookKeeper? On
> > > the website I see that we list all the version of BK back to v4.0.0
> > > (2011) Many of these older version had very little usage, but since we
> > > are starting to see many project using BK downstream maybe we should
> > > clarify the full lifecycle of BK
> > >
> > > Alessandro Luccaroni
> > > Platform Manager @ Diennea - MagNews
> > > Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519 Viale
> > > G.Marconi 30/14 - 48018 Faenza (RA) - Italy
> > >
> > >
> > > 
> > >
> > > CONFIDENTIALITY & PRIVACY NOTICE
> > > This e-mail (including any attachments) is strictly confidential and
> > > may also contain privileged information. If you are not the intended
> > > recipient you are not authorised to read, print, save, process or
> > > disclose this message. If you have received this message by mistake,
> > > please inform the sender immediately and destroy this e-mail, its
> > attachments and any copies.
> > > Any use, distribution, reproduction or disclosure by any person other
> > > than the intended recipient is strictly prohibited and the person
> > > responsible may incur in penalties.
> > > The use of this e-mail is only for professional purposes; there is no
> > > guarantee that the correspondence towards this e-mail will be read
> > > only by the recipient, because, under certain circumstances, there may
> > > be a need to access this email by third subjects belonging to the
> Company.
> > >
>
> 
>
> CONFIDENTIALITY & PRIVACY NOTICE
> This e-mail (including any attachments) is strictly confidential and may
> also contain privileged information. If you are not the intended recipient
> you are not authorised to read, print, save, process or disclose this
> message. If you have received this message by mistake, please inform the
> sender immediately and destroy this e-mail, its attachments and any copies.
> Any use, distribution, reproduction or disclosure by any person other than
> the intended recipient is strictly prohibited and the person responsible
> may incur in penalties.
> The use of this e-mail is only for professional purposes; there is no
> guarantee that the correspondence towards this e-mail will be read only by
> the recipient, because, under certain circumstances, there may be a need to
> access this email by third subjects belonging to the Company.
>

Re: Website update after 4.10, Time Based Release plan and EOL

[Sijie Guo]

Yes. We can probably relax the release window to be every 6 months. If the
growth of development increases again, we can revert it back to every 3
months.

Regarding EOL policy, we haven't really discussed how to proceed it. Most
of the systems I know are already using newer versions than 4.7.0. I think
we can consider deprecating the versions older than 4.7.0.

Thanks,
Sijie

On Wed, Nov 20, 2019 at 5:49 AM Alessandro Luccaroni - Diennea <
alessandro.luccar...@diennea.com> wrote:

> Hi BookKeepers,
> after the 4.10 release I see that the
> https://bookkeeper.apache.org/community/releases/ page needs some care.
> We are still listing 4.10 as the next feature release and we have skipped
> 3 release windows.
>
> I can see on Github that since the early 2019 the rate of contribution
> have slowed down a bit, so I think we should relax the Time Based Release
> Plan (new release every 6 month?).
>
> Apart from that, do we have an official EOL Policy for BookKeeper? On the
> website I see that we list all the version of BK back to v4.0.0 (2011)
> Many of these older version had very little usage, but since we are
> starting to see many project using BK downstream maybe we should clarify
> the full lifecycle of BK
>
> Alessandro Luccaroni
> Platform Manager @ Diennea - MagNews
> Tel.: (+39) 0546 066100 Int. 924 - Mob.: (+39) 393 7273519
> Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy
>
>
> 
>
> CONFIDENTIALITY & PRIVACY NOTICE
> This e-mail (including any attachments) is strictly confidential and may
> also contain privileged information. If you are not the intended recipient
> you are not authorised to read, print, save, process or disclose this
> message. If you have received this message by mistake, please inform the
> sender immediately and destroy this e-mail, its attachments and any copies.
> Any use, distribution, reproduction or disclosure by any person other than
> the intended recipient is strictly prohibited and the person responsible
> may incur in penalties.
> The use of this e-mail is only for professional purposes; there is no
> guarantee that the correspondence towards this e-mail will be read only by
> the recipient, because, under certain circumstances, there may be a need to
> access this email by third subjects belonging to the Company.
>

Re: Enhancing getBookieInfo

[Sijie Guo]

I am not sure it is a good idea to expose the stats via a RPC endpoint.

Because most of the stats collection systems (e.g. prometheus) are
collecting stats via an http endpoints.
I don't think we should duplicate the work that a stats library is good at.

Other thoughts inline.

On Sat, Nov 30, 2019 at 4:31 AM Enrico Olivelli  wrote:

> Hi all,
> In my company we are developing a visual tool for Bookkeeper Management and
> we would like to have more information about the status of a Bookie just by
> using RPC calls.
> We already have getBookieInfo that is only reporting about disk space
> usage.
>
> Examples of useful info:
> - link to metrics endpoint
> - last minor/major gc execution timestamp
> - last jvm start timestamp (uptime)
> - configuration file
>
> We already have the http endpoint but it comes with several disadvantages:
> - it is a secondary channel (so you have to use two channels to talk to the
> bookie)
>

Why not standardize the management endpoint in http endpoint and build the
dashboard on it?
It is a common approach for doing so. An RPC channel is more used for data
channel.


> - there is no way of getting the http address, as it is not advertised on
> metadata discovery service
>

We should fix this, no?


> - it does not support security (auth/tls..) yet
>

We should add TLS support and authentication to the http endpoint.

I guess rather than choosing to implement duplicated work in a PRC
endpoint, why can't we spend some time enhancing http endpoint?


>
> Given that I am mostly using Kerberos,
> going to the http way will need separate security configurations: for
> bookie RPC and for HTTP.
>

Jia implemented Kerberos for Pulsar http endpoint. Also HDFS supports
kerberos at its http endpoint as well.
So I don't think that is a problem as well.


>
> So that's why I would prefer to use the standard binary RPC protocol: only
> one client, only one auth, security is already ok, no additional discovery
> info.
>
> If you guys agree on the approach we can send a patch that includes the
> support for publishing more information on that RPC
>
> Cheers
> Enrico
>

Re: Missing BookKeeper tags on dockerhub

[Sijie Guo]

Enrico,

It is docker autobuild. We don't manage this part. I would suggest filling
an INFRA ticket to see what is the issue.

- Sijie

On Mon, Dec 2, 2019 at 3:59 AM Enrico Olivelli  wrote:

> @Sijie Guo   @Ivan Kelly 
> Do you have any idea ?
>
> I can't remember if we have ever switched to a new way of pushing such
> tags.
>
> Enrico
>
> Il giorno ven 29 nov 2019 alle ore 11:02 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
>
>> Hi,
>> It looks like dockerhub lost our latest tags: 4.9.2 and 4.10.0.
>>
>> https://hub.docker.com/r/apache/bookkeeper/tags
>>
>> IIRC when I released 4.10.0 the image was there, but I may be wrong.
>>
>> Shall we open a ticket to INFRA ?
>>
>> Enrico
>>
>

Re: Deprecate legacy API ? Which steps ?

[Sijie Guo]

+1 for deprecating the legacy API.

Sijie

On Wed, Nov 20, 2019 at 3:23 AM Enrico Olivelli  wrote:

> Hi,
> I think it is time to work in the direction of deprecating the "old" client
> API in favour of the new client API.
>
> Blockers:
> 1) Convert the documentation on the website to use only the new API
> 2) Let users of the new API use the 'Explicit LAC" feature
>
>
> for 2) I had sent in January a Pull Request
> https://github.com/apache/bookkeeper/pull/1901
>
> I agreed with Sijie that we should think more about the proposal and my
> patch was not the best we could do. As it was not a showstopper for me I
> stopped the work.
>
> Thoughts ?
>
> Enrico
>

Re: Board report draft for November 2019

[Sijie Guo]

FYI. I just posted the report.

Thanks Enrico for putting this together!

- Sijie

On Mon, Nov 11, 2019 at 8:42 PM Sijie Guo  wrote:

> Hi Enrico,
>
> Thank you for putting this together! Overall looks good to me.
>
> You can find the statistics here:
> https://reporter.apache.org/wizard/statistics?bookkeeper
>
> - Sijie
>
> On Thu, Nov 7, 2019 at 5:09 AM Enrico Olivelli 
> wrote:
>
>> Hello,
>> Below you can find the draft of the report to the ASF borad I am preparing
>> Any comment and correction is welcome from the community.
>> This time I used the wizard at
>> https://reporter.apache.org/wizard/?bookkeeper
>>
>> @Sijie Guo  I can't find mailing list activity
>> stats, can you help me ? It seems that "reporter" only show 'most notable
>> trends'  and not full data.
>>
>> I am not sure we should continue to talk about "companies": Yahoo,
>> Twitter, Salesforce
>>
>> Enrico
>>
>>
>> ## Description:
>> BookKeeper is a scalable, fault-tolerant, and low-latency storageservice
>> optimized for append-only workloads. It has been used as
>> a fundamental service to build high available and replicated services
>> in companies like Twitter, Yahoo and Salesforce. It is also the log
>> segment store for Apache DistributedLog and message store for Apache
>> Pulsar.
>>
>> Apache DistributedLog is a high-level API and service layer for
>> Apache BookKeeper, providing easier access to the BookKeeper
>> primitives. It is a subproject of Apache BookKeeper.
>>
>> ## Issues:
>> There are no issues requiring board attention at this time.
>>
>> ## Membership Data:
>> Apache BookKeeper was founded 2014-11-19 (5 years ago)
>> There are currently 21 committers and 16 PMC members in this project.
>> The Committer-to-PMC ratio is roughly 3:2.
>>
>> Community changes, past quarter:
>> - No new PMC members. Last addition was Charan Reddy G on 2019-07-24.
>> - No new committers. Last addition was Andrey Yegorov on 2018-02-09.
>>
>> ## Project Activity:
>> - 4.9.1 was released on April 7 2019
>> - 4.9.2 was released on May 16 2019
>> - 4.10.0: was released on November 6 2019
>> - The growth of Apache Pulsar community also help grow the adoption
>> of BookKeeper. This helps building the ecosystem around BookKeeper.
>> - The project is also extending its scope to cover long term distributed
>> storage and now bundles a new KV distributed database (StreamStorage).
>> - We released a new Python client for the StreamStorage service.
>> - We are working on a brand new CLI interface
>>
>> GitHub issues:
>> 19 issues opened on GitHub, past quarter (35% increase)
>> 35 issues closed on GitHub, past quarter (191% increase)
>>
>> GitHub PR activity:
>> 37 PRs opened on GitHub, past quarter (2% increase)
>> 26 PRs closed on GitHub, past quarter (-33% decrease)
>>
>> ## Community Health:
>> - During the last quarter there development slowed down a little and we
>> missed one scheduled release (we started a Time based release plan this
>> year).
>> - During the last months new users and contributors appeared expecially
>> from Apache Pulsar community and from other OSS projects that were born
>> recently and are based on Apache BookKeeper.
>> -Mailing list discussions are brisk, in particularly around the
>> active projects.
>>
>

Re: Committed python client version directly to master

[Sijie Guo]

LGTM

On Tue, Nov 12, 2019 at 4:50 AM Enrico Olivelli  wrote:

> Hi,
> as part of the release procedure I was changing the version of the python
> client manually to 4.11.0-SNAPSHOT
>
> I have pushed the change directly to the master branch
>
>
> https://github.com/apache/bookkeeper/commit/40f349ccff6840d53b6421fdcbf6c1ed1bb9ca1b
>
> I am sorry.
> Please any committer can second this change ?
>
> I feel it is not worth to revert the change, as it is part of the mechanics
> of the release.
> But we are following review-then-commit so this change should have been
> approved by another binding +1.
>
> Regards
> Enrico
>

Re: Board report draft for November 2019

[Sijie Guo]

Hi Enrico,

Thank you for putting this together! Overall looks good to me.

You can find the statistics here:
https://reporter.apache.org/wizard/statistics?bookkeeper

- Sijie

On Thu, Nov 7, 2019 at 5:09 AM Enrico Olivelli  wrote:

> Hello,
> Below you can find the draft of the report to the ASF borad I am preparing
> Any comment and correction is welcome from the community.
> This time I used the wizard at
> https://reporter.apache.org/wizard/?bookkeeper
>
> @Sijie Guo  I can't find mailing list activity stats,
> can you help me ? It seems that "reporter" only show 'most notable trends'
> and not full data.
>
> I am not sure we should continue to talk about "companies": Yahoo,
> Twitter, Salesforce
>
> Enrico
>
>
> ## Description:
> BookKeeper is a scalable, fault-tolerant, and low-latency storageservice
> optimized for append-only workloads. It has been used as
> a fundamental service to build high available and replicated services
> in companies like Twitter, Yahoo and Salesforce. It is also the log
> segment store for Apache DistributedLog and message store for Apache
> Pulsar.
>
> Apache DistributedLog is a high-level API and service layer for
> Apache BookKeeper, providing easier access to the BookKeeper
> primitives. It is a subproject of Apache BookKeeper.
>
> ## Issues:
> There are no issues requiring board attention at this time.
>
> ## Membership Data:
> Apache BookKeeper was founded 2014-11-19 (5 years ago)
> There are currently 21 committers and 16 PMC members in this project.
> The Committer-to-PMC ratio is roughly 3:2.
>
> Community changes, past quarter:
> - No new PMC members. Last addition was Charan Reddy G on 2019-07-24.
> - No new committers. Last addition was Andrey Yegorov on 2018-02-09.
>
> ## Project Activity:
> - 4.9.1 was released on April 7 2019
> - 4.9.2 was released on May 16 2019
> - 4.10.0: was released on November 6 2019
> - The growth of Apache Pulsar community also help grow the adoption
> of BookKeeper. This helps building the ecosystem around BookKeeper.
> - The project is also extending its scope to cover long term distributed
> storage and now bundles a new KV distributed database (StreamStorage).
> - We released a new Python client for the StreamStorage service.
> - We are working on a brand new CLI interface
>
> GitHub issues:
> 19 issues opened on GitHub, past quarter (35% increase)
> 35 issues closed on GitHub, past quarter (191% increase)
>
> GitHub PR activity:
> 37 PRs opened on GitHub, past quarter (2% increase)
> 26 PRs closed on GitHub, past quarter (-33% decrease)
>
> ## Community Health:
> - During the last quarter there development slowed down a little and we
> missed one scheduled release (we started a Time based release plan this
> year).
> - During the last months new users and contributors appeared expecially
> from Apache Pulsar community and from other OSS projects that were born
> recently and are based on Apache BookKeeper.
> -Mailing list discussions are brisk, in particularly around the
> active projects.
>

Re: [VOTE] Release 4.10.0, release candidate #0

[Sijie Guo]

+1 (binding)

- verified signatures, shasum
- build the source
- binary can run
- tag is good
- release note is good.

On Thu, Oct 31, 2019 at 7:46 PM Enrico Olivelli  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version 4.10.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.10.0-rc0" [4] with git sha
> 2f08377f5c56f96389fb3a8e51844844537e118b
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Enrico Olivelli
>
> [1]
>
> https://github.com/apache/bookkeeper/pull/2165/commits/0d91a95f306031d85c6a8195b0e6d73523cbfc0a
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.10.0-rc0/
> [3]
>
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1043/
> [4] https://github.com/apache/bookkeeper/tree/v4.10.0-rc0
>

Re: Apache Bookkeeper Checksum Validation

[Sijie Guo]

Hi Karan,

Thank you for your proposal. Can you also add your proposal as a BP to the
BP list? You can check the BP process here:
http://bookkeeper.apache.org/community/bookkeeper_proposals/

Thanks,
Sijie

On Fri, Sep 27, 2019 at 5:53 AM Karan Mehta 
wrote:

> Hello everyone,
>
> I wrote up a document here  for
> Apache Bookkeeper Checksum Validation for the issue
> . I have added certain
> options and highlighted the pros/cons of each design. I would like to hear
> everyone's thoughts on it. Feel free to comment on the doc to suggest
> ideas. Thanks for your inputs!
>
> --
> Karan Mehta
>
> 
>

Re: Upgrading Netty?

[Sijie Guo]

If we are close to a release, I would suggest holding on upgrading the
dependencies. We can upgrade in 4.11.

Thanks,
Sijie

On Wed, Aug 21, 2019 at 1:58 PM Enrico Olivelli  wrote:

> Ping
>
> Il ven 16 ago 2019, 19:03 Enrico Olivelli  ha
> scritto:
>
> > Hi all,
> > Is it worth to upgrade Netty before 4.10 release?
> > I am running clients and bookies with 4.1.36, we have 4.1.32 in community
> > pom.xml
> >
> > I think it is worth to update to latest and greatest 4.1.39 but I don't
> > have a way to run extensive testing
> >
> > Which version are you using in prod?
> >
> > Enrico
> >
>

Re: Cutting 4.10 release soon

[Sijie Guo]

On Wed, Aug 14, 2019 at 4:20 AM Enrico Olivelli  wrote:

> Hi folks,
> as Charan merged new tests for the clusterwise ledger scrutiny I would like
> to cut a new release and hopefully restart the timebased release schedule.
>

+1


>
> The only problem I must report before starting the procedure is that we are
> still not able to merge the change with the upgrade of Zookeeper dependency
> to 3.5.5.
> It seems that we that upgrade integration tests never pass on ASF CI (they
> pass locally on my machine)
>

Have you checked the logs on CI?



>
> There is also an open discussion about the behavior of the byte[] writer
> API
>

Does adding comments address the concern first? Because I don't see an easy
way to make byte[] array reusable after a sync operation.

- Sijie


>
> Thoughts?
>
> Enrico
>

Re: Time to think about 4.10.0 ?

[Sijie Guo]

+1 we should try to follow the time based release schedule.

On Wed, Jul 10, 2019 at 3:08 AM Enrico Olivelli  wrote:

> Hi,
> I feel we are no more following the time based release schedule.
>
> it is time to think about releasing 4.10.0 ?
>
> There is some ongoing work that it is worth to commit (Charan's BP,
> ZooKeeper client upgrade...)
>
> thoughts ?
>
> Enrico
>

Re: Cookie for FaultDomain Info?

[Sijie Guo]

+1 from me. `Cookie` was designed for keeping the informations that is
associated with a bookie (e.g. disk layouts, bookie id and etc).

I think it is making sense to have `FaultZoneId` stored as part of the
cookie.

- Sijie

On Sun, May 19, 2019 at 9:48 AM Venkateswara Rao Jujjuri 
wrote:

> In the current code, bookie to faultDomain mapping is supplied through
> different methods. Salesforce uses a script to read yaml file which
> contains racks/machines mapping. But I am wondering why can't we put this
> info in the Cookie? Assuming that these machines can never move across
> fault zones.
>
> Currently cookies contain  version, Host, JourlanDirs, ledgerDirs, and
> instanceId.
> If we add faultzoneId to it, it will be always available for everyone to
> look into.
> Is there any reason why it would be a bad idea?
>
> Thanks,
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>

Re: [DRAFT] BookKeeper Board Report of May

[Sijie Guo]

I forgot to mention here. I have already submitted the report.

- Sijie

On Wed, May 15, 2019 at 6:12 PM Jia Zhai  wrote:

> +1
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Thu, May 9, 2019 at 1:02 PM Enrico Olivelli 
> wrote:
>
> > +1
> >
> > Thanks
> > Enrico
> >
> > Il gio 9 mag 2019, 06:30 Matteo Merli  ha
> scritto:
> >
> > > +1
> > >
> > >
> > > --
> > > Matteo Merli
> > > 
> > >
> > > On Wed, May 8, 2019 at 6:47 PM Sijie Guo  wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Here is a draft for BookKeeper board report of May. Please take a
> look.
> > > >
> > > > If there is no objections, I will submit it today.
> > > >
> > > > ---
> > > >
> > > >
> > > > ## Description:
> > > >
> > > > BookKeeper is a scalable, fault-tolerant, and low-latency storage
> > > > service optimized for append-only workloads. It has been used as
> > > > a fundamental service to build high available and replicated services
> > > > in companies like Twitter, Yahoo and Salesforce. It is also the log
> > > > segment store for Apache DistributedLog and message store for Apache
> > > Pulsar.
> > > >
> > > > Apache DistributedLog is a high-level API and service layer for
> > > > Apache BookKeeper, providing easier access to the BookKeeper
> > > > primitives. It is a subproject of Apache BookKeeper.
> > > >
> > > > ## Issues:
> > > > There are no issues requiring board attention at this time.
> > > >
> > > > ## Activity:
> > > > - 4.9.0 was released on January 31 2019
> > > > - 4.8.2 was released on March 19 2019
> > > > - 4.9.1 was released on April 7 2019
> > > > - 4.9.2 is under release voting
> > > > - The growth of Apache Pulsar community also help grow the adoption
> > > > of BookKeeper.
> > > >  This helps building the ecosystem around BookKeeper.
> > > >
> > > > ## Health report:
> > > > - Development continues at a steady pace. We are merging multiple PRs
> > per
> > > > week on average.
> > > > - Mailing list and slack discussions are brisk, in particularly
> around
> > > the
> > > > active projects.
> > > >
> > > > ## PMC changes:
> > > >  - Currently 15 PMC members.
> > > >  - No new PMC members added in the last 3 months
> > > >  - Last PMC addition was Enrico Olivelli on Fri Feb 23 2018
> > > >
> > > > ## Committer base changes:
> > > >  - Currently 21 committers.
> > > >  - No new committers added in the last 3 months
> > > >  - Last committer addition was Andrey Yegorov at Sat Feb 10 2018
> > > >
> > > > ## Releases:
> > > > - 4.9.0 was released on January 31 2019
> > > > - 4.8.2 was released on March 19 2019
> > > > - 4.9.1 was released on April 7 2019
> > > >
> > > > ## Mailing list activity:
> > > >  - dev@bookkeeper.apache.org:
> > > > - 104 subscribers (down -1 in the last 3 months):
> > > > - 67 emails sent to list (112 in previous quarter)
> > > >
> > > >  - distributedlog-comm...@bookkeeper.apache.org:
> > > > - 12 subscribers (up 0 in the last 3 months):
> > > > - 0 emails sent to list (0 in previous quarter)
> > > >
> > > >  - distributedlog-...@bookkeeper.apache.org:
> > > > - 41 subscribers (up 0 in the last 3 months):
> > > > - 0 emails sent to list (0 in previous quarter)
> > > >
> > > >  - distributedlog-iss...@bookkeeper.apache.org:
> > > > - 9 subscribers (up 0 in the last 3 months):
> > > > - 6 emails sent to list (2 in previous quarter)
> > > >
> > > >  - distributedlog-u...@bookkeeper.apache.org:
> > > > - 26 subscribers (up 0 in the last 3 months):
> > > > - 0 emails sent to list (0 in previous quarter)
> > > >
> > > >  - iss...@bookkeeper.apache.org:
> > > > - 8 subscribers (up 0 in the last 3 months):
> > > > - 1825 emails sent to list (1867 in previous quarter)
> > > >
> > > >  - u...@bookkeeper.apache.org:
> > > > - 117 subscribers (up 0 in the last 3 months):
> > > > - 8 emails sent to list (11 in previous quarter)
> > > >
> > > >
> > > > 
> > > >
> > > > - Sijie
> > >
> >
>

[DRAFT] BookKeeper Board Report of May

[Sijie Guo]

Hi all,

Here is a draft for BookKeeper board report of May. Please take a look.

If there is no objections, I will submit it today.

---


## Description:

BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for append-only workloads. It has been used as
a fundamental service to build high available and replicated services
in companies like Twitter, Yahoo and Salesforce. It is also the log
segment store for Apache DistributedLog and message store for Apache Pulsar.

Apache DistributedLog is a high-level API and service layer for
Apache BookKeeper, providing easier access to the BookKeeper
primitives. It is a subproject of Apache BookKeeper.

## Issues:
There are no issues requiring board attention at this time.

## Activity:
- 4.9.0 was released on January 31 2019
- 4.8.2 was released on March 19 2019
- 4.9.1 was released on April 7 2019
- 4.9.2 is under release voting
- The growth of Apache Pulsar community also help grow the adoption
of BookKeeper.
 This helps building the ecosystem around BookKeeper.

## Health report:
- Development continues at a steady pace. We are merging multiple PRs per
week on average.
- Mailing list and slack discussions are brisk, in particularly around the
active projects.

## PMC changes:
 - Currently 15 PMC members.
 - No new PMC members added in the last 3 months
 - Last PMC addition was Enrico Olivelli on Fri Feb 23 2018

## Committer base changes:
 - Currently 21 committers.
 - No new committers added in the last 3 months
 - Last committer addition was Andrey Yegorov at Sat Feb 10 2018

## Releases:
- 4.9.0 was released on January 31 2019
- 4.8.2 was released on March 19 2019
- 4.9.1 was released on April 7 2019

## Mailing list activity:
 - dev@bookkeeper.apache.org:
- 104 subscribers (down -1 in the last 3 months):
- 67 emails sent to list (112 in previous quarter)

 - distributedlog-comm...@bookkeeper.apache.org:
- 12 subscribers (up 0 in the last 3 months):
- 0 emails sent to list (0 in previous quarter)

 - distributedlog-...@bookkeeper.apache.org:
- 41 subscribers (up 0 in the last 3 months):
- 0 emails sent to list (0 in previous quarter)

 - distributedlog-iss...@bookkeeper.apache.org:
- 9 subscribers (up 0 in the last 3 months):
- 6 emails sent to list (2 in previous quarter)

 - distributedlog-u...@bookkeeper.apache.org:
- 26 subscribers (up 0 in the last 3 months):
- 0 emails sent to list (0 in previous quarter)

 - iss...@bookkeeper.apache.org:
- 8 subscribers (up 0 in the last 3 months):
- 1825 emails sent to list (1867 in previous quarter)

 - u...@bookkeeper.apache.org:
- 117 subscribers (up 0 in the last 3 months):
- 8 emails sent to list (11 in previous quarter)




- Sijie

Re: Projects Can Apply Individually for Google Season of Docs

[Sijie Guo]

FYI. Jia has filed an application for BookKeeper. Once BookKeeper is
accepted as the project for SoD, he will bring the discussion back to the
dev mailing list.

- Sijie

On Thu, Apr 4, 2019 at 5:02 AM Sijie Guo  wrote:

> FYI.
>
> -- Forwarded message -
> From: Sijie Guo 
> Date: Wed, Apr 3, 2019 at 1:58 PM
> Subject: Fwd: Projects Can Apply Individually for Google Season of Docs
> To: Dev 
>
>
> FYI. Now projects can apply individually for Google Season of Docs. We
> should just apply this.
>
> - Sijie
>
>
> -- Forwarded message -
> From: 
> Date: Wed, Apr 3, 2019 at 1:55 PM
> Subject: Projects Can Apply Individually for Google Season of Docs
> To: d...@community.apache.org , <
> gene...@incubator.apache.org>, , <
> d...@beam.apache.org>, , 
>
>
> Hi All
>
> Initially the ASF as an organisation was planning to apply as a
> mentoring organisation for Google Season of Docs on behalf of all Apache
> projects but if accepted the maximum number of technical writers we
> could allocated is two. Two technical writers would probably not be
> enough to cover the potential demand from all our projects interested in
> participating.
>
> We've received feedback from Google that individual projects can apply.
> I will withdraw the ASF application so that any Apache project
> interested can apply individually for Season of Docs and so have the
> potential of being allocated a technical writer.
>
> Applications for Season of Docs is open now and closes on 23^rd April
> 2019. If your project would like to apply then please see the following
> link:
>
> https://developers.google.com/season-of-docs/docs/get-started/
>
> Good luck everyone!
>
> Thanks
> Sharan
>
>
>

Fwd: Projects Can Apply Individually for Google Season of Docs

[Sijie Guo]

FYI.

-- Forwarded message -
From: Sijie Guo 
Date: Wed, Apr 3, 2019 at 1:58 PM
Subject: Fwd: Projects Can Apply Individually for Google Season of Docs
To: Dev 


FYI. Now projects can apply individually for Google Season of Docs. We
should just apply this.

- Sijie


-- Forwarded message -
From: 
Date: Wed, Apr 3, 2019 at 1:55 PM
Subject: Projects Can Apply Individually for Google Season of Docs
To: d...@community.apache.org , <
gene...@incubator.apache.org>, , <
d...@beam.apache.org>, , 


Hi All

Initially the ASF as an organisation was planning to apply as a
mentoring organisation for Google Season of Docs on behalf of all Apache
projects but if accepted the maximum number of technical writers we
could allocated is two. Two technical writers would probably not be
enough to cover the potential demand from all our projects interested in
participating.

We've received feedback from Google that individual projects can apply.
I will withdraw the ASF application so that any Apache project
interested can apply individually for Season of Docs and so have the
potential of being allocated a technical writer.

Applications for Season of Docs is open now and closes on 23^rd April
2019. If your project would like to apply then please see the following
link:

https://developers.google.com/season-of-docs/docs/get-started/

Good luck everyone!

Thanks
Sharan

Re: Tags, version numbers and docker

[Sijie Guo]

I am fine with it if there is someone pushing the process forward.

- Sijie

On Thu, Mar 28, 2019 at 4:12 AM Jia Zhai  wrote:

> +1 for ivan's idea
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Thu, Mar 28, 2019 at 6:55 PM Enrico Olivelli 
> wrote:
>
> > Il giorno gio 28 mar 2019 alle ore 09:26 Ivan Kelly 
> > ha scritto:
> > >
> > > To clarify, what I intend to do here is.
> > >
> > > 1. Merge change from [1] to all live branches, so docker image will
> > > build directly from the voted on tag.
> > > 2. Get infra to add another autobuild rule that matches on
> > > release-([0-9.]+)\/docker, so that 4.8.2 and maybe 4.9.1 can be built
> > > with this change. This rule would not be used after that point.
> >
> >
> > +1
> >
> > >
> > > -Ivan
> > >
> > > [1]
> >
> https://github.com/ivankelly/bookkeeper/commit/e247ef705f055706604ba2f862c1006a8cf817e9
> > >
> > > On Wed, Mar 27, 2019 at 10:52 AM Ivan Kelly  wrote:
> > > >
> > > > > That’s a known issue. The auto build is controlled by ASF. We have
> > > > > discussed that before and came up the conclusion of current
> > approach. There
> > > > > is a BP to move dockerfile to a different repo. It just need
> someone
> > to
> > > > > complete the BP.
> > > >
> > > > This was a known issue a year ago. Nothing has moved on it, so I'm
> > > > trying to make movement now.
> > > >
> > > > > If you did so, you will not release 4.8.2 image, no?
> > > >
> > > > The image isn't part of the official release. At most it's a
> > > > convenience binary of the official release. However, at present it
> > > > isn't even that since it's generated from a tag which has not been
> +1d
> > > > by 3 PMC members.
> > > >
> > > > > Instead of doing a different way at the last phase of releasing a
> > release,
> > > > > I would suggest following the guide that was agreed by the
> > community, and
> > > > > work on the BP to move the dockerfile to a different repo in next
> > release.
> > > >
> > > > There's a fundamental release issue here. The official release is the
> > > > source tarball. The tag should reflect the contents of the source
> > > > tarball, and it should be possible to generate all binary convenience
> > > > packages from the source tarball.
> > > >
> > > > Things that do not match this criteria should not be presented as
> part
> > > > of the release as they have not been approved by 3 members of the
> PMC.
> > > >
> > > > I'm not going to put my signature on a tag which hasn't been voted
> on.
> > > > Currently whether tags constitute an official release artifact is
> > > > unresolved from a legal POV [1], but they are two clicks away from
> the
> > > > bookkeeper website frontpage so I think we should treat them as such.
> > > >
> > > > I will however fix the process for subsequent releases. The simplest
> > > > fix is to set the BK_VERSION from tag name. This can be done in a
> > > > build hook [2] which dockerhub autobuild[3] will pick up. Really
> > > > though we should build the tarball in a prehook so that the image can
> > > > also be generated for release candidates, so it can be tested and
> > > > voted on along with the rest of the convenience binaries.
> > > >
> > > > -Ivan
> > > >
> > > > [1] https://issues.apache.org/jira/browse/LEGAL-438
> > > > [2]
> >
> https://github.com/ivankelly/bookkeeper/commit/e247ef705f055706604ba2f862c1006a8cf817e9
> > > > [3]
> >
> https://cloud.docker.com/repository/registry-1.docker.io/ivankelly/bookkeeper/builds
> >
>

Re: Tags, version numbers and docker

[Sijie Guo]

On Tue, Mar 26, 2019 at 2:44 AM Ivan Kelly  wrote:

> Hi folks,
>
> Looking at doing the final tasks for the 4.8.2 release and stuck on
> the docker bit. It's not that I don't see what has been done before,
> but more that what is there is so so wrong.
>
> Take 4.8.1 release for example. The tarball for that release was cut
> from b4a2b1, yet the tag for that release is bf6149. bf6149 doesn't
> exist on branch-4.8.
>
> Instead, after b4a2b1 is cb8b76 which reverts 4.8.1 to 4.8.1-SNAPSHOT.
> So versions are going backwards in the branch.
>
> My bigger concern for now is the docker + tag bit. The tag should
> represent exactly what we released as artifacts.
>
> Who has control of the docker autobuild settings?  From reading the
> docks, you can set a regex and use a capture group to set the version.
> So for docker we could have additional tags, docker/release-4.8.2,
> which lives on the branch-4.8. Another alternative is to move the
> dockerfile into a different repo.


That’s a known issue. The auto build is controlled by ASF. We have
discussed that before and came up the conclusion of current approach. There
is a BP to move dockerfile to a different repo. It just need someone to
complete the BP.


>
> Anyhow, for now I'm going to set the tag to what was actually released.


If you did so, you will not release 4.8.2 image, no?

Instead of doing a different way at the last phase of releasing a release,
I would suggest following the guide that was agreed by the community, and
work on the BP to move the dockerfile to a different repo in next release.


>
> -Ivan
>

Re: Cutting 4.9.1

[Sijie Guo]

+1

On Wed, Mar 13, 2019 at 5:16 PM Enrico Olivelli  wrote:

> Hi guys,
> I need to cut 4.9.1 because we are seeing very often this issue
>
> https://github.com/apache/bookkeeper/commit/25c7506c0513351c533db643cb10c953d1e6d0b7
>
> Please tag any issue you want to merge into 4.9 branch.
>
> I will start the release process hopefully within the end of the week
>
> Please remember that Ivan started a VOTE thread for the 4.8 branch as well
>
> Cheers
> Enrico
>

Fwd: Google Season of Docs 2019

[Sijie Guo]

FYI

-- Forwarded message -
From: Kenneth Knowles 
Date: Wed, Mar 13, 2019 at 11:05 AM
Subject: Re: Google Season of Docs 2019
To: 
Cc: dev , , dev <
d...@beam.apache.org>, , <
d...@training.apache.org>


That's a lot of dev lists on one thread. Adding a couple more... :-)

Kenn

On Tue, Mar 12, 2019 at 7:17 PM Dave Fisher  wrote:

>
>
> > On Mar 12, 2019, at 7:01 PM, Huxing Zhang  wrote:
> >
> > Hi Dave,
> >
> > On Wed, Mar 13, 2019 at 2:14 AM Dave Fisher 
> wrote:
> >>
> >> Hi -
> >>
> >> Looks pretty cool.
> >
> > Yes, all the ASF projects could benefit from it. I think ASF should
> > apply it on behalf of all ASF projects, just like how the GSoC work.
> > Am I right?
>
> Yes! And the time is pretty quick. I’m hoping a someone over here will
> volunteer to help.
>
> Regards,
> Dave
>
>
> >
> >>
> >> Cc: to Apache Community Development.
> >>
> >> Regards,
> >> Dave
> >>
> >> Sent from my iPhone
> >>
> >>> On Mar 12, 2019, at 9:22 AM, Huxing Zhang  wrote:
> >>>
> >>> Hi,
> >>>
> >>> Google Season of Docs 2019[1] seems to be an interesting project,
> >>> which bring open source project and technical writer communities
> >>> together, just Like Google summer of code.
> >>>
> >>> I think the Dubbo can benefit from the project, especially the English
> >>> version of documentation could be improved.
> >>>
> >>> How do you think?
> >>>
> >>> [1] https://developers.google.com/season-of-docs/docs/timeline
> >>>
> >>> --
> >>> Best Regards！
> >>> Huxing
> >>
> >
> >
> > --
> > Best Regards！
> > Huxing
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Re: [VOTE] Release 4.8.2, release candidate #0

[Sijie Guo]

+1 (binding)

- verified source & binary package
- asc & sha512 are good
- artifacts are good
- tag is good

On Sat, Mar 9, 2019 at 2:28 AM Ivan Kelly  wrote:

> Hi everyone,
> Please review and vote on the release candidate #0 for the version
> 4.8.2, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed
> to dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.8.2-rc0" [4] with git sha
> f6adb282f88ab56afa3ef33b15d7acaf4b307eef
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes. Since it's late on
> friday, I won't tally the votes until this 17:00 UTC on wednesday
> 13th.
>
> Have a great weekend folks,
>
> Ivan
>
> [1] https://github.com/apache/bookkeeper/pull/1976
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.8.2-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1039/
> [4] https://github.com/apache/bookkeeper/tree/v4.8.2-rc0
>

Fwd: 4 Apache Events in 2019: DC Roadshow soon; next up Chicago, Las Vegas, and Berlin!

[Sijie Guo]

FYI. The CFP for ApacheCon North America is now open.



-- Forwarded message -
From: Rich Bowen 
Date: Wed, Mar 6, 2019 at 10:00 PM
Subject: 4 Apache Events in 2019: DC Roadshow soon; next up Chicago, Las
Vegas, and Berlin!
To: 


Dear Apache Enthusiast,

(You’re receiving this because you are subscribed to one or more user
mailing lists for an Apache Software Foundation project.)

TL;DR:
 * Apache Roadshow DC is in 3 weeks. Register now at
https://apachecon.com/usroadshowdc19/
 * Registration for Apache Roadshow Chicago is open.
http://apachecon.com/chiroadshow19
 * The CFP for ApacheCon North America is now open.
https://apachecon.com/acna19
 * Save the date: ApacheCon Europe will be held in Berlin, October 22nd
through 24th.  https://apachecon.com/aceu19


Registration is open for two Apache Roadshows; these are smaller events
with a more focused program and regional community engagement:

Our Roadshow event in Washington DC takes place in under three weeks, on
March 25th. We’ll be hosting a day-long event at the Fairfax campus of
George Mason University. The roadshow is a full day of technical talks
(two tracks) and an open source job fair featuring AWS, Bloomberg, dito,
GridGain, Linode, and Security University. More details about the
program, the job fair, and to register, visit
https://apachecon.com/usroadshowdc19/

Apache Roadshow Chicago will be held May 13-14th at a number of venues
in Chicago’s Logan Square neighborhood. This event will feature sessions
in AdTech, FinTech and Insurance, startups, “Made in Chicago”, Project
Shark Tank (innovations from the Apache Incubator), community diversity,
and more. It’s a great way to learn about various Apache projects “at
work” while playing at a brewery, a beercade, and a neighborhood bar.
Sign up today at https://www.apachecon.com/chiroadshow19/

We’re delighted to announce that the Call for Presentations (CFP) is now
open for ApacheCon North America in Las Vegas, September 9-13th! As the
official conference series of the ASF, ApacheCon North America will
feature over a dozen Apache project summits, including Cassandra,
Cloudstack, Tomcat, Traffic Control, and more. We’re looking for talks
in a wide variety of categories -- anything related to ASF projects and
the Apache development process. The CFP closes at midnight on May 26th.
In addition, the ASF will be celebrating its 20th Anniversary during the
event. For more details and to submit a proposal for the CFP, visit
https://apachecon.com/acna19/ . Registration will be opening soon.

Be sure to mark your calendars for ApacheCon Europe, which will be held
in Berlin, October 22-24th at the KulturBrauerei, a landmark of Berlin's
industrial history. In addition to innovative content from our projects,
we are collaborating with the Open Source Design community
(https://opensourcedesign.net/) to offer a track on design this year.
The CFP and registration will open soon at https://apachecon.com/aceu19/ .

Sponsorship opportunities are available for all events, with details
listed on each event’s site at http://apachecon.com/.

We look forward to seeing you!

Rich, for the ApacheCon Planners
@apachecon

Re: DL LogReader.readNext blocked forever

[Sijie Guo]

On Tue, Feb 26, 2019 at 6:37 PM Lothruin Mirwen 
wrote:

> Hi BookKeepers!
>
> I have an issue with DL LogReader.readNext(false) [ie: blocking]
>
> Such call should "block until return a record if there are records in the
> stream (aka catching up). Otherwise it would wait up to {@link
> DistributedLogConfiguration#getReadAheadWaitTime()} milliseconds and return
> null if there isn't any more records in the stream." (from javadoc).
>
> It normally works but with empty logs (no data ever written to DL/BK) it
> blocks forever. It seems that with no data it never "catchUp" and wait
> forever.
>

How do you create the empty logs? Did you just open a log writer without
writing records, or you use Namespace to create a log?

The difference is 1) first approach will have a log segment with zero
entries; 2) second approach might not even have any log segments.

I guess the difference between 1) and 2) cause the differences at handling
`readNext(false)`. If you can share more details about your setup, I have a
better idea of the direction at looking into the problem.


> As workaround I forcefully write a dummy record into the log. I'm missing
> something? Is there any missing configuration I need to provide to BK?
>
> Best regards
>
> Diego Salvi
>

Re: Community Meeting - 02/21/2019

[Sijie Guo]

Hi all,

I have archived the meeting notes:
https://cwiki.apache.org/confluence/display/BOOKKEEPER/2019-02-21+Meeting+Notes

I also moved the AI (action item) to next meeting agenda, so we can follow
up with those action items.

https://docs.google.com/document/d/1dEybUcT_D9Tdjr_H5Bu-QXm6IeDkRsEzypAjNXgfK7Y/edit#

Let me know if you have any comments or questions. Feel free to leave the
comments at google doc directly as well.

- Sijie



On Wed, Feb 20, 2019 at 8:39 PM Sijie Guo  wrote:

> Hi all,
>
> We will be resuming our regular bookkeeper community meeting starting this
> week.
>
> The meeting will be held at 8am PST on 02/21/2019.
>
> I am using a google doc for tracking the meeting notes. Please note the
> google doc is used for collaboration during the meeting process. All the
> meeting notes will be archived at BookKeeper confluence page according to
> ASF policy.
>
>
> https://docs.google.com/document/d/1dEybUcT_D9Tdjr_H5Bu-QXm6IeDkRsEzypAjNXgfK7Y/edit#
>
> Please checkout the google doc and add the items you wish to discuss at
> the end of agenda. The google doc is editable by everyone.
>
> Let me know if you have any questions/suggestions.
>
> Thanks,
> Sijie
>

Community Meeting - 02/21/2019

[Sijie Guo]

Hi all,

We will be resuming our regular bookkeeper community meeting starting this
week.

The meeting will be held at 8am PST on 02/21/2019.

I am using a google doc for tracking the meeting notes. Please note the
google doc is used for collaboration during the meeting process. All the
meeting notes will be archived at BookKeeper confluence page according to
ASF policy.

https://docs.google.com/document/d/1dEybUcT_D9Tdjr_H5Bu-QXm6IeDkRsEzypAjNXgfK7Y/edit#

Please checkout the google doc and add the items you wish to discuss at the
end of agenda. The google doc is editable by everyone.

Let me know if you have any questions/suggestions.

Thanks,
Sijie

Re: Community Meetings 2019

[Sijie Guo]

Hi all,

I have removed the old invitation and created new invitation on Apache
BookKeeper Community Calendar.

We will have two schedules in a monthly basis, one at 8 am PST and the
other one at 4 pm PST. I have also sent the invitation to dev@ mailing list.

You can check the "community meeting" page -
http://bookkeeper.apache.org/community/meeting/

Also I created a zoom meeting invitation. You can find all the details of
zoom meeting in calendar invitation. (zoom has better connectivities in
China).

Let me know if you have any questions.

Thanks,
Sijie

On Sun, Feb 17, 2019 at 3:18 PM Enrico Olivelli  wrote:

> Il giorno dom 17 feb 2019, 01:06 Sijie Guo  ha
> scritto:
>
> > > Other proposal is to setup
> > > two meetings in a month, one is convenient for US
> > > and other is convenient for Asia/Europe.
> >
> > I think that is a good proposal.
> >
> > We can do one at 8 am PST (Thursday) as before and one at 4 pm PST
> > (Thursday).
> > Let me know how do you guys think.
> >
> > Also I would suggest using zoom instead of google hangouts. Zoom has
> better
> > meeting quality
> > and be able to record meetings. If that's good to everyone, I can set one
> > up.
> >
>
>
> Sounds good.
> Can you update the original invitation and the website?
>
> Enrico
>
>
> > Thanks,
> > Sijie
> >
> > On Tue, Feb 12, 2019 at 1:11 AM Enrico Olivelli 
> > wrote:
> >
> > > Good.
> > >
> > > So we can 'meet' next week at 8 a.m. PST.
> > >
> > >
> > > The idea of weekly triage meeting works for me.
> > >
> > > Enrico
> > >
> > >
> > >
> > >
> > > Il giorno lun 11 feb 2019, 08:29 Venkateswara Rao Jujjuri <
> > > jujj...@gmail.com>
> > > ha scritto:
> > >
> > > > Having one standard time is good. But there needs to be participation
> > > too,
> > > > otherwise it won't be successful.
> > > > So lets find a good time that everyone can join. Other proposal is to
> > > setup
> > > > two meetings in a month, one is convenient for US
> > > > and other is convenient for Asia/Europe. Personally I am fine to take
> > > call
> > > > in the night if it is not between 6-9PM PST.
> > > > In the morning I am fine to take early calls even at 7 AM. I am just
> > > > stating my availability, and the goal should be larger participation
> > IMO.
> > > >
> > > > JV
> > > >
> > > > On Sun, Feb 10, 2019 at 9:14 PM Sijie Guo 
> wrote:
> > > >
> > > > > I think it is very difficult to find a good time slot for US,
> Europe
> > > and
> > > > > Asia.
> > > > >
> > > > > I would propose followings:
> > > > >
> > > > > - keep the regular community meetings at 8AM PST and make sure all
> > > > meeting
> > > > > notes are updated and shared via mailing lists. Release manager
> > should
> > > > > drive the community meeting.
> > > > > - additionally, if there are topics required ad-hoc calls, we can
> set
> > > up
> > > > > ad-hoc calls for the people required to be involved in the call.
> Also
> > > > make
> > > > > sure all meeting notes are shared to the community.
> > > > > - lastly, Ivan and me have been doing bug triages for Pulsar. We
> > might
> > > be
> > > > > starting doing same things for BookKeeper. It will be running
> weekly.
> > > We
> > > > > will document the bug triage process and share the meeting details.
> > > > > Everyone are recommended to join the triage as well.
> > > > >
> > > > > Let me know your thoughts.
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Fri, Feb 8, 2019 at 4:10 PM Enrico Olivelli <
> eolive...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > > I think it is time to resume Community Meetings in 2019.
> > > > > >
> > > > > > Currently we are running the meeting at 8 AM PST, every two
> weeks.
> > > > > >
> > > > > > We can decide to change the date, if we find it uncomfortable.
> > > > > >
> > > > > > Jia and I were proposing to have a variable time for the call,
> > > > > > depending on the needs or depending from the person who "calls"
> > for a
> > > > > > specific topic.
> > > > > >
> > > > > > I see that that time is not good for Jia and people from China.
> > > > > >
> > > > > > The goals of the community meetings are:
> > > > > > - Let community members 'meet' each other
> > > > > > - Discuss proposals
> > > > > > - Discuss about problems
> > > > > > - Share our projects/usages of BookKeeper/DL
> > > > > >
> > > > > > Every decision in ASF must be taken on ML, but we are humans and
> > > > > > humans "like" to talk and I find this meetings an important way
> of
> > > > > > "making community"
> > > > > >
> > > > > > Best regards
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Jvrao
> > > > ---
> > > > First they ignore you, then they laugh at you, then they fight you,
> > then
> > > > you win. - Mahatma Gandhi
> > > >
> > >
> >
>

Re: How to handle stale metadata?

[Sijie Guo]

On Tue, Feb 19, 2019 at 4:22 AM Venkateswara Rao Jujjuri 
wrote:

> Recently we ran into a situation where the LedgerMetadataListener never
> returned/detected metadata change. Due to this reader had stale metadata
> and tried to read from bookies that no longer have that ledger, hence
> NoSuchLedgerExistsException was returned to the caller.
>

Do the bookies have entries? Or does it change ensemble to cause an empty
fragment?
Those bookies respond NoSuchLedgerExistsException when talking to the empty
fragment.
Is that the case? I am not sure how NoSuchLedgerExistsException can be
propagated to the client.

Can you describe the sequence on how this happened?


>
> 1. I wonder if NoSuchLedgerExistsException is the right error here?


>- Client knows that the ledger exists in the metadata. It has valid
>handle. So ledger *Exists*.
>-  In this case it is stale metadata so a restart of client took care of
>the situation. But what if the ledger is in ZK, but missing from all
>bookies? This can be a durability or availability issue based on the
>bookies in the metadata are part of the cluster or not.
>- I think we need to have more sophisticated error handling here.
>Comments?
>





>
> 2. Having too many watches puts memory pressure on the client.
>
>- How about having an option to re-read the metadata on demand w/o
> watch?
>

+1 this has been in my todo list for a while. we should provide options to
do this either by watches or by re-read scheduling or both.


>   - Schedule a task to reread metadata on the first bookie failure with
>   NoSuchEntry/NoSuchLedger.
>   - If all three bookies fail, wait for the outstanding metadata read
>   to return before failing to user.
>   - If the metadata is read, and is different from the local copy,
>   reattempt the read.
>   - If metadata is not different, then fail with "some new error"
>   DataLossException or something?
>- This can cause latency if the metadata is changing a lot, but may be
>better than constant watches? It could be a configuration option.
>- We could even think of having both enabled if the reader is super
>conservative.
>
>
> Thoughts?
> JV
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>

Re: Community Meetings 2019

[Sijie Guo]

> Other proposal is to setup
> two meetings in a month, one is convenient for US
> and other is convenient for Asia/Europe.

I think that is a good proposal.

We can do one at 8 am PST (Thursday) as before and one at 4 pm PST
(Thursday).
Let me know how do you guys think.

Also I would suggest using zoom instead of google hangouts. Zoom has better
meeting quality
and be able to record meetings. If that's good to everyone, I can set one
up.

Thanks,
Sijie

On Tue, Feb 12, 2019 at 1:11 AM Enrico Olivelli  wrote:

> Good.
>
> So we can 'meet' next week at 8 a.m. PST.
>
>
> The idea of weekly triage meeting works for me.
>
> Enrico
>
>
>
>
> Il giorno lun 11 feb 2019, 08:29 Venkateswara Rao Jujjuri <
> jujj...@gmail.com>
> ha scritto:
>
> > Having one standard time is good. But there needs to be participation
> too,
> > otherwise it won't be successful.
> > So lets find a good time that everyone can join. Other proposal is to
> setup
> > two meetings in a month, one is convenient for US
> > and other is convenient for Asia/Europe. Personally I am fine to take
> call
> > in the night if it is not between 6-9PM PST.
> > In the morning I am fine to take early calls even at 7 AM. I am just
> > stating my availability, and the goal should be larger participation IMO.
> >
> > JV
> >
> > On Sun, Feb 10, 2019 at 9:14 PM Sijie Guo  wrote:
> >
> > > I think it is very difficult to find a good time slot for US, Europe
> and
> > > Asia.
> > >
> > > I would propose followings:
> > >
> > > - keep the regular community meetings at 8AM PST and make sure all
> > meeting
> > > notes are updated and shared via mailing lists. Release manager should
> > > drive the community meeting.
> > > - additionally, if there are topics required ad-hoc calls, we can set
> up
> > > ad-hoc calls for the people required to be involved in the call. Also
> > make
> > > sure all meeting notes are shared to the community.
> > > - lastly, Ivan and me have been doing bug triages for Pulsar. We might
> be
> > > starting doing same things for BookKeeper. It will be running weekly.
> We
> > > will document the bug triage process and share the meeting details.
> > > Everyone are recommended to join the triage as well.
> > >
> > > Let me know your thoughts.
> > >
> > > - Sijie
> > >
> > > On Fri, Feb 8, 2019 at 4:10 PM Enrico Olivelli 
> > > wrote:
> > >
> > > > Hi folks,
> > > > I think it is time to resume Community Meetings in 2019.
> > > >
> > > > Currently we are running the meeting at 8 AM PST, every two weeks.
> > > >
> > > > We can decide to change the date, if we find it uncomfortable.
> > > >
> > > > Jia and I were proposing to have a variable time for the call,
> > > > depending on the needs or depending from the person who "calls" for a
> > > > specific topic.
> > > >
> > > > I see that that time is not good for Jia and people from China.
> > > >
> > > > The goals of the community meetings are:
> > > > - Let community members 'meet' each other
> > > > - Discuss proposals
> > > > - Discuss about problems
> > > > - Share our projects/usages of BookKeeper/DL
> > > >
> > > > Every decision in ASF must be taken on ML, but we are humans and
> > > > humans "like" to talk and I find this meetings an important way of
> > > > "making community"
> > > >
> > > > Best regards
> > > >
> > > > Enrico
> > > >
> > >
> >
> >
> > --
> > Jvrao
> > ---
> > First they ignore you, then they laugh at you, then they fight you, then
> > you win. - Mahatma Gandhi
> >
>

Re: Community Meetings 2019

[Sijie Guo]

I think it is very difficult to find a good time slot for US, Europe and
Asia.

I would propose followings:

- keep the regular community meetings at 8AM PST and make sure all meeting
notes are updated and shared via mailing lists. Release manager should
drive the community meeting.
- additionally, if there are topics required ad-hoc calls, we can set up
ad-hoc calls for the people required to be involved in the call. Also make
sure all meeting notes are shared to the community.
- lastly, Ivan and me have been doing bug triages for Pulsar. We might be
starting doing same things for BookKeeper. It will be running weekly. We
will document the bug triage process and share the meeting details.
Everyone are recommended to join the triage as well.

Let me know your thoughts.

- Sijie

On Fri, Feb 8, 2019 at 4:10 PM Enrico Olivelli  wrote:

> Hi folks,
> I think it is time to resume Community Meetings in 2019.
>
> Currently we are running the meeting at 8 AM PST, every two weeks.
>
> We can decide to change the date, if we find it uncomfortable.
>
> Jia and I were proposing to have a variable time for the call,
> depending on the needs or depending from the person who "calls" for a
> specific topic.
>
> I see that that time is not good for Jia and people from China.
>
> The goals of the community meetings are:
> - Let community members 'meet' each other
> - Discuss proposals
> - Discuss about problems
> - Share our projects/usages of BookKeeper/DL
>
> Every decision in ASF must be taken on ML, but we are humans and
> humans "like" to talk and I find this meetings an important way of
> "making community"
>
> Best regards
>
> Enrico
>

Re: ExplicitLAC consolidation

[Sijie Guo]

On Mon, Feb 11, 2019 at 6:09 AM Venkateswara Rao Jujjuri 
wrote:

> Thanks for bringing this up. To start with, can we put all forms on LAC in
> a doc? i.e we have more than one way to read and write LAC.
> 1. What are the ways to send LAC to Bookies?
> 2. What are the ways bookies store the LAC?
> 3. What are the ways client can learn about LAC
> and the interfaces and configuration parameters around it, and what are the
> usecases of these?
>

+1 I would also like to see a doc (via BP on google doc) for this as well.

>
> I may be asking for more work here, but I think that way we can make more
> informed decision. What do you think Enrico?
> If you lead this, I will be more than happy to fill-in parts of this doc,
> and even review them.
>
> Thanks,
> JV
>
> On Sun, Feb 10, 2019 at 1:23 AM Enrico Olivelli 
> wrote:
>
> > Hi Bookkeepers,
> > I am trying to draw the best roadmap with the goal of consolidating
> > ExplicitLAC feature.
> > Currently I have two big topics:
> > - on the reader side I would like enable new API users to leverage
> > ExplicitLAC, transparently (no new configuration, now readflags, no
> > explicit API calls)
> > - on the writer side make ExplicitLAC work better with DEFERRED_SYNC
> (like
> > having a background force() together with the sending of ExplicitLAC)
> >
> > Currently I want to spend time mostly on the reader side, because it will
> > enable new clients to use ExplicitLAC.
> >
> > My current (new) idea is to add a new flag on readEntry() RPC with which
> > the client asks for the ExplicitLAC together with the entry.
> >
> > With this change we can support backward compatibility easily.
> > New clients will add that flag and they will be able to read the new
> > optional ExplicitLAC field.
> > Old bookies will ignore the flag.
> > Old clients won't ask for ExplicitLAC.
> >
> > If there is no ExplicitLAC this new feature won't add costs on the wire.
> >
> > After this discussion on the ML I will post a BP, if we agree that this
> > approach makes sense.
> >
> > I have started to draft a prototype yet, I would like to hear your
> opinion
> > (as usual I have very limited time)
> >
> > Regards
> > Enrico
> >
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>

Re: [DRAFT] BookKeeper Board report of Feb.

[Sijie Guo]

FYI. I have submitted the board report.

Thanks Jia for drafting the report and all the reviewers reviewing the
report.

Great community work!

- Sijie

On Fri, Feb 8, 2019 at 8:13 AM Sijie Guo  wrote:

> Good job, Jia. +1
>
> On Thu, Feb 7, 2019 at 4:11 PM Jia Zhai  wrote:
>
>> Hi all,
>>
>> Here is a draft for BookKeeper Board report of Feb. Please take a look.
>>
>> -
>>
>>
>> ## Description:
>>
>>
>> BookKeeper is a scalable, fault-tolerant, and low-latency storage
>>
>> service optimized for append-only workloads. It has been used as
>>
>> a fundamental service to build high available and replicated services
>>
>> in companies like Twitter, Yahoo and Salesforce. It is also the log
>>
>> segment store for Apache DistributedLog and message store for Apache
>> Pulsar.
>>
>>
>> Apache DistributedLog is a high-level API and service layer for
>>
>> Apache BookKeeper, providing easier access to the BookKeeper
>>
>> primitives. It is a subproject of Apache BookKeeper.
>>
>>
>> ## Issues:
>>
>>
>> There are no issues requiring board attention at this time.
>>
>>
>> ## Activity:
>>
>>
>>  - 4.9.0 was released on Sat, Feb 6, 2019
>>
>>  - The growth of Apache Pulsar community also help grow the adoption
>>
>> of BookKeeper. This helps building the ecosystem around BookKeeper.
>>
>>
>> ## Health report:
>>
>>
>>  - Development continues at a steady pace. We are merging multiple PRs per
>>
>> week on average.
>>
>>  - Mailing list and slack discussions are brisk, in particularly around
>> the
>>
>> active projects.
>>
>>
>> ## PMC changes:
>>
>>
>>  - Currently 15 PMC members.
>>
>>  - No new PMC members added in the last 3 months
>>
>>  - Last PMC addition was Enrico Olivelli on Fri Feb 23 2018
>>
>>
>> ## Committer base changes:
>>
>>
>>  - Currently 21 committers.
>>
>>  - No new committers added in the last 3 months
>>
>>  - Last committer addition was Andrey Yegorov at Fri Feb 09 2018
>>
>>
>> ## Releases:
>>
>>
>>  - 4.9.0 was released on Sat, Feb 6, 2019
>>
>>  - 4.7.3 was released on Fri, Dec 7, 2018
>>
>>  - 4.8.1 was released on Fri, Nov 30, 2018
>>
>>
>> ## Mailing list activity:
>>
>>
>>  - Discussions mostly run on dev@ mailing list, distributedlog lists
>>
>> are mostly unused, now that the project as been merged completely with
>>
>> BookKeeper and users started to use the version bundled with
>>
>> BookKeeper releases.
>>
>>
>>
>>  - dev@bookkeeper.apache.org:
>>
>> - 105 subscribers (up 2 in the last 3 months):
>>
>> - 119 emails sent to list (121 in previous quarter)
>>
>>
>>
>>  - distributedlog-comm...@bookkeeper.apache.org:
>>
>> - 12 subscribers (up 0 in the last 3 months):
>>
>> - 0 emails sent to list (0 in previous quarter)
>>
>>
>>
>>  - distributedlog-...@bookkeeper.apache.org:
>>
>> - 41 subscribers (up 0 in the last 3 months):
>>
>> - 0 emails sent to list (0 in previous quarter)
>>
>>
>>
>>  - distributedlog-iss...@bookkeeper.apache.org:
>>
>> - 9 subscribers (up 0 in the last 3 months):
>>
>> - 2 emails sent to list (4 in previous quarter)
>>
>>
>>
>>  - distributedlog-u...@bookkeeper.apache.org:
>>
>> - 26 subscribers (up 0 in the last 3 months):
>>
>> - 0 emails sent to list (0 in previous quarter)
>>
>>
>>
>>  - iss...@bookkeeper.apache.org:
>>
>> - 8 subscribers (up 0 in the last 3 months):
>>
>> - 1994 emails sent to list (2433 in previous quarter)
>>
>>
>>
>>  - u...@bookkeeper.apache.org:
>>
>> - 118 subscribers (up 3 in the last 3 months):
>>
>> - 14 emails sent to list (17 in previous quarter)
>>
>>
>>
>>
>> -
>>
>>
>>
>>
>> Best Regards.
>>
>>
>> Jia Zhai
>>
>> Beijing, China
>>
>> Mobile: 15810491983
>>
>

Re: [DISCUSS] Release 4.10.0 Planning

[Sijie Guo]

I have also updated the release schedule :
http://bookkeeper.apache.org/community/releases/



On Sun, Feb 3, 2019 at 8:33 PM Enrico Olivelli  wrote:

> New items:
> - make our binary packages work with jdk11
> - consider moving docker images to jdk11
>
> Enrico
>
> Il giorno sab 2 feb 2019, 01:08 Sijie Guo  ha scritto:
>
> > Thanks, Enrico.
> >
> > I have sent out a PR to update our release schedule page.
> > https://github.com/apache/bookkeeper/pull/1932
> >
> > - Sijie
> >
> > On Sun, Jan 27, 2019 at 3:30 PM Enrico Olivelli 
> > wrote:
> >
> > > Il giorno dom 27 gen 2019, 06:00 Sijie Guo  ha
> > scritto:
> > >
> > > > Hi all,
> > > >
> > > > Since we are in the progress of releasing 4.9.0, I think it is a good
> > > time
> > > > to start the discussion of what we are going to do for 4.10.0.
> > > >
> > > > Here are a few list of tasks I have in my mind:
> > > >
> > > > - Complete a few ongoing BPs
> > > >   * BP-27: New BookKeeper CLI `bkctl`.
> > > >   * BP-36: Stats annotations
> > > >   * BP-37: Configuration management
> > > > - Code refactoring
> > > >   * Split the bookkeeper-server module. Move common stuffs to
> `common`,
> > > > split client and server into separate modules
> > > > - New Proposals
> > > >   * Use UUID for bookie registration instead of ip to improve
> > bookkeeper
> > > > usability in cloud environments
> > > >   * Thin client without interacting with zookeeper
> > > >
> > > > Feel free to comment with the tasks you would like to include in
> > 4.10.0.
> > > >
> > >
> > > Great agenda.
> > > I would like to move forward the work I am doing around Explicit LAC:
> > > - let new API users use ExplicitLAC on the reader side
> > > - integrate ExplicitLAC with force() on the writer side
> > >
> > >
> > > Is it time for a hangout?
> > >
> > > Enrico
> > >
> > >
> > > > - Sijie
> > > >
> > >
> >
>

Re: [DRAFT] BookKeeper Board report of Feb.

[Sijie Guo]

Good job, Jia. +1

On Thu, Feb 7, 2019 at 4:11 PM Jia Zhai  wrote:

> Hi all,
>
> Here is a draft for BookKeeper Board report of Feb. Please take a look.
>
> -
>
>
> ## Description:
>
>
> BookKeeper is a scalable, fault-tolerant, and low-latency storage
>
> service optimized for append-only workloads. It has been used as
>
> a fundamental service to build high available and replicated services
>
> in companies like Twitter, Yahoo and Salesforce. It is also the log
>
> segment store for Apache DistributedLog and message store for Apache
> Pulsar.
>
>
> Apache DistributedLog is a high-level API and service layer for
>
> Apache BookKeeper, providing easier access to the BookKeeper
>
> primitives. It is a subproject of Apache BookKeeper.
>
>
> ## Issues:
>
>
> There are no issues requiring board attention at this time.
>
>
> ## Activity:
>
>
>  - 4.9.0 was released on Sat, Feb 6, 2019
>
>  - The growth of Apache Pulsar community also help grow the adoption
>
> of BookKeeper. This helps building the ecosystem around BookKeeper.
>
>
> ## Health report:
>
>
>  - Development continues at a steady pace. We are merging multiple PRs per
>
> week on average.
>
>  - Mailing list and slack discussions are brisk, in particularly around the
>
> active projects.
>
>
> ## PMC changes:
>
>
>  - Currently 15 PMC members.
>
>  - No new PMC members added in the last 3 months
>
>  - Last PMC addition was Enrico Olivelli on Fri Feb 23 2018
>
>
> ## Committer base changes:
>
>
>  - Currently 21 committers.
>
>  - No new committers added in the last 3 months
>
>  - Last committer addition was Andrey Yegorov at Fri Feb 09 2018
>
>
> ## Releases:
>
>
>  - 4.9.0 was released on Sat, Feb 6, 2019
>
>  - 4.7.3 was released on Fri, Dec 7, 2018
>
>  - 4.8.1 was released on Fri, Nov 30, 2018
>
>
> ## Mailing list activity:
>
>
>  - Discussions mostly run on dev@ mailing list, distributedlog lists
>
> are mostly unused, now that the project as been merged completely with
>
> BookKeeper and users started to use the version bundled with
>
> BookKeeper releases.
>
>
>
>  - dev@bookkeeper.apache.org:
>
> - 105 subscribers (up 2 in the last 3 months):
>
> - 119 emails sent to list (121 in previous quarter)
>
>
>
>  - distributedlog-comm...@bookkeeper.apache.org:
>
> - 12 subscribers (up 0 in the last 3 months):
>
> - 0 emails sent to list (0 in previous quarter)
>
>
>
>  - distributedlog-...@bookkeeper.apache.org:
>
> - 41 subscribers (up 0 in the last 3 months):
>
> - 0 emails sent to list (0 in previous quarter)
>
>
>
>  - distributedlog-iss...@bookkeeper.apache.org:
>
> - 9 subscribers (up 0 in the last 3 months):
>
> - 2 emails sent to list (4 in previous quarter)
>
>
>
>  - distributedlog-u...@bookkeeper.apache.org:
>
> - 26 subscribers (up 0 in the last 3 months):
>
> - 0 emails sent to list (0 in previous quarter)
>
>
>
>  - iss...@bookkeeper.apache.org:
>
> - 8 subscribers (up 0 in the last 3 months):
>
> - 1994 emails sent to list (2433 in previous quarter)
>
>
>
>  - u...@bookkeeper.apache.org:
>
> - 118 subscribers (up 3 in the last 3 months):
>
> - 14 emails sent to list (17 in previous quarter)
>
>
>
>
> -
>
>
>
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: 15810491983
>

[ANNOUNCE] Apache BookKeeper 4.9.0 released

[Sijie Guo]

The Apache BookKeeper team is proud to announce Apache BookKeeper version
4.9.0.

Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage
service optimized for
real-time workloads. It has been used for a fundamental service to build
reliable services.
It is also the log segment store for Apache DistributedLog and the message
store for Apache Pulsar.

This is the 16th release of the Apache BookKeeper.

The 4.9.0 release incorporates hundreds of bug fixes, improvements,
and features since previous major release, 4.8.0, which was released four
months ago.
It is a new milestone in Apache BookKeeper community.

There are a few big changes around metadata management, such as refactoring
ledger metatadata
in ledger handle to make it immutable, storing ledger metadata in binary
format and a new metadata
driver implementation based on Etcd. Additionally, there are huge
improvements around memory
management, tooling, and documentation.

For BookKeeper release details and downloads, visit:

http://bookkeeper.apache.org/releases/

BookKeeper 4.9.0 Release Notes are at:

http://bookkeeper.apache.org/docs/4.9.0/overview/releaseNotes/

We would like to thank the contributors that made the release possible.

Regards,

The BookKeeper Team

Re: [DISCUSS] Release 4.10.0 Planning

[Sijie Guo]

Thanks, Enrico.

I have sent out a PR to update our release schedule page.
https://github.com/apache/bookkeeper/pull/1932

- Sijie

On Sun, Jan 27, 2019 at 3:30 PM Enrico Olivelli  wrote:

> Il giorno dom 27 gen 2019, 06:00 Sijie Guo  ha scritto:
>
> > Hi all,
> >
> > Since we are in the progress of releasing 4.9.0, I think it is a good
> time
> > to start the discussion of what we are going to do for 4.10.0.
> >
> > Here are a few list of tasks I have in my mind:
> >
> > - Complete a few ongoing BPs
> >   * BP-27: New BookKeeper CLI `bkctl`.
> >   * BP-36: Stats annotations
> >   * BP-37: Configuration management
> > - Code refactoring
> >   * Split the bookkeeper-server module. Move common stuffs to `common`,
> > split client and server into separate modules
> > - New Proposals
> >   * Use UUID for bookie registration instead of ip to improve bookkeeper
> > usability in cloud environments
> >   * Thin client without interacting with zookeeper
> >
> > Feel free to comment with the tasks you would like to include in 4.10.0.
> >
>
> Great agenda.
> I would like to move forward the work I am doing around Explicit LAC:
> - let new API users use ExplicitLAC on the reader side
> - integrate ExplicitLAC with force() on the writer side
>
>
> Is it time for a hangout?
>
> Enrico
>
>
> > - Sijie
> >
>

[RESULT] [VOTE] Release 4.9.0, release candidate #1

[Sijie Guo]

Hi all,

I am happy to announce the vote for 4.9.0 rc1 has passed with 5 +1 binding
votes and 0 -1 votes.

5 +1 bindings are:

Enrico Olivelli
Ivan Kelly
Jia Zhai
Matteo Merli
Sijie Guo

I will do the remaining tasks for completing 4.9.0 release.

Thanks,
Sijie

On Sun, Jan 27, 2019 at 12:38 PM Sijie Guo  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 4.9.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.9.0-rc1" [4] with git sha
> b0e3adfea04e7e65512cca54e8b63d197abf910c
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/1910
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.0-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1038/
> [4] https://github.com/apache/bookkeeper/tree/v4.9.0-rc1
>

Re: [VOTE] Release 4.9.0, release candidate #1

[Sijie Guo]

Include my +1 here as well.

- Sijie

On Sun, Jan 27, 2019 at 12:38 PM Sijie Guo  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 4.9.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.9.0-rc1" [4] with git sha
> b0e3adfea04e7e65512cca54e8b63d197abf910c
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/bookkeeper/pull/1910
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.0-rc1/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1038/
> [4] https://github.com/apache/bookkeeper/tree/v4.9.0-rc1
>

[DISCUSS] Release 4.10.0 Planning

[Sijie Guo]

Hi all,

Since we are in the progress of releasing 4.9.0, I think it is a good time
to start the discussion of what we are going to do for 4.10.0.

Here are a few list of tasks I have in my mind:

- Complete a few ongoing BPs
  * BP-27: New BookKeeper CLI `bkctl`.
  * BP-36: Stats annotations
  * BP-37: Configuration management
- Code refactoring
  * Split the bookkeeper-server module. Move common stuffs to `common`,
split client and server into separate modules
- New Proposals
  * Use UUID for bookie registration instead of ip to improve bookkeeper
usability in cloud environments
  * Thin client without interacting with zookeeper

Feel free to comment with the tasks you would like to include in 4.10.0.

- Sijie

[VOTE] Release 4.9.0, release candidate #1

[Sijie Guo]

Hi everyone,

Please review and vote on the release candidate #1 for the version 4.9.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* Release notes [1]
* The official Apache source and binary distributions to be deployed to
dist.apache.org [2]
* All artifacts to be deployed to the Maven Central Repository [3]
* Source code tag "v4.9.0-rc1" [4] with git sha
b0e3adfea04e7e65512cca54e8b63d197abf910c

BookKeeper's KEYS file contains PGP keys we used to sign this release:
https://dist.apache.org/repos/dist/release/bookkeeper/KEYS

Please download these packages and review this release candidate:

- Review release notes
- Download the source package (verify shasum, and asc) and follow the
instructions to build and run the bookkeeper service.
- Download the binary package (verify shasum, and asc) and follow the
instructions to run the bookkeeper service.
- Review maven repo, release tag, licenses, and any other things you think
it is important to a release.

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] https://github.com/apache/bookkeeper/pull/1910
[2] https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.0-rc1/
[3]
https://repository.apache.org/content/repositories/orgapachebookkeeper-1038/
[4] https://github.com/apache/bookkeeper/tree/v4.9.0-rc1

[Cancel] [VOTE] Release 4.9.0, release candidate #0

[Sijie Guo]

Cancel rc0.

Will send out rc1 soon.

- Sijie

On Mon, Jan 21, 2019 at 6:27 PM Sijie Guo  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #0 for the version 0.4.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * Release notes [1]
> * The official Apache source and binary distributions to be deployed to
> dist.apache.org [2]
> * All artifacts to be deployed to the Maven Central Repository [3]
> * Source code tag "v4.9.0-rc0" [4] with git sha
> fc513dc3cb5817e5516aa0fd26115b35cc84b24a
>
> BookKeeper's KEYS file contains PGP keys we used to sign this release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>
> Please download these packages and review this release candidate:
>
> - Review release notes
> - Download the source package (verify shasum, and asc) and follow the
> instructions to build and run the bookkeeper service.
> - Download the binary package (verify shasum, and asc) and follow the
> instructions to run the bookkeeper service.
> - Review maven repo, release tag, licenses, and any other things you think
> it is important to a release.
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Sijie Guo
>
> [1] https://github.com/apache/bookkeeper/pull/1910
> [2]
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.9.0-rc0/
> [3]
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1037/
> [4] https://github.com/apache/bookkeeper/tree/v4.9.0-rc0
>

Re: EnsemblePlacementPolicy exposes third party API "Pair" from commons-lang3 in a public API

[Sijie Guo]

On Tue, Jan 22, 2019 at 8:40 AM Enrico Olivelli  wrote:

> Hi all,
> while reviewing 4.9 release I found this problem around a change about
> EnsemblePlacementPolicy
>
> this is the issue
> https://github.com/apache/bookkeeper/issues/1914
>
> The problem is that in public API we should not use third party
> classes in order to preserve compatibility with incompatible changes
> of the third party library.
>
> We already had such problems in the past.
>
> I think the best way to address this problem is to introduce one
> specific class in BookKeeper, maybe an inner class of
> EnsemblePlacementPolicy.
>

why not just introduce a Pair like class in bookkeeper-common module?
instead of an inner class for EnsemblePlacementPolicy.


>
> If we agree on this solution I can send the patch, it is very
> straightforward
>
> Regards
> Enrico
>

Re: Bookkeeper 4.9...

[Sijie Guo]

Sure. We can do the documentation and release in parallel.

Sijie

On Tue, Jan 15, 2019 at 2:30 PM Enrico Olivelli  wrote:

> Awesome.
>
>
> I have created another important task for the new metadaserviceuri
> configuration
>
> https://github.com/apache/bookkeeper/issues/1897
>
> We can release 4.9 without such docs by if someone in the community has
> some cycle it will be a great step forward for the adoption of this new
> features
>
> Cheers
>
>
> Enrico
>
>
>
> Il mar 15 gen 2019, 03:19 Sijie Guo  ha scritto:
>
> > As discussed in the pull request, the ExplicitLAC related change will go
> to
> > 4.10.
> >
> > I have cleaned up all the 4.9.0 issues. The pending issue for 4.9.0 is
> the
> > binary metadata format.
> > Once that is merged, I will start the release for 4.9.0.
> >
> > - Sijie
> >
> > On Fri, Dec 28, 2018 at 5:43 PM Enrico Olivelli 
> > wrote:
> >
> > > I would like to include transparent support for ExplicitLAC on the
> reader
> > > side.
> > > https://github.com/apache/bookkeeper/pull/1572
> > >
> > > I have to finalize the patch by cleaning up (re-adding) tests but
> > > before spending time in a hurry I would like to know if the idea as
> > > enough consensus to get on 4.9, otherwise I can work on it for 4.10.
> > > It is a client side change, no read protocol changes, which I would
> > > like to postpone to future releases.
> > >
> > > Enrico
> > >
> > > Il giorno gio 27 dic 2018 alle ore 11:04 Enrico Olivelli
> > >  ha scritto:
> > > >
> > > > Sijie,
> > > > If you prefer I can take this.
> > > > Most of the work is from you and Ivan
> > > >
> > > > No problem for me.
> > > > Thank you
> > > > Enrico
> > > >
> > > > Il gio 27 dic 2018, 10:53 Sijie Guo  ha scritto:
> > > >>
> > > >> Since no one volunteers for being release manager, I will take this
> > > round
> > > >> then.
> > > >>
> > > >> - Sijie
> > > >>
> > > >> On Sat, Dec 22, 2018 at 2:13 PM Enrico Olivelli <
> eolive...@gmail.com>
> > > wrote:
> > > >>
> > > >> > Hi Bookkeepers !
> > > >> > Any plan to volunteer for being release manager of 4.9 ?
> > > >> >
> > > >> > I will be happy to help any one who wants to try for the first
> > time...
> > > >> >
> > > >> > Cheers
> > > >> > Enrico
> > > >> > --
> > > >> >
> > > >> >
> > > >> > -- Enrico Olivelli
> > > >> >
> > > >
> > > > --
> > > >
> > > >
> > > > -- Enrico Olivelli
> > >
> >
> --
>
>
> -- Enrico Olivelli
>

Re: Bookkeeper 4.9...

[Sijie Guo]

As discussed in the pull request, the ExplicitLAC related change will go to
4.10.

I have cleaned up all the 4.9.0 issues. The pending issue for 4.9.0 is the
binary metadata format.
Once that is merged, I will start the release for 4.9.0.

- Sijie

On Fri, Dec 28, 2018 at 5:43 PM Enrico Olivelli  wrote:

> I would like to include transparent support for ExplicitLAC on the reader
> side.
> https://github.com/apache/bookkeeper/pull/1572
>
> I have to finalize the patch by cleaning up (re-adding) tests but
> before spending time in a hurry I would like to know if the idea as
> enough consensus to get on 4.9, otherwise I can work on it for 4.10.
> It is a client side change, no read protocol changes, which I would
> like to postpone to future releases.
>
> Enrico
>
> Il giorno gio 27 dic 2018 alle ore 11:04 Enrico Olivelli
>  ha scritto:
> >
> > Sijie,
> > If you prefer I can take this.
> > Most of the work is from you and Ivan
> >
> > No problem for me.
> > Thank you
> > Enrico
> >
> > Il gio 27 dic 2018, 10:53 Sijie Guo  ha scritto:
> >>
> >> Since no one volunteers for being release manager, I will take this
> round
> >> then.
> >>
> >> - Sijie
> >>
> >> On Sat, Dec 22, 2018 at 2:13 PM Enrico Olivelli 
> wrote:
> >>
> >> > Hi Bookkeepers !
> >> > Any plan to volunteer for being release manager of 4.9 ?
> >> >
> >> > I will be happy to help any one who wants to try for the first time...
> >> >
> >> > Cheers
> >> > Enrico
> >> > --
> >> >
> >> >
> >> > -- Enrico Olivelli
> >> >
> >
> > --
> >
> >
> > -- Enrico Olivelli
>

Re: Bookkeeper 4.9...

[Sijie Guo]

Since no one volunteers for being release manager, I will take this round
then.

- Sijie

On Sat, Dec 22, 2018 at 2:13 PM Enrico Olivelli  wrote:

> Hi Bookkeepers !
> Any plan to volunteer for being release manager of 4.9 ?
>
> I will be happy to help any one who wants to try for the first time...
>
> Cheers
> Enrico
> --
>
>
> -- Enrico Olivelli
>