Re: [VOTE] Release 4.14.2, release candidate #1

[Flavio Junqueira]

+1

- Verified signature and digest
- Built locally and with Pravega
- Ran unit tests
- Ran local smoke tests with standalone
- Checked NOTICE and LICENSE files

-Flavio

> On 20 Aug 2021, at 11:06, Yong Zhang  wrote:
> 
> +1
> 
> - verify the source package shasum, and asc
> - verify binary package shasum, and asc
> - run apache-rat check
> - run standalone service and do simpletest
> 
> On Wed, 18 Aug 2021 at 10:52, Yong Zhang  wrote:
> 
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version 4.14.2,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>> 
>> The complete staging area is available for your review, which includes:
>> * Release notes [1]
>> * The official Apache source and binary distributions to be deployed to
>> dist.apache.org [2]
>> * All artifacts to be deployed to the Maven Central Repository [3]
>> * Source code tag "release-4.5.0" [4] with git sha [5]
>> 
>> BookKeeper's KEYS file contains PGP keys we used to sign this release:
>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>> 
>> Please download these packages and review this release candidate:
>> 
>> - Review release notes
>> - Download the source package (verify shasum, and asc) and follow the
>> instructions to build and run the bookkeeper service.
>> - Download the binary package (verify shasum, and asc) and follow the
>> instructions to run the bookkeeper service.
>> - Review maven repo, release tag, licenses, and any other things you think
>> it is important to a release.
>> 
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Release Manager
>> 
>> [1] https://github.com/apache/bookkeeper/pull/2765
>> [2]
>> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.14.2-rc1/
>> [3]
>> https://repository.apache.org/content/repositories/orgapachebookkeeper-1057/
>> [4] https://github.com/apache/bookkeeper/releases/tag/v4.14.2-rc1
>> [5] 07214020bc0c4f46d6572328322a1f9db29fe87a
>> 

Re: [DISCUSS] Release 4.14.2

[Flavio Junqueira]

It sounds like there are more vulnerabilities that can be addressed with 
upgrades:

https://github.com/apache/bookkeeper/issues/2511 


Do we want to proceed with 4.14.2 and consider a 4.14.3 that addresses other 
vulnerabilities or try to address as many as we are aware of? I'm asking 
because I'm already seeing an RC out.

Thanks,
-Flavio

> On 17 Aug 2021, at 07:59, Sijie Guo  wrote:
> 
> +1
> 
> On Thu, Aug 12, 2021 at 11:59 PM Yong Zhang  wrote:
>> 
>> Hi,
>> 
>> We have changed the BouncyCastle at this PR
>> https://github.com/apache/bookkeeper/pull/2631,
>> which introduces an Incompatible issue. Detail:
>> https://github.com/apache/pulsar/issues/10937.
>> 
>> This also blocks the user upgrade their charts to pulsar 2.8.0
>> https://github.com/apache/pulsar-helm-chart/pull/130
>> 
>> We have fixed it by https://github.com/apache/bookkeeper/pull/2740,
>> so I want to start a new release of bookkeeper for unblocking the users.
>> 
>> If there are no objections, I'll move forward with the patch release.
>> 
>> Thanks,
>> Yong

Re: Running without journal

[Flavio Junqueira]

+1, it makes sense to enable bookies to run without duplicating IOs for entry 
data. I'm curious to see how you justify removing the ledger as opposed to 
removing the ledger storage and preserving the journal. I suspect that the 
random reads against the ledger storage matter more to you than the sequential 
writes, and you're possibly able to make it perform well enough with SSD and 
even NVMe drives.

I should wait for your write up rather than speculate. Looking forward to 
seeing the BP.

-Flavio  

> On 3 May 2021, at 16:52, Enrico Olivelli  wrote:
> 
> Il giorno lun 3 mag 2021 alle ore 16:30 Jack Vanlightly
> mailto:jvanligh...@splunk.com.invalid>> ha 
> scritto:
>> 
>> Hi all,
>> 
>> At Splunk we have defined and implemented changes to BookKeeper to allow
>> bookies to run without the journal. The motivation for this work is to
>> allow BookKeeper to be run with lower operating costs while still offering
>> decent data safety guarantees.
>> 
>> Before submitting the work as a PR we'd like to formalise the proposed
>> changes in a BP where we state our motivation, explain the protocol
>> changes, the work on formally verifying the proposal and be open to comment.
>> 
>> We'll create a BP this week if that sounds good to you all.
> 
> Great to hear that !
> 
> Thanks
> Enrico
> 
>> 
>> Thanks
>> Jack
>> 
>> --
>> *Jack Vanlightly*
>> Principal Software Engineer
>> Splunk Inc.
>> jvanligh...@splunk.com  > >
>> Barcelona

Re: Clean build and NOTICE (was: Re: [VOTE] Release 4.13.0, release candidate #0)

[Flavio Junqueira]

I ran another build overnight and this time it finished successfully. It does 
look like there are a few flaky tests.

As for whether the NOTICE file issue is a blocker, the ASF documentation is not 
very clear about the year in the copyright:

  https://infra.apache.org/licensing-howto.html

but from my past discussions with legal in this company and others, I get that 
it is important, especially when an issue becomes known. I'd rather be cautious 
and fix it before releasing, but I see a bunch of +1 votes already, and 
everything else I checked so far looks good. There are flaky tests, but they 
don't seem to be specific to this release. 

-Flavio

> On 24 Feb 2021, at 10:01, Henry Saputra  wrote:
> 
> Hmm I did not see those errors
> 
> Oh, and I have sent PR for the NOTICE file here:
> https://github.com/apache/bookkeeper/pull/2621
> 
> Wonder if they happen due to "flaky" tests
> 
> - Henry
> 
> On Tue, Feb 23, 2021 at 2:15 PM Flavio Junqueira  wrote:
> 
>> A couple of things:
>> 
>> 1- I haven't been able to get a clean build, I'll list the tests failures
>> I observed below.
>> 2- We need to update the year range in the NOTICE file to say 2021 instead
>> of 2020.
>> 
>> -Flavio
>> 
>> 
>> [ERROR]
>> org.apache.bookkeeper.client.SlowBookieTest.testSlowBookieAndFastFailOn
>> Time elapsed: 3.384 s  <<< FAILURE!
>> java.lang.AssertionError: There should be no missing fragments
>> expected:<0> but was:<1>
>>at org.junit.Assert.fail(Assert.java:88)
>>at org.junit.Assert.failNotEquals(Assert.java:834)
>>at org.junit.Assert.assertEquals(Assert.java:645)
>>at
>> org.apache.bookkeeper.client.SlowBookieTest.doBackpressureTest(SlowBookieTest.java:293)
>>at
>> org.apache.bookkeeper.client.SlowBookieTest.testSlowBookieAndFastFailOn(SlowBookieTest.java:197)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>at java.lang.reflect.Method.invoke(Method.java:498)
>>at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>>at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>>at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>>at
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>>at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>>at
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>>at
>> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
>>at
>> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>at java.lang.Thread.run(Thread.java:748)
>> 
>> 
>> [ERROR]
>> org.apache.bookkeeper.client.TestReadLastConfirmedLongPoll.testReadLACLongPollWhenSomeBookiesDown[0]
>> Time elapsed: 5.726 s  <<< FAILURE!
>> java.lang.AssertionError
>>at org.junit.Assert.fail(Assert.java:86)
>>at org.junit.Assert.assertTrue(Assert.java:41)
>>at org.junit.Assert.assertTrue(Assert.java:52)
>>at
>> org.apache.bookkeeper.client.TestReadLastConfirmedLongPoll.testReadLACLongPollWhenSomeBookiesDown(TestReadLastConfirmedLongPoll.java:184)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>at java.lang.reflect.Method.invoke(Method.java:498)
>>at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>>at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>>at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>>at
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>>at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>>at
>> org

Clean build and NOTICE (was: Re: [VOTE] Release 4.13.0, release candidate #0)

[Flavio Junqueira]

A couple of things:

1- I haven't been able to get a clean build, I'll list the tests failures I 
observed below.
2- We need to update the year range in the NOTICE file to say 2021 instead of 
2020.

-Flavio


[ERROR] org.apache.bookkeeper.client.SlowBookieTest.testSlowBookieAndFastFailOn 
 Time elapsed: 3.384 s  <<< FAILURE!
java.lang.AssertionError: There should be no missing fragments expected:<0> but 
was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.bookkeeper.client.SlowBookieTest.doBackpressureTest(SlowBookieTest.java:293)
at 
org.apache.bookkeeper.client.SlowBookieTest.testSlowBookieAndFastFailOn(SlowBookieTest.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)


[ERROR] 
org.apache.bookkeeper.client.TestReadLastConfirmedLongPoll.testReadLACLongPollWhenSomeBookiesDown[0]
  Time elapsed: 5.726 s  <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.bookkeeper.client.TestReadLastConfirmedLongPoll.testReadLACLongPollWhenSomeBookiesDown(TestReadLastConfirmedLongPoll.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)


[ERROR] 
org.apache.bookkeeper.bookie.CompactionByBytesTest.testCompactionFailureShouldNotResultInDuplicatedData
  Time elapsed: 16.374 s  <<< FAILURE!
java.lang.AssertionError: Entry log file ([0,1,2].log should have been 
compacted in ledgerDirectory: /tmp/bookie6403976059727115417test
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at 
org.apache.bookkeeper.bookie.CompactionTest.testCompactionFailureShouldNotResultInDuplicatedData(CompactionTest.java:1360)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

Thanks for the feedback, JV, see comments interspersed:

> On 18 Jan 2021, at 22:54, Venkateswara Rao Jujjuri  wrote:
> 
> On Mon, Jan 18, 2021 at 10:53 AM Sijie Guo  <mailto:guosi...@gmail.com>> wrote:
> 
>>> One concern for me in this thread is case (3). I'd expect a client that
>> doesn't crash to not give up, and eventually replace the bookie if it is
>> unresponsive.
>> 
>> The current implementation doesn't retry replacing a bookie if an entry is
>> already acknowledged (receiving AQ responses). It relies on inspection to
>> repair the hole.
>> 
> 
> Exactly. It is not even practical to do this as with the current code.

I'd be interested in knowing more precisely what makes you say it.

> Once the Qa meets we move the LAC. So
> 
> Ensemble  B0  B1 B2 LAC
> Entry:0   W  W   W  -1
> 1W  WNR0   (NR: No Response)
> 2W  WNR1
> Now B1 failed with network error where write fails immediately
> 3  when attempted to write it gets error immediately and
> attempts ensemble change.
>I think this is wrong. Why we treat errors after Qa is
> different from before reaching Qa.
>   What is stopping the code from waiting to see if Qa is
> met or not before attempting ensemble change.? @Sijie Guo


If I understand what you are saying, then we could end up in the situation that 
we never try to replace the faulty bookie because all entries get AQ replies 
from B0 and B1(you say that B1 failed, but I think you meant B2 based on the 
example). There needs to be a trigger for the bookie replacement despite 
entries receiving AQ replies.

Actually, this point makes me wonder whether one alternative to the back 
pressure discussion in this thread would be to replace a bookie based on the 
number of entries queued in the bookie client. If a bookie client is 
accumulating many entries for a bookie compared to others in the ensemble, then 
we could declare it unhealthy and trigger a replacement. Is this a suitable 
approach?


> mailto:guosi...@gmail.com>> ?
> Ensemble B0  B10  B2LAC
> 3   WWNR   2
> 
> Since we changed ensemble if entry 1 and 2 fails with timeout we can't go
> back and retroactively change the ensemble
> 
> 
> 
>> In case (1), the client crashed and the ledger will be recovered by some
>> reader. For all entries that have been acknowledged, including e, I'd
>> expect them to be readable from the closed ledger. Each one of these
>> entries that haven't been written to bookie b should be written there as
>> part of the recovery process.
>> 
> I don't think this can ever happen because we have OSE hashed by ledgerId.
> We can't receive and process any responses before we send out to all Qw
> bookies.

I'm not sure what you're referring to as not being possible. If an entry e has 
been acknowledged to the application, then the last entry once closed must be 
greater or equal to the id of e, right? You might be referring to something 
else?

> 
> Not sure what is the consensus reached on Issue#1063
> <https://github.com/apache/bookkeeper/commit/9d09a9c2a64b745271ef1c6dad9e5ab3ab3f2a5c
>  
> <https://github.com/apache/bookkeeper/commit/9d09a9c2a64b745271ef1c6dad9e5ab3ab3f2a5c>>.
> If it appears to be a problem let's have a quick call, maybe that is easy
> to resolve.
> 

As part of this thread, Jack, Enrico and I have set some time this Friday to 
talk. We scheduled for 4pm CET / 10am EST. Would you and Sijie be interested in 
joining? If so, ping me separately so that I can send you the zoom link. In 
general, anyone interested should feel free to join.

-Flavio

> 
>> So the memory pressure is not coming from retrying. It is straight that the
>> bookkeeper client references the sendBuffers until it receives any
>> responses from the slow bookie. The bookkeeper client allows enqueuing
>> addEntry operations because the operations meet the AQ requirements. Pulsar
>> does add `maxPendingPublishdRequestsPerConnection` mechanism to throttle
>> the add operations. But this won't work as bookkeeper will notify the
>> callbacks once the operations meet the AQ requirements. But there is a huge
>> amount of memory (throughput * timeout period) referenced by a slow bookie.
>> Hence we have to add a memory-based throttling mechanism as Matteo
>> suggested.
>> 
>> If we want to add the retry logic to replace a bookie, this will add more
>> pressure to the memory. But it can still be solved by a memory-based
>> back-

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

> Based on my understanding, Jack wants the behavior on recovering an entry
> does not have enough replicas to be deterministic. i.e. If the entry does
> not have enough replicas, we can always exclude the entry. Jack, did I get
> you right?

I see, if that's the case, then part of the problem here is that there is 
uncertainty in some cases about the state of an entry. We might not be able to 
read enough copies to determine for sure that an entry has been sufficiently 
replicated and consequently might have been acknowledged to the application. To 
be conservative and avoid violating safety, we make such an entry part of the 
closed ledger.

-Flavio

> On 18 Jan 2021, at 19:55, Sijie Guo  wrote:
> 
> On Mon, Jan 18, 2021 at 10:18 AM Flavio Junqueira  <mailto:f...@apache.org>> wrote:
> 
>>>>> Regarding recovery reads, recovery read doesn't need to be
>> deterministic.
>>>>> For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
>>>>> either including it or excluding it in the sealed ledger is correct
>>>>> behavior. The bookkeeper client guarantees that once a ledger is
>> sealed,
>>>>> the entries in the sealed ledger can always be read and can be read
>>>>> consistently.
>>>> 
>>>>> I am not sure it is a problem unless I misunderstand it.
>>>> 
>>>> It is true that it doesn't violate any safety property, but it is a
>> strange
>>>> check to me. It looks like an implementation artefact rather than an
>>>> explicit protocol design choice. But not a huge deal.
>>>> 
>>> 
>>> It was discussed in the earlier days as a design choice for this
>> protocol.
>>> 
>>> If we want to make it deterministic, we might need to consider what is
>> the
>>> performance penalty.
>> 
>> 
>> I don't quite follow the observation about a deterministic check. The
>> example that Sijie provides makes sense to me if I understand it right as
>> the entry does not have enough replicas, so it can go either way when the
>> ledger is close. But, that assumes that no later entry has been
>> acknowledged, otherwise we have a data loss if we skip the entry and
>> consequently have a problem with the protocol. If anyone cares to explain
>> the deterministic check referred to, I'd appreciate.
>> 
> 
> Based on my understanding, Jack wants the behavior on recovering an entry
> does not have enough replicas to be deterministic. i.e. If the entry does
> not have enough replicas, we can always exclude the entry. Jack, did I get
> you right?
> 
> - Sijie
> 
> 
>> 
>> -Flavio
>> 
>>> On 18 Jan 2021, at 18:51, Sijie Guo  wrote:
>>> 
>>> Jack,
>>> 
>>> Thank you for your replies! That's good as there are not violations of
>>> bookkeeper protocol.
>>> 
>>> Comments inline.
>>> 
>>> On Mon, Jan 18, 2021 at 3:20 AM Jack Vanlightly
>>> mailto:jvanligh...@splunk.com.invalid> 
>>> <mailto:jvanligh...@splunk.com.invalid 
>>> <mailto:jvanligh...@splunk.com.invalid>>>
>> wrote:
>>> 
>>>>> Did you guys see any issues with the ledger auditor?
>>>> 
>>>>> The active writer can't guarantee it writing entries to WQ because it
>> can
>>>>> crash during retrying adding entries to (WQ - AQ) bookies.
>>>> 
>>>> The need to repair AQ replicated entries is clear and the auditor is one
>>>> such strategy. Ivan has also worked on a self-healing bookie strategy
>> where
>>>> each bookie itself is able to detect these holes and is able to obtain
>> the
>>>> missing entries itself. The detection of these holes using this
>> strategy is
>>>> more efficient as it only requires network calls for the ledger metadata
>>>> scanning (to zk) and the missing entry reads (to other bookies). The
>>>> auditor as I understand it, reads all entries of all ledgers from all
>>>> bookies (of an entries ensemble) meaning these entries cross the
>> network.
>>>> Using the auditor approach is likely to be run less frequently due to
>> the
>>>> network cost.
>>>> 
>>> 
>>> Agreed on the efficiency part. I think the Salesforce team introduced the
>>> Disk Scrubber to solve that problem already unless I confused something
>>> there.
>>> 
>>> +JV Jujjuri mailto:vjujj...@salesforce.com> 
>>> <mailto:vjujj...

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

Thanks for the feedback, Sijie:

> On 18 Jan 2021, at 19:53, Sijie Guo  wrote:
> 
>> One concern for me in this thread is case (3). I'd expect a client that
>> doesn't crash to not give up, and eventually replace the bookie if it is
>> unresponsive.
>> 
> The current implementation doesn't retry replacing a bookie if an entry is
> already acknowledged (receiving AQ responses). It relies on inspection to
> repair the hole.

Ok, that's good information, let me think a bit more about it. I'd like to 
understand why we can't keep a pending add op reference until it is fully 
replicated, which as I understand would enable bookie replacement for entries 
with fewer than WQ acks.

> 
> So the memory pressure is not coming from retrying. It is straight that the
> bookkeeper client references the sendBuffers until it receives any
> responses from the slow bookie. The bookkeeper client allows enqueuing
> addEntry operations because the operations meet the AQ requirements.

I see, the entries queuing in the bookie client are inducing memory pressure in 
the presence of a slow bookie.

> Pulsar
> does add `maxPendingPublishdRequestsPerConnection` mechanism to throttle
> the add operations. But this won't work as bookkeeper will notify the
> callbacks once the operations meet the AQ requirements. But there is a huge
> amount of memory (throughput * timeout period) referenced by a slow bookie.
> Hence we have to add a memory-based throttling mechanism as Matteo
> suggested.

Thanks for pointing to Mateo's message, I reviewed it again. He actually makes 
two observations:

1- It is difficult to throttle from outside the bookkeeper client because the 
application using it does not have visibility into what has been fully 
replicated. A back pressure mechanism internal to the bookie (and possibly 
configurable) might be necessary.
2- There is some Pulsar work (PIP-74) that could be leveraged to throttle from 
outside the bookkeeper client based on memory limits.

> 
> If we want to add the retry logic to replace a bookie, this will add more
> pressure to the memory. But it can still be solved by a memory-based
> back-pressure mechansim.
> 

I don't know much about (2), but I'll have a look to form an opinion. At a high 
level, it seems reasonable. We might still want to consider doing (1) to 
simplify the job of the application.

-Flavio  

> Thanks,
> Sijie
> 
> On Mon, Jan 18, 2021 at 8:10 AM Flavio Junqueira  wrote:
> 
>> In the scenario that WQ > AQ, a client acknowledges the add of an entry e
>> to the application once it receives AQ bookie acks. Say now that the client
>> is not able to write a copy of e to at least one bookie b, it could be
>> because:
>> 
>> 1- The client crashed before it is able to do it
>> 2- Bookie b crashed
>> 3- The client gave up trying
>> 
>> In case (1), the client crashed and the ledger will be recovered by some
>> reader. For all entries that have been acknowledged, including e, I'd
>> expect them to be readable from the closed ledger. Each one of these
>> entries that haven't been written to bookie b should be written there as
>> part of the recovery process.
>> 
>> In case (2), the client is not able to write entry e to the crashed bookie
>> b, so it will replace the bookie and write e to the new bookie. I see in
>> this discussion that there is an option to disable bookie replacement, I'm
>> ignoring that for this discussion.
>> 
>> In case (3), the client say discards the entry after adding successfully
>> to AQ bookies, and gives up at some point because it can't reach the
>> bookie. The client maybe replaces bookie b or bookie b eventually comes
>> back and the client proceeds with the adds. In either case, there is a hole
>> that can only be fixed by inspecting the ledger.
>> 
>> One concern for me in this thread is case (3). I'd expect a client that
>> doesn't crash to not give up, and eventually replace the bookie if it is
>> unresponsive. But, that certainly leads to the memory pressure problem that
>> was also mentioned in the thread, for which one potential direction also
>> mentioned is to apply back pressure.
>> 
>> Thanks,
>> -Flavio
>> 
>>> On 18 Jan 2021, at 12:20, Jack Vanlightly 
>> wrote:
>>> 
>>>> Did you guys see any issues with the ledger auditor?
>>> 
>>>> The active writer can't guarantee it writing entries to WQ because it
>> can
>>>> crash during retrying adding entries to (WQ - AQ) bookies.
>>> 
>>> The need to repair AQ replicated entries is clear and the auditor i

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

 
>>>> This leaves a "hole" as the entry is now replicated only to 2 bookies,
>>> 
>>> We do have one hole when ensemble change is enabled and WQ > AQ. That
>> was a
>>> known behavior. But the hole will be repaired by the ledger auditor as JV
>>> said. Did you guys see any issues with the ledger auditor?
>>> 
>>>> I'd think that we guarantee that an entry that is acknowledged is
>>> eventually written WQ ways and that it is observable by readers when the
>>> ledger is closed.
>>> 
>>> To Flavio's question, we don't guarantee (and can't guarantee) that the
>>> active writer will eventually write the entries to WQ. For the active
>>> writers, we only guarantee entries are written to AQ. The ledger auditor
>> is
>>> to ensure all the entries are written to WQ.
>>> 
>>> The active writer can't guarantee it writing entries to WQ because it can
>>> crash during retrying adding entries to (WQ - AQ) bookies.
>>> 
>>>> A single successful read is enough. However
>>> there is an odd behavior in that as soon as (WQ-AQ)+1 reads fail with
>>> explicit NoSuchEntry/Ledger, the read is considered failed and the ledger
>>> recovery process ends there. This means that given the responses
>>> b1->success, b2->NoSuchLedger, b3->NoSuchLedger, whether the read is
>>> considered successful is non-deterministic.
>>> 
>>> Regarding recovery reads, recovery read doesn't need to be deterministic.
>>> For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
>>> either including it or excluding it in the sealed ledger is correct
>>> behavior. The bookkeeper client guarantees that once a ledger is sealed,
>>> the entries in the sealed ledger can always be read and can be read
>>> consistently.
>>> 
>>> I am not sure it is a problem unless I misunderstand it.
>>> 
>>> - Sijie
>>> 
>>> On Fri, Jan 15, 2021 at 7:43 AM Jack Vanlightly
>>>  wrote:
>>> 
>>>> Let's set up a call and create any issues from that. I have already
>>> created
>>>> the patches in our (Splunk) fork and it might be easiest or not to wait
>>>> until we re-sync up with the open source repo. We can include the fixes
>>> in
>>>> the discussion.
>>>> 
>>>> Jack
>>>> 
>>>> On Fri, Jan 15, 2021 at 4:33 PM Flavio Junqueira 
>> wrote:
>>>> 
>>>>> [ External sender. Exercise caution. ]
>>>>> 
>>>>> Hi Jack,
>>>>> 
>>>>> Thanks for getting back.
>>>>> 
>>>>>> What's the best way to share the TLA+ findings?
>>>>> 
>>>>> Would you be able to share the spec? I'm ok with reading TLA+.
>>>>> 
>>>>> As for sharing your specific findings, I'd suggest one of the
>>> following:
>>>>> 
>>>>> 1- Create an email thread describing the scenarios that trigger a
>> bug.
>>>>> 2- Create issues, one for each problem you found.
>>>>> 3- Create a discussion on the project Slack, perhaps a channel
>> specific
>>>>> for it.
>>>>> 4- Set up a zoom call to present and discuss with the community.
>>>>> 
>>>>> Option 2 is ideal from a community perspective, but we can also set
>> up
>>> a
>>>>> call inviting everyone and create issues out of that discussion. We
>> can
>>>> in
>>>>> fact set up a call even if we create the issues ahead of time.
>>>>> 
>>>>> Does it make sense?
>>>>> 
>>>>> -Flavio
>>>>> 
>>>>>> On 15 Jan 2021, at 16:14, Jack Vanlightly >>> .INVALID>
>>>>> wrote:
>>>>>> 
>>>>>> Hi Flavio,
>>>>>> 
>>>>>>>> This is an example of a scenario corresponding to what we suspect
>>> is
>>>> a
>>>>>> bug introduced earlier, but Enrico is arguing that this is not the
>>>>> intended
>>>>>> behavior, and at this point, I agree.
>>>>>> 
>>>>>>>> By the time a successful callback is received, the client might
>>> only
>>>>>> have replicated AQ ways, so the guarantee can only be at that point
>>> of
>>>>&g

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

>> A single successful read is enough. However
>> there is an odd behavior in that as soon as (WQ-AQ)+1 reads fail with
>> explicit NoSuchEntry/Ledger, the read is considered failed and the ledger
>> recovery process ends there. This means that given the responses
>> b1->success, b2->NoSuchLedger, b3->NoSuchLedger, whether the read is
>> considered successful is non-deterministic.
>> 
>> Regarding recovery reads, recovery read doesn't need to be deterministic.
>> For the entry with (b1->success, b2->NoSuchLedger, b3->NoSuchLedger),
>> either including it or excluding it in the sealed ledger is correct
>> behavior. The bookkeeper client guarantees that once a ledger is sealed,
>> the entries in the sealed ledger can always be read and can be read
>> consistently.
>> 
>> I am not sure it is a problem unless I misunderstand it.
>> 
>> - Sijie
>> 
>> On Fri, Jan 15, 2021 at 7:43 AM Jack Vanlightly
>>  wrote:
>> 
>>> Let's set up a call and create any issues from that. I have already
>> created
>>> the patches in our (Splunk) fork and it might be easiest or not to wait
>>> until we re-sync up with the open source repo. We can include the fixes
>> in
>>> the discussion.
>>> 
>>> Jack
>>> 
>>> On Fri, Jan 15, 2021 at 4:33 PM Flavio Junqueira  wrote:
>>> 
>>>> [ External sender. Exercise caution. ]
>>>> 
>>>> Hi Jack,
>>>> 
>>>> Thanks for getting back.
>>>> 
>>>>> What's the best way to share the TLA+ findings?
>>>> 
>>>> Would you be able to share the spec? I'm ok with reading TLA+.
>>>> 
>>>> As for sharing your specific findings, I'd suggest one of the
>> following:
>>>> 
>>>> 1- Create an email thread describing the scenarios that trigger a bug.
>>>> 2- Create issues, one for each problem you found.
>>>> 3- Create a discussion on the project Slack, perhaps a channel specific
>>>> for it.
>>>> 4- Set up a zoom call to present and discuss with the community.
>>>> 
>>>> Option 2 is ideal from a community perspective, but we can also set up
>> a
>>>> call inviting everyone and create issues out of that discussion. We can
>>> in
>>>> fact set up a call even if we create the issues ahead of time.
>>>> 
>>>> Does it make sense?
>>>> 
>>>> -Flavio
>>>> 
>>>>> On 15 Jan 2021, at 16:14, Jack Vanlightly >> .INVALID>
>>>> wrote:
>>>>> 
>>>>> Hi Flavio,
>>>>> 
>>>>>>> This is an example of a scenario corresponding to what we suspect
>> is
>>> a
>>>>> bug introduced earlier, but Enrico is arguing that this is not the
>>>> intended
>>>>> behavior, and at this point, I agree.
>>>>> 
>>>>>>> By the time a successful callback is received, the client might
>> only
>>>>> have replicated AQ ways, so the guarantee can only be at that point
>> of
>>>>> being able to tolerate AQ - 1 crashes. The ledger configuration
>> states
>>>> that
>>>>> the application wants to have WQ copies >> of each entry, though. I'd
>>>>> expect a ledger to have WQ copies of each entry up to the final entry
>>>>> number when it is closed. Do you see it differently?
>>>>> 
>>>>> I also agree and was pretty surprised when I discovered the
>> behaviour.
>>> It
>>>>> is not something that users expect and I think we need to correct it.
>>> So
>>>>> I'm with you.
>>>>> 
>>>>> What's the best way to share the TLA+ findings?
>>>>> 
>>>>> Jack
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Jan 15, 2021 at 3:59 PM Flavio Junqueira 
>>> wrote:
>>>>> 
>>>>>> [ External sender. Exercise caution. ]
>>>>>> 
>>>>>>> Let's say we have WQ 3 and AQ 2. An add (e100) has reached AQ and
>> the
>>>>>>> confirm callback to the client is called and the LAC is set to
>>> 100.Now
>>>>>> the
>>>>>>> 3rd bookie times out. Ensemble change is executed and all pending
>>> adds
>>>>

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

Hi Jack,

Thanks for getting back.

> What's the best way to share the TLA+ findings?

Would you be able to share the spec? I'm ok with reading TLA+. 

As for sharing your specific findings, I'd suggest one of the following:

1- Create an email thread describing the scenarios that trigger a bug.
2- Create issues, one for each problem you found.
3- Create a discussion on the project Slack, perhaps a channel specific for it.
4- Set up a zoom call to present and discuss with the community. 

Option 2 is ideal from a community perspective, but we can also set up a call 
inviting everyone and create issues out of that discussion. We can in fact set 
up a call even if we create the issues ahead of time.

Does it make sense?

-Flavio

> On 15 Jan 2021, at 16:14, Jack Vanlightly  
> wrote:
> 
> Hi Flavio,
> 
>>> This is an example of a scenario corresponding to what we suspect is a
> bug introduced earlier, but Enrico is arguing that this is not the intended
> behavior, and at this point, I agree.
> 
>>> By the time a successful callback is received, the client might only
> have replicated AQ ways, so the guarantee can only be at that point of
> being able to tolerate AQ - 1 crashes. The ledger configuration states that
> the application wants to have WQ copies >> of each entry, though. I'd
> expect a ledger to have WQ copies of each entry up to the final entry
> number when it is closed. Do you see it differently?
> 
> I also agree and was pretty surprised when I discovered the behaviour. It
> is not something that users expect and I think we need to correct it. So
> I'm with you.
> 
> What's the best way to share the TLA+ findings?
> 
> Jack
> 
> 
> 
> 
> 
> On Fri, Jan 15, 2021 at 3:59 PM Flavio Junqueira  wrote:
> 
>> [ External sender. Exercise caution. ]
>> 
>>> Let's say we have WQ 3 and AQ 2. An add (e100) has reached AQ and the
>>> confirm callback to the client is called and the LAC is set to 100.Now
>> the
>>> 3rd bookie times out. Ensemble change is executed and all pending adds
>> that
>>> are above the LAC of 100 are replayed to another bookie, meaning that the
>>> entry e100 is not replayed to another bookie, causing this entry to meet
>>> the rep factor of only AQ.
>> 
>> This is an example of a scenario corresponding to what we suspect is a bug
>> introduced earlier, but Enrico is arguing that this is not the intended
>> behavior, and at this point, I agree.
>> 
>>> This is alluded to in the docs as they state
>>> that AQ is also the minimum guaranteed replication factor.
>> 
>> By the time a successful callback is received, the client might only have
>> replicated AQ ways, so the guarantee can only be at that point of being
>> able to tolerate AQ - 1 crashes. The ledger configuration states that the
>> application wants to have WQ copies of each entry, though. I'd expect a
>> ledger to have WQ copies of each entry up to the final entry number when it
>> is closed. Do you see it differently?
>> 
>>> I'd be happy to set up a meeting to discuss the spec and its findings.
>> 
>> 
>> That'd be great, I'm interested.
>> 
>> -Flavio
>> 
>>> On 15 Jan 2021, at 15:30, Jack Vanlightly 
>> wrote:
>>> 
>>>> No you cannot miss data, if the client is not able to find a bookie that
>>> is
>>>> able to answer with the entry it receives an error.
>>> 
>>> Let's say we have WQ 3 and AQ 2. An add (e100) has reached AQ and the
>>> confirm callback to the client is called and the LAC is set to 100. Now
>> the
>>> 3rd bookie times out. Ensemble change is executed and all pending adds
>> that
>>> are above the LAC of 100 are replayed to another bookie, meaning that the
>>> entry e100 is not replayed to another bookie, causing this entry to meet
>>> the rep factor of only AQ. This is alluded to in the docs as they state
>>> that AQ is also the minimum guaranteed replication factor.
>>> 
>>>> The recovery read fails if it is not possible to read every entry from
>> at
>>>> least AQ bookies  (that is it allows WQ-QA read failures),
>>>> and it does not hazard to "repair" (truncate) the ledger if it does not
>>>> find enough bookies.
>>> 
>>> This is not quite accurate. A single successful read is enough. However
>>> there is an odd behaviour in that as soon as (WQ-AQ)+1 reads fail with
>>> explicit NoSuchEntry/Ledger, the read is considered failed and the ledger
>>> recovery proc

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

> Let's say we have WQ 3 and AQ 2. An add (e100) has reached AQ and the
> confirm callback to the client is called and the LAC is set to 100.Now the
> 3rd bookie times out. Ensemble change is executed and all pending adds that
> are above the LAC of 100 are replayed to another bookie, meaning that the
> entry e100 is not replayed to another bookie, causing this entry to meet
> the rep factor of only AQ.

This is an example of a scenario corresponding to what we suspect is a bug 
introduced earlier, but Enrico is arguing that this is not the intended 
behavior, and at this point, I agree. 

> This is alluded to in the docs as they state
> that AQ is also the minimum guaranteed replication factor.

By the time a successful callback is received, the client might only have 
replicated AQ ways, so the guarantee can only be at that point of being able to 
tolerate AQ - 1 crashes. The ledger configuration states that the application 
wants to have WQ copies of each entry, though. I'd expect a ledger to have WQ 
copies of each entry up to the final entry number when it is closed. Do you see 
it differently?

>  I'd be happy to set up a meeting to discuss the spec and its findings. 


That'd be great, I'm interested.

-Flavio

> On 15 Jan 2021, at 15:30, Jack Vanlightly  
> wrote:
> 
>> No you cannot miss data, if the client is not able to find a bookie that
> is
>> able to answer with the entry it receives an error.
> 
> Let's say we have WQ 3 and AQ 2. An add (e100) has reached AQ and the
> confirm callback to the client is called and the LAC is set to 100. Now the
> 3rd bookie times out. Ensemble change is executed and all pending adds that
> are above the LAC of 100 are replayed to another bookie, meaning that the
> entry e100 is not replayed to another bookie, causing this entry to meet
> the rep factor of only AQ. This is alluded to in the docs as they state
> that AQ is also the minimum guaranteed replication factor.
> 
>> The recovery read fails if it is not possible to read every entry from at
>> least AQ bookies  (that is it allows WQ-QA read failures),
>> and it does not hazard to "repair" (truncate) the ledger if it does not
>> find enough bookies.
> 
> This is not quite accurate. A single successful read is enough. However
> there is an odd behaviour in that as soon as (WQ-AQ)+1 reads fail with
> explicit NoSuchEntry/Ledger, the read is considered failed and the ledger
> recovery process ends there. This means that given the responses
> b1->success, b2->NoSuchLedger, b3->NoSuchLedger, whether the read is
> considered successful is non-deterministic. If the response from b1 is
> received last, then the read is already considered failed, otherwise the
> read succeeds.
> 
> I have come to the above conclusions through my reverse engineering process
> for creating the TLA+ specification. I still have pending to
> reproduce the AQ rep factor behaviour via some tests, but have verified via
> tests the conclusion about ledger recovery reads.
> 
> Note that I have found two defects with the BookKeeper protocol, most
> notably data loss due to that fencing does not prevent further successful
> adds. Currently the specification and associated documentation is on a
> private Splunk repo, but I'd be happy to set up a meeting to discuss the
> spec and its findings.
> 
> Best
> Jack
> 
> 
> On Fri, Jan 15, 2021 at 11:51 AM Enrico Olivelli 
> wrote:
> 
>> [ External sender. Exercise caution. ]
>> 
>> Jonathan,
>> 
>> Il giorno gio 14 gen 2021 alle ore 20:57 Jonathan Ellis <
>> jbel...@apache.org>
>> ha scritto:
>> 
>>> On 2021/01/11 08:31:03, Jack Vanlightly wrote:
 Hi,
 
 I've recently modelled the BookKeeper protocol in TLA+ and can confirm
>>> that
 once confirmed, that an entry is not replayed to another bookie. This
 leaves a "hole" as the entry is now replicated only to 2 bookies,
>>> however,
 the new data integrity check that Ivan worked on, when run periodically
 will be able to repair that hole.
>>> 
>>> Can I read from the bookie with a hole in the meantime, and silently miss
>>> data that it doesn't know about?
>>> 
>> 
>> No you cannot miss data, if the client is not able to find a bookie that is
>> able to answer with the entry it receives an error.
>> 
>> The ledger has a known tail (LastAddConfirmed entry) and this value is
>> stored on ledger metadata once the ledger is "closed".
>> 
>> When the ledger is still open, that is when the writer is writing to it,
>> the reader is allowed to read only up to the LastAddConfirmed entry
>> this LAC value is returned to the reader using a piggyback mechanism,
>> without reading from metadata.
>> The reader cannot read beyond the latest position that has been confirmed
>> to the writer by AQ bookies.
>> 
>> We have a third case, the 'recovery read'.
>> A reader starts a "recovery read" when you want to recover a ledger that
>> has been abandoned by a dead writer
>> or when you are a new leader (Pulsar Bundle Owner) or you want to fen

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

Right, good catch, Enrico. The issue (#1063) description says: 

> PendingAddOp:maybeRecycle()->recycle() keeps the buffer until writeComplete() 
> is called for each bookie write. We need to keep this buffer only until it is 
> successfully
> transferred by netty. In the current code, the write is retired only if
> disableEnsembleChangeFeature is enabled. Otherwise, there is no point in 
> keeping
> this buffer around.

JV, the author of the PR, says also the following to Sijie:

> toSend buffer is not needed for retries as we discussed on slack.


I don't know what the reason is. JV, Sijie, it has been a while back, but 
perhaps you guys can elaborate? Specifically, I'm trying to understand what is 
the guarantee that BK is currently offering for a configuration in which WQ > 
AQ. I'd think that we guarantee that an entry that is acknowledged is 
eventually written WQ ways and that it is observable by readers when the ledger 
is closed.

-Flavio

> On 14 Jan 2021, at 18:34, Enrico Olivelli  wrote:
> 
> Flavio
> 
> Il giorno gio 14 gen 2021 alle ore 17:56 Flavio Junqueira 
> ha scritto:
> 
>> Using your example, the PendindAddOp should remain active until there are
>> 3 copies of the add entry. The client can ack back once it receives two
>> positive acks from bookies, but it shouldn't declare the add entry done at
>> that point.
>> 
> 
> Probably this behaviour has been broken by this commit
> https://github.com/apache/bookkeeper/commit/9d09a9c2a64b745271ef1c6dad9e5ab3ab3f2a5c
> 
> My understanding is that as soon as we reach AQ we are discarding the
> "toSend" buffer and we cannot retry the write anymore
> 
> Enrico
> 
> 
>> 
>> There is the case that the third bookie is slow, but it could have failed
>> altogether, in which case the entry needs to be replicated in a new bookie.
>> 
>> -Flavio
>> 
>> On 13 Jan 2021, at 17:28, Enrico Olivelli  wrote:
>> 
>> 
>> 
>> Il giorno mer 13 gen 2021 alle ore 17:05 Flavio Junqueira 
>> ha scritto:
>> 
>>> We should work on some kind of back-pressure mechanism for the client,
>>> but I am not sure about which kind of support we should provide at BK level
>>> 
>>> 
>>> Is there an issue for this? If there isn't, then perhaps we can start
>>> that way.
>>> 
>>> And as soon as the application is notified of the result of the write
>>> (success or failure) we are releasing the reference to the payload (as I
>>> have shown in this email thread),
>>> so in theory the application has full control over the retained memory
>>> and it can apply its own memory management mechanisms
>>> 
>>> 
>>> Say we have a ledger for which WQ > AQ. Are you saying that if AQ bookies
>>> reply, then it is possible that the entry is not going to be written to
>>> |WQ| - |AQ| bookies because the entry data might have been reclaimed by the
>>> application? The contract as I understand it is that an entry is to be
>>> replicated |WQ| ways, even though the application is willing to receive a
>>> confirmation after |AQ| bookie responses.
>>> 
>>> What am I missing?
>>> 
>> 
>> If I am not wrong in reading PendingAddOp code currently we do it this
>> way, say we run with 3-3-2:
>> - enqueue the write request to the 3 PerChannelBookieClients
>> - as soon as we receive 2 confirmations we trigger the callback and
>> discard the payload
>> 
>> so if the first 2 confirmations arrive before we write to the socket
>> (enqueue the payload on Netty channel actually) of the third bookie, we are
>> not sending the entry to the 3rd bookie at all.
>> This should not happen because we serialize the operations per-ledger (by
>> sticking them to one thread), so you cannot process the incoming acks from
>> the first two bookies while executing PendingAddOp write loop.
>> So we are giving a chance to every bookie to receive the entry, if it is
>> in good health (bookie, network...)
>> 
>> Enrico
>> 
>> 
>> 
>>> 
>>> -Flavio
>>> 
>>> On 13 Jan 2021, at 11:30, Enrico Olivelli  wrote:
>>> 
>>> Flavio
>>> 
>>> Il giorno mar 12 gen 2021 alle ore 17:26 Flavio Junqueira 
>>> ha scritto:
>>> 
>>>> I have observed the issue that Matteo describes and I also attributed
>>>> the problem to the absence of a back pressure mechanism in the client.
>>>> Issue #2497 was not about that, though. There was some corruption going on
>>>> that was leading to

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

Using your example, the PendindAddOp should remain active until there are 3 
copies of the add entry. The client can ack back once it receives two positive 
acks from bookies, but it shouldn't declare the add entry done at that point. 

There is the case that the third bookie is slow, but it could have failed 
altogether, in which case the entry needs to be replicated in a new bookie.

-Flavio 

> On 13 Jan 2021, at 17:28, Enrico Olivelli  wrote:
> 
> 
> 
> Il giorno mer 13 gen 2021 alle ore 17:05 Flavio Junqueira  <mailto:f...@apache.org>> ha scritto:
>> We should work on some kind of back-pressure mechanism for the client, but I 
>> am not sure about which kind of support we should provide at BK level
> 
> Is there an issue for this? If there isn't, then perhaps we can start that 
> way.
> 
>> And as soon as the application is notified of the result of the write 
>> (success or failure) we are releasing the reference to the payload (as I 
>> have shown in this email thread),
>> so in theory the application has full control over the retained memory and 
>> it can apply its own memory management mechanisms 
> 
> Say we have a ledger for which WQ > AQ. Are you saying that if AQ bookies 
> reply, then it is possible that the entry is not going to be written to |WQ| 
> - |AQ| bookies because the entry data might have been reclaimed by the 
> application? The contract as I understand it is that an entry is to be 
> replicated |WQ| ways, even though the application is willing to receive a 
> confirmation after |AQ| bookie responses.
> 
> What am I missing?
> 
> If I am not wrong in reading PendingAddOp code currently we do it this way, 
> say we run with 3-3-2:
> - enqueue the write request to the 3 PerChannelBookieClients
> - as soon as we receive 2 confirmations we trigger the callback and discard 
> the payload
> 
> so if the first 2 confirmations arrive before we write to the socket (enqueue 
> the payload on Netty channel actually) of the third bookie, we are not 
> sending the entry to the 3rd bookie at all.
> This should not happen because we serialize the operations per-ledger (by 
> sticking them to one thread), so you cannot process the incoming acks from 
> the first two bookies while executing PendingAddOp write loop.
> So we are giving a chance to every bookie to receive the entry, if it is in 
> good health (bookie, network...)
> 
> Enrico  
> 
>  
> 
> -Flavio  
> 
>> On 13 Jan 2021, at 11:30, Enrico Olivelli > <mailto:eolive...@gmail.com>> wrote:
>> 
>> Flavio
>> 
>> Il giorno mar 12 gen 2021 alle ore 17:26 Flavio Junqueira > <mailto:f...@apache.org>> ha scritto:
>> I have observed the issue that Matteo describes and I also attributed the 
>> problem to the absence of a back pressure mechanism in the client. Issue 
>> #2497 was not about that, though. There was some corruption going on that 
>> was leading to the server receiving garbage.
>> 
>> Correct, #2497 is not about the topic of this email, I just mentioned it 
>> because the discussion started from that comment from Matteo.
>> 
>> We should work on some kind of back-pressure mechanism for the client, but I 
>> am not sure about which kind of support we should provide at BK level
>> 
>> Regarding the writer side of this story and memory usage,
>> we are not performing copies of the original payload that the caller is 
>> passing, in case of a ByteBuf
>> see PendingAddOp
>> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java#L263
>>  
>> <https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java#L263>
>> and here, we simply wrap it in a ByteBufList 
>> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/checksum/DigestManager.java#L116
>>  
>> <https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/checksum/DigestManager.java#L116>
>> 
>> And as soon as the application is notified of the result of the write 
>> (success or failure) we are releasing the reference to the payload (as I 
>> have shown in this email thread),
>> so in theory the application has full control over the retained memory
>> and it can apply its own memory management mechanisms
>> 
>> 
>> Enrico
>> 
>> 
>> -Flavio 
>> 
>> > On 8 Jan 2021, at 22:47, Matteo Merli > > <mailto:mme...@apache.org>> wrote:
>> > 
>> > On Fri, Jan

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

> We should work on some kind of back-pressure mechanism for the client, but I 
> am not sure about which kind of support we should provide at BK level

Is there an issue for this? If there isn't, then perhaps we can start that way.

> And as soon as the application is notified of the result of the write 
> (success or failure) we are releasing the reference to the payload (as I have 
> shown in this email thread),
> so in theory the application has full control over the retained memory and it 
> can apply its own memory management mechanisms 

Say we have a ledger for which WQ > AQ. Are you saying that if AQ bookies 
reply, then it is possible that the entry is not going to be written to |WQ| - 
|AQ| bookies because the entry data might have been reclaimed by the 
application? The contract as I understand it is that an entry is to be 
replicated |WQ| ways, even though the application is willing to receive a 
confirmation after |AQ| bookie responses.

What am I missing?

-Flavio  

> On 13 Jan 2021, at 11:30, Enrico Olivelli  wrote:
> 
> Flavio
> 
> Il giorno mar 12 gen 2021 alle ore 17:26 Flavio Junqueira  <mailto:f...@apache.org>> ha scritto:
> I have observed the issue that Matteo describes and I also attributed the 
> problem to the absence of a back pressure mechanism in the client. Issue 
> #2497 was not about that, though. There was some corruption going on that was 
> leading to the server receiving garbage.
> 
> Correct, #2497 is not about the topic of this email, I just mentioned it 
> because the discussion started from that comment from Matteo.
> 
> We should work on some kind of back-pressure mechanism for the client, but I 
> am not sure about which kind of support we should provide at BK level
> 
> Regarding the writer side of this story and memory usage,
> we are not performing copies of the original payload that the caller is 
> passing, in case of a ByteBuf
> see PendingAddOp
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java#L263
>  
> <https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/PendingAddOp.java#L263>
> and here, we simply wrap it in a ByteBufList 
> https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/checksum/DigestManager.java#L116
>  
> <https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/checksum/DigestManager.java#L116>
> 
> And as soon as the application is notified of the result of the write 
> (success or failure) we are releasing the reference to the payload (as I have 
> shown in this email thread),
> so in theory the application has full control over the retained memory
> and it can apply its own memory management mechanisms
> 
> 
> Enrico
> 
> 
> -Flavio 
> 
> > On 8 Jan 2021, at 22:47, Matteo Merli  > <mailto:mme...@apache.org>> wrote:
> > 
> > On Fri, Jan 8, 2021 at 8:27 AM Enrico Olivelli  > <mailto:eolive...@gmail.com>> wrote:
> >> 
> >> Hi Matteo,
> >> in this comment you are talking about an issue you saw when WQ is greater 
> >> that AQ
> >> https://github.com/apache/bookkeeper/issues/2497#issuecomment-734423246 
> >> <https://github.com/apache/bookkeeper/issues/2497#issuecomment-734423246>
> >> 
> >> IIUC you are saying that if one bookie is slow the client continues to 
> >> accumulate references to the entries that still have not received the 
> >> confirmation from it.
> >> I think that this is correct.
> >> 
> >> Have you seen problems in production related to this scenario ?
> >> Can you tell more about them ?
> > 
> > Yes, for simplicity, assume e=3, w=3, a=2.
> > 
> > If one bookie is slow (not down, just slow), the BK client will the
> > acks to the user that the entries are written after the first 2 acks.
> > In the meantime, it will keep waiting for the 3rd bookie to respond.
> > If the bookie responds within the timeout, the entries can now be
> > dropped from memory, otherwise the write will timeout internally and
> > it will get replayed to a new bookie.
> > 
> > In both cases, the amount of memory used in the client will max at
> > "throughput" * "timeout". This can be a large amount of memory and
> > easily cause OOM errors.
> > 
> > Part of the problem is that it cannot be solved from outside the BK
> > client, since there's no visibility on what entries have 2 or 3 acks
> > and therefore it's not possible to apply backp

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

I have observed the issue that Matteo describes and I also attributed the 
problem to the absence of a back pressure mechanism in the client. Issue #2497 
was not about that, though. There was some corruption going on that was leading 
to the server receiving garbage.

-Flavio 

> On 8 Jan 2021, at 22:47, Matteo Merli  wrote:
> 
> On Fri, Jan 8, 2021 at 8:27 AM Enrico Olivelli  wrote:
>> 
>> Hi Matteo,
>> in this comment you are talking about an issue you saw when WQ is greater 
>> that AQ
>> https://github.com/apache/bookkeeper/issues/2497#issuecomment-734423246
>> 
>> IIUC you are saying that if one bookie is slow the client continues to 
>> accumulate references to the entries that still have not received the 
>> confirmation from it.
>> I think that this is correct.
>> 
>> Have you seen problems in production related to this scenario ?
>> Can you tell more about them ?
> 
> Yes, for simplicity, assume e=3, w=3, a=2.
> 
> If one bookie is slow (not down, just slow), the BK client will the
> acks to the user that the entries are written after the first 2 acks.
> In the meantime, it will keep waiting for the 3rd bookie to respond.
> If the bookie responds within the timeout, the entries can now be
> dropped from memory, otherwise the write will timeout internally and
> it will get replayed to a new bookie.
> 
> In both cases, the amount of memory used in the client will max at
> "throughput" * "timeout". This can be a large amount of memory and
> easily cause OOM errors.
> 
> Part of the problem is that it cannot be solved from outside the BK
> client, since there's no visibility on what entries have 2 or 3 acks
> and therefore it's not possible to apply backpressure. Instead,
> there should be a backpressure mechanism in the BK client itself to
> prevent this kind of issue.
> One possibility there could be to use the same approach as described
> in 
> https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits,
> giving a max memory limit per BK client instance and throttling
> everything after the quota is reached.
> 
> 
> Matteo

Re: Unbounded memory usage for WQ > AQ ?

[Flavio Junqueira]

Hi Jack,

> I've recently modelled the BookKeeper protocol in TLA+ and can confirm that
> once confirmed, that an entry is not replayed to another bookie.

Should I assume that you modeled it after the code? Otherwise, what did you use 
as a reference? Is the TLA+ spec available anywhere? It sounds like a good 
development.

> once confirmed, that an entry is not replayed to another bookie.


I'd like to understand this a bit better. I think this is saying that if I have 
an entry e that is written to AQ < WQ, and at least one bookie b in the ledger 
ensemble crashes before it writes e, then e is considered confirmed and when b 
is replaced with b' for the ledger, e is not replicated on b'.

If that's the case, then isn't it a bug?

>  the new data integrity check that Ivan worked on, when run periodically

> will be able to repair that hole.


This is good, but I'm not sure this is a replacement for a proper fix.

Please let me know if I'm missing anything.

-Flavio 

> On 11 Jan 2021, at 09:31, Jack Vanlightly  
> wrote:
> 
> Hi,
> 
> I've recently modelled the BookKeeper protocol in TLA+ and can confirm that
> once confirmed, that an entry is not replayed to another bookie. This
> leaves a "hole" as the entry is now replicated only to 2 bookies, however,
> the new data integrity check that Ivan worked on, when run periodically
> will be able to repair that hole.
> 
> Jack
> 
> On Sat, Jan 9, 2021 at 1:06 AM Venkateswara Rao Jujjuri 
> wrote:
> 
>> [ External sender. Exercise caution. ]
>> 
>> On Fri, Jan 8, 2021 at 2:29 PM Matteo Merli 
>> wrote:
>> 
>>> On Fri, Jan 8, 2021 at 2:15 PM Venkateswara Rao Jujjuri
>>>  wrote:
 
> otherwise the write will timeout internally and it will get replayed
>>> to a
 new bookie.
 If Qa is met and the writes of Qw-Qa fail after we send the success to
>>> the
 client, why would the write replayed on a new bookie?
>>> 
>>> I think the original intention was to avoid having 1 bookie with a
>>> "hole" in the entries sequence. If you then lose one of the 2 bookies,
>>> it would be difficult to know which entries need to be recovered.
>>> 
>> 
>> @Matteo Merli   I don't believe we retry the write
>> on bookie if Qa is satisfied and the write to a bookie timedout.
>> Once the entry is ack'ed to the client we move the LAC and can't
>> retroactively change the active segment's ensemble.
>> 
>>> will get replayed to a new bookie
>> This will happen only if we are not able to satisfy Qa and go through
>> ensemble changes.
>> We change the ensemble and tetry write only if bookie write fails before
>> satisfying Qa.
>> We have added a new feature called handling "delayed write failure", but
>> that happens only for
>> new entries not retroactively.
>> 
>> I may be missing something here, and not understanding your point.
>> 
>> Thanks,
>> JV
>> 
>> 
>> 
>> 
>> --
>> Jvrao
>> ---
>> First they ignore you, then they laugh at you, then they fight you, then
>> you win. - Mahatma Gandhi
>> 

Re: [VOTE] Apache BookKeeper 4.11.1 release candidate 1

[Flavio Junqueira]

+1

- built from sources (it did show a few flaky tests, though)
- checked LICENSE and NOTICE
- checked that dependencies is staging repo are correctly resolved
- verified digest and signature

-Flavio

> On 12 Oct 2020, at 13:52, Enrico Olivelli  wrote:
> 
> UP !
> 
> Enrico
> 
> Il giorno mar 6 ott 2020 alle ore 17:02 Enrico Olivelli 
> ha scritto:
> 
>> +1 (binding)
>> - built and run tests on JDK8 on Linux Fedora
>> - verified signatures and hashes
>> - run smoke tests with the built signatures
>> 
>> Regards
>> Enrico
>> 
>> 
>> Il giorno mar 6 ott 2020 alle ore 14:52 Enrico Olivelli <
>> eolive...@gmail.com> ha scritto:
>> 
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version
>>> 4.11.1, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>> 
>>> The complete staging area is available for your review, which includes:
>>> * Release notes [1]
>>> * The official Apache source and binary distributions to be deployed to
>>> dist.apache.org [2]
>>> * All artifacts to be deployed to the Maven Central Repository [3]
>>> * Source code tag "v4.11.1-rc1" [4] with git sha
>>> 15a4e97c7ae7a383e80f5b6e58ffcf379e22abda
>>> 
>>> BookKeeper's KEYS file contains PGP keys we used to sign this release:
>>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>>> 
>>> Please download these packages and review this release candidate:
>>> 
>>> - Review release notes
>>> - Download the source package (verify shasum, and asc) and follow the
>>> instructions to build and run the bookkeeper service.
>>> - Download the binary package (verify shasum, and asc) and follow the
>>> instructions to run the bookkeeper service.
>>> - Review maven repo, release tag, licenses, and any other things you think
>>> it is important to a release.
>>> 
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>> 
>>> Thanks,
>>> Enrico Olivelli
>>> 
>>> [1] https://github.com/apache/bookkeeper/pull/2418
>>> [2]
>>> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.11.1-rc1/
>>> [3]
>>> https://repository.apache.org/content/repositories/orgapachebookkeeper-1046/
>>> [4] https://github.com/apache/bookkeeper/tree/v4.11.1-rc1
>>> 
>> 

Re: [VOTE] Release 4.11.1, release candidate #0

[Flavio Junqueira]

-1. The notice file says `Copyright 2011-2018`, and we need to update the years 
so that the range is `2011-2020`:

https://www.apache.org/legal/src-headers.html#notice 


I'm wondering additionally whether there is anything that we bundle that 
requires updating the NOTICE file.

   https://infra.apache.org/licensing-howto.html

Otherwise, I have been able to build successfully and check that it resolves 
the dependencies in the stating repo.

-Flavio

> On 2 Oct 2020, at 12:24, Enrico Olivelli  wrote:
> 
> Up !
> 
> Enrico
> 
> Il giorno mer 30 set 2020 alle ore 08:35 Enrico Olivelli <
> eolive...@gmail.com> ha scritto:
> 
>> Thank you guys again,
>> we need at least one more binding vote from a PMC member (unfortunately
>> Rajan's vote is not "binding" yet in BookKeeper project)
>> 
>> Enrico
>> 
>> Il giorno mar 29 set 2020 alle ore 14:17 Jia Zhai 
>> ha scritto:
>> 
>>> +1 (binding)
>>> 
>>> Environment: macOS 10.15.6, jdk 8.
>>> 
>>> - verified packages checksum and signatures.
>>> 
>>> - the source package build and test all run successfully.
>>> 
>>> - in both binary package(server & all), after change standalone.conf of
>>> allowloopback=true, 'bin/bookkeeper standalone' and 'bin/bookkeeper shell
>>> bookiesanity' runs well.
>>> 
>>> 
>>> 
>>> Best Regards.
>>> 
>>> 
>>> Jia Zhai
>>> 
>>> Beijing, China
>>> 
>>> Mobile: +86 15810491983
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Sep 28, 2020 at 3:52 PM Enrico Olivelli 
>>> wrote:
>>> 
 Hi,
 thanks to everyone who voted up to now
 
 we need at least 3 votes from PMC members in order to approve the
>>> release
 
 please take a little time to validate the release
 
 Enrico
 
 Il giorno dom 27 set 2020 alle ore 04:33 Rajan Dhabalia <
 rdhaba...@apache.org> ha scritto:
 
> +1 (binding)
> 
> - Environment: MacOS 10.15.6, jdk8
> - verified packages and checksum.
> - verified src build and tests.
> - verified bin with standalone server and bookiesanity is passing.
> 
> Thanks,
> Rajan
> 
> On Fri, Sep 25, 2020 at 8:40 AM Enrico Olivelli 
> wrote:
> 
>> Hi everyone,
>> Please review and vote on the release candidate #0 for the version
> 4.11.1,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific
>>> comments)
>> 
>> The complete staging area is available for your review, which
>>> includes:
>> * Release notes [1]
>> * The official Apache source and binary distributions to be
>>> deployed to
>> dist.apache.org [2]
>> * All artifacts to be deployed to the Maven Central Repository [3]
>> * Source code tag "v4.11.1-rc0" [4] with git sha
>> 8a92ffc984494a291667784d0b54ba251eba0451
>> 
>> BookKeeper's KEYS file contains PGP keys we used to sign this
>>> release:
>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>> 
>> Please download these packages and review this release candidate:
>> 
>> - Review release notes
>> - Download the source package (verify shasum, and asc) and follow
>>> the
>> instructions to build and run the bookkeeper service.
>> - Download the binary package (verify shasum, and asc) and follow
>>> the
>> instructions to run the bookkeeper service.
>> - Review maven repo, release tag, licenses, and any other things you
> think
>> it is important to a release.
>> 
>> The vote will be open for at least 72 hours. It is adopted by
>>> majority
>> approval, with at least 3 PMC affirmative votes.
>> 
>> Thanks,
>> Enrico Olivelli
>> 
>> [1] https://github.com/apache/bookkeeper/pull/2418
>> [2]
>> 
 
>>> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.11.1-rc0/
>> [3]
>> 
>> 
> 
 
>>> https://repository.apache.org/content/repositories/orgapachebookkeeper-1045/
>> [4] https://github.com/apache/bookkeeper/tree/v4.11.1-rc0
>> 
> 
 
>>> 
>> 

Build passing while tests are failing?

[Flavio Junqueira]

I was checking the builds of some of the PRs in the queue, and I noticed at 
least one in which there are test failures while the build indicates success:

 
https://github.com/apache/bookkeeper/runs/1174551641?check_suite_focus=true 


Check under "Run bookie tests". It seems to be incorrectly saying that the 
build is passing, am I misreading it or is it really a problem?

Thanks,
-Flavio

Re: Log truncation and sync up when bookie fails and rejoins

[Flavio Junqueira]

Hi Umesh,

The BookKeeper protocol is closer to register protocols than it is to consensus 
protocols, although we do need consensus for agreement on the closed state of a 
ledger. A ledger has a single writer, and either the writer closes it naturally 
or a reader needs to close it by running a recovery protocol that includes 
fencing the bookies. That's from the client perspective. From the bookie 
perspective, if a bookie of a ledger ensemble crashes while a ledger is being 
written to, then it is replaced and the history of the ledger is updated in the 
ledger metadata according to the last add confirmed by the crashed bookie. If 
the bookie crashes after the ledger is closed, then auto-recovery re-replicates 
the data.

You also ask for pointers. Are you looking for code pointers?

-Flavio 

> On 28 Jan 2020, at 10:15, Unmesh Joshi  wrote:
> 
> Hi,
> 
> In case of partial failures while implementing Replicated Log, there are
> few requirements which need to be fulfilled to sync logs on multiple nodes
> in case of node failure. e.g. In RAFT, if a node fails, there is a sync up
> that happens with communication from leader to push all the entries and
> truncation in case of some conflicting entries. Same happens with with
> Kafka when followers start re pulling data after startup. Consistency need
> to be maintained, by maintaining something like commitIndex or highwater
> mark to have the index till which the majority of the servers have
> acknowledged write.
> 
> In Apache BookKeeper I see that the consistency part is described well with
> last add confirmed description in below documentation,
> https://bookkeeper.apache.org/archives/docs/r4.4.0/bookkeeperProtocol.html
> 
> But who takes care of updating a particular Bookie in case it crashses (or
> temporarily partitioned) and rejoins the cluster? Is there any
> initialization protocol that is run at startup?
> In Kafka, we have something like Controller which managers this
> initialization, (or in RAFT its the current leader which will manage it)
> I am going through the code to understand, but any pointers or hints will
> be very helpful
> 
> Thanks,
> Unmesh

Re: [Draft] bookkeeper board report - August 2017

[Flavio Junqueira]

Thanks for putting this together, a few comments:

- Yahoo! Pulsar is now an Apache Incubator project, it is better to refer to it 
as Apache Pulsar instead.
- "the 4.5 release is also there. It is due in this August" -> "release 4.5 is 
almost complete and it is due in August". 
- "at Mar 2017" -> "in Mar 2017"
- The board typically looks for signs indicating that there are potential 
committers being considered or that are promising. For example, I'm not seeing 
the number of new contributors, which is a loose indicator of the potential for 
new committers. I suggest we say that there are a few promising candidates 
based on the growth of the community and contributions from joining developers. 
We don't have to list names.
- It might also be useful to expand on the mailing list activity. For example, 
we could refer to the number of merges to indicate that all the mailing list 
activity is inducing a very good number of merged contributions.

-Flavio


> On 01 Aug 2017, at 09:59, Jia Zhai  wrote:
> 
> Hi all,
> Here is a draft for our August report,  Please help review and comments on
> it.
> 
> Thanks a lot.
> -Jia
> 
> ==
> ## Description:
> 
> BookKeeper is a distributed, reliable, and high performance
> logging service. It has been used as a fundamental service to build
> high available and replicated services in companies like Twitter,
> Yahoo and Salesforce. It is also the log segment store for Apache
> DistributedLog (incubating) and message store for Yahoo Pulsar.
> 
> ## Issues:
> 
> There are no issues requiring board attention at this time.
> 
> ## Activity:
> 
> - DistributedLog graduates as a sub-project of BookKeeper. Consolidate the
> development with log stream library over bookkeeper.
> - BookKeeper moves the development to github - use github for tracking
> issues, moved to gitbox.
> - New contributions for a new bookkeeper website.
> 
> ## Health report:
> 
> - After a year of development among developers from multiple different
> companies, the 4.5 release is also there. It is due in this August, it
> includes exciting features like security support, netty 4 upgrade,
> weight-based placement policy, long poll reads.
> - We moved to gitbox to reduce the gap for new contributors to engage with
> the community.
> - DL graduates as a sub-project to consolidate the development efforts
> around log stream library.
> 
> ## PMC changes:
> 
> - Currently 9 PMC members.
> - JV Jujjuri was added to the PMC on Tue Jun 27 2017
> 
> ## Committer base changes:
> 
> - Currently 14 committers.
> - No new committers added in the last 3 months
> - Last committers addition was Enrico Olivelli and Charan Reddy G at Mar
> 2017
> 
> ## Releases:
> 
> - Last release was 4.4.0 on Sun May 15 2016
> 
> ## Mailing list activity:
> 
> Consolidate efforts coming from DL graduation as sub-project. More
> contributor/committer engagement cause the increased of mailing list
> activities.
> 
> - dev@bookkeeper.apache.org:
>- 90 subscribers (up 7 in the last 3 months):
>- 2706 emails sent to list (875 in previous quarter)
> 
> - iss...@bookkeeper.apache.org:
>- 6 subscribers (up 0 in the last 3 months)
> 
> - u...@bookkeeper.apache.org:
>- 100 subscribers (down -1 in the last 3 months):
>- 27 emails sent to list (23 in previous quarter)
> 
> 
> ## JIRA activity:
> - 60 JIRA tickets created in the last 3 months
> - 88 JIRA tickets closed/resolved in the last 3 months
> 
> - 85 Github issues created in the last 3 months
> - 42 Github issues closed/resolved in the last 3 months
> - 83 Github Pull Requests created in the last 3 months
> - 75 Github Pull Requests closed/resolved in the last 3 months

Re: Git protected branches

[Flavio Junqueira]

It's kind of odd that we have to go through infra to do this. Such a 
configuration feels like should be under the control of committers or at least 
the PMC.

Thanks for doing it in any case, Enrico.

-Flavio

> On 13 Jul 2017, at 01:55, Jia Zhai  wrote:
> 
> 👍
> 
> On Thu, Jul 13, 2017 at 5:34 AM, Venkateswara Rao Jujjuri > wrote:
> 
>> Awesome. !!
>> 
>> On Tue, Jul 11, 2017 at 6:06 AM, Enrico Olivelli 
>> wrote:
>> 
>>> The ticket has been closed.
>>> The protection is on
>>> 
>>> -- Enrico
>>> 
>>> 2017-07-07 11:22 GMT+02:00 Enrico Olivelli :
>>> 
 
 
 2017-07-07 1:51 GMT+02:00 Sijie Guo :
 
> +1 for disable force push to master
> 
 
 
 this is the issue:
 https://issues.apache.org/jira/browse/INFRA-14535
 
 I will send updates
 
 - Enrico
 
 
> 
> On Thu, Jul 6, 2017 at 1:16 PM, Enrico Olivelli 
> wrote:
> 
>> Other thoughts?
>> If there is no objection I would like to enable the force push
>> block.
> Next
>> week
>> 
>> Enrico
>> 
>> Il mar 4 lug 2017, 10:28 Enrico Olivelli  ha
> scritto:
>> 
>>> 2017-07-04 5:51 GMT+02:00 Jia Zhai :
 Agree to have them protected.
 Currently, seems these branches are not protected, at least
>> master
>> branch
 is not. The merge button is still available even "Jenkins: Maven
> clean
 install — Build finished." in some of PRs
 
 We may first need to make the test stable.
>>> 
>>> I think that is it enough to only protect against "force push",
>>> which
>>> makes impossible for anyone to 'rewrite' the "public history" of
>> the
>>> project
>>> Other checks are performed by QA test and by the committer who
>> takes
>>> responsibility for pushing the patch in the repo
>>> 
>>> 
>>> -- Enrico
>>> 
>>> 
 
 On Tue, Jul 4, 2017 at 5:10 AM, Enrico Olivelli <
> eolive...@gmail.com>
>>> wrote:
 
> Hi,
> I don't know current configuration but I would like to ensure
>>> that
> our
> release branches, that is master and branch-xx are 'protected'.
> I mean at least that 'force push' must be forbidden to
>> everyone.
> 
> In github it is simple to enable such protection, see
> https://help.github.com/articles/about-protected-branches/
> 
> I don't want to try btyy performing a force push. Maybe we can
>>> ask
>> infra
> about current configuration
> 
> Thoughts?
> 
> Enrico
> --
> 
> 
> -- Enrico Olivelli
> 
>>> 
>> --
>> 
>> 
>> -- Enrico Olivelli
>> 
> 
 
 
>>> 
>> 
>> 
>> 
>> --
>> Jvrao
>> ---
>> First they ignore you, then they laugh at you, then they fight you, then
>> you win. - Mahatma Gandhi
>> 

Re: [ANNOUNCE] JV Jujjuri joins the Apache BookKeeper PMC

[Flavio Junqueira]

Congrats, JV!

-F

> On 07 Jul 2017, at 00:11, Charan Reddy G  wrote:
> 
> Congrats JV! 
> 
> On Thu, Jul 6, 2017 at 9:29 AM, Enrico Olivelli  > wrote:
> Congrats JV !
> 
> Il gio 6 lug 2017, 18:24 Sijie Guo  > ha scritto:
> 
> > On behalf of the Apache BookKeeper PMC I am pleased to announce that JV
> > Jujjuri has accepted our invitation to become a PMC member on the Apache
> > BookKeeper project.
> >
> > JV has been an active contributor in many areas, including driving
> > community meetings, reviewing patches, helping with triaging bugs and
> > organizing the bookkeeper meetups. He has the ability and passion to move
> > the development of bookkeeper forward and has been speaking of Apache
> > BookKeeper in different conferences and meetups.
> >
> > Please join me in thanking JV for his contributions to date and
> > anticipation of many more contributions.
> >
> > Welcome to the PMC, JV!
> >
> > - Sijie
> >
> --
> 
> 
> -- Enrico Olivelli
> 

Re: [DISCUSS] BookKeeper - A High Performance and Low Latency Storage Service

[Flavio Junqueira]

I like the idea of updating the characterization of the project. It makes sense 
that the project shapes up over time according to use cases, and the 
description of years ago is not necessarily a good fit. I'm not sure if you are 
looking for suggestions for the text or just feedback on what BK has become. My 
only suggestion in addition to what everyone has said for the description 
itself is to make sure that it starts with a single crispy sentence of what BK 
is, like you suggested, something along the lines of append-only, fault 
tolerant, scalable, and low latency storage. I often find myself wanting some 
sentence that describes succinctly a project, in general, not specifically BK. 
We can then complement it with layers of more detail.

With respect to WAL vs. not only WAL, the problem I have faced over time with 
this is that the various ways of using an append-only abstraction are fairly 
nuanced. Logging is an overloaded term and qualifiers like "change" or 
"write-ahead" or "transaction" are not necessarily perceived in the same way, 
so I'm in favor of moving away from WAL as a primary way of characterizing BK 
to avoid any confusion around capabilities it provides.

I think it has been more than 6-7 years depending on how you count. The first 
commit is from 2011, but we had been doing it for at least a couple of years in 
Yahoo!. I'm personally very happy to see how it developed and also happy to see 
all the effort to consolidate. It is great to see the community growing and 
thriving.

-Flavio

> On 04 Jul 2017, at 17:17, Sijie Guo  wrote:
> 
> +1 agree.
> 
> On Jul 3, 2017 7:21 PM, "Venkateswara Rao Jujjuri" 
> wrote:
> 
>> Everything said on this thread is important and accurate. The description
>> on the website must be a story rather than a blurb.
>> We should talk about BK's strengths as Enrico pointed out, and because of
>> its versatility it became fundamental building block
>> for various other technologies and usecases. IMO, the entire story is very
>> powerful and appealing for BK.
>> 
>> On Mon, Jul 3, 2017 at 7:55 AM, Sijie Guo  wrote:
>> 
>>> On Mon, Jul 3, 2017 at 1:35 AM, Enrico Olivelli 
>>> wrote:
>>> 
 2017-07-03 7:00 GMT+02:00 Sijie Guo :
> Hi all,
> 
> It has been almost 6-7 years since Apache BookKeeper was born. Apache
> BookKeeper has already grown beyond a WAL system. Both Twitter and
>>> Yahoo
> have used it as their storage foundation for their messaging systems,
> Salesforce is using it for storage service. We also started talking
 Apache
> BookKeeper as a storage service since 2016 ([1][2]).
> 
> I am thinking of changing the description of Apache BookKeeper from a
>>> WAL
> system to "a High Performance and Low Latency Storage Service (that
> optimized for immutable/append-only data)" in the new website that we
>>> are
> building for BP-11
> .
> This will help us to bring more use cases/adoptions to the project
>> and
 help
> grow the community.
> 
> Any thoughts?
 
 My two cents
 
 Honestly when I found BookKeeper I was very happy because I found an
 "original" building block to build replicated state machines.
 I think that the main soul of BK is exactly to be a WAL and this is
 really "original".
 
 From my point of view the "key features" of BookKeeper are "Fencing"
 and "Last-Add-Confirmed protocol"
 
 BookKeeper is really good at storing data, but IMHO it is because it
 has been designed and implemented by very skilled engineers,
 BookKeeper needs to be "fast", because in order to provide a fast WAL
 you have to give an ultra-fast storage, because the essence of a WAL
 is  "durability" and usually "durable" comes together with 'sync' and
 so with 'slow' .
 
 I am not a "marketing expert" but IMHO we should stress on the
 distinctive features of BK in respect to other softwares.
 
 I am not against the proposed change but as an user I wanted to point
 that I happy with BK because it is the most powerful distributed WAL
 (and maybe it is the unique in the opensource/free world)
 
 I would like to write in the website that BookKeeper is the real
 answer to whom who are looking for a distributed WAL.
>>> 
>>> 
>>> Agree, we should make a clear case for distributed WAL.
>>> 
>>> It is worth just putting down all the use cases that BookKeeper has
>>> supported.
>>> 
>>> - WAL (e.g. HDFS NameNode)
>>> - Message Store (e.g. Apache Pulsar, Twitter Pub/Sub via DistributedLog)
>>> - Offset/Cursor Store (e.g. Apache Pulsar stores cursors in ledgers)
>>> - Object/Blob Store (e.g. in replicated state machine, storing state
>>> machine snapshots in ledgers. We used this pattern at distributedlog
>> based
>>> replicated state machines.)
>>> - ...
>>> 
>>> They are not all typical WAL use cases. But the comm

community meeting?

[Flavio Junqueira]

Is there a meeting scheduled for today? The page says that the next one is on 
Jun 15.

-Flavio

[jira] [Commented] (BOOKKEEPER-1099) Make bookie automatically create folders on new machine.

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059431#comment-16059431
 ] 

Flavio Junqueira commented on BOOKKEEPER-1099:
--

I agree with [~hustlmsp]. If you know what you're doing and want to embed the 
formatting, then it's fine, but having it done automatically and by default 
could lead to undesirable behavior and it touches the bookkeeper metadata. 

> Make bookie automatically create folders on new machine.
> 
>
> Key: BOOKKEEPER-1099
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1099
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Zhimeng Shi
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When bookie start and found it need run bookie format to create folders it 
> should do that automatically. That way will make new machine deployment and 
> machine replacement process much simpler without manually call the 
> bookieformat command.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Official Bookkeeper Docker Image

[Flavio Junqueira]

I haven't tested exhaustively, but as a first approximation, it works for me. I 
was able to get bookies up and running following the steps in the repo 
description, with the caveat that I had to use a different image name. I have 
also left some comments in the PR, but it looks like you found an error and 
will raise a new PR, is that right?

-Flavio

> On 21 Jun 2017, at 18:25, Francesco Caliumi - Diennea 
>  wrote:
> 
> I realized that in the PR there is an error. I will re-file ASAP the correct 
> PR (with probably the centos build in dir "4.4.0").
> 
> Sorry for the inconvenience.
> 
> On Wed, 2017-06-21 at 20:40 +0800, Jia Zhai wrote:
> 
> will take a look. Thanks a lot.
> 
> On Wed, Jun 21, 2017 at 3:38 PM, Francesco Caliumi - Diennea <
> francesco.cali...@diennea.com> wrote:
> 
> 
> 
> I will file a PR ASAP (naming the new directory "docker").
> 
> @Jia: Have you tried the new version of the build? It should resolve the
> zk metaformat issue.
> 
> On Tue, 2017-06-20 at 18:13 -0700, Sijie Guo wrote:
> 
> Jia,
> 
> Thank you for your clarification.
> 
> Francesco,
> 
> It might be good for you to send a pull request by putting your docker
> files to directory 'docker'. So people who has docker experiences can
> review it.
> 
> - Sijie
> 
> On Tue, Jun 20, 2017 at 5:37 PM, Jia Zhai 
> mailto:zhaiji...@gmail.com> aiji...@gmail.com>> wrote:
> 
> 
> 
> Regarding push docker images into apache account, there was an update in
> our bi-weekly-meeting notes, which including example of others.
> From my view, the first step is to have a "docker" dir, which is mentioned
> by sijie, in bookkeeper repo, to contains Docker Build related files.
> We could do it once we have our docker dir in the repo.
> 
> And regarding the docker use case, some use case may use it on K8S and
> DCOS, so automatic deploy would be a requirement.
> 
> Thanks.
> 
> On Wed, Jun 21, 2017 at 6:29 AM, Sijie Guo 
> mailto:guosi...@gmail.com> si...@gmail.com>> wrote:
> 
> 
> 
> On Tue, Jun 20, 2017 at 1:46 PM, Enrico Olivelli 
> mailto:eolive...@gmail.com>
> >
> wrote:
> 
> 
> 
> Il mar 20 giu 2017, 19:45 Sijie Guo 
> mailto:guosi...@gmail.com> si...@gmail.com>> ha scritto:
> 
> 
> 
> Hi Francesco,
> 
> - regarding the gpg, if we make pushing the new docker image as part
> 
> 
> 
> 
> 
> 
> of
> 
> 
> 
> 
> the
> 
> 
> release procedure, we should be able to support that.
> - regarding the directory name, is 'docker' enough? any reason that
> 
> 
> 
> 
> 
> 
> you
> 
> 
> 
> 
> 
> 
> prefix with 'bookkeeper-docker‘?
> 
> Also, does anyone follow up with INFRA - how does pushing docker
> 
> 
> 
> 
> 
> 
> image
> 
> 
> to
> 
> 
> 
> 
> apache account work?
> 
> 
> 
> 
> I can do it, but if Jia prefers to pick up this task for me is ok as
> 
> 
> 
> 
> well
> 
> 
> 
> 
> 
> 
> 
> 
> Oh, I remembered. If Jia is picking up this, we can wait for him.
> 
> 
> 
> 
> 
> Enrico
> 
> 
> 
> 
> - Sijie
> 
> 
> On Tue, Jun 20, 2017 at 5:09 AM, Francesco Caliumi - Diennea <
> francesco.cali...@diennea.com>
> wrote:
> 
> 
> 
> I wrote the first draft of the proposal: https://cwiki.apache.org/
> confluence/display/BOOKKEEPER/BP-10+-+Official+Bookkeeper+
> 
> 
> 
> 
> 
> 
> Docker+Image
> 
> 
> 
> 
> 
> 
> 
> Any suggestion would be appreciated.
> 
> On Thu, 2017-06-01 at 12:56 +, Francesco Caliumi - Diennea
> 
> 
> 
> 
> 
> 
> 
> 
> wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks Sijie, I will re-read docker hub guidelines and write the
> 
> 
> 
> 
> proposal
> 
> 
> 
> 
> accordingly.
> 
> Alas, I had no time for refreshing the topic, so I think that
> 
> 
> 
> 
> discussing
> 
> 
> 
> 
> it is useless in today meeting, but I think that in the next one I
> 
> 
> 
> 
> 
> 
> will
> 
> 
> 
> 
> be
> 
> 
> ready.
> 
> On Wed, 2017-05-31 at 13:12 -0700, Sijie Guo wrote:
> 
> Francesco,
> 
> I granted you the permission for confluence page.
> 
> I am okay with putting the docker build files in the main repo and
> configuring the jenkins build to automatically push the docker
> 
> 
> 
> 
> 
> 
> 
> 
> images
> 
> 
> 
> 
> to
> 
> 
> 
> 
> docker hub for snapshots.
> 
> - Sijie
> 
> On Wed, May 31, 2017 at 12:07 AM, Francesco Caliumi - Diennea <
> francesco.cali...@diennea.com 
> 
>  
> 
> 
> 
> > wrote:
> 
> 
> 
> 
> 
> Hi Sijie, the issue is https://issues.apache.org/
> jira/browse/BOOKKEEPER-974. There is also a meeting note (
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> 2017-2-10+Meeting+notes) that introduced the need to choose between
> keeping docker build files in a seperate repo (like zookkeeper
> 
> 
> 
> 
> 
> 
> 
> 
> does)
> 
> 
> or
>

Re: Official Bookkeeper Docker Image

[Flavio Junqueira]

Either centos or debian should work for us. Actually, another data point that I 
got from a colleague is that only centos works well with dcos-commons according 
to his experience, which others might not care, but we use with our stuff.

And, thanks for the clarification on configuration parameters.

-Flavio

> On 20 Jun 2017, at 17:41, Francesco Caliumi - Diennea 
>  wrote:
> 
> Hi Flavio,
> 
> On Tue, 2017-06-20 at 15:40 +0200, Flavio Junqueira wrote:
> 
> Hey Francesco,
> 
> Thanks for putting this together. I have a couple of comments from our side:
> 
> - I'm not sure whether this is an issue for others, and would actually be 
> interested in hearing whether it is or not, but the use of Alpine is a deal 
> breaker for us because of the GPL lawsuit stories around busybox.
> 
> 
> I didn't know about the GPL lawsuit and I don't even know about the possibile 
> implications. One possibility is to create an alternate build based on debian 
> or centos and keep alpine one with labels x.x.x-alpine (e.g 4.4.0-alpine) 
> like many projects do. I am curious about the community's views about this 
> matter.
> 
> 
> - Will we be setting parameters via environment variables?
> 
> 
> Via enviornment variables and via direct configuration files mounted on the 
> container. See https://github.com/caiok/bookkeeper-docker#configuration 
> <https://github.com/caiok/bookkeeper-docker#configuration>
> 
> 
> -Flavio
> 
> 
> 
> On 20 Jun 2017, at 14:09, Francesco Caliumi - Diennea 
>  <mailto:francesco.cali...@diennea.com><mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com>>> wrote:
> 
> I wrote the first draft of the proposal: 
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-10+-+Official+Bookkeeper+Docker+Image
>  
> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-10+-+Official+Bookkeeper+Docker+Image><https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-10+-+Official+Bookkeeper+Docker+Image
>  
> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-10+-+Official+Bookkeeper+Docker+Image>>
> 
> Any suggestion would be appreciated.
> 
> On Thu, 2017-06-01 at 12:56 +, Francesco Caliumi - Diennea wrote:
> 
> Thanks Sijie, I will re-read docker hub guidelines and write the proposal 
> accordingly.
> 
> Alas, I had no time for refreshing the topic, so I think that discussing it 
> is useless in today meeting, but I think that in the next one I will be ready.
> 
> On Wed, 2017-05-31 at 13:12 -0700, Sijie Guo wrote:
> 
> Francesco,
> 
> I granted you the permission for confluence page.
> 
> I am okay with putting the docker build files in the main repo and
> configuring the jenkins build to automatically push the docker images to
> docker hub for snapshots.
> 
> - Sijie
> 
> On Wed, May 31, 2017 at 12:07 AM, Francesco Caliumi - Diennea <
> francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com><mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com>> <mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com>><mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com><mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com>>><mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com><mailto:francesco.cali...@diennea.com 
> <mailto:francesco.cali...@diennea.com>>>> wrote:
> 
> 
> 
> Hi Sijie, the issue is https://issues.apache.org/ 
> <https://issues.apache.org/> <https://issues.apache.org/ 
> <https://issues.apache.org/>>
> jira/browse/BOOKKEEPER-974. There is also a meeting note (
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/ 
> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/><https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>  <https://cwiki.apache.org/confluence/display/BOOKKEEPER/>>
> 2017-2-10+Meeting+notes) that introduced the need to choose between
> keeping docker build files in a seperate repo (like zookkeeper does) or put
> them in bookkeeper main repo.
> 
> I need to refresh my mind on the matter anyway, I'll try to do it before
> next meeting.
> 
> My user on confluence is "caio".
> 
> Francesco.
> 
> On Tue, 2017-05-30 at 10:48 -0700, Sijie Guo wrote:
> 
> Ah, sorry my bad. Do you mind pointing me the JIRA/pull request that needs
> responses from me?
> 
> What is your confluence wiki handle? I can grant you the edit permission.
> 
> - Sijie
> 
> On Mon, May 29, 2017 at 12:44 AM, F

Re: Official Bookkeeper Docker Image

[Flavio Junqueira]

Hey Francesco,

Thanks for putting this together. I have a couple of comments from our side:

- I'm not sure whether this is an issue for others, and would actually be 
interested in hearing whether it is or not, but the use of Alpine is a deal 
breaker for us because of the GPL lawsuit stories around busybox.
- Will we be setting parameters via environment variables?

-Flavio
 
> On 20 Jun 2017, at 14:09, Francesco Caliumi - Diennea 
>  wrote:
> 
> I wrote the first draft of the proposal: 
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/BP-10+-+Official+Bookkeeper+Docker+Image
>  
> 
> 
> Any suggestion would be appreciated.
> 
> On Thu, 2017-06-01 at 12:56 +, Francesco Caliumi - Diennea wrote:
> 
> Thanks Sijie, I will re-read docker hub guidelines and write the proposal 
> accordingly.
> 
> Alas, I had no time for refreshing the topic, so I think that discussing it 
> is useless in today meeting, but I think that in the next one I will be ready.
> 
> On Wed, 2017-05-31 at 13:12 -0700, Sijie Guo wrote:
> 
> Francesco,
> 
> I granted you the permission for confluence page.
> 
> I am okay with putting the docker build files in the main repo and
> configuring the jenkins build to automatically push the docker images to
> docker hub for snapshots.
> 
> - Sijie
> 
> On Wed, May 31, 2017 at 12:07 AM, Francesco Caliumi - Diennea <
> francesco.cali...@diennea.com 
>  > >> wrote:
> 
> 
> 
> Hi Sijie, the issue is https://issues.apache.org/ 
> jira/browse/BOOKKEEPER-974. There is also a meeting note (
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/ 
> 
> 2017-2-10+Meeting+notes) that introduced the need to choose between
> keeping docker build files in a seperate repo (like zookkeeper does) or put
> them in bookkeeper main repo.
> 
> I need to refresh my mind on the matter anyway, I'll try to do it before
> next meeting.
> 
> My user on confluence is "caio".
> 
> Francesco.
> 
> On Tue, 2017-05-30 at 10:48 -0700, Sijie Guo wrote:
> 
> Ah, sorry my bad. Do you mind pointing me the JIRA/pull request that needs
> responses from me?
> 
> What is your confluence wiki handle? I can grant you the edit permission.
> 
> - Sijie
> 
> On Mon, May 29, 2017 at 12:44 AM, Francesco Caliumi - Diennea <
> francesco.cali...@diennea.com 
>  > > >>
> wrote:
> 
> 
> 
> Hi all,
> some months ago I wrote a prototype of a docker image that follows the
> standard rules of official docker images.
> 
> The work was suspended waiting for some responses from Sijie, but I think
> it would be easier to follow the thread if it is tied to a bookkeeper
> proposal on Confluence.
> 
> @Sijie: could you give me edit access to confluence in order to write down
> the state of the work and the open points that need a decision from the
> community?
> 
> Thanks,
> Francesco.
> 
> 
> --
> 
> Francesco Caliumi
> Developer @ Diennea - MagNews
> Tel.: (+39) 0546 066100 - Int. 266
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
> 
> [Magnews.it]
> 
> [Linkedin]
> [Twitter]   [Facebook] <
> http://www.facebook.com/pages/MagNews/197617841797>  [Newsletter] <
> http://www.magnews.it/it/iscriviti-alla-newsletter>
> 
> 
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> email marketing! http://www.magnews.it/newsletter/
> 
> The information in this email is confidential and may be legally
> privileged. If you are not the intended recipient please notify the sender
> immediately and destroy this email. Any unauthorized, direct or indirect,
> disclosure, copying, storage, distribution or other use is strictly
> forbidden.
> 
> 
> 
> --
> 
> Francesco Caliumi
> Developer @ Diennea - MagNews
> Tel.: (+39) 0546 066100 - Int. 266
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
> 
> [Magnews.it]
> 
> [Linkedin]
> [Twitter]   [Facebook] <
> http://www.facebook.com/pages/MagNews/197617841797>  [Newsletter] <
> http://www.magnews.it/it/iscriviti-alla-newsletter>
> 
> 
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> email marketing! http://www.magnews.it/newsletter/
> 
> 

Re: Jenkins Pull-Request Build

[Flavio Junqueira]

Yeah, I think it is fine to delete that job. Given this line in the config:

curl http://people.apache.org/~ivank/debug.diff | patch -p0

I suspect Ivan created it for debugging. Does anyone else see a problem with 
deleting this job:

https://builds.apache.org/view/All/job/bookkeeper-debug/ 


-Flavio

> On 09 Jun 2017, at 15:59, Enrico Olivelli  wrote:
> 
> There is a bookkeeper-debug job which I think it should be dropped
> can you do it Sijie or Flavio ?
> 
> If you are OK I think I can do it by myself
> 
> -- Enrico
> 
> 2017-06-01 20:17 GMT+02:00 Sijie Guo :
>> Yeah. I updated the description for the master job.
>> 
>> (yes. this wiki page is cloned from DL.)
>> 
>> On Thu, Jun 1, 2017 at 11:14 AM, Enrico Olivelli 
>> wrote:
>> 
>>> Great!
>>> There is a reference to distributed log I think.
>>> 
>>> Maybe you should mention that the master job now deploys snapshots to
>>> repository.apache.org
>>> 
>>> Enrico
>>> 
>>> Il gio 1 giu 2017, 20:10 Sijie Guo  ha scritto:
>>> 
 Cleaned up the jenkins jobs.
 
 Also put up a wiki page documenting this -
 
 https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>> Continuous+Integration+Setup
 
 - Sijie
 
 On Thu, Jun 1, 2017 at 11:05 AM, Enrico Olivelli 
 wrote:
 
> Yep.
> Consider deleting the 'look for patches' job as we are only using
>>> github
> now
> 
> Il gio 1 giu 2017, 19:56 Sijie Guo  ha scritto:
> 
>> since there is no objection coming up, deleting the broken
>> "bookkeeper-master-git-pullrequest" jenkins job.
>> 
>> - Sijie
>> 
>> On Thu, Jun 1, 2017 at 12:50 AM, Sijie Guo 
>>> wrote:
>> 
>>> Hi all,
>>> 
>>> Currently, the bookkeeper-master-git-pullrequest [1] build is
>>> broken
> due
>>> to failing to find the "mvn" command. This build is using
>>> customized
>> shell
>>> script to execute the maven command. Not sure if it is related to
>>> any
>>> environment changes in Jenkins.
>>> 
>>> I re-created a new job to do the pull request build
>> "bookkeeper-precommit-pullrequest"
>>> [2] using jenkins' maven build rather than using customized shell
> script.
>>> It seems to be working well.
>>> 
>>> If there is no objections, I am going to delete [1]
>> bookkeeper-master-git-pullrequest.
>>> Any thoughts?
>>> 
>>> [1] https://builds.apache.org/job/bookkeeper-master-git-
>>> pullrequest
>>> [2] https://builds.apache.org/job/bookkeeper-precommit-
>>> pullrequest/
>>> 
>>> - Sijie
>>> 
>> 
> --
> 
> 
> -- Enrico Olivelli
> 
 
>>> --
>>> 
>>> 
>>> -- Enrico Olivelli
>>> 

Re: GitHub permissions

[Flavio Junqueira]

Agreed that it is best to have the Apache pre-commit build run successfully 
before merging.

-Flavio
 
> On 09 Jun 2017, at 11:12, Enrico Olivelli  wrote:
> 
> 2017-06-09 11:06 GMT+02:00 Flavio Junqueira :
>> I just checked the configuration in jenkins and the retention of a build is 
>> currently 14 days and no max number of jobs. That should be sufficient, 
>> otherwise we might end up overwhelming jenkins.
>> 
> 
> Yet I think that actual configuration is ok from this point of view.
> 
> As we have decided to start using github issues more permissions
> should be given to committers or at least to PMCs
> 
> I think that PR are indeed "issues" for GitHub
> 
> If you think it is not the case I'm fine anyway.
> 
> Usually I run tests on my laptop before approving or merging patches
> 
> 
>> -Flavio
>> 
>>> On 09 Jun 2017, at 10:58, Flavio Junqueira  wrote:
>>> 
>>> I can increase retention if that helps.
>>> 
>>> -Flavio
>>> 
>>>> On 09 Jun 2017, at 10:51, Enrico Olivelli  wrote:
>>>> 
>>>> Hi,
>>>> I would like to ask infra permissions to edit pull requests of other
>>>> users, at least "close"/"reopen" to force automatic QA.
>>>> 
>>>> We have a short backlog of PR QA builds so after some time it is not
>>>> possible to see old tests results
>>>> 
>>>> Maybe this permission can be extended automatically to every committer
>>>> of the project.
>>>> 
>>>> I think that not every of us (committers) has bound his github account
>>>> to the apache id, in fact when you are "linked" in the comments on PRs
>>>> you are listed as "member", otherwise as "contributor")
>>>> 
>>>> Thoughts ?
>>>> 
>>>> Enrico
>>> 
>> 

Re: GitHub permissions

[Flavio Junqueira]

I just checked the configuration in jenkins and the retention of a build is 
currently 14 days and no max number of jobs. That should be sufficient, 
otherwise we might end up overwhelming jenkins.

-Flavio

> On 09 Jun 2017, at 10:58, Flavio Junqueira  wrote:
> 
> I can increase retention if that helps.
> 
> -Flavio
> 
>> On 09 Jun 2017, at 10:51, Enrico Olivelli  wrote:
>> 
>> Hi,
>> I would like to ask infra permissions to edit pull requests of other
>> users, at least "close"/"reopen" to force automatic QA.
>> 
>> We have a short backlog of PR QA builds so after some time it is not
>> possible to see old tests results
>> 
>> Maybe this permission can be extended automatically to every committer
>> of the project.
>> 
>> I think that not every of us (committers) has bound his github account
>> to the apache id, in fact when you are "linked" in the comments on PRs
>> you are listed as "member", otherwise as "contributor")
>> 
>> Thoughts ?
>> 
>> Enrico
> 

Re: GitHub permissions

[Flavio Junqueira]

I can increase retention if that helps.

-Flavio

> On 09 Jun 2017, at 10:51, Enrico Olivelli  wrote:
> 
> Hi,
> I would like to ask infra permissions to edit pull requests of other
> users, at least "close"/"reopen" to force automatic QA.
> 
> We have a short backlog of PR QA builds so after some time it is not
> possible to see old tests results
> 
> Maybe this permission can be extended automatically to every committer
> of the project.
> 
> I think that not every of us (committers) has bound his github account
> to the apache id, in fact when you are "linked" in the comments on PRs
> you are listed as "member", otherwise as "contributor")
> 
> Thoughts ?
> 
> Enrico

Re: Releases 4.5.0 and 4.4.1

[Flavio Junqueira]

Ok, thanks for the clarification. I was having a look at the issues you pointed 
out:

- BK-670: That's quite an old issue! There are 5 sub-tasks, 2 resolved, 1 patch 
available, and 2 unresolved. I can see that you have reviewers for the patch 
available one, but let me know if you need help reviewing it. For the remaining 
two, is there anyone looking at them?
- BK-575: There is one reviewer assigned for the remaining sub-task. I can also 
review if you want me to.

-Flavio

> On 08 Jun 2017, at 12:39, Sijie Guo  wrote:
> 
> BOOKKEEPER-670 (long poll) and BOOKKEEPER-575 (SSL) are the two remaining
> big features for 4.5.0.  The timeline for 4.5.0 for completing
> BOOKKEEPER-670 is in 2 weeks. The SSL one is almost done, waiting for
> Kishore pushing a new pull request.
> 
> We don't do 4.4.1 because we are converging Twitter/Yahoo/Salesforce
> branches in 4.5.0, unless there is a critical bug coming up. We defer
> creating 4.4.1 until it is really needed.
> 
> - Sijie
> 
> On Thu, Jun 8, 2017 at 2:44 AM, Flavio Junqueira  wrote:
> 
>> Hi there,
>> 
>> I wanted to check about the status of our releases. According to jira,
>> there are only 3 blockers and 2 of them are documentation issues for 4.5.0.
>> Does it mean that the we should be able to release 4.5.0 shortly? There are
>> already 116 issues resolved for 4.5.0 also according to jira.
>> 
>> I don't see an unreleased version 4.4.1 on jira, are we not doing a bug
>> fix release for the 4.4 branch? I can't believe that the 4.4 branch has no
>> bugs. :-)
>> 
>> I'd appreciate some feedback here.
>> 
>> -Flavio

Releases 4.5.0 and 4.4.1

[Flavio Junqueira]

Hi there,

I wanted to check about the status of our releases. According to jira, there 
are only 3 blockers and 2 of them are documentation issues for 4.5.0. Does it 
mean that the we should be able to release 4.5.0 shortly? There are already 116 
issues resolved for 4.5.0 also according to jira.

I don't see an unreleased version 4.4.1 on jira, are we not doing a bug fix 
release for the 4.4 branch? I can't believe that the 4.4 branch has no bugs. :-)

I'd appreciate some feedback here.

-Flavio 

Re: [VOTE] Use Github Issues for Issue Tracking

[Flavio Junqueira]

Sure, feel free to close the vote. Thanks for bearing with me.

-Flavio

> On 06 Jun 2017, at 18:07, Sijie Guo  wrote:
> 
> On Tue, Jun 6, 2017 at 12:41 AM, Flavio Junqueira  <mailto:f...@apache.org>> wrote:
> 
>> 
>>> On 05 Jun 2017, at 18:30, Sijie Guo  wrote:
>>> 
>>> On Mon, Jun 5, 2017 at 2:53 AM, Flavio Junqueira  wrote:
>>> 
>>>> Ok, so why don't we make clear what we are voting for?
>>> 
>>> If this vote is approved, then the outcome is that the community is going
>>>> to work on a plan to move to github issues, and that plan will be voted?
>>> 
>>> 
>>> I believed that I made the clarification on what to vote in this thread.
>>> 
>> 
>> This is probably where we are not converging. I don't know what passing
>> this vote entails other than a wish to move to github issues in the case it
>> passes. I'd much rather vote on a proposal that includes the precise steps
>> so that we can assess whether that's a good move or not. But wait, you say
>> something interesting next...
>> 
>>> Implementing the new github workflow would be a separated bookkeeper
>>> proposal and that implementation will follow bookkeeper proposal process
>> to
>>> vote, which is a lazy majority vote from committers.
>>> 
>>> Anyone are welcome to work on implementing the workflow.
>>> 
>> 
>> I see, we aren't really voting on a change or committing to anything, you
>> just want to assess preferences.
>> 
>>> 
>>>> Or is it going to be considered a code change and require just a +1
>> from a
>>>> committer? The reason I'm insisting is that I like the idea of moving to
>>>> github issues, but I'm not sure what we are voting for precisely.
>>>> 
>>>> As for the approval process, it is not clear from the list of actions
>>>> which one is the one to take as not a single one is a good match. When
>> this
>>>> is the case, the binding votes should be the PMC ones, and given that
>> this
>>>> is a pretty significant change, I'd be more comfortable with lazy
>>>> consensus. Also, note that the shared resources of the project are
>>>> responsibility of the PMC. While we can do it in the open to gather
>>>> feedback from the community, and in fact, I'd say that this makes a lot
>> of
>>>> sense to do it, it is the PMC responsibility.
>>>> 
>>> 
>>> Thanks for pointing it out. Yes, the binding votes for this will be
>> active
>>> PMC members.
>>> 
>>> Although I have a different personal opinion on the responsibility on
>>> shared resources. My feel is the shared resources are more developer
>>> resources, which committers/developers should own more responsibilities.
>> We
>>> can discuss it separately.
>> 
>> It is not really my preference, this is in the bylaws of the project. It
>> is a responsibility of the PMC to maintain shared resources. I think that
>> the issue tracking application is a shared resource.
>> 
>>> 
>>> 
>>>> My recommendation is that we clarify what we are voting for and call a
>>>> second vote. I'd happy to help out if it comes to that.
>>>> 
>>> 
>>> Let me know if the clarification is good to you or not.
>> 
>> Possibly, if this vote is about declaring intention rather than
>> committing, then I'm totally good with that.
>> 
> 
> Flavio, if you are good with that, I would like to close this vote, so that
> we can move forward to make it happen.
> 
> If you have a proposal on the workflow, feel free to raise it up as a BP.
> 
> - Sijie
> 
> 
>> 
>> -Flavio
>> 
>>> 
>>> 
>>>> 
>>>> Thanks,
>>>> -Flavio
>>>> 
>>>> 
>>>>> On 03 Jun 2017, at 19:29, Sijie Guo  wrote:
>>>>> 
>>>>> On Sat, Jun 3, 2017 at 7:39 AM, Flavio Junqueira 
>> wrote:
>>>>> 
>>>>>> Thanks for pushing this through, Sijie. I have a couple of concerns
>>>> about
>>>>>> this proposal:
>>>>>> 
>>>>>> 1- There is a proposal, but I'm not seeing a workflow defined for
>> github
>>>>>> issues. Should we define one and make that part of the vote?
>>>>>> 
>>>>> 
>>>>> Defining a workflow requires efforts an

Re: [VOTE] Use Github Issues for Issue Tracking

[Flavio Junqueira]

> On 05 Jun 2017, at 18:30, Sijie Guo  wrote:
> 
> On Mon, Jun 5, 2017 at 2:53 AM, Flavio Junqueira  wrote:
> 
>> Ok, so why don't we make clear what we are voting for?
> 
> If this vote is approved, then the outcome is that the community is going
>> to work on a plan to move to github issues, and that plan will be voted?
> 
> 
> I believed that I made the clarification on what to vote in this thread.
> 

This is probably where we are not converging. I don't know what passing this 
vote entails other than a wish to move to github issues in the case it passes. 
I'd much rather vote on a proposal that includes the precise steps so that we 
can assess whether that's a good move or not. But wait, you say something 
interesting next...

> Implementing the new github workflow would be a separated bookkeeper
> proposal and that implementation will follow bookkeeper proposal process to
> vote, which is a lazy majority vote from committers.
> 
> Anyone are welcome to work on implementing the workflow.
> 

I see, we aren't really voting on a change or committing to anything, you just 
want to assess preferences.

> 
>> Or is it going to be considered a code change and require just a +1 from a
>> committer? The reason I'm insisting is that I like the idea of moving to
>> github issues, but I'm not sure what we are voting for precisely.
>> 
>> As for the approval process, it is not clear from the list of actions
>> which one is the one to take as not a single one is a good match. When this
>> is the case, the binding votes should be the PMC ones, and given that this
>> is a pretty significant change, I'd be more comfortable with lazy
>> consensus. Also, note that the shared resources of the project are
>> responsibility of the PMC. While we can do it in the open to gather
>> feedback from the community, and in fact, I'd say that this makes a lot of
>> sense to do it, it is the PMC responsibility.
>> 
> 
> Thanks for pointing it out. Yes, the binding votes for this will be active
> PMC members.
> 
> Although I have a different personal opinion on the responsibility on
> shared resources. My feel is the shared resources are more developer
> resources, which committers/developers should own more responsibilities. We
> can discuss it separately.

It is not really my preference, this is in the bylaws of the project. It is a 
responsibility of the PMC to maintain shared resources. I think that the issue 
tracking application is a shared resource.

> 
> 
>> My recommendation is that we clarify what we are voting for and call a
>> second vote. I'd happy to help out if it comes to that.
>> 
> 
> Let me know if the clarification is good to you or not.

Possibly, if this vote is about declaring intention rather than committing, 
then I'm totally good with that.

-Flavio

> 
> 
>> 
>> Thanks,
>> -Flavio
>> 
>> 
>>> On 03 Jun 2017, at 19:29, Sijie Guo  wrote:
>>> 
>>> On Sat, Jun 3, 2017 at 7:39 AM, Flavio Junqueira  wrote:
>>> 
>>>> Thanks for pushing this through, Sijie. I have a couple of concerns
>> about
>>>> this proposal:
>>>> 
>>>> 1- There is a proposal, but I'm not seeing a workflow defined for github
>>>> issues. Should we define one and make that part of the vote?
>>>> 
>>> 
>>> Defining a workflow requires efforts and putting attentions into the
>>> community. The vote carries the proposal from last sync up to achieve a
>>> consensus in the community - "shall we try out github issues". If the
>>> community agrees on this, we can call for volunteers to drive the
>>> discussion on defining a workflow and help with logistics (e.g. works
>> with
>>> INFRA team) to make it happen.
>>> 
>>> If the community isn't interested in moving forward, it doesn't make any
>>> sense to put the efforts on it.
>>> 
>>> 
>>>> 2- I'm not sure what the binding votes are and what the approval process
>>>> is. Are we following the project bylaws here?
>>>> 
>>> 
>>> http://bookkeeper.apache.org/bylaws.html
>>> 
>>> This is related to development/release workflow, is counted as "Release
>>> Plan". It follows the "Lazy majority" approval process, any votes from
>>> committers are binding votes.
>>> 
>>> The voting period is 3 days.
>>> 
>>> - Sijie
>>> 
>>> 
>>>> 
>>>> Thanks,
>>>> -Flavio
>>>> 

Re: Publish SNAPSHOT artifacts on Maven Central

[Flavio Junqueira]

I have used Jenkins extensively, but not the job builder you're referring to, 
which I think is this one:

https://docs.openstack.org/infra/jenkins-job-builder/ 


I like Travis too, but it is indeed more limited as Sijie says. If all you need 
is to run nightly and PR builds, then Travis will do it just fine.

-Flavio

> On 05 Jun 2017, at 15:31, Enrico Olivelli  wrote:
> 
> I took a look at jenkins job builder, it seems a great tool.
> Two points before getting started:
> 1) do anyone ever used this tool or something like that?
> 2) it needs calling the API and it is possible to delete all jenkins jobs
> with a single command, surely there will be some kind of permissions to be
> set on jenkins side, but maybe we should ask to infra team if there is
> already some other user of such tool in ASF
> 
> Travis looks really more simple from this point of view at first glance
> 
> Enrico
> 
> Il sab 3 giu 2017, 08:38 Enrico Olivelli  ha scritto:
> 
>> 
>> 
>> Il sab 3 giu 2017, 02:24 Sijie Guo  ha scritto:
>> 
>>> On Sat, Apr 22, 2017 at 2:43 AM, Ivan Kelly  wrote:
>>> 
> I volunteer to setup the jenkins job and deal with infra
 Is there already a nightly job? I remember in the past there was only
 a precommit job. It would be good to have a nightly job so that flakes
 can be easily tracked. It's hard to do this with a precommit, as the
 failures may be in the patch that's being tested.
 
 I'd also recommend using jenkins job builder [1] for the job setup.
 This way the job specification can be pushed to the bookkeeper repo.
 
>>> 
>>> I actually liked the idea to also have the job specification in bookkeeper
>>> repo. So we also manages the job specification changes in same review
>>> process. anyone is interested in exploring this?
>>> 
>> 
>> Sure I will try
>> 
>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
 
 -Ivan
 
 
 [1] https://docs.openstack.org/infra/jenkins-job-builder/
 
>>> 
>> --
>> 
>> 
>> -- Enrico Olivelli
>> 
> -- 
> 
> 
> -- Enrico Olivelli

Re: [DISCUSS] Move Bookkeeper Website from CMS to Jekyll (or other static generators)

[Flavio Junqueira]

I don't have a super strong opinion here, but I'm not sure I understand the 
concern. The textile files are stored in the repo, so any doc changes should be 
reviewed and committed as any other code change, no? Granted that it is in the 
hands of a committer to push the changes to the web site, which isn't very 
friendly.

In the case we move out of CMS, where would the site be hosted?

-Flavio

> On 03 Jun 2017, at 21:42, Enrico Olivelli  wrote:
> 
> Our site is written using Textile, I found this
> https://github.com/jekyll/jekyll-textile-converter maybe the switch to
> Jekyll will be easy
> 
> The other problem will be to switch the cms, maybe just a request to
> infra to switch to github pages will be enough
> 
> Enrico
> 
> 2017-06-03 19:15 GMT+02:00 Sijie Guo :
>> I don't think there is any enforcements from Apache INFRA side. You can use
>> any technology for hosting website and documentation. I do see a lot of
>> projects using Jekyll-like solutions for the website, where they typically
>> have a separate XXX-site git repo and use gitpubsub (which is just a simply
>> git push) for publishing the content.
>> 
>> For DL, originally the website was generated by internally tool called
>> DocBird. When we open sourced DL, we push the generated static content to
>> gh-pages and uses github pages for hosting the content. After we moved to
>> incubator, we changed to use Jekyll to generate the static content and add
>> the generated content on asf-site branch.
>> 
>> For me, I don't care what technologies we are using. I'd like a simpler
>> workflow, same/similar as the source code workflow and every changes should
>> be under same/similar review process. Any git-based, github-friendly
>> solution would be preferred here. If we agree on moving, we should call for
>> volunteers to help with this.
>> 
>> - Sijie
>> 
>> On Sat, Jun 3, 2017 at 4:24 AM, Enrico Olivelli  wrote:
>> 
>>> It has been some time since you made this proposal, on some ticket.
>>> At the moment I did not make any concrete proposal because I wanted to
>>> study how to make the conversion.
>>> I am in favour of switching to a more popular sokution like jekyll and
>>> maybe markdown language
>>> Using git will be good as well. It will be more integrated.
>>> 
>>> I am not an expert I think we need some volunteer toto carry on the
>>> migration.
>>> 
>>> On the infra side it would be good to listen to experiences from other
>>> apache projects. On new DL site what technology are you using?
>>> Kafka website has been restyled some month ago, maybe we can take a look
>>> 
>>> -- Enrico
>>> 
>>> Il sab 3 giu 2017, 02:21 Sijie Guo  ha scritto:
>>> 
 I'd like to raise another discussion about moving bookkeeper website from
 CMS to other static generators (e.g. Jekyll, Hugo).
 
 BookKeeper uses Apache CMS for generating the documentation and website
 [1]. The website source code is hosted at a svn repo, which now becomes
 obsolete from
 our current review/workflow. I also heard committers complaining about
>>> the
 steps to get a change out.
 
 I think it is the time to also think of moving the website away from CMS
>>> to
 a more Github friendly solution.
 
 We should consider follows for the new solution:
 
 - have similar review flow as the main source code (github pull
>>> requests).
 - developers can easy to folk and run/validate their changes locally, and
 maybe also easier for the other reviews to verify.
 
 Any thoughts?
 
 [1] :
 
 https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>> Building+the+website+and+documentation
 
>>> --
>>> 
>>> 
>>> -- Enrico Olivelli
>>> 

Re: [VOTE] Use Github Issues for Issue Tracking

[Flavio Junqueira]

Ok, so why don't we make clear what we are voting for? If this vote is 
approved, then the outcome is that the community is going to work on a plan to 
move to github issues, and that plan will be voted? Or is it going to be 
considered a code change and require just a +1 from a committer? The reason I'm 
insisting is that I like the idea of moving to github issues, but I'm not sure 
what we are voting for precisely.

As for the approval process, it is not clear from the list of actions which one 
is the one to take as not a single one is a good match. When this is the case, 
the binding votes should be the PMC ones, and given that this is a pretty 
significant change, I'd be more comfortable with lazy consensus. Also, note 
that the shared resources of the project are responsibility of the PMC. While 
we can do it in the open to gather feedback from the community, and in fact, 
I'd say that this makes a lot of sense to do it, it is the PMC responsibility.

My recommendation is that we clarify what we are voting for and call a second 
vote. I'd happy to help out if it comes to that.

Thanks,
-Flavio


> On 03 Jun 2017, at 19:29, Sijie Guo  wrote:
> 
> On Sat, Jun 3, 2017 at 7:39 AM, Flavio Junqueira  wrote:
> 
>> Thanks for pushing this through, Sijie. I have a couple of concerns about
>> this proposal:
>> 
>> 1- There is a proposal, but I'm not seeing a workflow defined for github
>> issues. Should we define one and make that part of the vote?
>> 
> 
> Defining a workflow requires efforts and putting attentions into the
> community. The vote carries the proposal from last sync up to achieve a
> consensus in the community - "shall we try out github issues". If the
> community agrees on this, we can call for volunteers to drive the
> discussion on defining a workflow and help with logistics (e.g. works with
> INFRA team) to make it happen.
> 
> If the community isn't interested in moving forward, it doesn't make any
> sense to put the efforts on it.
> 
> 
>> 2- I'm not sure what the binding votes are and what the approval process
>> is. Are we following the project bylaws here?
>> 
> 
> http://bookkeeper.apache.org/bylaws.html
> 
> This is related to development/release workflow, is counted as "Release
> Plan". It follows the "Lazy majority" approval process, any votes from
> committers are binding votes.
> 
> The voting period is 3 days.
> 
> - Sijie
> 
> 
>> 
>> Thanks,
>> -Flavio
>> 
>>> On 01 Jun 2017, at 21:59, Sijie Guo  wrote:
>>> 
>>> Per the community meeting
>>> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>> 2017-06-01+Meeting+notes>
>>> this morning, JV proposed to start use Github issues for issue tracking
>> for
>>> a few months and see how does it work out. I am starting this email
>> thread
>>> to vote for this.
>>> 
>>> The vote will be:
>>> 
>>> - Start using Github issues/pull requests for issue tracking for 3
>> months.
>>> - During this 3 months, we will continue using both JIRA and Github
>> issues.
>>> - After 3 months, if the community decides whether we should continue
>> using
>>> Github issues or moving from JIRA to Github issues. (The final decision
>>> will be a separate vote in 3 months)
>>> 
>>> Please vote +1 if in favor of trying out Github issues and -1 if not.
>>> 
>>> See below thread and community meeting notes
>>> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>> 2017-06-01+Meeting+notes>
>>> for reference:
>>> 
>>> http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/201705.mbox/%
>> 3CCAO2yDyYKmUiSfGfkGCKtfP8mmQtcJubGoMO-KsWsjM9_3pOd0Q%40mail.gmail.com%3E
>>> 
>>> - Sijie
>> 
>> 

Re: [VOTE] Use Github Issues for Issue Tracking

[Flavio Junqueira]

Thanks for pushing this through, Sijie. I have a couple of concerns about this 
proposal:

1- There is a proposal, but I'm not seeing a workflow defined for github 
issues. Should we define one and make that part of the vote?
2- I'm not sure what the binding votes are and what the approval process is. 
Are we following the project bylaws here?

Thanks,
-Flavio

> On 01 Jun 2017, at 21:59, Sijie Guo  wrote:
> 
> Per the community meeting
> 
> this morning, JV proposed to start use Github issues for issue tracking for
> a few months and see how does it work out. I am starting this email thread
> to vote for this.
> 
> The vote will be:
> 
> - Start using Github issues/pull requests for issue tracking for 3 months.
> - During this 3 months, we will continue using both JIRA and Github issues.
> - After 3 months, if the community decides whether we should continue using
> Github issues or moving from JIRA to Github issues. (The final decision
> will be a separate vote in 3 months)
> 
> Please vote +1 if in favor of trying out Github issues and -1 if not.
> 
> See below thread and community meeting notes
> 
> for reference:
> 
> http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/201705.mbox/%3CCAO2yDyYKmUiSfGfkGCKtfP8mmQtcJubGoMO-KsWsjM9_3pOd0Q%40mail.gmail.com%3E
> 
> - Sijie

Re: [VOTE] Use Github Issues for Issue Tracking

[Flavio Junqueira]

Thanks for pushing this through, Sijie. I have a couple of concerns about this 
proposal:

1- There is a proposal, but I'm not seeing a workflow defined for github 
issues. Should we define one and make that part of the vote?
2- I'm not sure what the binding votes are and what the approval process is. 
Are we following the project bylaws here?

Thanks,
-Flavio

> On 01 Jun 2017, at 21:59, Sijie Guo  wrote:
> 
> Per the community meeting
> 
> this morning, JV proposed to start use Github issues for issue tracking for
> a few months and see how does it work out. I am starting this email thread
> to vote for this.
> 
> The vote will be:
> 
> - Start using Github issues/pull requests for issue tracking for 3 months.
> - During this 3 months, we will continue using both JIRA and Github issues.
> - After 3 months, if the community decides whether we should continue using
> Github issues or moving from JIRA to Github issues. (The final decision
> will be a separate vote in 3 months)
> 
> Please vote +1 if in favor of trying out Github issues and -1 if not.
> 
> See below thread and community meeting notes
> 
> for reference:
> 
> http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/201705.mbox/%3CCAO2yDyYKmUiSfGfkGCKtfP8mmQtcJubGoMO-KsWsjM9_3pOd0Q%40mail.gmail.com%3E
> 
> - Sijie

Re: [DISCUSS] Issue Tracking - Jira or Github Issues

[Flavio Junqueira]

I have seen some large projects relying on Github Issues, Docker being one of 
them. I have recently been using it in the Pravega project and I do find that 
it doesn't offer right up front some of the features that jira offers. For 
example, it doesn't give you the ability of creating a workflow, although what 
we have done and have seen others doing it to create labels to represent steps 
of a workflow. We ended up overloading the use of labels, but it looks decent 
with the colors and such.

I also find a bit confusing the relationship between issues and pull requests 
at times. We have been trying to enforce that each pull request requires at 
least one issue, but sometimes it feels unnatural because you also have space 
for a description and the ability to comment in a pull request.

I'm not sure what the story is for github issues and apache infra, though. The 
information I have is the same as Bobby's. Does anyone have a pointer?

-Flavio

> On 26 May 2017, at 17:06, Bobby Evans  wrote:
> 
> Apache does have a requirement that community discussions and especially 
> votes are stored on apache servers.  This is often done by linking different 
> systems together (like pull requests to JIRA) or by having a fire-hose of 
> changes from the external system sent to some apache mailing list that it can 
> archive.
> I have not used github issues much but from what I have done it does not look 
> even close to being as full featured as JIRA. So my vote would be to ask 
> people to use JIRA, but not ignore the github issues. 
> 
> - Bobby
> 
> On Friday, May 26, 2017, 9:57:43 AM CDT, Sijie Guo  
> wrote:Hi all,
> 
> Currently we are using Jira for issue tracking and using Github for
> managing pull requests. For a new developer, he has to create two accounts
> in order to engage with BookKeeper community. I am thinking - shall we also
> move the issue tracking to use Github Issues (which I believe Apache Infra
> supports that now)? So most of the development activities will happen in
> Github.
> 
> Another reason I asked this - I saw a Github issue was created.
> https://github.com/apache/bookkeeper/issues/165 I believe we somehow
> requested to change the permissions to allow creating Github issues before.
> 
> Any thoughts?
> 
> - Sijie

Re: [discuss] moving the community meeting to Friday

[Flavio Junqueira]

Well, I keep seeing that "Requesting to join the call..." and no further 
progress.

-Flavio

> On 10 Feb 2017, at 16:18, Enrico Olivelli  wrote:
> 
> Francesco and Bobby are in troubles too I think
> 
> 2017-02-10 17:08 GMT+01:00 Venkateswara Rao Jujjuri :
> 
>> Hmm others joined. Did you use this?
>> 
>> https://hangouts.google.com/hangouts/_/salesforce.com/bk-community-sa
>> 
>> On Fri, Feb 10, 2017 at 8:06 AM, Flavio Junqueira  wrote:
>> 
>>> It says: "Requesting to join the video call".
>>> 
>>> -Flavio
>>> 
>>>> On 10 Feb 2017, at 16:01, Venkateswara Rao Jujjuri 
>>> wrote:
>>>> 
>>>> Thanks!
>>>> 
>>>> Please join this link.
>>>> 
>>>> https://hangouts.google.com/hangouts/_/salesforce.com/bk-community-sa
>>>> 
>>>> On Fri, Feb 10, 2017 at 7:16 AM, Enrico Olivelli 
>>>> wrote:
>>>> 
>>>>> I have just created the page for the meeting of today
>>>>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>>>> 2017-2-10+Meeting+notes
>>>>> 
>>>>> 
>>>>> 2017-02-09 17:18 GMT+01:00 Sijie Guo :
>>>>> 
>>>>>> Thank you JV.
>>>>>> 
>>>>>> Sijie
>>>>>> 
>>>>>> On Feb 9, 2017 8:12 AM, "Venkateswara Rao Jujjuri" <
>> jujj...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Correct. Its tomorrow 8 AM PST and use this video call link for
>>>>> tomorrow.
>>>>>>> https://goo.gl/zBJvgs
>>>>>>> 
>>>>>>> JV
>>>>>>> 
>>>>>>> On Thu, Feb 9, 2017 at 8:09 AM, Bobby Evans
>>>>> >>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> So is the meeting happening on Friday then?  Because it looks like
>> no
>>>>>> one
>>>>>>>> is around to let people into the hangout.  Is
>> https://goo.gl/6UZR1w
>>>>>> the
>>>>>>>> right URL?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> - Bobby
>>>>>>>> 
>>>>>>>> On Thursday, February 9, 2017, 4:26:12 AM CST, Francesco Caliumi -
>>>>>>> Diennea
>>>>>>>>  wrote:If the meeting is going to
>>>>> take
>>>>>>>> place this Friday 8 AM, I could present
>>>>>>>> bookkeeper docker image (how to build, configure and use it), if
>>>>>>>> someone is interested.
>>>>>>>> 
>>>>>>>> We could then discuss whether to put build files in bookkeeper
>>>>>>>> repository or let them reside in a separate one and who will be in
>>>>>>>> charge of updating images when new versions are released.
>>>>>>>> 
>>>>>>>> - Francesco
>>>>>>>> 
>>>>>>>> On Wed, 2017-02-08 at 21:33 -0800, Venkateswara Rao Jujjuri wrote:
>>>>>>>>> May be it is too late; I am ok moving it Friday 8 AM
>>>>>>>>> 
>>>>>>>>> @Sijie what would you think? We also need Agenda
>>>>>>>>> 
>>>>>>>>> On Wed, Feb 8, 2017 at 9:13 AM, Enrico Olivelli <
>>>>> eolive...@gmail.com
>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Sorry for bothering everybody, this week we are going to meet on
>>>>>>>>>> Thursday?
>>>>>>>>>> 
>>>>>>>>>> This is the link to the discussion page
>>>>>>>>>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>>>>> Community+Me
>>>>>>>>>> etings
>>>>>>>>>> 
>>>>>>>>>> Cheers
>>>>>>>>>> Enrico
>>>>>>>>>> 
>>>>>>>>>> Il mar 7 feb 2017, 15:46 Bobby Evans >>>>> 
>>>>>>>>>> ha
>>>>>>>>>> scritto:
>>>>>>>>>&

Re: [discuss] moving the community meeting to Friday

[Flavio Junqueira]

It says: "Requesting to join the video call".

-Flavio

> On 10 Feb 2017, at 16:01, Venkateswara Rao Jujjuri  wrote:
> 
> Thanks!
> 
> Please join this link.
> 
> https://hangouts.google.com/hangouts/_/salesforce.com/bk-community-sa
> 
> On Fri, Feb 10, 2017 at 7:16 AM, Enrico Olivelli 
> wrote:
> 
>> I have just created the page for the meeting of today
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>> 2017-2-10+Meeting+notes
>> 
>> 
>> 2017-02-09 17:18 GMT+01:00 Sijie Guo :
>> 
>>> Thank you JV.
>>> 
>>> Sijie
>>> 
>>> On Feb 9, 2017 8:12 AM, "Venkateswara Rao Jujjuri" 
>>> wrote:
>>> 
 Correct. Its tomorrow 8 AM PST and use this video call link for
>> tomorrow.
 https://goo.gl/zBJvgs
 
 JV
 
 On Thu, Feb 9, 2017 at 8:09 AM, Bobby Evans
>> >>> 
 wrote:
 
> So is the meeting happening on Friday then?  Because it looks like no
>>> one
> is around to let people into the hangout.  Is https://goo.gl/6UZR1w
>>> the
> right URL?
> 
> 
> - Bobby
> 
> On Thursday, February 9, 2017, 4:26:12 AM CST, Francesco Caliumi -
 Diennea
>  wrote:If the meeting is going to
>> take
> place this Friday 8 AM, I could present
> bookkeeper docker image (how to build, configure and use it), if
> someone is interested.
> 
> We could then discuss whether to put build files in bookkeeper
> repository or let them reside in a separate one and who will be in
> charge of updating images when new versions are released.
> 
> - Francesco
> 
> On Wed, 2017-02-08 at 21:33 -0800, Venkateswara Rao Jujjuri wrote:
>> May be it is too late; I am ok moving it Friday 8 AM
>> 
>> @Sijie what would you think? We also need Agenda
>> 
>> On Wed, Feb 8, 2017 at 9:13 AM, Enrico Olivelli <
>> eolive...@gmail.com
 
>> wrote:
>> 
>>> 
>>> Sorry for bothering everybody, this week we are going to meet on
>>> Thursday?
>>> 
>>> This is the link to the discussion page
>>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>> Community+Me
>>> etings
>>> 
>>> Cheers
>>> Enrico
>>> 
>>> Il mar 7 feb 2017, 15:46 Bobby Evans >> 
>>> ha
>>> scritto:
>>> 
 
 Yes I would love to join too.  We missed the last meeting
 invite.  In
 general we can make time most days, especially if it is in the
 morning.
 
 
 - Bobby
 
 On Tuesday, February 7, 2017, 5:21:49 AM CST, Enrico Olivelli <
 eolive...@gmail.com> wrote:Any news ?
 This week the meeting will be on Friday ?
 
 @Bobby
 It would be great if some of your new team at Yahoo could join
 the
>>> meeting
 
 
 2017-01-31 7:44 GMT+01:00 Enrico Olivelli >> :
 
> 
> For me Friday is ok too. And 8am pt is ideal for me.
> 
> I hope that more guus can attend these useful meetings
> 
> Enrico
> 
> Il mar 31 gen 2017, 02:17 Sijie Guo  ha
> scritto:
> 
>> 
>> It was raised in the other email thread. I'd like to pull
>> that out to
>> discuss here. Is anyone good on moving the community
>> meeting
>> to Friday
>> same
>> time (8am pst)?
>> 
>> please give us your options and we can try to coordinate
>> here.
>> 
>> - Sijie
>> 
> --
> 
> 
> -- Enrico Olivelli
> 
 
>>> --
>>> 
>>> 
>>> -- Enrico Olivelli
>>> 
>> 
>> 
>> 
> --
> Francesco Caliumi
> Developer @ Diennea - MagNews
> Tel.: (+39) 0546 066100 - Int. 266
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
> 
> 
> 
> 
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital
>> ed
> email marketing! http://www.magnews.it/newsletter/
> 
> The information in this email is confidential and may be legally
> privileged. If you are not the intended recipient please notify the
 sender
> immediately and destroy this email. Any unauthorized, direct or
>>> indirect,
> disclosure, copying, storage, distribution or other use is strictly
> forbidden.
> 
 
 
 
 --
 Jvrao
 ---
 First they ignore you, then they laugh at you, then they fight you,
>> then
 you win. - Mahatma Gandhi
 
>>> 
>> 
> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: Community Meetings in this month

[Flavio Junqueira]

Is there any chance we can move his meeting to Friday same time? I have a 
constraint I can get around on Thursdays, and I'd like to join occasionally.

Thanks,
-Flavio

> On 26 Jan 2017, at 18:22, Enrico Olivelli  wrote:
> 
> Very good.
> Thanks
> 
> Il gio 26 gen 2017, 18:58 Venkateswara Rao Jujjuri  ha
> scritto:
> 
>> Minutes published.
>> 
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/2017-1-26+Meeting
>> 
>> On Thu, Jan 26, 2017 at 7:41 AM, Enrico Olivelli 
>> wrote:
>> 
>>> I am sorry I think I cannot attend the meeting of today. My little sons
>> are
>>> ill and I have to do some baby sitting...
>>> 
>>> About the merge of security PRs. I am confident that the patch on ZK ACLs
>>> is quite simple and it is open enough to let us implement more fine
>> grained
>>> acls in the future.
>>> The patch on the auth plugin is the base for SSL and JAAS, I think are
>> only
>>> waiting for Yahoo ACK as they are the olny known users of this feature
>> and
>>> we are breaking compatibility.
>>> 
>>> About SSL: I think it is the first good step and mutual auth will be easy
>>> to implement as a second step
>>> 
>>> About JAAS: it is just a prototype and requires code clean up at least
>>> 
>>> I hope we can agree on all of those and merge as soon as possible so that
>>> we can move on merging the other important changes on LAC
>>> 
>>> Enrico
>>> 
>>> Il mer 25 gen 2017, 21:57 Sijie Guo  ha scritto:
>>> 
 Sorry for late response. Thanks JV for driving this.
 
 I added another item to the agenda. I'd like to discuss how to merge
>> the
 security features into 4.5.0.
 
 - Sijie
 
 On Wed, Jan 18, 2017 at 8:54 AM, Venkateswara Rao Jujjuri <
 jujj...@gmail.com
> wrote:
 
> I have created a page for 1/26 meeting with Agenda section.
> Please add your agenda items to that.
> 
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>> 2017-1-26+Meeting
> 
> 
> 
> On Wed, Jan 18, 2017 at 7:46 AM, Enrico Olivelli <
>> eolive...@gmail.com>
> wrote:
> 
>> Next meeting ?
>> Do we have an agenda or proposals to discute ?
>> 
>> 
>> 
>> 2016-12-21 9:13 GMT+01:00 Enrico Olivelli - Diennea <
>> enrico.olive...@diennea.com>:
>> 
>>> +1
>>> 
>>> Il giorno mar, 20/12/2016 alle 16.58 -0800, Sijie Guo ha scritto:
>>> 
>>> Hi all,
>>> 
>>> Since it is a holiday month, couple of folks might already had
>>> travel
>>> plans. Going to cancel the community meetings this month. Any
> objections?
>>> 
>>> - Sijie
>>> 
>>> 
>>> --
>>> 
>>> Enrico Olivelli Software Development Manager @Diennea Tel.: (+39)
 0546
>>> 066100 - Int. 925 Viale G.Marconi 30/14 - 48018 Faenza (RA)
>>> MagNews -
>>> E-mail Marketing Solutions http://www.magnews.it Diennea -
>> Digital
>>> Marketing Solutions http://www.diennea.com
>>> 
>>> 
>>> 
>>> Iscriviti alla nostra newsletter per rimanere aggiornato su
>> digital
 ed
>>> email marketing! http://www.magnews.it/newsletter/
>>> 
>>> The information in this email is confidential and may be legally
>>> privileged. If you are not the intended recipient please notify
>> the
>> sender
>>> immediately and destroy this email. Any unauthorized, direct or
> indirect,
>>> disclosure, copying, storage, distribution or other use is
>> strictly
>>> forbidden.
>>> 
>> 
> 
> 
> 
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you,
>>> then
> you win. - Mahatma Gandhi
> 
 
>>> --
>>> 
>>> 
>>> -- Enrico Olivelli
>>> 
>> 
>> 
>> 
>> --
>> Jvrao
>> ---
>> First they ignore you, then they laugh at you, then they fight you, then
>> you win. - Mahatma Gandhi
>> 
> -- 
> 
> 
> -- Enrico Olivelli

Re: Helping with BookKeeper at Yahoo

[Flavio Junqueira]

Good to see you on this list, Bobby.

-Flavio

> On 12 Jan 2017, at 16:42, Bobby Evans  wrote:
> 
> Hi,
> 
> My name is Bobby Evans and I wanted to introduce myself to everyone.  My
> team is going to start helping out the Pulsar team here at Yahoo in
> maintaining/improving BookKeeper.  So if you see some new names on JIRAs
> that is who we are.  Also if you have some suggestions on bug fixes,
> documentation updates, or other things that we can do to get started with
> the community please let us know.
> 
> Thanks,
> 
> Bobby

[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15738007#comment-15738007
 ] 

Flavio Junqueira commented on BOOKKEEPER-974:
-

yeah, we don't. some volunteer did it on his own, so the "official" label is a 
bit tricky because it isn't official from a community perspective. it would be 
much better if it were community driven, like this community is trying to do.

> make pushing a docker image as part of the release procedure
> 
>
> Key: BOOKKEEPER-974
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Jia Zhai
>
> make pushing a docker image as part of the release procedure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Next Community Meeting?

[Flavio Junqueira]

I can't find the information to connect... :-(


> On 30 Nov 2016, at 19:46, Sijie Guo  wrote:
> 
> Thank you all for the proposed agenda. Let's continue the community meeting
> this week with  following agenda:
> 
> - DCOS support (Jia)
> - Long Poll Reads
> - Multi Journals
> - Release 4.5
> 
> We will discuss the placement policy on Dec 15th.
> 
> Sijie
> 
> On Nov 30, 2016 11:12 AM, "Venkateswara Rao Jujjuri" 
> wrote:
> 
>> Yeah it will be awesome if Jia can present. But I am going to miss. :( I
>> will ask someone at Salesforce to record it. :)
>> 
>> On Wed, Nov 30, 2016 at 10:54 AM, Enrico Olivelli 
>> wrote:
>> 
>>> I'm sorry. The next meeting will be on 1 dec or on 8 dec?
>>> I am a bit confused.
>>> 
>>> thank you Jia it will be very interesting
>>> 
>>> Enrico
>>> 
>>> Il mer 30 nov 2016, 12:57 Jia Zhai  ha scritto:
>>> 
>>>> +1.
>>>> It is my pleasure to introduce the DCOS/Mesos support work.
>>>> 
>>>> Thanks a lot.
>>>> -Jia
>>>> 
>>>> 
>>>> On Wed, Nov 30, 2016 at 5:44 PM, Flavio Junqueira 
>>> wrote:
>>>> 
>>>>> It'd be awesome if Jia could present the DCOS/Mesos support work. The
>>>>> other points are good with me too.
>>>>> 
>>>>> -Flavio
>>>>> 
>>>>>> On 30 Nov 2016, at 07:45, Enrico Olivelli 
>>> wrote:
>>>>>> 
>>>>>> Some ideas for next meetings:
>>>>>> - Jia work on DS/OS and Docker support
>>>>>> - Recap on Longpoll Read & Piggyback Support + Salesforce LAC
>>>>> improvements
>>>>>> - Recap on Multiple journals and other improvements for Yahoo stuff
>>>>>> - Recap on auto-recovery
>>>>>> - 4.5 release
>>>>>> 
>>>>>> in the future I will propose something about removing ZooKeeper in
>>>>>> order to have a single JVM networkless Bookeeper, but I'm not
>> ready,
>>> I
>>>>>> will file a BP as soon as possible
>>>>>> 
>>>>>> Maybe I missed many other meetings and so I would like to know more
>>>>>> about new features.
>>>>>> 
>>>>>> If there is no real stuff to discuss maybe we can postpone the
>>> meeting
>>>>>> to next week as JV and Salesforce guys will be able to attend and
>> to
>>>>>> present their work
>>>>>> 
>>>>>> I see that this new release is full of new great features, is it
>>> worth
>>>>>> to change to 5.0.0 ?
>>>>>> 
>>>>>> 2016-11-30 3:36 GMT+01:00 Sijie Guo :
>>>>>>> I am fine with moving Rithin's topic to Dec 15th.
>>>>>>> 
>>>>>>> I don't have other stuff to discuss unless there is other
>> proposals
>>>>> coming
>>>>>>> up for discussion.
>>>>>>> 
>>>>>>> - Sijie
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Nov 29, 2016 at 4:39 PM, Venkateswara Rao Jujjuri <
>>>>> jujj...@gmail.com
>>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I have a conflict this Thursday and I can't make it either, and I
>>>> can't
>>>>>>>> make next week too. :(
>>>>>>>> So two options:
>>>>>>>> 1. Continue Dec 1st call with different agenda.
>>>>>>>> 2. Move Rithin's topic to Dec 15th.
>>>>>>>> 
>>>>>>>> On Tue, Nov 29, 2016 at 4:17 PM, Sijie Guo 
>>> wrote:
>>>>>>>> 
>>>>>>>>> works for me. I will update the schedule.
>>>>>>>>> 
>>>>>>>>> - Sijie
>>>>>>>>> 
>>>>>>>>> On Tue, Nov 29, 2016 at 3:56 PM, Rithin Shetty <
>> rit...@gmail.com>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Sijie,
>>>>>>>>>> 
>>>>>>>>>>   Unfortunately, I'll be OOO this Thursday. Can we move this
>>> topic
>>>>> to
>>>>>>>>>> next week

Re: Next Community Meeting?

[Flavio Junqueira]

It'd be awesome if Jia could present the DCOS/Mesos support work. The other 
points are good with me too.

-Flavio

> On 30 Nov 2016, at 07:45, Enrico Olivelli  wrote:
> 
> Some ideas for next meetings:
> - Jia work on DS/OS and Docker support
> - Recap on Longpoll Read & Piggyback Support + Salesforce LAC improvements
> - Recap on Multiple journals and other improvements for Yahoo stuff
> - Recap on auto-recovery
> - 4.5 release
> 
> in the future I will propose something about removing ZooKeeper in
> order to have a single JVM networkless Bookeeper, but I'm not ready, I
> will file a BP as soon as possible
> 
> Maybe I missed many other meetings and so I would like to know more
> about new features.
> 
> If there is no real stuff to discuss maybe we can postpone the meeting
> to next week as JV and Salesforce guys will be able to attend and to
> present their work
> 
> I see that this new release is full of new great features, is it worth
> to change to 5.0.0 ?
> 
> 2016-11-30 3:36 GMT+01:00 Sijie Guo :
>> I am fine with moving Rithin's topic to Dec 15th.
>> 
>> I don't have other stuff to discuss unless there is other proposals coming
>> up for discussion.
>> 
>> - Sijie
>> 
>> 
>> On Tue, Nov 29, 2016 at 4:39 PM, Venkateswara Rao Jujjuri >> wrote:
>> 
>>> I have a conflict this Thursday and I can't make it either, and I can't
>>> make next week too. :(
>>> So two options:
>>> 1. Continue Dec 1st call with different agenda.
>>> 2. Move Rithin's topic to Dec 15th.
>>> 
>>> On Tue, Nov 29, 2016 at 4:17 PM, Sijie Guo  wrote:
>>> 
>>>> works for me. I will update the schedule.
>>>> 
>>>> - Sijie
>>>> 
>>>> On Tue, Nov 29, 2016 at 3:56 PM, Rithin Shetty  wrote:
>>>> 
>>>>> Hi Sijie,
>>>>> 
>>>>>Unfortunately, I'll be OOO this Thursday. Can we move this topic to
>>>>> next week ? I'll have the wiki updated by then. Sorry about the late
>>>>> notice.
>>>>> 
>>>>> --Rithin
>>>>> 
>>>>> On Tue, Nov 29, 2016 at 3:12 PM, Flavio Junqueira <
>>>>> fpjunque...@yahoo.com.invalid> wrote:
>>>>> 
>>>>>> The wiki page is actually empty...
>>>>>> 
>>>>>> -Flavio
>>>>>> 
>>>>>>> On 29 Nov 2016, at 18:58, Sijie Guo  wrote:
>>>>>>> 
>>>>>>> Just a reminder - We will have a community meeting on 12/1. We will
>>>>>>> discuss BP-2:
>>>>>>> Resource aware data placement
>>>>>>> <https://cwiki.apache.org/confluence/display/BOOKKEEPER/
>>>>>> BP-2+-+Resource+aware+data+placement>
>>>>>>> .
>>>>>>> 
>>>>>>> - Sijie
>>>>>>> 
>>>>>>> On Wed, Nov 16, 2016 at 10:49 AM, Matteo Merli <
>>>> matteo.me...@gmail.com
>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> On Tue, Nov 15, 2016 at 12:22 PM Yiming Zang
>>>>> >>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> SGTM +1
>>>>>>>>> 
>>>>>>>>> On Tue, Nov 15, 2016 at 10:34 AM, Venkateswara Rao Jujjuri <
>>>>>>>>> jujj...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Sounds good.
>>>>>>>>>> 
>>>>>>>>>> On Tue, Nov 15, 2016 at 1:41 AM Enrico Olivelli <
>>>>> eolive...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Ok for me.
>>>>>>>>>>> 
>>>>>>>>>>> Enrico
>>>>>>>>>>> 
>>>>>>>>>>> Il mar 15 nov 2016, 10:39 Sijie Guo  ha
>>>> scritto:
>>>>>>>>>>> 
>>>>>>>>>>>> We will continue discussing the 'resource aware data
>>> placement'
>>>> in
>>>>>>>>> the
>>>>>>>>>>> next
>>>>>>>>>>>> community meeting.
>>>>>>>>>>>> 
>>>>>>>>>>>> However, I think it is 11/24. I guess most of the people might
>>>>>>>>> already
>>>>>>>>>>> have
>>>>>>>>>>>> some plan around that time. Shall we move the meeting one week
>>>>>>>> after,
>>>>>>>>>>> which
>>>>>>>>>>>> is Dec 1st?
>>>>>>>>>>>> 
>>>>>>>>>>>> Does that sound good?
>>>>>>>>>>>> 
>>>>>>>>>>>> - Sijie
>>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> -- Enrico Olivelli
>>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Sent from iPhone
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Jvrao
>>> ---
>>> First they ignore you, then they laugh at you, then they fight you, then
>>> you win. - Mahatma Gandhi
>>> 

Re: Next Community Meeting?

[Flavio Junqueira]

The wiki page is actually empty...

-Flavio

> On 29 Nov 2016, at 18:58, Sijie Guo  wrote:
> 
> Just a reminder - We will have a community meeting on 12/1. We will
> discuss BP-2:
> Resource aware data placement
> 
> .
> 
> - Sijie
> 
> On Wed, Nov 16, 2016 at 10:49 AM, Matteo Merli 
> wrote:
> 
>> +1
>> 
>> On Tue, Nov 15, 2016 at 12:22 PM Yiming Zang 
>> wrote:
>> 
>>> SGTM +1
>>> 
>>> On Tue, Nov 15, 2016 at 10:34 AM, Venkateswara Rao Jujjuri <
>>> jujj...@gmail.com> wrote:
>>> 
 Sounds good.
 
 On Tue, Nov 15, 2016 at 1:41 AM Enrico Olivelli 
 wrote:
 
> Ok for me.
> 
> Enrico
> 
> Il mar 15 nov 2016, 10:39 Sijie Guo  ha scritto:
> 
>> We will continue discussing the 'resource aware data placement' in
>>> the
> next
>> community meeting.
>> 
>> However, I think it is 11/24. I guess most of the people might
>>> already
> have
>> some plan around that time. Shall we move the meeting one week
>> after,
> which
>> is Dec 1st?
>> 
>> Does that sound good?
>> 
>> - Sijie
>> 
> --
> 
> 
> -- Enrico Olivelli
> 
 --
 Sent from iPhone
 
>>> 
>> 

Re: A bookkeeper DCOS/mesos package

[Flavio Junqueira]

Very nice, Jia, thanks for the update. I suggest creating a jira issue to track 
what work remains to be done in this front. I'm particularly interested in 
having a way forward to update the package as we release new versions. I'd say 
that automating as much as possible should be a goal.

-Flavio

> On 22 Nov 2016, at 04:03, Jia Zhai  wrote:
> 
> FYI. Bookkeeper package have been merged into the public Universe, now it
> is here:
> universe <https://github.com/mesosphere/universe>/repo
> <https://github.com/mesosphere/universe/tree/version-3.x/repo>/packages
> <https://github.com/mesosphere/universe/tree/version-3.x/repo/packages>/B
> <https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/B>/
> bookkeeper
> <https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/B/bookkeeper>
> /*0*/ .
> And the example is under-review here:
> https://github.com/dcos/examples/pull/7
> 
> Hi Flavio and JV,
> Regarding Jira ticket, There is currently no Jira Ticket. And I will open
> one if it is useful.
> 
> Regarding the Sync issue, that maybe need a manual work, since it
> referenced to a hard-coded bookkeeper docker image. From my view,  In the
> future, we may also need to matain an official bookkeeper docker image, and
> this package will reference to it.
> 
> Regarding container issue, Mesos support both Docker container and Mesos
> container.  This BK package using a Docker type.
> The working flow of this package is some thing like this: 1, have a DCOS
> env. 2, on the GUI goto  "Universe" on the left of the window,  you would
> find bookkeeper there, click it to install.  That's all to deploy a bookie
> on one of your DCOS slave-agent.
> Here is some docs related to DCOS/Mesos:
> https://dcos.io/docs/1.8/
> https://mesosphere.github.io/marathon/docs/
> 
> Thanks a lot.
> -Jia
> 
> 
> 
> On Mon, Nov 21, 2016 at 1:55 AM, Venkateswara Rao Jujjuri > wrote:
> 
>> This is really great news. I have few questions being a naive to Mesos.
>> 1. Can we have a doc which explains how to do this even if they have no
>> experience with Mesos?
>> 2. I would like to expand on Flavio's question. How do we keep these two
>> projects in sync? When either side releases new versions.
>> 3. In the pull request I see lot of references to Docker. Does it mean this
>> runs both on Docker container and Mesos container?
>> 4. Will it be too much effort to run this on vanilla Docker?
>> 
>> This is really great and thanks a lot Jia.
>> 
>> - JV
>> 
>> 
>> On Sun, Nov 20, 2016 at 7:31 AM, Flavio Junqueira  wrote:
>> 
>>> Is there a jira for this? Or are you posting this here just so that the
>>> community is aware, Jia?
>>> 
>>> One question about this packaging: what happens when there is a new
>>> release of bookkeeper, is there anything we need to update? Is it worth
>>> adding to the release process?
>>> 
>>> Thanks,
>>> -Flavio
>>> 
>>>> On 08 Nov 2016, at 19:43, Enrico Olivelli  wrote:
>>>> 
>>>> Great! I hope it will give more visibility to BookKeeper!
>>>> 
>>>> Thank you Jia
>>>> Enrico
>>>> 
>>>> Il mar 8 nov 2016, 18:52 Sijie Guo  ha scritto:
>>>> 
>>>>> That's awesome! Great job, Jia.
>>>>> 
>>>>> I will take a closer look :D
>>>>> 
>>>>> - Sijie
>>>>> 
>>>>> On Tue, Nov 8, 2016 at 5:52 AM, Jia Zhai  wrote:
>>>>> 
>>>>>> Hi Guys,
>>>>>> There is a PR against bookkeeper DCOS package, this is a very first
>>>>>> version, I will enhance it while using it. welcome comments on it.
>>>>>> https://github.com/mesosphere/universe/pull/797
>>>>>> 
>>>>>> There is also an example PR there for how this package should be
>> used:
>>>>>> https://github.com/dcos/examples/pull/7
>>>>>> 
>>>>>> This bookkeeper package could easily deploy one bookie on a DCOS
>>>>>> slave-agent. And easily scale bookie instances to deploy on several
>>> DCOS
>>>>>> slaves, and export bookie service at: "slave-agent-ip:3181".
>>>>>> It is basically a marathon app, but use dcos package to make an
>> easier
>>>>>> usage.
>>>>>> With this package, it is now need one command(or few clicks in GUI)
>> to
>>>>>> deploy one bookie on your DCOS environment:
>>>>>> $dcos package install bookkeeper
>>>>>> 
>>>>>> Here are links for more info about DCOS/Mesos:
>>>>>> Mesos: http://mesos.apache.org/documentation/latest/
>>>>>> DCOS: https://dcos.io/docs/1.8/
>>>>>> marathon: https://mesosphere.github.io/marathon/docs/
>>>>>> 
>>>>>> Best Regards.
>>>>>> -Jia
>>>>>> 
>>>>> 
>>>> --
>>>> 
>>>> 
>>>> -- Enrico Olivelli
>>> 
>>> 
>> 
>> 
>> --
>> Jvrao
>> ---
>> First they ignore you, then they laugh at you, then they fight you, then
>> you win. - Mahatma Gandhi
>> 

Re: A bookkeeper DCOS/mesos package

[Flavio Junqueira]

Is there a jira for this? Or are you posting this here just so that the 
community is aware, Jia?

One question about this packaging: what happens when there is a new release of 
bookkeeper, is there anything we need to update? Is it worth adding to the 
release process?

Thanks,
-Flavio

> On 08 Nov 2016, at 19:43, Enrico Olivelli  wrote:
> 
> Great! I hope it will give more visibility to BookKeeper!
> 
> Thank you Jia
> Enrico
> 
> Il mar 8 nov 2016, 18:52 Sijie Guo  ha scritto:
> 
>> That's awesome! Great job, Jia.
>> 
>> I will take a closer look :D
>> 
>> - Sijie
>> 
>> On Tue, Nov 8, 2016 at 5:52 AM, Jia Zhai  wrote:
>> 
>>> Hi Guys,
>>> There is a PR against bookkeeper DCOS package, this is a very first
>>> version, I will enhance it while using it. welcome comments on it.
>>> https://github.com/mesosphere/universe/pull/797
>>> 
>>> There is also an example PR there for how this package should be used:
>>> https://github.com/dcos/examples/pull/7
>>> 
>>> This bookkeeper package could easily deploy one bookie on a DCOS
>>> slave-agent. And easily scale bookie instances to deploy on several DCOS
>>> slaves, and export bookie service at: "slave-agent-ip:3181".
>>> It is basically a marathon app, but use dcos package to make an easier
>>> usage.
>>> With this package, it is now need one command(or few clicks in GUI) to
>>> deploy one bookie on your DCOS environment:
>>> $dcos package install bookkeeper
>>> 
>>> Here are links for more info about DCOS/Mesos:
>>> Mesos: http://mesos.apache.org/documentation/latest/
>>> DCOS: https://dcos.io/docs/1.8/
>>> marathon: https://mesosphere.github.io/marathon/docs/
>>> 
>>> Best Regards.
>>> -Jia
>>> 
>> 
> -- 
> 
> 
> -- Enrico Olivelli

[jira] [Commented] (BOOKKEEPER-963) Allow to use multiple journals in bookie

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15681315#comment-15681315
 ] 

Flavio Junqueira commented on BOOKKEEPER-963:
-

I'm missing a bit more motivation for this feature. We already allow multiple 
ledger directories, so is it the case that you're bottlenecked on the journal 
and you want to have multiple journals per bookie? Could you elaborate a bit 
further on your motivation, the hardware you're targeting and such? [~mmerli]

> Allow to use multiple journals in bookie
> 
>
> Key: BOOKKEEPER-963
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-963
> Project: Bookkeeper
>  Issue Type: Improvement
>Reporter: Matteo Merli
>Assignee: Matteo Merli
> Fix For: 4.5.0
>
>
> By configuring multiple journals, we can take advantage of the IO of multiple
> disks to increase the write throughput of a single bookie.
> Each journal will have its own journal and sync threads and writes will be
> assigned to a particular journal by hashing on the ledger id.
> In addition to using multiple physical disks, there can improvements even by
> using multiple journal on a single SSD device, because these disks can handle
> well multiple concurrent writes in different blocks of the disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BOOKKEEPER-882) Document that BookKeeper servers will shutdown on losing quorum

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15636338#comment-15636338
 ] 

Flavio Junqueira commented on BOOKKEEPER-882:
-

What is quorum referring to here, is it a zookeeper quorum?

> Document that BookKeeper servers will shutdown on losing quorum
> ---
>
> Key: BOOKKEEPER-882
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-882
> Project: Bookkeeper
>  Issue Type: Documentation
>  Components: bookkeeper-server
>Affects Versions: 4.3.1
>Reporter: Lucas Bradstreet
>
> We have adopted BookKeeper as part of our state management in Onyx 0.8.0 
> http://www.onyxplatform.org. As part of testing Onyx 0.8.0 we have begun 
> testing with Jepsen, and our first step was testing BookKeeper without any 
> interaction with Onyx.
> We have discovered that the BookKeeper servers automatically shutdown without 
> retry upon losing a connection to a quorum of nodes, which is the expected 
> behaviour according to the code.
> It is not obvious from the link at http://bookkeeper.apache.org/ under 
> "Documentation / Admin guide", that this is the expected behaviour.
> I believe it would be beneficial if this was discussed in the admin 
> documentation.
> Cheers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Discuss] 64 bits ledger id

[Flavio Junqueira]

Agreed that the probability of collision is low, but it exists and there is no 
way to check if it kicks in if we don't have that information centralized. For 
wraparounds, we at least have a way of detecting that we reached the maximum 
value, but granted that if we reach the maximum value, then we may have 
duplicates as we reset the counter.

Assuming that we still keep ledger metadata in the metadata service and it is 
keyed by ledger id, we can check whether the ledger id is taken upon writing 
the ledger metadata, can't we?

-Flavio 

> On 26 Oct 2016, at 16:41, Venkateswara Rao Jujjuri  wrote:
> 
> Flavio,
> 
> In theory you are right. But in practice it is virtually unique.
> RFC 4122 (https://www.ietf.org/rfc/rfc4122.txt) talks about multiple ways
> to generate virtually unique ID.
> Even in the sequence-id we have a theoretical chance of wrapping. So if we
> are worried about duplicates,
> we need a way to handle it. But in both cases (64 bit long or 128 bit uuid)
> collisions are impractical.
> 
> Also I am not advocating to get rid of metadata server completely, Just
> saying for this purpose we don't need to.
> Thanks,
> JV
> 
> 
> On Wed, Oct 26, 2016 at 3:43 AM, Flavio Junqueira  wrote:
> 
>> Hi JV,
>> 
>> Do I understand correctly that you're proposing to generate 128-bit UUIDs
>> locally to use as ledger ids? If so, I'm concerned that even if the
>> probability of collision is low, there is still a chance of having
>> collisions with no way to verify in the case you aren't relying on a
>> metadata server.
>> 
>> I feel that I'm missing something here because you can't completely get
>> rid of the metadata server without getting in trouble or at least without
>> deeper changes. I'd appreciate if you could clarify, please.
>> 
>> -Flavio
>> 
>>> On 26 Oct 2016, at 00:39, Venkateswara Rao Jujjuri 
>> wrote:
>>> 
>>> We are using 64 bit ledger ids internally right now, but the ledger id is
>>> supported by the application/caller.
>>> We have extended Hierarchical ledger manger to Long hierarchical ledger
>>> manger for this.
>>> 
>>> Ultimately we would like to move to 128 bit UUID as the ledger id. That
>>> makes ledgers unique without
>>> the need of centralized ZK/metadata server.
>>> 
>>> On Tue, Oct 25, 2016 at 4:11 PM, Sijie Guo  wrote:
>>> 
>>>> *Problem: *
>>>> 
>>>> Currently the ledger id is long, which it should be 64-bits. However
>>>> currently bookkeeper only can generate 32-bits ledger id as zookeeper's
>>>> sequence znode only produce 32-bits.
>>>> 
>>>> This problem was basically raised before at BOOKKEEPER-421. Jiannan has
>>>> already done fair amount of work on this and there were several patches
>> for
>>>> it.
>>>> 
>>>> This email thread is to start the discussion for 64-bits ledger id
>> support
>>>> in bookkeeper.
>>>> 
>>>> *Discuss*:
>>>> 
>>>> Based on bookkeeper-421, the changes will relatively happen in following
>>>> places. Assume the metadata store is ZooKeeper.
>>>> 
>>>> 
>>>>  1. How to generate 64-bits ledger id? (64 Bits Ledger ID Generation
>>>>  <https://issues.apache.org/jira/browse/BOOKKEEPER-552>)
>>>>  2. How to store the 64-bits ledger id in zookeeper? (New LedgerManager
>>>>  for 64 Bits Ledger ID Management in ZooKeeper
>>>>  <https://issues.apache.org/jira/browse/BOOKKEEPER-553>)
>>>>  3. How can the garbage collect handle correctly with 64-bits ledger
>> id?
>>>> (
>>>>  https://issues.apache.org/jira/browse/BOOKKEEPER-553?
>>>> focusedCommentId=13558192&page=com.atlassian.jira.
>>>> plugin.system.issuetabpanels:comment-tabpanel#comment-13558192
>>>>  )
>>>>  4. How can we upgrade current HierarchicalLedgerManager to support
>>>>  64-bits. [??]
>>>> 
>>>> Feel free to take a look at those tickets and make any proposals.
>>>> 
>>>> - Sijie
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Jvrao
>>> ---
>>> First they ignore you, then they laugh at you, then they fight you, then
>>> you win. - Mahatma Gandhi
>> 
>> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: bookkeeper dev/sync meeting

[Flavio Junqueira]

It might be worth discussion the next release, who is going to be the release 
manager, important issues, etc.

-Flavio

> On 26 Oct 2016, at 00:50, Sijie Guo  wrote:
> 
> I setup a bi-weekly sync-up meeting on Thursday 8am-9am
> https://goo.gl/6UZR1w
> 
> We will start from this week (10/27/2016). The agenda of this meeting will
> be
> 
> - Discuss Improvement Proposal - *64bits (even 128bits) ledger id support*.
> - Review and address concern for open pull requests.
> - Open discussion.
> 
> Feel free to join the meeting if you are interested. We tried to find a
> time slot that is good for most of the committers in the community. If
> Thursday 8am-9am doesn't work for you and you are really interested in the
> topic of this week, feel free to ping this thread and we can make
> arrangement.
> 
> - Sijie
> 
> 
> On Thu, Oct 13, 2016 at 9:48 PM, Sijie Guo  wrote:
> 
>> Flavio, agreed.
>> 
>> We are thinking of running the meeting in following form:
>> 
>> - we will try to run a bi-weekly community syncup meeting to discussion
>> pull requests, jiras and such. If nothing to discuss, we will cancel.
>> - we will collect the topics to discuss prior to the syncup meeting and
>> send the agenda out.
>> - we are looking for meeting on Thursday, so that we can send the agenda
>> out early that week.
>> - we will have notes and keep them on the wiki
>> 
>> I also liked the idea of KIP. There are quite a few improvements that come
>> up recently. We can also try to adopt the process of KIP (I can start a
>> different thread for that).
>> 
>> - Security, Authentication and Authorization
>> - Ledger Id beyond 32 bits (64 bits, 128 bits)
>> - Ensemble placement for tier storage
>> - Disk corruption detection and repair
>> - ..
>> (JV also has a list of improvements to add :))
>> 
>> - Sijie
>> 
>> On Thu, Oct 13, 2016 at 2:26 AM, Flavio Junqueira  wrote:
>> 
>>> I think it is great to have such meetings. The Kafka community, for
>>> example, organizes meetings to discuss KIPs:
>>> 
>>> https://goo.gl/OK0Rhi <https://goo.gl/OK0Rhi>
>>> 
>>> and if we have such meetings, I'd suggest we make sure to propose an
>>> agenda beforehand so that all people participated can join. Also, from an
>>> ASF perspective, please keep in mind that no decisions can be made offline.
>>> All project-related decisions need to happen via the mailing list or jira.
>>> The goal of the meeting should be for coordination purposes and potentially
>>> technical discussion around the project, issues, and code.
>>> 
>>> -Flavio
>>> 
>>>> On 13 Oct 2016, at 08:32, Sijie Guo  wrote:
>>>> 
>>>> Matteo, JV and me have tried running weekly syncup meetings for a while.
>>>> We'd like to extend it as the community syncup meeting - for technical
>>>> discussions, community matters and any sync-ups. The meeting is
>>> typically
>>>> comprised of 2 parts - the first part is going through any open pull
>>>> requests and newly open jiras and the second part is technical
>>> discussions
>>>> around issues and features and community matters (like meetups and
>>> events).
>>>> 
>>>> We'd like to know if any other contributors are interested in
>>>> participating. If so, what are the best time to meet and how often it
>>>> should be?
>>>> 
>>>> - Sijie
>>> 
>>> 
>> 

Re: [Discuss] 64 bits ledger id

[Flavio Junqueira]

Hi JV,

Do I understand correctly that you're proposing to generate 128-bit UUIDs 
locally to use as ledger ids? If so, I'm concerned that even if the probability 
of collision is low, there is still a chance of having collisions with no way 
to verify in the case you aren't relying on a metadata server.

I feel that I'm missing something here because you can't completely get rid of 
the metadata server without getting in trouble or at least without deeper 
changes. I'd appreciate if you could clarify, please.

-Flavio
 
> On 26 Oct 2016, at 00:39, Venkateswara Rao Jujjuri  wrote:
> 
> We are using 64 bit ledger ids internally right now, but the ledger id is
> supported by the application/caller.
> We have extended Hierarchical ledger manger to Long hierarchical ledger
> manger for this.
> 
> Ultimately we would like to move to 128 bit UUID as the ledger id. That
> makes ledgers unique without
> the need of centralized ZK/metadata server.
> 
> On Tue, Oct 25, 2016 at 4:11 PM, Sijie Guo  wrote:
> 
>> *Problem: *
>> 
>> Currently the ledger id is long, which it should be 64-bits. However
>> currently bookkeeper only can generate 32-bits ledger id as zookeeper's
>> sequence znode only produce 32-bits.
>> 
>> This problem was basically raised before at BOOKKEEPER-421. Jiannan has
>> already done fair amount of work on this and there were several patches for
>> it.
>> 
>> This email thread is to start the discussion for 64-bits ledger id support
>> in bookkeeper.
>> 
>> *Discuss*:
>> 
>> Based on bookkeeper-421, the changes will relatively happen in following
>> places. Assume the metadata store is ZooKeeper.
>> 
>> 
>>   1. How to generate 64-bits ledger id? (64 Bits Ledger ID Generation
>>   )
>>   2. How to store the 64-bits ledger id in zookeeper? (New LedgerManager
>>   for 64 Bits Ledger ID Management in ZooKeeper
>>   )
>>   3. How can the garbage collect handle correctly with 64-bits ledger id?
>> (
>>   https://issues.apache.org/jira/browse/BOOKKEEPER-553?
>> focusedCommentId=13558192&page=com.atlassian.jira.
>> plugin.system.issuetabpanels:comment-tabpanel#comment-13558192
>>   )
>>   4. How can we upgrade current HierarchicalLedgerManager to support
>>   64-bits. [??]
>> 
>> Feel free to take a look at those tickets and make any proposals.
>> 
>> - Sijie
>> 
> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: bookkeeper dev/sync meeting

[Flavio Junqueira]

I think it is great to have such meetings. The Kafka community, for example, 
organizes meetings to discuss KIPs:

https://goo.gl/OK0Rhi 

and if we have such meetings, I'd suggest we make sure to propose an agenda 
beforehand so that all people participated can join. Also, from an ASF 
perspective, please keep in mind that no decisions can be made offline. All 
project-related decisions need to happen via the mailing list or jira. The goal 
of the meeting should be for coordination purposes and potentially technical 
discussion around the project, issues, and code.

-Flavio
 
> On 13 Oct 2016, at 08:32, Sijie Guo  wrote:
> 
> Matteo, JV and me have tried running weekly syncup meetings for a while.
> We'd like to extend it as the community syncup meeting - for technical
> discussions, community matters and any sync-ups. The meeting is typically
> comprised of 2 parts - the first part is going through any open pull
> requests and newly open jiras and the second part is technical discussions
> around issues and features and community matters (like meetups and events).
> 
> We'd like to know if any other contributors are interested in
> participating. If so, what are the best time to meet and how often it
> should be?
> 
> - Sijie

Re: BOOKKEEPER-612 RegionAwarePlacement Policy Progress

[Flavio Junqueira]

Enrico,

The PR for BK-612 has been reviewed, right? We need to wait for Sijie to show 
up because he has been the main committer reviewing it, I'd rather not merge 
without his +1. Perhaps he is busy or away, so let's wait.

-Flavio

> On 23 Sep 2016, at 08:06, Enrico Olivelli  wrote:
> 
> (I'm re-sending this email from another account, as I think the prev
> message went to spam folder)
> 
> Hi BookKeepers,
> 
> I would like to ping on BOOKKEEPER-612, which has a pending PR.
> 
> It is actually blocking the work on
> https://issues.apache.org/jira/browse/BOOKKEEPER-912 Allow
> EnsemblePlacementPolicy to choose bookies using ledger custom data
> (multitenancy support) as this is changes the signature of methods on
> EnsemblePlacementPolicy.
> 
> Does anyone have time to review  so that it can be committed ?
> 
> For me BOOKKEEEPER-912 is a very important feature and I would like to
> work on it for 4.5.0
> 
> This is the issue
> https://issues.apache.org/jira/browse/BOOKKEEPER-612
> 
> This is the PR from Robin
> https://github.com/apache/bookkeeper/pull/56
> 
> 
> Thank you very much
> 
> Enrico

[jira] [Created] (BOOKKEEPER-947) Update workflow information on the web site

[Flavio Junqueira (JIRA)]

Flavio Junqueira created BOOKKEEPER-947:
---

 Summary: Update workflow information on the web site
 Key: BOOKKEEPER-947
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-947
 Project: Bookkeeper
  Issue Type: Documentation
Reporter: Flavio Junqueira
Assignee: Flavio Junqueira


We need to update the information on how to contribute on the web site, in 
particular, how to submit patches. The wiki has been updated:

https://cwiki.apache.org/confluence/display/BOOKKEEPER/Contributing+to+BookKeeper

but not the web site:

http://bookkeeper.apache.org/svn.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Stale documentation?

[Flavio Junqueira]

I think this page is stale:

http://bookkeeper.apache.org/svn.html 

It says that the project does not accept pull requests at the moment, which I 
believe it does, and also refers to the review board. Bot to mention that the 
document is called svn.html. :-)

Is there a jira to update it?

-Flavio 

Re: Proposed board report [August]

[Flavio Junqueira]

Hey Sijie,

Thanks for putting together. Have a look at https://reporter.apache.org/ 


It'll tell you precisely the additional information you're missing.

-Flavio

> On 01 Aug 2016, at 07:31, Sijie Guo  wrote:
> 
> Hello folks,
> 
> We have a board report due this month. Here is the draft.
> 
> Let me know if have anything else to add.
> 
> 
> 
> BookKeeper is a distributed, reliable, and high performance
> logging service. It has been used as a fundamental service to build
> high available and replicated services in large companies like Twitter,
> Yahoo and Salesforce. It is also the log segment store for Apache
> DistributedLog (incubating).
> 
> = Project Status =
> 
> The 4.4.0 has been release on May 16th. The community is working on
> releasing 4.5.0. The 4.5.0 release is targeted on merging major changes
> from different companies, including Twitter, Yahoo and Salesforce.
> 
> = Releases =
> 
> Our last release was 4.4.0, released on 2016-05-16. The community is
> working on 4.5.0 release.
> 
> = Community Status =
> 
> The last PMC member added was Matteo Merli (mmerli) on the 27th May 2016.
> 
> The last committers added were Jia Zhai (zhaijia) and JV (Jujjuri) on the
> 14th Jun 2016.
> 
> Salesforce held the last bookkeeper meetup on the 28th Jun 2016.
> 
> No infrastructure issues.
> 
> 73 subscribers in dev@bookkeeper.apache.org
> 93 subscribers in u...@bookkeeper.apache.org
> 
> 931 issues opened to date, 14 since 2016-05-18
> 653 issues resolved to date, 11 since 2016-05-18
> 61 people have reported issues, 7 since 2016-05-18
> 34 people have contributed patches, 7 since 2016-05-18
> 

Re: [ANNOUNCE] New BookKeeper Committer: JV Jujjuri

[Flavio Junqueira]

Congrats, JV!

-Flavio

> On 14 Jun 2016, at 08:21, Arun M. Krishnakumar  wrote:
> 
> Congrats JV!
> 
> Thanks,
> Arun
> 
>> On Jun 14, 2016, at 12:04 AM, Charan Reddy G  wrote:
>> 
>> Congrats JV!
>> 
>> Thanks,
>> Charan
>> 
>>> On Tuesday, June 14, 2016, Sijie Guo  wrote:
>>> 
>>> The Apache BookKeeper PMC recently extended committer karma to JV Jujjuri
>>> and he has accepted. JV is the main driver of bookkeeper production usage
>>> at Salesforce. He is active in the community, involving not only code
>>> contribution but also discussion and code reviews. We are looking forward
>>> to more contributions from him.
>>> 
>>> Congratulations and welcome onboard, JV!
>>> 

Re: [ANNOUNCE] Matteo Merli joins the Apache BookKeeper PMC

[Flavio Junqueira]

Congrats, Matteo!

-Flavio

> On 27 May 2016, at 17:14, Sijie Guo  wrote:
> 
> On behalf of the Apache BookKeeper PMC I am pleased to announce that Matteo 
> Merli has accepted our invitation to become a PMC member on the Apache 
> BookKeeper project. Matteo has been an active contributor in many areas,
> including recently release 4.3.2 and 4.4.0 as release manager. Please join me 
> in thanking Matteo for his contributions to date and anticipation of many 
> more contributions.
> 
> Welcome to the PMC, Matteo!
> 

Re: Confluence documentation

[Flavio Junqueira]

Thanks, Enrico! The wiki is more informal in the sense that the documentation 
there isn't really specific to versions and such. We are more strict about the 
official project documentation. Go ahead and keep populating that page, we can 
iterate over the content as you make progress.

-Flavio

> On 26 May 2016, at 16:26, Enrico Olivelli  wrote:
> 
> Hi,
> I have just created the API page at
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/API
> 
> I'm going to write directly on it,  would it be better to share a draft
> version somewhere else ?
> Docs need review before being published, but confluence does not allow this
> kind of facility
> 
> The position of the page is on top of the list, maybe it would be better on
> another place
> 
> 
> Thanks
> Enrico

Re: [ANNOUNCE] Apache BookKeeper 4.4.0 released

[Flavio Junqueira]

Great job, Matteo and community! Thank you.

-Flavio

> On 16 May 2016, at 16:15, Matteo Merli  wrote:
> 
> The Apache BookKeeper team is proud to announce Apache BookKeeper version
> 4.4.0.
> 
> This is the first feature release of Apache BookKeeper as an Apache Top
> Level
> Project.
> 
> This release contains a total of 94 Jira tickets fixed and brings several
> bookie
> reliability and operability improvements, along with a long list of
> bugfixes.
> 
> Please refer to the release notes for full details.
> http://bookkeeper.apache.org/docs/r4.4.0/releaseNotes.html
> 
> The release can be downloaded at:
> 
> http://bookkeeper.apache.org/releases.html
> 
> We would like to thank the contributors that made the release possible.
> 
> Regards,
> 
> The BookKeeper Team

Re: [VOTE] BookKeeper 4.4.0 Release Candidate 0

[Flavio Junqueira]

+1, I have checked NOTICE and LICENSE. I have run the rat tool to make sure we 
aren't missing anything with respect to licenses. I have run tests.

The one minor thing is that the NOTICE file says 2015 rather than 2016. We need 
to update it for the next release, unless there is something else and we need 
to do another RC.

-Flavio 


> On 05 May 2016, at 17:49, Matteo Merli  wrote:
> 
> This is the first release candidate for Apache BookKeeper, version 4.4.0.
> 
> It fixes the following issues:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311293&version=12325566
> 
> *** Please download, test and vote by May 10th 2016, 10:00 GMT.
> 
> Note that we are voting upon the source (tag), binaries are provided for
> convenience.
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.4.0-candidate-0/
> 
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachebookkeeper-1010/
> 
> The tag to be voted upon:
> release-4.4.0 (8bb39b187d1b681cf88651b825933d046545d34d)
> 
> BookKeeper's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
> 
> Please download the the source package, and follow the README to build
> and run a bookkeeper service.
> 
> Thank you,
> Matteo

RC Highlights (was: Re: [VOTE] BookKeeper 4.4.0 Release Candidate 0)

[Flavio Junqueira]

Please try to avoid discussions on the vote thread. It is fine to raise issues 
about the RC and such, but raise it in a separate e-mail thread.

-Flavio

> On 11 May 2016, at 07:41, Enrico Olivelli  wrote:
> 
> I think that it will be more useful for users the #2 option. Maybe
> highlights can be done sharing some blog post.
> 
> My most important question is if there is a special migration procedure for
> bookie data or is it enough to update the jars?
> 
> Enrico
> 
> Il Ven 6 Mag 2016 18:02 Matteo Merli  ha scritto:
> 
>> Enrico,
>> I think it's a good point.
>> 
>> So far, the closest thing we have is this twiki page:
>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Roadmap
>> But it's kind of not very visible for users and outdated. We should keep
>> that page for feature planning for future releases.
>> 
>> I think we could either:
>> 1. Add a highlights sections in the release notes page
>> 2. Create a "History" page in documenation where we put the highlights for
>> each release
>> 
>> I'd say I prefer option #2. What do you guys think?
>> 
>> Matteo
>> 
>> On Fri, May 6, 2016 at 2:25 AM Enrico Olivelli 
>> wrote:
>> 
>>> +1 (non binding)
>>> 
>>> Ran unit tests on Linux/Oracle jdk 1.8.0_77
>>> 
>>> Ran internal Majordodo Unit tests
>>> 
>>> -- Enrico
>>> 
>>> 2016-05-05 18:49 GMT+02:00 Matteo Merli :
>>> 
 This is the first release candidate for Apache BookKeeper, version
>> 4.4.0.
 
 It fixes the following issues:
 
 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311293&version=12325566
 
 *** Please download, test and vote by May 10th 2016, 10:00 GMT.
 
 Note that we are voting upon the source (tag), binaries are provided
>> for
 convenience.
 
 Source and binary files:
 
 
>>> 
>> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.4.0-candidate-0/
 
 Maven staging repo:
 
 
>>> 
>> https://repository.apache.org/content/repositories/orgapachebookkeeper-1010/
 
 The tag to be voted upon:
 release-4.4.0 (8bb39b187d1b681cf88651b825933d046545d34d)
 
 BookKeeper's KEYS file containing PGP keys we use to sign the release:
 https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
 
 Please download the the source package, and follow the README to build
 and run a bookkeeper service.
 
 Thank you,
 Matteo
 
>>> 
>> 
> -- 
> 
> 
> -- Enrico Olivelli

Re: Proposed board report [May]

[Flavio Junqueira]

Much better, thanks for the update. I suggest you omit the Hedwig part, it will 
take more explanation to be clear and it is really just mechanics of the 
project.

-Flavio
 
> On 11 May 2016, at 00:26, Sijie Guo  wrote:
> 
> Updated:
> 
> 
> BookKeeper is a distributed, reliable, and high performance
> logging service.
> 
> = Project Status =
> 
> Development on 4.4.0 release is completed. The first release candidate has
> been sent out for voting. We expect to announce 4.4.0 release by end of
> this week. We will start on release 4.5.0 after 4.4.0 is release.
> 
> There are various improvements on storage and client in 4.4.0. An advanced
> ledger handle is introduced in 4.4.0, to provide API to add entries with
> user supplied entry ids. Details will be noted in 4.4.0 release note.
> 
> We voted to remove Hedwig from 4.4.0 release as this project has become
> inactive for a while.
> 
> = Releases =
> 
> Our last release was 4.3.2, released on 2015-11-30. The first release
> candidate of 4.4.0 was out for voting last week.
> 
> = Community Status =
> 
> The last committer added was Matteo Merli (mmerli) on
> the 6th Jul 2015.
> 
> No infrastructure issues.
> 
> Two BookKeeper Talks at ApacheCon this year: "Apache BookKeeper at Twitter"
> from Leigh Stewart, "Low latency storage service using BookKeeper" from JV.
> 
> New projects using BookKeeper were open sourced:
> 
> - Majordodo: A distributed resource manager : http://majordodo.org/
> - DistributedLog: A replicated log service:
> https://github.com/twitter/distributedlog
> 
> 71 subscribers in dev@bookkeeper.apache.org
> 88 subscribers in u...@bookkeeper.apache.org
> 
> 916 issues opened to date, 26 since 2016-03-03
> 641 issues resolved to date, 28 since 2016-03-03
> 59 people have reported issues, 7 since 2016-03-03
> 33 people have contributed patches, 7 since 2016-03-03
> 
> 
> On Tue, May 10, 2016 at 3:42 AM, Flavio Junqueira  wrote:
> 
>> Thanks for putting this together, Sijie. A couple of thoughts on the
>> report:
>> 
>> - Let's mention recent use cases that have been present in blog posts,
>> talks, or new open source: Majordodo, DistributedLog, Salesforce storage.
>> - Let's mention the talks at ApacheCon
>> - It might be a good idea to mention what is new and interesting in 4.4.0.
>> 
>> -Flavio
>> 
>>> On 10 May 2016, at 11:39, Rakesh Radhakrishnan 
>> wrote:
>>> 
>>> +1, lgtm
>>> 
>>> Regards,
>>> Rakesh
>>> 
>>> On Tue, May 10, 2016 at 11:28 AM, Venkateswara Rao Jujjuri <
>>> jujj...@gmail.com> wrote:
>>> 
>>>> LGTM
>>>> 
>>>> On Mon, May 9, 2016 at 8:45 PM, Matteo Merli  wrote:
>>>> 
>>>>> Looks good to me
>>>>> 
>>>>> On Mon, May 9, 2016 at 7:06 PM Sijie Guo  wrote:
>>>>> 
>>>>>> Hello folks:
>>>>>> 
>>>>>> We have a board report due this month. Here is the draft.
>>>>>> 
>>>>>> Let me know if have anything to add. I have to submit this before
>> 5/11.
>>>>>> 
>>>>>> 
>>>>>> BookKeeper is a distributed, reliable, and high performance
>>>>>> logging service.
>>>>>> 
>>>>>> = Project Status =
>>>>>> 
>>>>>> Development on 4.4.0 release is completed. The first release candidate
>>>>> has
>>>>>> been sent out for voting. We expect to announce 4.4.0 release by end
>> of
>>>>>> this week. We will start on release 4.5.0 after 4.4.0 is release.
>>>>>> 
>>>>>> We voted to remove Hedwig from 4.4.0 release as this project has
>> become
>>>>>> inactive for a while.
>>>>>> 
>>>>>> = Releases =
>>>>>> 
>>>>>> Our last release was 4.3.2, released on 2015-11-30. The first release
>>>>>> candidate of 4.4.0 was out for voting last week.
>>>>>> 
>>>>>> = Community Status =
>>>>>> 
>>>>>> The last committer added was Matteo Merli (mmerli) on
>>>>>> the 6th Jul 2015.
>>>>>> 
>>>>>> No infrastructure issues.
>>>>>> 
>>>>>> 71 subscribers in dev@bookkeeper.apache.org
>>>>>> 88 subscribers in u...@bookkeeper.apache.org
>>>>>> 
>>>>>> 916 issues opened to date, 26 since 2016-03-03
>>>>>> 641 issues resolved to date, 28 since 2016-03-03
>>>>>> 59 people have reported issues, 7 since 2016-03-03
>>>>>> 33 people have contributed patches, 7 since 2016-03-03
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Jvrao
>>>> ---
>>>> First they ignore you, then they laugh at you, then they fight you, then
>>>> you win. - Mahatma Gandhi
>>>> 
>> 
>> 

Re: Proposed board report [May]

[Flavio Junqueira]

Thanks for putting this together, Sijie. A couple of thoughts on the report:

- Let's mention recent use cases that have been present in blog posts, talks, 
or new open source: Majordodo, DistributedLog, Salesforce storage.
- Let's mention the talks at ApacheCon
- It might be a good idea to mention what is new and interesting in 4.4.0.

-Flavio

> On 10 May 2016, at 11:39, Rakesh Radhakrishnan  
> wrote:
> 
> +1, lgtm
> 
> Regards,
> Rakesh
> 
> On Tue, May 10, 2016 at 11:28 AM, Venkateswara Rao Jujjuri <
> jujj...@gmail.com> wrote:
> 
>> LGTM
>> 
>> On Mon, May 9, 2016 at 8:45 PM, Matteo Merli  wrote:
>> 
>>> Looks good to me
>>> 
>>> On Mon, May 9, 2016 at 7:06 PM Sijie Guo  wrote:
>>> 
 Hello folks:
 
 We have a board report due this month. Here is the draft.
 
 Let me know if have anything to add. I have to submit this before 5/11.
 
 
 BookKeeper is a distributed, reliable, and high performance
 logging service.
 
 = Project Status =
 
 Development on 4.4.0 release is completed. The first release candidate
>>> has
 been sent out for voting. We expect to announce 4.4.0 release by end of
 this week. We will start on release 4.5.0 after 4.4.0 is release.
 
 We voted to remove Hedwig from 4.4.0 release as this project has become
 inactive for a while.
 
 = Releases =
 
 Our last release was 4.3.2, released on 2015-11-30. The first release
 candidate of 4.4.0 was out for voting last week.
 
 = Community Status =
 
 The last committer added was Matteo Merli (mmerli) on
 the 6th Jul 2015.
 
 No infrastructure issues.
 
 71 subscribers in dev@bookkeeper.apache.org
 88 subscribers in u...@bookkeeper.apache.org
 
 916 issues opened to date, 26 since 2016-03-03
 641 issues resolved to date, 28 since 2016-03-03
 59 people have reported issues, 7 since 2016-03-03
 33 people have contributed patches, 7 since 2016-03-03
 
 
>>> 
>> 
>> 
>> 
>> --
>> Jvrao
>> ---
>> First they ignore you, then they laugh at you, then they fight you, then
>> you win. - Mahatma Gandhi
>> 

Re: Logo

[Flavio Junqueira]

If you're interested in past presentations of BK (and we used the logo), check 
here:

https://cwiki.apache.org/confluence/display/BOOKKEEPER/BookKeeper+papers+and+presentations
 
<https://cwiki.apache.org/confluence/display/BOOKKEEPER/BookKeeper+papers+and+presentations>

In particular, there is this one that uses the logo extensively:

Serving millions of journals with Apache BookKeeper 
<https://cwiki.apache.org/confluence/download/attachments/27832576/bk-hadoop-summit-2013.pdf?version=1&modificationDate=1363841022000&api=v2>

-Flavio

> On 07 May 2016, at 01:10, Venkateswara Rao Jujjuri  wrote:
> 
> Ah. Great. Log on our website is so tiny (http://bookkeeper.apache.org/ 
> <http://bookkeeper.apache.org/>)
> hence it did not register me.
> 
> Thanks for pointing out.
> 
> On Fri, May 6, 2016 at 4:08 PM, Flavio Junqueira  <mailto:f...@apache.org>> wrote:
> 
>> Hey JV,
>> 
>> We've picked the one you see on the main web site. In that jira, there is
>> one for white background:
>> 
>> https://issues.apache.org/jira/secure/attachment/12506291/bookeper_wht.png 
>> <https://issues.apache.org/jira/secure/attachment/12506291/bookeper_wht.png>
>> <
>> https://issues.apache.org/jira/secure/attachment/12506291/bookeper_wht.png 
>> <https://issues.apache.org/jira/secure/attachment/12506291/bookeper_wht.png>
>>> 
>> 
>> and one for black background:
>> 
>> https://issues.apache.org/jira/secure/attachment/12506662/bookeper_blk.png 
>> <https://issues.apache.org/jira/secure/attachment/12506662/bookeper_blk.png>
>> <
>> https://issues.apache.org/jira/secure/attachment/12506662/bookeper_blk.png 
>> <https://issues.apache.org/jira/secure/attachment/12506662/bookeper_blk.png>
>>> 
>> 
>> The other ones were candidates we generated during the selection process.
>> 
>> Does it make sense?
>> 
>> -Flavio
>> 
>> 
>>> On 06 May 2016, at 23:54, Venkateswara Rao Jujjuri 
>> wrote:
>>> 
>>> I think we need to pick one and move forward. I want to use it in my
>>> upcoming presentation at ApacheCon. :)
>>> 
>>> On Fri, May 6, 2016 at 2:41 PM, Flavio P JUNQUEIRA 
>> wrote:
>>> 
>>>> Check it here:
>>>> 
>>>> https://issues.apache.org/jira/browse/BOOKKEEPER-31
>>>> 
>>>> -Flavio
>>>> On 6 May 2016 22:02, "Venkateswara Rao Jujjuri" 
>> wrote:
>>>> 
>>>>> Do we have a logo for BookKeeper Project?
>>>>> I see this, and looks good too,  is this our logo?
>>>>> 
>>>>> https://svn.apache.org/repos/asf/bookkeeper/site/trunk/content/images/
>>>>> <
>>>>> 
>>>> 
>> https://www.google.com/url?q=https%3A%2F%2Fsvn.apache.org%2Frepos%2Fasf%2Fbookkeeper%2Fsite%2Ftrunk%2Fcontent%2Fimages%2Fbookkeeper.png&sa=D&sntz=1&usg=AFQjCNHt9rdiNGF78w_Uikdw7vQKX84h0w
>>>>>> 
>>>>> 
>>>>> --
>>>>> Jvrao
>>>>> ---
>>>>> First they ignore you, then they laugh at you, then they fight you,
>> then
>>>>> you win. - Mahatma Gandhi
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Jvrao
>>> ---
>>> First they ignore you, then they laugh at you, then they fight you, then
>>> you win. - Mahatma Gandhi
>> 
>> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: Jenkins build is back to normal : bookkeeper-master #1371

[Flavio Junqueira]

Very happy to see that the build is mostly back to normal. The last build that 
failed seems to be due to a jenkins issue. Great job everyone!

-Flavio

> On 07 May 2016, at 14:14, Apache Jenkins Server  
> wrote:
> 
> See 
> 

Re: Logo

[Flavio Junqueira]

Hey JV,

We've picked the one you see on the main web site. In that jira, there is one 
for white background:

https://issues.apache.org/jira/secure/attachment/12506291/bookeper_wht.png 


and one for black background:

https://issues.apache.org/jira/secure/attachment/12506662/bookeper_blk.png 


The other ones were candidates we generated during the selection process.

Does it make sense?

-Flavio


> On 06 May 2016, at 23:54, Venkateswara Rao Jujjuri  wrote:
> 
> I think we need to pick one and move forward. I want to use it in my
> upcoming presentation at ApacheCon. :)
> 
> On Fri, May 6, 2016 at 2:41 PM, Flavio P JUNQUEIRA  wrote:
> 
>> Check it here:
>> 
>> https://issues.apache.org/jira/browse/BOOKKEEPER-31
>> 
>> -Flavio
>> On 6 May 2016 22:02, "Venkateswara Rao Jujjuri"  wrote:
>> 
>>> Do we have a logo for BookKeeper Project?
>>> I see this, and looks good too,  is this our logo?
>>> 
>>> https://svn.apache.org/repos/asf/bookkeeper/site/trunk/content/images/
>>> <
>>> 
>> https://www.google.com/url?q=https%3A%2F%2Fsvn.apache.org%2Frepos%2Fasf%2Fbookkeeper%2Fsite%2Ftrunk%2Fcontent%2Fimages%2Fbookkeeper.png&sa=D&sntz=1&usg=AFQjCNHt9rdiNGF78w_Uikdw7vQKX84h0w
 
>>> 
>>> --
>>> Jvrao
>>> ---
>>> First they ignore you, then they laugh at you, then they fight you, then
>>> you win. - Mahatma Gandhi
>>> 
>> 
> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: [VOTE] BookKeeper 4.4.0 Release Candidate 0

[Flavio Junqueira]

We typically only share the release notes and let people figure out what 
matters to them. It is up to the release manager to decide whether there is 
something important to highlight or not.

I agree that testing the upgrade path is pretty important. If you have tried, 
feel free to report back with either success or failure (hopefully success).

-Flavio

> On 06 May 2016, at 09:53, Enrico Olivelli  wrote:
> 
> Maybe it would be useful to draft a "news and noteworthy" page to summarize
> changes.
> 
> My primary concern is about the upgrade from 4.3.X to 4.4.0
> 
> Other interesting changes for me
> 
> Upgrading:
> https://issues.apache.org/jira/browse/BOOKKEEPER-870 Change the default
> value for bookie settings.
> https://issues.apache.org/jira/browse/BOOKKEEPER-810 Allow to configure TCP
> connect timeout
> 
> New API:
> https://issues.apache.org/jira/browse/BOOKKEEPER-793 Move to Java7
> https://issues.apache.org/jira/browse/BOOKKEEPER-880 Make LedgerHandle
> implement AutoCloseable
> https://issues.apache.org/jira/browse/BOOKKEEPER-867 New Client API to
> allow applications pass-in EntryId
> https://issues.apache.org/jira/browse/BOOKKEEPER-886 Allow to disable
> ledgers operation throttling
> https://issues.apache.org/jira/browse/BOOKKEEPER-879 Record ledger creation
> time
> 
> New docs:
> https://issues.apache.org/jira/browse/BOOKKEEPER-801 Bookkeeper client
> tutorial
> https://issues.apache.org/jira/browse/BOOKKEEPER-803 Guide for making a
> replicated log out of ledgers
> 
> Other notable enhancements:
> https://issues.apache.org/jira/browse/BOOKKEEPER-855 handle session expire
> event in bookie
> https://issues.apache.org/jira/browse/BOOKKEEPER-901 Add an authentication
> framework
> 
> 
> 2016-05-05 18:49 GMT+02:00 Matteo Merli :
> 
>> This is the first release candidate for Apache BookKeeper, version 4.4.0.
>> 
>> It fixes the following issues:
>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311293&version=12325566
>> 
>> *** Please download, test and vote by May 10th 2016, 10:00 GMT.
>> 
>> Note that we are voting upon the source (tag), binaries are provided for
>> convenience.
>> 
>> Source and binary files:
>> 
>> https://dist.apache.org/repos/dist/dev/bookkeeper/bookkeeper-4.4.0-candidate-0/
>> 
>> Maven staging repo:
>> 
>> https://repository.apache.org/content/repositories/orgapachebookkeeper-1010/
>> 
>> The tag to be voted upon:
>> release-4.4.0 (8bb39b187d1b681cf88651b825933d046545d34d)
>> 
>> BookKeeper's KEYS file containing PGP keys we use to sign the release:
>> https://dist.apache.org/repos/dist/release/bookkeeper/KEYS
>> 
>> Please download the the source package, and follow the README to build
>> and run a bookkeeper service.
>> 
>> Thank you,
>> Matteo
>> 

Re: BookKeeper release 4.4 plan

[Flavio Junqueira]

There is no pending issue for 4.4.0 according to jira, so the plan sounds good 
to me, +1.

-Flavio

> On 04 May 2016, at 20:10, Venkateswara Rao Jujjuri  wrote:
> 
> +1.
> 
> On Wed, May 4, 2016 at 9:04 AM, Sijie Guo  wrote:
> 
>> No objections. +1 for the release plan.
>> 
>> Sijie
>> 
>> On Wednesday, May 4, 2016, Matteo Merli  wrote:
>> 
>>> Hi all,
>>> 
>>> we should have all the changes planned for 4.4 release already merged in
>>> master and build seems to be good.
>>> 
>>> If there are no objections or suggestions for Jiras that you think should
>>> be considered for 4.4 inclusion, I will prepare a release candidate to
>>> submit for voting.
>>> 
>>> Thanks,
>>> Matteo
>>> 
>> 
> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: Another project using Bookkeeper - Majordodo

[Flavio Junqueira]

Thanks, Enrico. I have granted you permission to upload attachments.

-Flavio

> On 03 May 2016, at 10:24, Enrico Olivelli  wrote:
> 
> This is the draft
> https://cwiki.apache.org/confluence/display/BOOKKEEPER/Majordodo
> 
> Actually I cannot upload images, I put a direct link to GitHub, it needs to
> be uploaded to Confluence
> 
> Does any one can review ?
> Uma suggested that it is too verbose
> 
> Thanks
> @Diennea we love BookKeeper!
> 
> 
> 
> 2016-05-03 10:20 GMT+02:00 Enrico Olivelli :
> 
>> Now it works !
>> 
>> I'll file a draft of how Majordodo uses BookKeeper as soon as possible,
>> doing a summary of https://github.com/eolivelli/majordodo-docs-drafts
>> 
>> Thank you very much.
>> 
>> 
>> 2016-05-03 10:18 GMT+02:00 Flavio Junqueira :
>> 
>>> Try again, Enrico, please.
>>> 
>>> -Flavio
>>> 
>>>> On 03 May 2016, at 09:04, Enrico Olivelli  wrote:
>>>> 
>>>> I'm sorry, I cannot  edit/create pages at
>>>> https://cwiki.apache.org/confluence/display/BOOKKEEPER,
>>>> 
>>>> I even tried to log out and log in, but no result
>>>> 
>>>> Thanks
>>>> 
>>>> 2016-05-03 9:12 GMT+02:00 Flavio Junqueira :
>>>> 
>>>>> Enrico,
>>>>> 
>>>>> Please check that you have edit access to the wiki.
>>>>> 
>>>>> -Flavio
>>>>> 
>>>>>> On 01 May 2016, at 13:11, Enrico Olivelli 
>>> wrote:
>>>>>> 
>>>>>> Any news on this?
>>>>>> 
>>>>>> Il Mar 26 Apr 2016 03:06 Sijie Guo  ha scritto:
>>>>>> 
>>>>>>> Flavio,
>>>>>>> 
>>>>>>> I don't think I have the admin access to the space 'bookkeeper'. do
>>> you?
>>>>>>> 
>>>>>>> - Sijie
>>>>>>> 
>>>>>>> On Mon, Apr 25, 2016 at 2:04 PM, Flavio Junqueira 
>>>>> wrote:
>>>>>>> 
>>>>>>>> Sijie,
>>>>>>>> 
>>>>>>>> Are you able to grant access to Enrico?
>>>>>>>> 
>>>>>>>> -Flavio
>>>>>>>> 
>>>>>>>>> On 19 Apr 2016, at 08:08, Enrico Olivelli 
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> My username is eolivelli at cwiki.apache.org
>>>>>>>>> Thanks
>>>>>>>>> 
>>>>>>>>> -- Enrico
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 2016-04-19 17:05 GMT+02:00 Flavio Junqueira
>>>>>>>> :
>>>>>>>>> 
>>>>>>>>>> Have you created a wiki account already? I don't seem to be able
>>> to
>>>>>>>> grant
>>>>>>>>>> access to the wiki. Sijie, are you able to add people to the
>>>>>>> bookkeeper
>>>>>>>>>> wiki?
>>>>>>>>>> 
>>>>>>>>>> -Flavio
>>>>>>>>>> 
>>>>>>>>>>> On 19 Apr 2016, at 07:25, Enrico Olivelli 
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> Some time ago I wrote some more notes about the usage of
>>> BookKeeper
>>>>>>> in
>>>>>>>>>>> Majordodo.
>>>>>>>>>>> This is the doc
>>>>>>>>>>> https://github.com/eolivelli/majordodo-docs-drafts
>>>>>>>>>>> May I get write access to the Wiki in order to write down my
>>>>> summary,
>>>>>>>>>>> As there is no version control or review facility in Confluence
>>> how
>>>>>>>> can I
>>>>>>>>>>> work?
>>>>>>>>>>> 
>>>>>>>>>>> - Enrico
>>>>>>>>>>> 
>>>>>>>>>>> Il Mer 2 Mar 2016 12:16 Enrico Olivelli  ha
>>>>>>>>>> scritto:
>>>>>>>>>>> 
>>>>>>>>>>>> Here 

Re: Another project using Bookkeeper - Majordodo

[Flavio Junqueira]

Try again, Enrico, please.

-Flavio

> On 03 May 2016, at 09:04, Enrico Olivelli  wrote:
> 
> I'm sorry, I cannot  edit/create pages at
> https://cwiki.apache.org/confluence/display/BOOKKEEPER,
> 
> I even tried to log out and log in, but no result
> 
> Thanks
> 
> 2016-05-03 9:12 GMT+02:00 Flavio Junqueira :
> 
>> Enrico,
>> 
>> Please check that you have edit access to the wiki.
>> 
>> -Flavio
>> 
>>> On 01 May 2016, at 13:11, Enrico Olivelli  wrote:
>>> 
>>> Any news on this?
>>> 
>>> Il Mar 26 Apr 2016 03:06 Sijie Guo  ha scritto:
>>> 
>>>> Flavio,
>>>> 
>>>> I don't think I have the admin access to the space 'bookkeeper'. do you?
>>>> 
>>>> - Sijie
>>>> 
>>>> On Mon, Apr 25, 2016 at 2:04 PM, Flavio Junqueira 
>> wrote:
>>>> 
>>>>> Sijie,
>>>>> 
>>>>> Are you able to grant access to Enrico?
>>>>> 
>>>>> -Flavio
>>>>> 
>>>>>> On 19 Apr 2016, at 08:08, Enrico Olivelli 
>> wrote:
>>>>>> 
>>>>>> My username is eolivelli at cwiki.apache.org
>>>>>> Thanks
>>>>>> 
>>>>>> -- Enrico
>>>>>> 
>>>>>> 
>>>>>> 2016-04-19 17:05 GMT+02:00 Flavio Junqueira
>>>>> :
>>>>>> 
>>>>>>> Have you created a wiki account already? I don't seem to be able to
>>>>> grant
>>>>>>> access to the wiki. Sijie, are you able to add people to the
>>>> bookkeeper
>>>>>>> wiki?
>>>>>>> 
>>>>>>> -Flavio
>>>>>>> 
>>>>>>>> On 19 Apr 2016, at 07:25, Enrico Olivelli 
>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> Some time ago I wrote some more notes about the usage of BookKeeper
>>>> in
>>>>>>>> Majordodo.
>>>>>>>> This is the doc
>>>>>>>> https://github.com/eolivelli/majordodo-docs-drafts
>>>>>>>> May I get write access to the Wiki in order to write down my
>> summary,
>>>>>>>> As there is no version control or review facility in Confluence how
>>>>> can I
>>>>>>>> work?
>>>>>>>> 
>>>>>>>> - Enrico
>>>>>>>> 
>>>>>>>> Il Mer 2 Mar 2016 12:16 Enrico Olivelli  ha
>>>>>>> scritto:
>>>>>>>> 
>>>>>>>>> Here is a first simple post about Majordodo and BookKeeper
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> http://eolivelli.blogspot.it/2016/03/majordodo-distributed-resource-manager.html
>>>>>>>>> 
>>>>>>>>> as soon as I can I will provide some more detailed content about
>>>>>>> Majordodo
>>>>>>>>> internals to put on BookKeeper wiki
>>>>>>>>> 
>>>>>>>>> -- Enrico
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 2016-02-24 7:43 GMT+01:00 Venkateswara Rao Jujjuri <
>>>> jujj...@gmail.com
>>>>>> :
>>>>>>>>> 
>>>>>>>>>> Exciting to see this. As Sijie suggested use case details will be
>>>>>>> awesome.
>>>>>>>>>> 
>>>>>>>>>> -JV
>>>>>>>>>> 
>>>>>>>>>> On Wednesday, February 24, 2016, Enrico Olivelli <
>>>>> eolive...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Thank you for adding Majordodo to Bookkeeper wiki.
>>>>>>>>>>> 
>>>>>>>>>>> Documentation on Majordodo website is still very incomplete we
>>>> will
>>>>>>>>>>> describe more completely the usage of Bookkeeper in Majordodo
>>>>>>>>>> internals. If
>>>>>>>>>>> you want I can write some post as soon

Re: Another project using Bookkeeper - Majordodo

[Flavio Junqueira]

Enrico, 

Please check that you have edit access to the wiki.

-Flavio

> On 01 May 2016, at 13:11, Enrico Olivelli  wrote:
> 
> Any news on this?
> 
> Il Mar 26 Apr 2016 03:06 Sijie Guo  ha scritto:
> 
>> Flavio,
>> 
>> I don't think I have the admin access to the space 'bookkeeper'. do you?
>> 
>> - Sijie
>> 
>> On Mon, Apr 25, 2016 at 2:04 PM, Flavio Junqueira  wrote:
>> 
>>> Sijie,
>>> 
>>> Are you able to grant access to Enrico?
>>> 
>>> -Flavio
>>> 
>>>> On 19 Apr 2016, at 08:08, Enrico Olivelli  wrote:
>>>> 
>>>> My username is eolivelli at cwiki.apache.org
>>>> Thanks
>>>> 
>>>> -- Enrico
>>>> 
>>>> 
>>>> 2016-04-19 17:05 GMT+02:00 Flavio Junqueira
>>> :
>>>> 
>>>>> Have you created a wiki account already? I don't seem to be able to
>>> grant
>>>>> access to the wiki. Sijie, are you able to add people to the
>> bookkeeper
>>>>> wiki?
>>>>> 
>>>>> -Flavio
>>>>> 
>>>>>> On 19 Apr 2016, at 07:25, Enrico Olivelli 
>> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> Some time ago I wrote some more notes about the usage of BookKeeper
>> in
>>>>>> Majordodo.
>>>>>> This is the doc
>>>>>> https://github.com/eolivelli/majordodo-docs-drafts
>>>>>> May I get write access to the Wiki in order to write down my summary,
>>>>>> As there is no version control or review facility in Confluence how
>>> can I
>>>>>> work?
>>>>>> 
>>>>>> - Enrico
>>>>>> 
>>>>>> Il Mer 2 Mar 2016 12:16 Enrico Olivelli  ha
>>>>> scritto:
>>>>>> 
>>>>>>> Here is a first simple post about Majordodo and BookKeeper
>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> http://eolivelli.blogspot.it/2016/03/majordodo-distributed-resource-manager.html
>>>>>>> 
>>>>>>> as soon as I can I will provide some more detailed content about
>>>>> Majordodo
>>>>>>> internals to put on BookKeeper wiki
>>>>>>> 
>>>>>>> -- Enrico
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 2016-02-24 7:43 GMT+01:00 Venkateswara Rao Jujjuri <
>> jujj...@gmail.com
>>>> :
>>>>>>> 
>>>>>>>> Exciting to see this. As Sijie suggested use case details will be
>>>>> awesome.
>>>>>>>> 
>>>>>>>> -JV
>>>>>>>> 
>>>>>>>> On Wednesday, February 24, 2016, Enrico Olivelli <
>>> eolive...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Thank you for adding Majordodo to Bookkeeper wiki.
>>>>>>>>> 
>>>>>>>>> Documentation on Majordodo website is still very incomplete we
>> will
>>>>>>>>> describe more completely the usage of Bookkeeper in Majordodo
>>>>>>>> internals. If
>>>>>>>>> you want I can write some post as soon as I can
>>>>>>>>> 
>>>>>>>>> Cheers
>>>>>>>>> Enrico
>>>>>>>>> 
>>>>>>>>> Il giorno Mar 23 Feb 2016 19:55 Sijie Guo >>>>>>> >
>>>>>>>>> ha scritto:
>>>>>>>>> 
>>>>>>>>>> Yup. I just updated the poweredby page.
>>>>>>>>>> 
>>>>>>>>>> - Sijie
>>>>>>>>>> 
>>>>>>>>>> On Tue, Feb 23, 2016 at 10:42 AM, Flavio Junqueira <
>> f...@apache.org
>>>>>>>>> > wrote:
>>>>>>>>>> 
>>>>>>>>>>> You should probably update the poweredby page too, Sijie.
>>>>>>>>>>> 
>>>>>>>>>>> -Flavio
>>>>>>>>>>> 
>>>>>>>>>>>> On 23 Feb 2016, at 18:11, Sijie Guo >>>>>>> >
>>>&g

[jira] [Commented] (BOOKKEEPER-923) OutOfMemoryError during build on Jenkins

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257111#comment-15257111
 ] 

Flavio Junqueira commented on BOOKKEEPER-923:
-

You're right that the stack trace doesn't include anything related to 
BookKeeper, but it could be accidental: BK fills up the heap and when surefire 
tries to produce a report, it finds the memory full. I'm a little reluctant 
about increasing the max heap size without fully understanding why this is 
happening. 

> OutOfMemoryError during build on Jenkins
> 
>
> Key: BOOKKEEPER-923
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-923
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: build
>Affects Versions: 4.4.0
> Environment: ASF Jenkins platform
>Reporter: Enrico Olivelli
>Priority: Critical
>
> I'm seeing this java.lang.OutOfMemoryError error on Jenkins
> > https://builds.apache.org/job/bookkeeper-master/1349/
> I'm afraid that new version uses more memory than previous one,
> is there some commit which changed memory consumption ?
> {code}
> ---
>  T E S T S
> ---
> Running org.apache.bookkeeper.metastore.TestMetaStore
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.386 sec
> Running org.apache.bookkeeper.proto.TestDeathwatcher
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.38 sec
> Running org.apache.bookkeeper.proto.TestBackwardCompatCMS42
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.631 sec
> Running org.apache.bookkeeper.proto.TestPerChannelBookieClient
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.231 sec
> Running org.apache.bookkeeper.proto.TestBKStats
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 sec
> Running org.apache.bookkeeper.client.RoundRobinDistributionScheduleTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.235 sec
> Running org.apache.bookkeeper.client.TestWatchEnsembleChange
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.548 sec
> Running org.apache.bookkeeper.client.TestAddEntryQuorumTimeout
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.19 sec
> Running org.apache.bookkeeper.client.TestRackawareEnsemblePlacementPolicy
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.259 sec
> Running org.apache.bookkeeper.client.BookKeeperTest
> Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.856 sec
> Running org.apache.bookkeeper.client.TestReadTimeout
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.936 sec
> Running org.apache.bookkeeper.client.SlowBookieTest
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.236 sec
> Running org.apache.bookkeeper.client.TestTryReadLastConfirmed
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.28 sec
> Running org.apache.bookkeeper.client.LocalBookKeeperTest
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.26 sec
> Running org.apache.bookkeeper.client.UpdateLedgerCmdTest
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.712 sec
> Running org.apache.bookkeeper.client.TestFencing
> Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.622 sec
> Running org.apache.bookkeeper.client.BookieWriteLedgerTest
> Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.367 sec
> Running org.apache.bookkeeper.client.ListLedgersTest
> Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.221 sec
> Running org.apache.bookkeeper.client.BookKeeperCloseTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.428 sec
> Running org.apache.bookkeeper.client.BookieRecoveryTest
> Tests run: 72, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.937 sec
> Running org.apache.bookkeeper.client.TestBookieHealthCheck
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.448 sec
> Running org.apache.bookkeeper.client.UpdateLedgerOpTest
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.666 sec
> Running org.apache.bookkeeper.client.TestLedgerFragmentReplication
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.914 sec
> Running org.apache.bookkeeper.client.TestBookieWatcher
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.92 sec
> Running org.apache.bookkeeper.clien

Re: OutOfMemoryError error on Jenkins

[Flavio Junqueira]

I'm sorry, I dropped the ball on this thread. Here is one issue that we had 
some time back with out of memory:

https://issues.apache.org/jira/browse/BOOKKEEPER-4 
<https://issues.apache.org/jira/browse/BOOKKEEPER-4>

The fix that solved the issue is linked to that jira.

-Flavio

> On 19 Apr 2016, at 09:26, Venkateswara Rao Jujjuri  wrote:
> 
>> We have seen OOM exceptions before, but I can't remember seeing it for
> these tests. We currently have this in Jenkins: -Xmx2048m
> 
> Flavio,
> Can you please give more information on the previous OOMs? and how did we
> took care of them? i.e do you have to tweak any
> JVM parameters? or bug fixes went in? or they are still lurking? If so,
> what are the situations we may see them?
> 
> On Tue, Apr 19, 2016 at 7:34 AM, Flavio Junqueira <
> fpjunque...@yahoo.com.invalid> wrote:
> 
>> We have seen OOM exceptions before, but I can't remember seeing it for
>> these tests. We currently have this in Jenkins: -Xmx2048m
>> 
>> Maybe you want to create a jira and report it?
>> 
>> -Flavio
>> 
>>> On 18 Apr 2016, at 15:04, Enrico Olivelli - Diennea <
>> enrico.olive...@diennea.com> wrote:
>>> 
>>> I'm seeing this java.lang.OutOfMemoryError error on Jenkins
>>> https://builds.apache.org/job/bookkeeper-master/1349/
>>> 
>>> did you see this  ?
>>> 
>>> 
>>> 
>>> Running org.apache.bookkeeper.replication.TestReplicationWorker
>>> Tests run: 27, Failures: 3, Errors: 3, Skipped: 0, Time elapsed: 135.658
>> sec <<< FAILURE!
>>> Running org.apache.bookkeeper.replication.AuditorPeriodicCheckTest
>>> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError:
>> Java heap space
>>>   at java.util.Arrays.copyOf(Arrays.java:2367)
>>>   at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>>>   at
>> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>>>   at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>>>   at java.lang.StringBuffer.append(StringBuffer.java:237)
>>>   at
>> org.apache.maven.surefire.report.ConsoleOutputFileReporter.writeMessage(ConsoleOutputFileReporter.java:115)
>>>   at
>> org.apache.maven.surefire.report.MulticastingReporter.writeMessage(MulticastingReporter.java:101)
>>>   at
>> org.apache.maven.surefire.report.TestSetRunListener.writeTestOutput(TestSetRunListener.java:99)
>>>   at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:132)
>>>   at
>> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
>>>   at java.lang.Thread.run(Thread.java:745)
>>> Exception in thread "Thread-215" java.lang.OutOfMemoryError: Java heap
>> space
>>>   at java.util.Arrays.copyOf(Arrays.java:2367)
>>>   at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>>>   at
>> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>>>   at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
>>>   at java.lang.StringBuffer.append(StringBuffer.java:322)
>>>   at java.io.BufferedReader.readLine(BufferedReader.java:351)
>>>   at java.io.BufferedReader.readLine(BufferedReader.java:382)
>>>   at
>> org.codehaus.plexus.util.cli.StreamPumper.run(StreamPumper.java:129)
>>> 
>>> 
>>> 
>>> --
>>> Enrico Olivelli
>>> Software Development Manager @Diennea
>>> Tel.: (+39) 0546 066100 - Int. 925
>>> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>>> 
>>> MagNews - E-mail Marketing Solutions
>>> http://www.magnews.it
>>> Diennea - Digital Marketing Solutions
>>> http://www.diennea.com
>>> 
>>> 
>>> 
>>> 
>>> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
>> email marketing! http://www.magnews.it/newsletter/
>>> 
>>> The information in this email is confidential and may be legally
>> privileged. If you are not the intended recipient please notify the sender
>> immediately and destroy this email. Any unauthorized, direct or indirect,
>> disclosure, copying, storage, distribution or other use is strictly
>> forbidden.
>> 
>> 
> 
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

Re: Another project using Bookkeeper - Majordodo

[Flavio Junqueira]

Sijie,

Are you able to grant access to Enrico?

-Flavio

> On 19 Apr 2016, at 08:08, Enrico Olivelli  wrote:
> 
> My username is eolivelli at cwiki.apache.org
> Thanks
> 
> -- Enrico
> 
> 
> 2016-04-19 17:05 GMT+02:00 Flavio Junqueira :
> 
>> Have you created a wiki account already? I don't seem to be able to grant
>> access to the wiki. Sijie, are you able to add people to the bookkeeper
>> wiki?
>> 
>> -Flavio
>> 
>>> On 19 Apr 2016, at 07:25, Enrico Olivelli  wrote:
>>> 
>>> Hi,
>>> Some time ago I wrote some more notes about the usage of BookKeeper in
>>> Majordodo.
>>> This is the doc
>>> https://github.com/eolivelli/majordodo-docs-drafts
>>> May I get write access to the Wiki in order to write down my summary,
>>> As there is no version control or review facility in Confluence how can I
>>> work?
>>> 
>>> - Enrico
>>> 
>>> Il Mer 2 Mar 2016 12:16 Enrico Olivelli  ha
>> scritto:
>>> 
>>>> Here is a first simple post about Majordodo and BookKeeper
>>>> 
>>>> 
>> http://eolivelli.blogspot.it/2016/03/majordodo-distributed-resource-manager.html
>>>> 
>>>> as soon as I can I will provide some more detailed content about
>> Majordodo
>>>> internals to put on BookKeeper wiki
>>>> 
>>>> -- Enrico
>>>> 
>>>> 
>>>> 
>>>> 2016-02-24 7:43 GMT+01:00 Venkateswara Rao Jujjuri :
>>>> 
>>>>> Exciting to see this. As Sijie suggested use case details will be
>> awesome.
>>>>> 
>>>>> -JV
>>>>> 
>>>>> On Wednesday, February 24, 2016, Enrico Olivelli 
>>>>> wrote:
>>>>> 
>>>>>> Thank you for adding Majordodo to Bookkeeper wiki.
>>>>>> 
>>>>>> Documentation on Majordodo website is still very incomplete we will
>>>>>> describe more completely the usage of Bookkeeper in Majordodo
>>>>> internals. If
>>>>>> you want I can write some post as soon as I can
>>>>>> 
>>>>>> Cheers
>>>>>> Enrico
>>>>>> 
>>>>>> Il giorno Mar 23 Feb 2016 19:55 Sijie Guo >>>> >
>>>>>> ha scritto:
>>>>>> 
>>>>>>> Yup. I just updated the poweredby page.
>>>>>>> 
>>>>>>> - Sijie
>>>>>>> 
>>>>>>> On Tue, Feb 23, 2016 at 10:42 AM, Flavio Junqueira >>>>> > wrote:
>>>>>>> 
>>>>>>>> You should probably update the poweredby page too, Sijie.
>>>>>>>> 
>>>>>>>> -Flavio
>>>>>>>> 
>>>>>>>>> On 23 Feb 2016, at 18:11, Sijie Guo >>>> >
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Yup, Enrico, do you mind adding your use case to the wiki?
>>>>>>>>> 
>>>>>>>>> On Tue, Feb 23, 2016 at 9:56 AM, Uma gangumalla <
>>>>>> umamah...@apache.org >
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Exited to see this. Great news. How about adding the details in
>>>>> Wiki
>>>>>>>>>> 
>>>>>>>>>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/PoweredBy
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Uma
>>>>>>>>>> 
>>>>>>>>>> On Tue, Feb 23, 2016 at 3:47 AM, Enrico Olivelli - Diennea <
>>>>>>>>>> enrico.olive...@diennea.com > wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> I would like to share with Bookkeepers the general availability
>>>>> of
>>>>>>>>>>> Majordodo http://majordodo.org/.
>>>>>>>>>>> 
>>>>>>>>>>> Majordodo is our distributed resource manager built on top of
>>>>>>>> Bookkeeper.
>>>>>>>>>>> It is designed for multi-tenancy and for handling millions of
>>>>>>>> micro-tasks
>>>>>>>>>>> (today our largest deployment spawns about 80 millions tasks per
>>>>>> day
>>>>>>>> on a
>>>>>>>>>>> 100 nodes cluster)
>>>>>>>>>>> 
>>>>>>>>>>> This is how Bookkeeper-based replication works
>>>>>>>>>>> https://majordodo.readme.io/docs/broker-replication
>>>>>>>>>>> 
>>>>>>>>>>> Thank you all for working on Bookkeeper
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Enrico Olivelli
>>>>>>>>>>> Software Development Manager @Diennea
>>>>>>>>>>> Tel.: (+39) 0546 066100 - Int. 925
>>>>>>>>>>> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>>>>>>>>>>> 
>>>>>>>>>>> MagNews - E-mail Marketing Solutions
>>>>>>>>>>> http://www.magnews.it
>>>>>>>>>>> Diennea - Digital Marketing Solutions
>>>>>>>>>>> http://www.diennea.com
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Iscriviti alla nostra newsletter per rimanere aggiornato su
>>>>> digital
>>>>>>> ed
>>>>>>>>>>> email marketing! http://www.magnews.it/newsletter/
>>>>>>>>>>> 
>>>>>>>>>>> The information in this email is confidential and may be legally
>>>>>>>>>>> privileged. If you are not the intended recipient please notify
>>>>> the
>>>>>>>>>> sender
>>>>>>>>>>> immediately and destroy this email. Any unauthorized, direct or
>>>>>>>> indirect,
>>>>>>>>>>> disclosure, copying, storage, distribution or other use is
>>>>> strictly
>>>>>>>>>>> forbidden.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> 
>>>>>> -- Enrico Olivelli
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Sent from iPhone
>>>>> 
>>>> 
>>>> --
>>> 
>>> 
>>> -- Enrico Olivelli
>> 
>> 

Re: Another project using Bookkeeper - Majordodo

[Flavio Junqueira]

Have you created a wiki account already? I don't seem to be able to grant 
access to the wiki. Sijie, are you able to add people to the bookkeeper wiki?

-Flavio

> On 19 Apr 2016, at 07:25, Enrico Olivelli  wrote:
> 
> Hi,
> Some time ago I wrote some more notes about the usage of BookKeeper in
> Majordodo.
> This is the doc
> https://github.com/eolivelli/majordodo-docs-drafts
> May I get write access to the Wiki in order to write down my summary,
> As there is no version control or review facility in Confluence how can I
> work?
> 
> - Enrico
> 
> Il Mer 2 Mar 2016 12:16 Enrico Olivelli  ha scritto:
> 
>> Here is a first simple post about Majordodo and BookKeeper
>> 
>> http://eolivelli.blogspot.it/2016/03/majordodo-distributed-resource-manager.html
>> 
>> as soon as I can I will provide some more detailed content about Majordodo
>> internals to put on BookKeeper wiki
>> 
>> -- Enrico
>> 
>> 
>> 
>> 2016-02-24 7:43 GMT+01:00 Venkateswara Rao Jujjuri :
>> 
>>> Exciting to see this. As Sijie suggested use case details will be awesome.
>>> 
>>> -JV
>>> 
>>> On Wednesday, February 24, 2016, Enrico Olivelli 
>>> wrote:
>>> 
>>>> Thank you for adding Majordodo to Bookkeeper wiki.
>>>> 
>>>> Documentation on Majordodo website is still very incomplete we will
>>>> describe more completely the usage of Bookkeeper in Majordodo
>>> internals. If
>>>> you want I can write some post as soon as I can
>>>> 
>>>> Cheers
>>>> Enrico
>>>> 
>>>> Il giorno Mar 23 Feb 2016 19:55 Sijie Guo >> >
>>>> ha scritto:
>>>> 
>>>>> Yup. I just updated the poweredby page.
>>>>> 
>>>>> - Sijie
>>>>> 
>>>>> On Tue, Feb 23, 2016 at 10:42 AM, Flavio Junqueira >>> > wrote:
>>>>> 
>>>>>> You should probably update the poweredby page too, Sijie.
>>>>>> 
>>>>>> -Flavio
>>>>>> 
>>>>>>> On 23 Feb 2016, at 18:11, Sijie Guo >> >
>>>> wrote:
>>>>>>> 
>>>>>>> Yup, Enrico, do you mind adding your use case to the wiki?
>>>>>>> 
>>>>>>> On Tue, Feb 23, 2016 at 9:56 AM, Uma gangumalla <
>>>> umamah...@apache.org >
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Exited to see this. Great news. How about adding the details in
>>> Wiki
>>>>>>>> 
>>>>>>>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/PoweredBy
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Uma
>>>>>>>> 
>>>>>>>> On Tue, Feb 23, 2016 at 3:47 AM, Enrico Olivelli - Diennea <
>>>>>>>> enrico.olive...@diennea.com > wrote:
>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> I would like to share with Bookkeepers the general availability
>>> of
>>>>>>>>> Majordodo http://majordodo.org/.
>>>>>>>>> 
>>>>>>>>> Majordodo is our distributed resource manager built on top of
>>>>>> Bookkeeper.
>>>>>>>>> It is designed for multi-tenancy and for handling millions of
>>>>>> micro-tasks
>>>>>>>>> (today our largest deployment spawns about 80 millions tasks per
>>>> day
>>>>>> on a
>>>>>>>>> 100 nodes cluster)
>>>>>>>>> 
>>>>>>>>> This is how Bookkeeper-based replication works
>>>>>>>>> https://majordodo.readme.io/docs/broker-replication
>>>>>>>>> 
>>>>>>>>> Thank you all for working on Bookkeeper
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Enrico Olivelli
>>>>>>>>> Software Development Manager @Diennea
>>>>>>>>> Tel.: (+39) 0546 066100 - Int. 925
>>>>>>>>> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>>>>>>>>> 
>>>>>>>>> MagNews - E-mail Marketing Solutions
>>>>>>>>> http://www.magnews.it
>>>>>>>>> Diennea - Digital Marketing Solutions
>>>>>>>>> http://www.diennea.com
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Iscriviti alla nostra newsletter per rimanere aggiornato su
>>> digital
>>>>> ed
>>>>>>>>> email marketing! http://www.magnews.it/newsletter/
>>>>>>>>> 
>>>>>>>>> The information in this email is confidential and may be legally
>>>>>>>>> privileged. If you are not the intended recipient please notify
>>> the
>>>>>>>> sender
>>>>>>>>> immediately and destroy this email. Any unauthorized, direct or
>>>>>> indirect,
>>>>>>>>> disclosure, copying, storage, distribution or other use is
>>> strictly
>>>>>>>>> forbidden.
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> --
>>>> 
>>>> 
>>>> -- Enrico Olivelli
>>>> 
>>> 
>>> 
>>> --
>>> Sent from iPhone
>>> 
>> 
>> --
> 
> 
> -- Enrico Olivelli

Re: OutOfMemoryError error on Jenkins

[Flavio Junqueira]

We have seen OOM exceptions before, but I can't remember seeing it for these 
tests. We currently have this in Jenkins: -Xmx2048m

Maybe you want to create a jira and report it?

-Flavio

> On 18 Apr 2016, at 15:04, Enrico Olivelli - Diennea 
>  wrote:
> 
> I'm seeing this java.lang.OutOfMemoryError error on Jenkins
> https://builds.apache.org/job/bookkeeper-master/1349/
> 
> did you see this  ?
> 
> 
> 
> Running org.apache.bookkeeper.replication.TestReplicationWorker
> Tests run: 27, Failures: 3, Errors: 3, Skipped: 0, Time elapsed: 135.658 sec 
> <<< FAILURE!
> Running org.apache.bookkeeper.replication.AuditorPeriodicCheckTest
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java 
> heap space
>at java.util.Arrays.copyOf(Arrays.java:2367)
>at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>at java.lang.StringBuffer.append(StringBuffer.java:237)
>at 
> org.apache.maven.surefire.report.ConsoleOutputFileReporter.writeMessage(ConsoleOutputFileReporter.java:115)
>at 
> org.apache.maven.surefire.report.MulticastingReporter.writeMessage(MulticastingReporter.java:101)
>at 
> org.apache.maven.surefire.report.TestSetRunListener.writeTestOutput(TestSetRunListener.java:99)
>at 
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:132)
>at 
> org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
>at java.lang.Thread.run(Thread.java:745)
> Exception in thread "Thread-215" java.lang.OutOfMemoryError: Java heap space
>at java.util.Arrays.copyOf(Arrays.java:2367)
>at 
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>at 
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>at 
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
>at java.lang.StringBuffer.append(StringBuffer.java:322)
>at java.io.BufferedReader.readLine(BufferedReader.java:351)
>at java.io.BufferedReader.readLine(BufferedReader.java:382)
>at org.codehaus.plexus.util.cli.StreamPumper.run(StreamPumper.java:129)
> 
> 
> 
> --
> Enrico Olivelli
> Software Development Manager @Diennea
> Tel.: (+39) 0546 066100 - Int. 925
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
> 
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it
> Diennea - Digital Marketing Solutions
> http://www.diennea.com
> 
> 
> 
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email 
> marketing! http://www.magnews.it/newsletter/
> 
> The information in this email is confidential and may be legally privileged. 
> If you are not the intended recipient please notify the sender immediately 
> and destroy this email. Any unauthorized, direct or indirect, disclosure, 
> copying, storage, distribution or other use is strictly forbidden.

Github commits

[Flavio Junqueira]

I was having a look at BK-904 and couldn't determine from jira or the pull 
request who reviewed and merged. By looking at the log, I can see it was 
Matteo. 

For committers: please +1 any issue before merging it either on the pull 
request or jira. 

-Flavio 

[jira] [Commented] (BOOKKEEPER-917) LocalBookKeeperTest seems to be silently failing

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233468#comment-15233468
 ] 

Flavio Junqueira commented on BOOKKEEPER-917:
-

[~eolivelli] would you mind starting a pull request for this issue? it is 
important that we resolve this soon because the QA builds are getting stuck on 
this test and failing consistently as a consequence.

> LocalBookKeeperTest seems to be silently failing
> 
>
> Key: BOOKKEEPER-917
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-917
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Affects Versions: 4.4.0
>Reporter: Flavio Junqueira
>Assignee: Enrico Olivelli
>Priority: Critical
> Fix For: 4.4.0
>
>
> I've noticed this while inspecting the output in jenkins:
> {noformat}
> Running org.apache.bookkeeper.client.BookKeeperCloseTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.074 sec
> Running org.apache.bookkeeper.client.LocalBookKeeperTest
> Running org.apache.bookkeeper.meta.GcLedgersTest
> {noformat}
> It sounds like {{LocalBookKeeperTest}} is failing silently. Is it hanging and 
> timing out?
> https://builds.apache.org/job/bookkeeper-master-git-pullrequest/57/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BOOKKEEPER-917) LocalBookKeeperTest seems to be silently failing

[Flavio Junqueira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flavio Junqueira updated BOOKKEEPER-917:

Assignee: Enrico Olivelli  (was: Matteo Merli)

> LocalBookKeeperTest seems to be silently failing
> 
>
> Key: BOOKKEEPER-917
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-917
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>Affects Versions: 4.4.0
>Reporter: Flavio Junqueira
>Assignee: Enrico Olivelli
>Priority: Critical
> Fix For: 4.4.0
>
>
> I've noticed this while inspecting the output in jenkins:
> {noformat}
> Running org.apache.bookkeeper.client.BookKeeperCloseTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.074 sec
> Running org.apache.bookkeeper.client.LocalBookKeeperTest
> Running org.apache.bookkeeper.meta.GcLedgersTest
> {noformat}
> It sounds like {{LocalBookKeeperTest}} is failing silently. Is it hanging and 
> timing out?
> https://builds.apache.org/job/bookkeeper-master-git-pullrequest/57/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BOOKKEEPER-917) LocalBookKeeperTest seems to be silently failing

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230455#comment-15230455
 ] 

Flavio Junqueira commented on BOOKKEEPER-917:
-

Right, but it doesn't pass even if I comment out the {{LOG.error}} line :

{noformat}
public void channelOpen(ChannelHandlerContext ctx,
ChannelStateEvent e) throws Exception {
LOG.info("Channel open {}", ctx.getChannel());
SocketAddress remote  = ctx.getChannel().getRemoteAddress();
if (remote instanceof InetSocketAddress) {
authProvider = 
authProviderFactory.newProvider((InetSocketAddress)remote,
new AuthHandshakeCompleteCallback());
} else {
//LOG.error("Unknown socket type {} for {}", remote.getClass(), 
remote);
}
super.channelOpen(ctx, e);
}
{noformat}

There is something else broken with the test.

> LocalBookKeeperTest seems to be silently failing
> 
>
> Key: BOOKKEEPER-917
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-917
> Project: Bookkeeper
>  Issue Type: Bug
>  Components: bookkeeper-server
>    Affects Versions: 4.4.0
>Reporter: Flavio Junqueira
>Priority: Critical
> Fix For: 4.4.0
>
>
> I've noticed this while inspecting the output in jenkins:
> {noformat}
> Running org.apache.bookkeeper.client.BookKeeperCloseTest
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.074 sec
> Running org.apache.bookkeeper.client.LocalBookKeeperTest
> Running org.apache.bookkeeper.meta.GcLedgersTest
> {noformat}
> It sounds like {{LocalBookKeeperTest}} is failing silently. Is it hanging and 
> timing out?
> https://builds.apache.org/job/bookkeeper-master-git-pullrequest/57/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: AuthZ and AuthN

[Flavio Junqueira]

Perhaps we can leverage the work in BOOKKEEPER-588 to implement authorization?

-Flavio

> On 07 Apr 2016, at 16:36, Venkateswara Rao Jujjuri  wrote:
> 
> Team,
> 
> Current BK uses clear text passwords. Do we have any plans of adding
> Authorization and encrypted passwords?
> 
> -- 
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi

[jira] [Updated] (BOOKKEEPER-916) Placement policy to accomodate different types of ledger storage

[Flavio Junqueira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flavio Junqueira updated BOOKKEEPER-916:

Summary: Placement policy to accomodate different types of ledger storage  
(was: Placement policy to accomodate different types of ledger strorages)

> Placement policy to accomodate different types of ledger storage
> 
>
> Key: BOOKKEEPER-916
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-916
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-client, bookkeeper-server
>Affects Versions: 4.3.2
>Reporter: Venkateswararao Jujjuri (JV)
>Assignee: Venkateswararao Jujjuri (JV)
>
> As we start to use bookkeeper as long term storage, it may not be right use 
> of resources to keep all copies of entry (write ensemble) on efficient 
> storage. This feature is to come up with an intelligent placement that 
> distributes entry copies across different classes of storage.
> Simply put, say we have SSD based ledger storage and HDD based ledger storage 
> on each system. Instead of putting all copies of entries either on SSD or on 
> HDD, this placement policy maintains one copy on SSD and others on HDD.
> - Have at least one copy on SSD and others on HDD.
>- Writer need to be aware of this classification
>- Replication logic need to be aware of this logic.
> - While reading attempt to read from SSD first.
>   - Reader also need to be aware of this logic.
> This will push bookkeeper  towards the long term storage, also can be a 
> stepping store towards introducing storage tiers in the future.
> This has dependency/relation to
>  https://issues.apache.org/jira/browse/BOOKKEEPER-912.
> https://issues.apache.org/jira/browse/BOOKKEEPER-915



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BOOKKEEPER-912) Allow EnsemblePlacementPolicy to choose bookies using ledger custom data (multitenancy support)

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229993#comment-15229993
 ] 

Flavio Junqueira commented on BOOKKEEPER-912:
-

I think one of the difficulties here is to keep bookie "labels" updated. First, 
you need to think how to you'd label bookies, If you're labeling according to 
tenant, then would you pre-select bookies and label them when the tenant is 
created in the system? 

Second, if you keep this information in ZK, then you'd need to fetch all 
bookies and create a small inverted index locally mapping labels to bookies. 
Ideally, we could have ZK filtering by label when requesting children. Perhaps 
a feature to think about for ZK if the idea sounds good.

Third, what happens when labels change? Do we tell all clients through 
notifications? There is also the possibility that labels of a bookie conflict. 
For the sake of example, let's say that a bookie has labels A and B, but 
whatever client that uses label A can 't use label B. There might be rules that 
need to be verified when assigning labels to bookies.

So those are some thoughts on the issue. Otherwise, I think the multi-tenancy 
direction is great. 

> Allow EnsemblePlacementPolicy to choose bookies using ledger custom data 
> (multitenancy support)
> ---
>
> Key: BOOKKEEPER-912
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-912
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-client
>Affects Versions: 4.4.0
>Reporter: Enrico Olivelli
> Fix For: 4.5.0
>
>
> I would like to restrict the set of bookies to be used for a specific ledger. 
> Actually a single EnsemblePlacementPolicy is used for all the ledgers.
> This is because I want to create a ledger only using a dedicated set of 
> machines/bookies which are dedicated to the 'tenant' for which I'm creating 
> the ledger.
> We can add an optional (byte[]) parameter to asyncCreateLedger which in turn 
> is to be passed to the configured EnsemblePlacementPolicy which in turn will 
> be able to decide which are the most suitable bookies for the tenant.
> This parameter must be stored on ledger metadata, in order to be used in 
> replaceBookie. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BOOKKEEPER-917) LocalBookKeeperTest seems to be silently failing

[Flavio Junqueira (JIRA)]

Flavio Junqueira created BOOKKEEPER-917:
---

 Summary: LocalBookKeeperTest seems to be silently failing
 Key: BOOKKEEPER-917
 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-917
 Project: Bookkeeper
  Issue Type: Bug
  Components: bookkeeper-server
Affects Versions: 4.3.2
Reporter: Flavio Junqueira
Priority: Critical
 Fix For: 4.4.0


I've noticed this while inspecting the output in jenkins:

{noformat}
Running org.apache.bookkeeper.client.BookKeeperCloseTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.074 sec
Running org.apache.bookkeeper.client.LocalBookKeeperTest
Running org.apache.bookkeeper.meta.GcLedgersTest
{noformat}

It sounds like {{LocalBookKeeperTest}} is failing silently. Is it hanging and 
timing out?

https://builds.apache.org/job/bookkeeper-master-git-pullrequest/57/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BOOKKEEPER-916) Placement policy to accomodate different types of ledger strorages

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229964#comment-15229964
 ] 

Flavio Junqueira commented on BOOKKEEPER-916:
-

I really like the discussion about storage tiers. I'd like us to have a deeper 
discussion about it before endeavoring into a feature implementation. Here are 
some thoughts.

Long term storage and storage tiers have different aspects and implications. 
For long term storage, we probably want to have say archival bookies and have 
regular bookies replicating data there when they go cold rather than have 
clients directly writing into the cold storage layer. I'm assuming that the 
writes into cold storage are not necessarily fast and consequently you don't 
want to have it in the critical path of addEntry requests. Related to this 
observation is how often ledger data goes cold and that one needs to store long 
term. In the applications I've seen, it doesn't happen often, ledgers are 
rather short-lived, but I'd like to hear opinions.

On storage tiers, I assume we may want to be able to offer different ways of 
storing the data to client applications based on performance constraints, 
reliability, isolation. I think that making a distinction between SSDs and HDDs 
is a good one, and one tier we might want to care about because it might speed 
up things quite a bit for us is non-volatile RAM. 

> Placement policy to accomodate different types of ledger strorages
> --
>
> Key: BOOKKEEPER-916
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-916
> Project: Bookkeeper
>  Issue Type: New Feature
>  Components: bookkeeper-client, bookkeeper-server
>Affects Versions: 4.3.2
>Reporter: Venkateswararao Jujjuri (JV)
>Assignee: Venkateswararao Jujjuri (JV)
>
> As we start to use bookkeeper as long term storage, it may not be right use 
> of resources to keep all copies of entry (write ensemble) on efficient 
> storage. This feature is to come up with an intelligent placement that 
> distributes entry copies across different classes of storage.
> Simply put, say we have SSD based ledger storage and HDD based ledger storage 
> on each system. Instead of putting all copies of entries either on SSD or on 
> HDD, this placement policy maintains one copy on SSD and others on HDD.
> - Have at least one copy on SSD and others on HDD.
>- Writer need to be aware of this classification
>- Replication logic need to be aware of this logic.
> - While reading attempt to read from SSD first.
>   - Reader also need to be aware of this logic.
> This will push bookkeeper  towards the long term storage, also can be a 
> stepping store towards introducing storage tiers in the future.
> This has dependency/relation to
>  https://issues.apache.org/jira/browse/BOOKKEEPER-912.
> https://issues.apache.org/jira/browse/BOOKKEEPER-915



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BOOKKEEPER-880) Make LedgerHandle implement AutoCloseable

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229918#comment-15229918
 ] 

Flavio Junqueira commented on BOOKKEEPER-880:
-

Fine with me, I've changed the fix version.

> Make LedgerHandle implement AutoCloseable
> -
>
> Key: BOOKKEEPER-880
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-880
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-client
>Affects Versions: 4.3.1
>Reporter: Enrico Olivelli
>Priority: Trivial
> Fix For: 4.4.0
>
>
> It would be useful for simple clients to have LedgerHandle implement 
> AutoCloseable.maybe Bookkeeper class too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BOOKKEEPER-880) Make LedgerHandle implement AutoCloseable

[Flavio Junqueira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BOOKKEEPER-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flavio Junqueira updated BOOKKEEPER-880:

Fix Version/s: 4.4.0

> Make LedgerHandle implement AutoCloseable
> -
>
> Key: BOOKKEEPER-880
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-880
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-client
>Affects Versions: 4.3.1
>Reporter: Enrico Olivelli
>Priority: Trivial
> Fix For: 4.4.0
>
>
> It would be useful for simple clients to have LedgerHandle implement 
> AutoCloseable.maybe Bookkeeper class too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Precommit test failures

[Flavio Junqueira]

I've noticed that the following tests are failing consistently in Jenkins

Test Result (7 failures / +7)
org.apache.bookkeeper.replication.TestReplicationWorker.testRWZKSessionLost[0]
org.apache.bookkeeper.replication.TestReplicationWorker.testRWShutdownOnLocalBookieReadonlyTransition[0]
org.apache.bookkeeper.replication.TestReplicationWorker.testRWZKSessionLost[1]
org.apache.bookkeeper.replication.TestReplicationWorker.testRWShutdownOnLocalBookieReadonlyTransition[1]
org.apache.bookkeeper.replication.TestReplicationWorker.testRWZKSessionLost[2]
org.apache.bookkeeper.replication.TestReplicationWorker.testRWShutdownOnLocalBookieReadonlyTransition[2]
org.apache.bookkeeper.test.ReadOnlyBookieTest.testBookieShouldTurnWritableFromReadOnly


Is there a jira for this and is there anyone looking into it? We need to fix 
those.

-Flavio

[jira] [Commented] (BOOKKEEPER-870) Change the default value for bookie settings.

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222907#comment-15222907
 ] 

Flavio Junqueira commented on BOOKKEEPER-870:
-

[~si...@apache.org] do you want to create a pull request for this so that I 
merge it?

> Change the default value for bookie settings.
> -
>
> Key: BOOKKEEPER-870
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-870
> Project: Bookkeeper
>  Issue Type: Documentation
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.4.0
>
> Attachments: BOOKKEEPER-870.patch
>
>
> It would be good to tune the default settings for bookies.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: BookKeeper 4.4.0

[Flavio Junqueira]

You could review code, for example, any of the patch available issues here:

https://issues.apache.org/jira/browse/BOOKKEEPER-895?jql=project%20%3D%20BOOKKEEPER%20AND%20fixVersion%20%3D%204.4.0%20AND%20status%20%3D%20%22Patch%20Available%22%20ORDER%20BY%20priority%20DESC
 
<https://issues.apache.org/jira/browse/BOOKKEEPER-895?jql=project%20=%20BOOKKEEPER%20AND%20fixVersion%20=%204.4.0%20AND%20status%20=%20%22Patch%20Available%22%20ORDER%20BY%20priority%20DESC>

-Flavio

> On 31 Mar 2016, at 09:26, Enrico Olivelli - Diennea 
>  wrote:
> 
> Sure,
> how can I help concretely ?
> 
> I'm waiting for the version to be RC to try it on our dev/qa environments at 
> Diennea, but as far as I know there are a lot of merges to be done on 4.4.0 
> and any test won't be so useful
> 
> - Enrico
> 
> Il giorno gio, 31/03/2016 alle 09.15 +0100, Flavio Junqueira ha scritto:
>> The status according to jira is the following:
>> 
>> - No blocker
>> - 5 patch available
>> - 17 open issues
>> 
>> https://issues.apache.org/jira/browse/BOOKKEEPER/fixforversion/12325566/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-issues-panel
>>  
>> <https://issues.apache.org/jira/browse/BOOKKEEPER/fixforversion/12325566/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-issues-panel>
>> 
>> Do you want to help with reviews, Enrico?
>> 
>> -Flavio
>> 
>>> On 31 Mar 2016, at 09:12, Enrico Olivelli - Diennea 
>>> mailto:enrico.olive...@diennea.com>> wrote:
>>> 
>>> Hi all,
>>> Is there any news for 4.4.0 release ?
>>> 
>>> https://issues.apache.org/jira/browse/BOOKKEEPER/fixforversion/12325566/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>  
>>> <https://issues.apache.org/jira/browse/BOOKKEEPER/fixforversion/12325566/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel>
>>> 
>>> 
>>> 
>>> --
>>> Enrico Olivelli
>>> Software Development Manager @Diennea
>>> Tel.: (+39) 0546 066100 - Int. 925
>>> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>>> 
>>> MagNews - E-mail Marketing Solutions
>>> http://www.magnews.it
>>> Diennea - Digital Marketing Solutions
>>> http://www.diennea.com
>>> 
>>> 
>>> 
>>> 
>>> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed 
>>> email marketing! http://www.magnews.it/newsletter/
>>> 
>>> The information in this email is confidential and may be legally 
>>> privileged. If you are not the intended recipient please notify the sender 
>>> immediately and destroy this email. Any unauthorized, direct or indirect, 
>>> disclosure, copying, storage, distribution or other use is strictly 
>>> forbidden.
>> 
>  -- 
> Enrico Olivelli 
> Software Development Manager @Diennea 
> Tel.: (+39) 0546 066100 - Int. 925 
> Viale G.Marconi 30/14 - 48018 Faenza (RA) 
> 
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it
> Diennea - Digital Marketing Solutions
> http://www.diennea.com
> 
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email 
> marketing! http://www.magnews.it/newsletter/
> 
> The information in this email is confidential and may be legally privileged. 
> If you are not the intended recipient please notify the sender immediately 
> and destroy this email. Any unauthorized, direct or indirect, disclosure, 
> copying, storage, distribution or other use is strictly forbidden.

Re: BookKeeper 4.4.0

[Flavio Junqueira]

The status according to jira is the following:

- No blocker
- 5 patch available
- 17 open issues

https://issues.apache.org/jira/browse/BOOKKEEPER/fixforversion/12325566/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-issues-panel
 


Do you want to help with reviews, Enrico?

-Flavio

> On 31 Mar 2016, at 09:12, Enrico Olivelli - Diennea 
>  wrote:
> 
> Hi all,
> Is there any news for 4.4.0 release ?
> 
> https://issues.apache.org/jira/browse/BOOKKEEPER/fixforversion/12325566/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> 
> 
> 
> --
> Enrico Olivelli
> Software Development Manager @Diennea
> Tel.: (+39) 0546 066100 - Int. 925
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
> 
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it
> Diennea - Digital Marketing Solutions
> http://www.diennea.com
> 
> 
> 
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email 
> marketing! http://www.magnews.it/newsletter/
> 
> The information in this email is confidential and may be legally privileged. 
> If you are not the intended recipient please notify the sender immediately 
> and destroy this email. Any unauthorized, direct or indirect, disclosure, 
> copying, storage, distribution or other use is strictly forbidden.