Pulsar Summit San Francisco 2022 CFP is open!

Hi, all the Pulsar community members, Pulsar Summit San Francisco 2022 CFP is open now! It will be our *first-ever in-person Pulsar Summit *in San Francisco! We're glad to invite you to share your experience and add up to the wonderfulness. As a speaker, you will receive the chance to

[ANNOUNCE] New Committer: Yong Zhang

Zhang! Sijie Guo on behalf of Apache BookKeeper PMC

[ANNOUNCE] Introducing Apache Pulsar Hackathon 2021!!!

Hi, Pulsar and BookKeeper communities, The adoption of Apache Pulsar and BookKeeper is accelerating as organizations around the world pursue cloud-native technologies. To build on this momentum, we are excited to announce the first-ever Apache Pulsar Hackathon 2021, *taking place on May 6-7th*.

[ANNOUNCE] Pulsar Summit North America 2021 Call-for-speakers and Registration is now open

Dear Apache Pulsar community, We are excited to announce that Pulsar Summit North America 2021 will be hosted virtually on June 16th - 17th. Call-for-speakers and attendee registration are now open for the conference! Pulsar Summit is the conference dedicated to Apache Pulsar, and the messaging

Announcing Pulsar Summit Virtual Conference 2020 | June 17-18

Hi all, Over the past few weeks, after having to reschedule Pulsar Summit San Francisco 2020, we have been working on delivering the event in an online format instead. Today, we are excited to announce the Pulsar Summit Virtual Conference, happening on June 17-18! Join the speakers from Splunk,

[ANNOUNCE] Pulsar Summit San Francisco 2020 Call for Presentation extended!

can find more detailed track descriptions and the form to submit a proposal on the Pulsar Summit website at --> https://pulsar-summit.org/call-for-presentations/ --> https://sessionize.com/pulsar-summit-san-francisco-2020/ As usual, accepted speakers get a free conference pass. Best regards,

[ANNOUNCE] Apache Pulsar 2.5.0 released

The Apache Pulsar team is proud to announce Apache Pulsar version 2.5.0. Pulsar is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for

Re: Bookeeper exception on pods restart

* bookie > shutdown…. > > In which case, what is the way to ensure a graceful termination of > bookies, so we don’t lose/corrupt any data? > > > > -Thanks, > > Prajakta > > > > *From:* Sijie Guo > *Sent:* Tuesday, December 3, 2019 1:53 PM > *To:* Sharda, Ra

Re: Bookeeper exception on pods restart

we should run this on a running environment to recover from > the failure? > > “bin/bookkeeper shell ledger -ledgeridformat long -m [ledger-id]" > ---------- > *From:* Sijie Guo > *Sent:* Tuesday, December 3, 2019 1:10 PM > *To:* Sharda, Ravi > *Cc:* Enrico Oliv

Re: Bookeeper exception on pods restart

-- > *From:* Enrico Olivelli > *Sent:* Monday, December 2, 2019 7:01 PM > *To:* user > *Cc:* Sijie Guo ; Flavio Junqueira > *Subject:* Re: Bookeeper exception on pods restart > > > [EXTERNAL EMAIL] > > > sh-4.2# ./bookkeeper shell ledger -m

Re: Bookeeper exception on pods restart

rStorage.getEntry(SortedLedgerStorage.java:153) > > at org.apache.bookkeeper.bookie.L > --- > sh-4.2# ./bookkeeper shell readledger --ledgerid 56 > ERROR: invalid value for option ledgerid : 56 > Must specify a ledger id > > --- > I didn't know how to check that the log fi

Re: Invitation to Apache Pulsar Meetup | Shanghai on 11/16

Enrico, We will be recording the event and also post the slides. We will also post the user stories after the presentations. Thanks, Sijie On Wed, Oct 30, 2019 at 9:23 PM Enrico Olivelli wrote: > Thank you Jennifer > > Are you going to record the event and share it? > I am from Europe and so

Fwd: 4 Apache Events in 2019: DC Roadshow soon; next up Chicago, Las Vegas, and Berlin!

FYI. The CFP for ApacheCon North America is now open. -- Forwarded message - From: Rich Bowen Date: Wed, Mar 6, 2019 at 10:00 PM Subject: 4 Apache Events in 2019: DC Roadshow soon; next up Chicago, Las Vegas, and Berlin! To: Dear Apache Enthusiast, (You’re receiving this

[ANNOUNCE] Apache BookKeeper 4.9.0 released

The Apache BookKeeper team is proud to announce Apache BookKeeper version 4.9.0. Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads. It has been used for a fundamental service to build reliable services. It is also the log segment

[ANNOUNCE] Apache BookKeeper 4.7.3 released

The Apache BookKeeper team is proud to announce Apache BookKeeper version 4.7.3. Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads. It has been used for a fundamental service to build reliable services. It is also the log segment

Re: Community meeting tomorrow

On Wed, Nov 14, 2018 at 10:26 AM Enrico Olivelli wrote: > Hi BookKeepers ! > tomorrow we will 'meet' on hangout for the usual Community Meeting. > It is the place to discuss current projects around BookKeeper. > > https://bookkeeper.apache.org/community/meeting/ > > We are going to release

Re: Missing Apache BookKeeper 4.8.0 artifacts from Maven Central

That was an email from Oct 4 On Fri, Nov 9, 2018 at 5:01 AM Ivan Kelly wrote: > Wow. Suprised we're only hearing about it now > > -Ivan > > On Thu, Oct 4, 2018 at 1:14 PM, Enrico Olivelli > wrote: > > Now it is okay > > https://search.maven.org/search?q=g:org.apache.bookkeeper > > > > Cheers >

Re: Migrating from zkServers to metadataServiceURI

On Tue, Oct 9, 2018 at 1:53 PM Enrico Olivelli wrote: > > > Il mar 9 ott 2018, 22:45 Sijie Guo ha scritto: > >> >> >> On Mon, Oct 8, 2018 at 1:51 AM Enrico Olivelli >> wrote: >> >>> Il giorno lun 8 ott 2018 alle ore 09:59 Sijie Guo >>

Re: Migrating from zkServers to metadataServiceURI

On Sun, Oct 7, 2018 at 11:51 PM Enrico Olivelli wrote: > Hi, > I am going to migrate a bunch of applications from the deprecated > zkServers to metadataServiceURI. > I think that the change is not really straightforward. > why? the metadataServiceURI should be "zk://". For example, if your

Re: [ANNOUNCE] Apache BookKeeper 4.8.0 released

Congrats! Well done, Enrico. - Sijie On Fri, Sep 28, 2018 at 4:27 AM Enrico Olivelli wrote: > The Apache BookKeeper team is proud to announce Apache BookKeeper version > 4.8.0. > > Apache BookKeeper is a scalable, fault-tolerant, and low-latency > storage service optimized for > real-time

[ANNOUNCE] Apache BookKeeper 4.7.2 released

The Apache BookKeeper team is proud to announce Apache BookKeeper version 4.7.2. Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads. It has been used for a fundamental service to build reliable services. It is also the log segment

Re: Is every message has status for acknowledge in ack way of individually

+ us...@pulsar.incubator.apache.org A short answer is: yes, every message has the status for acknowledgement. pulsar uses a bookkeeper ledger for storing the changes for message acknowledgements. for cumulative acknowledgment, pulsar stores the updates for the acked message id (you can think it

Re: Ack mode of individual pulsar

I can't find source code in package org.apache.bookkeeper.mledger.proto > > -- > li.peng...@zhaopin.com.cn > > > *From:* Sijie Guo > *Date:* 2018-07-12 00:40 > *To:* user ; us...@pulsar.incubator.apache.org > *Subject:* Re: Ack mode of individual pulsar >

Re: Ack mode of individual pulsar

+users@pulsar In the context of pulsar, when a consumer acks a message, it doesn’t remove entries from bookkeeper. Instead, for each subscription at pulsar, pulsar brokers use a separate ledger for storing the acknowledgments for each subscription. When a broker receives individual acks, it

Re: Problem of Prometheus Stats Providers

ot "Resource not found"。 > > My config file is upload. > > -- > li.peng...@zhaopin.com.cn > > > *From:* Sijie Guo > *Date:* 2018-07-09 15:30 > *To:* user > *Subject:* Re: Re: Problem of Prometheus Stats Providers > Okay. So

Re: Re: Problem of Prometheus Stats Providers

> > # The http server port to listen on. Default value is 8080. > httpServerPort=8080 > > # The http server class > httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer > > > > ------ > li.peng...@zhaopin.com.cn > > > *From:

Re: Problem of Prometheus Stats Providers

Interesting, I have tried these settings at my laptop. I am able to curl the metrics : "curl -s localhost:8000/metrics". Can you clarify a few things: - what version of bookkeeper are you using? - did you happen to enable bookie http server? set httpServerEnabled to be true - Sijie On Sun, Jul

Re: latency of bookkeeper

er write latency. try use the settings I pointed out above. > > > 原始邮件 > *发件人:* li.penghui > *收件人:* user > *发送时间:* 2018年7月6日(周五) 17:56 > *主题:* Re: latency of bookkeeper > > Can I close the WAL if i can tolerate any enries lost. if WAL can be > closed, i can use bookk

Re: latency of bookkeeper

On Fri, Jul 6, 2018 at 2:58 AM li.penghui wrote: > But i should return the entry append success or fail for every entry. > If you can design your application leveraging asynchronous calls, you can return entry append success or fail without being blocked at waiting. > > Can I close the WAL if

Re: latency of bookkeeper

I think your question is a bit not clear, latency and throughput are two kind of different metrics. Your question seems to be asking for high throughput. Anyway, I will try to explain the performance tradeoff between latency and throughput and hope that helps. Bookkeeper by default fsync the

Re: Expected workload for DBLedgerStorage ?

On Wed, Jun 20, 2018 at 12:45 PM Enrico Olivelli wrote: > > > Il mer 20 giu 2018, 16:52 Sijie Guo ha scritto: > >> On Wed, Jun 20, 2018 at 4:09 AM Enrico Olivelli >> wrote: >> >>> Hi, >>> I am experimenting DBLedgerStorage with some project

[ANNOUNCE] Apache BookKeeper 4.7.1 released

The Apache BookKeeper team is proud to announce Apache BookKeeper version 4.7.1. Apache BookKeeper is a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads. It has been used for a fundamental service to build reliable services. It is also the log segment

Re: Help with bad errors on 4.6.1

Sent out a PR for the issues that I observed: https://github.com/apache/bookkeeper/pull/1240 On Thu, Mar 8, 2018 at 10:47 PM, Sijie Guo <guosi...@gmail.com> wrote: > So the problem here is: > > - a corrupted request failed the V3 request decoder, so bookie switched to > use

Re: Help with bad errors on 4.6.1

So the problem here is: - a corrupted request failed the V3 request decoder, so bookie switched to use v2 request decoder. Once the switch happen, the bookie will always use v2 request decoder decoding v3 request. then all your v3 requests will be failing with unknown op and trigger the bytebuf

Re: Help with bad errors on 4.6.1

<eolive...@gmail.com> wrote: > > > Il gio 8 mar 2018, 20:52 Sijie Guo <guosi...@gmail.com> ha scritto: > >> On Thu, Mar 8, 2018 at 7:42 AM, Enrico Olivelli <eolive...@gmail.com> >> wrote: >> >>> >>> >>> 2018-03-08 14:

Re: Help with bad errors on 4.6.1

On Thu, Mar 8, 2018 at 7:42 AM, Enrico Olivelli wrote: > > > 2018-03-08 14:50 GMT+01:00 Ivan Kelly : > >> It just occurred to me that this could be a problem with the recycler. >> If we recycle a buffer too early, but then keep using it, another user >>

[ANNOUNCE] Enrico Olivelli and Jia Zhai join the Apache BookKeeper PMC

On behalf of the Apache BookKeeper PMC I am pleased to announce that Enrico Olivelli and Jia Zhai have accepted our invitation to become a PMC member on the Apache BookKeeper project. Please join me in thanking Enrico and Jia for their contributions to date and anticipation of many more

[ANNOUNCE] New BookKeeper Committer: Sam Just

The Apache BookKeeper PMC recently extended committer karma to Sam Just and he has accepted. Sam Just has been contributing bookkeeper changes to the community for a while. He is the main person who is leading the efforts of pushing salesforce changes to open source. He has been active in code

Re: [ANNOUNCE] Apache BookKeeper 4.6.0 released

Jia, congrats for you first release! Thanks everyone for making contributions to 4.6.0 release. - Sijie On Mon, Dec 25, 2017 at 11:23 PM, Jia Zhai wrote: > The Apache BookKeeper team is proud to announce Apache BookKeeper version > 4.6.0. > > Apache BookKeeper is a

[ANNOUNCE] New BookKeeper Committer: Yiming Zang

The Apache BookKeeper PMC recently extended committer karma to Yiming Zang and he has accepted. Yiming Zang has been contributing bookkeeper & distributedlog changes to the community for a while. He is active in code contributions, discussion and code reviews. We are looking forward to more

[ANNOUNCE] New BookKeeper Committer: Yiming Zang

The Apache BookKeeper PMC recently extended committer karma to Yiming Zang and he has accepted. Yiming Zang has been contributing bookkeeper & distributedlog changes to the community for a while. He is active in code contributions, discussion and code reviews. We are looking forward to more

Re: HTTP / REST API?

tioned, > write ledger seems need more thinking. > > Thanks. > -Jia > > On Wed, Nov 8, 2017 at 3:18 AM, Sijie Guo <guosi...@gmail.com> wrote: > >> + Jia >> >> I think when Jia did BP-17 initially, he actually had the apis for >> ledgers for writin

Re: log compaction of entries

On Tue, Nov 7, 2017 at 12:18 PM, Istvan Soos <istvan.s...@gmail.com> wrote: > On Tue, Nov 7, 2017 at 8:09 PM, Sijie Guo <guosi...@gmail.com> wrote: > > But I would to learn more about your use case and to see how we can > support > > you. > > It is a nice featu

Re: log compaction of entries

On Tue, Nov 7, 2017 at 10:47 AM, Istvan Soos wrote: > On the website [0] I gather that data compaction is mostly about > cleaning up after we delete a ledger. Is there a feature or plan to > implement entry-level compaction, e.g. to have an ID that uniquely > identifies an

Re: HTTP / REST API?

Hi Istvan, This feature is coming in the next `4.6.0` release (it should be available in 2 weeks or so). We have an issue to document this feature. At the same time, you can checkout this BP for details -

Re: ConsoleProxyWriter exception

edback. > > Thanks > > --Filippo > > 2017-09-11 18:43 GMT+02:00 Sijie Guo <guosi...@gmail.com>: > > Hi Filippo, > > > > I think the problem was introduced by recent changes on upgrading > bookkeeper > > to 4.5.0 and using ByteBuf for better memory manage

Re: Publish SNAPSHOT jar

Enrico, Thank you for pointing it out. - Sijie On Tue, Aug 22, 2017 at 11:35 PM, Enrico Olivelli <eolive...@gmail.com> wrote: > On mer 23 ago 2017, 03:56 Jia Zhai <zhaiji...@gmail.com> wrote: > > >  > > > > On Wed, Aug 23, 2017 at 9:02 AM, Sijie Guo <

Re: BookKeeper docker image is avaliable under apache now

in the ensemble" >> >> >> >> On Fri, Aug 11, 2017 at 6:04 AM, Francesco Caliumi - Diennea < >> francesco.cali...@diennea.com> wrote: >> >> > Good, happy to finally see it in docker hub! Thank you all! >> > >> > On Fri, 201

Re: [ANNOUNCE] Apache BookKeeper 4.5.0 released

Awesome! Well done! This is a great milestone to the bookkeeper community! Thanks JV for doing the release! Thanks everyone for making this happen. Sijie On Aug 11, 2017 9:56 AM, "JV Jujjuri" wrote: > The Apache BookKeeper team is proud to announce Apache BookKeeper

Re: InterleavedLedgerStorage vs SortedLedgerStorage

cache for caching, better I/O isolation. But if you only have a few ledgers or you rarely read from the ledgers, the memtables in Sorted might bring in overhead for you. - Sijie > Enrico > > Il mer 19 lug 2017, 11:12 Sijie Guo <guosi...@gmail.com> ha scritto: > >> Curren

Re: InterleavedLedgerStorage vs SortedLedgerStorage

Currently we don't have such documentation about these two ledger storages. We are working on improving the documentation as part of BP-11. - Sijie On Wed, Jul 19, 2017 at 3:34 PM, Enrico Olivelli wrote: > Hi, > is there any doc about the differences and usecases for >

Re: BookKeeper#openLedgerNoRecovery hangs

On Wed, Jul 19, 2017 at 4:04 PM, Enrico Olivelli wrote: > Hi, > in some internal benchmarks we are experiencing openLedgerNoRecovery calls > which remain hung. > I see that basically that function calls ZookKeeper#getData. > > Does anyone have an idea of how it can happen ?

Re: BookKeeper and CheckStyle

> bookkeeper-server) >>> I started to do the second part (bookkeeper-server), but It is a huge >>> change. It contains almost 5000 issues. >>> I'm thinking about how to slice it up to small steps. >>> >> >> Thank you David, >> Yep, the idea i

[ANNOUNCE] JV Jujjuri joins the Apache BookKeeper PMC

On behalf of the Apache BookKeeper PMC I am pleased to announce that JV Jujjuri has accepted our invitation to become a PMC member on the Apache BookKeeper project. JV has been an active contributor in many areas, including driving community meetings, reviewing patches, helping with triaging bugs

Re: BookKeeper and CheckStyle

e second part (bookkeeper-server), but It is a huge >>>> change. It contains almost 5000 issues. >>>> I'm thinking about how to slice it up to small steps. >>>> >>> >>> Thank you David, >>> Yep, the idea is to create a single patch per packa

[ANNOUNCE] New BookKeeper Committer: Enrico Olivelli & Charan Reddy

The Apache BookKeeper PMC recently extended committer karma to Enrico Olivelli & Charan Reddy and they have accepted. Charan Reddy and Enrico Olivelli have been contributing bookkeeper changes to the community for a while. They are active in code contributions, discussion and code reviews. We are

High Performance Logging with Apache BookKeeper

FYI. https://www.linux.com/news/event/apachecon/2017/4/high-performance-logging-apache-bookkeeper This is the follow article for Apache BookKeeper talk at #vault. - Sijie

Re: New BookKeeper Proposal "Allow Reads outside the LAC Protocol"

gt; So, the tool keeps reading the leader, past the LAC until it reaches a > > NoEntry error. > > > > Thank you Matteo > That's exacly what I meant. But I would like to port it to the standard > client (non-admin) > I will refer to this work in the BP > > > >

Re: New BookKeeper Proposal "Allow Reads outside the LAC Protocol"

) or use readUncommittedEntries (to read potentially *uncommitted* entries. So the correctness is managed by application itself (out side of the logic). - Sijie On Thu, Mar 30, 2017 at 8:55 AM, Sijie Guo <guosi...@gmail.com> wrote: > + dev@ and moved user@ to cc > > On Thu, Mar 30,

Re: New BookKeeper Proposal "Allow Reads outside the LAC Protocol"

+ dev@ and moved user@ to cc On Thu, Mar 30, 2017 at 8:26 AM, Enrico Olivelli wrote: > I would like to share with the community this BookKeeper proposal > > https://cwiki.apache.org/confluence/display/BOOKKEEPER/ > BP-5+Allow+reads+outside+the+LAC+Protocol > > I will submit

Apache BookKeeper Talk at #lfvault

FYI. This is the slides of the bookkeeper talk JV and me gave at #lfvault. You can check it out if you are interested in. https://vault2017.sched.com/event/A4wM/apache-bookkeeper-a-high-performance-and-low-latency-cloud-storage-service-sijie-guo-twitter http://www.slideshare.net/hustlmsp/apache

Re: Write guarantees and the LAC Protocol

On Wed, Mar 15, 2017 at 9:17 AM, Enrico Olivelli wrote: > Hi BookKeepers, > I'm trying to understand some basic aspect of BookKeeper > > I have a strange case, and I would like to have some opinion from any > "expert". > > I am using BookKeeper as a transaction log for a

Re: Bookkeeper Recovery Issues

of these logs while >> some data movement seems to be occurring, but it's really weird that it's >> constantly trying to connect to the failed node. >> Couldn't it realize it's down because it's not shown as available on >> zookeeper? >> >> We also see a couple of

Re: BookKeeper as low-latency distributed data storage

In general, reading an entry from bookie is pretty fast as most of the index pages will be cached in memory, so it typically only needs one disk access. You might see evaluated latency when index pages begin swapping in and out. So ideally I would suggest using small index page size and large

Re: [ANNOUNCE] Apache BookKeeper 4.4.0 released

Great job! Matteo and the community! Sijie On Monday, May 16, 2016, Matteo Merli wrote: > The Apache BookKeeper team is proud to announce Apache BookKeeper version > 4.4.0. > > This is the first feature release of Apache BookKeeper as an Apache Top > Level > Project. > >

Re: hint about ledger removal

for my understanding, this seems to be a coordination case, you are coordinating the bookkeeper clients to delete ledgers after meeting a certain condition. zookeeper is a good candidate to do such coordination than bookkeeper. as bookkeeper is more focus on replication and consistency on

Re: R: Cannot compile 4.3.2 tag

> -- Enrico > > > *Da:* Matteo Merli [mailto:mme...@yahoo-inc.com] > *Inviato:* martedì 17 novembre 2015 23:41 > *A:* user@bookkeeper.apache.org; Sijie Guo <si...@apache.org> > *Cc:* Enrico Olivelli - Diennea <enrico.olive...@diennea.com> > *Oggetto:* Re: Can

Re: Cannot compile 4.3.2 tag

e I was testing. > > Sorry about that, fixing and releasing rc1 > > Matteo > > > Sent from Yahoo Mail for iPhone <https://yho.com/footer0> > > > > > On Tuesday, November 17, 2015, 11:04 AM, Sijie Guo <si...@apache.org> > wrote: > > T

Re: DistributedLog, a high-performance replicated log service built over BookKeeper

://www.periscope.tv/w/1YpKkXOadpXGj (second part) - Sijie On Wed, Sep 16, 2015 at 9:33 AM, Sijie Guo <si...@apache.org> wrote: > Thanks. Yup, we will record it, and also try periscope it. > > - Sijie > > On Wed, Sep 16, 2015 at 1:26 AM, Ivan Kelly <iv...@apache.org> wrote: > &

Re: Apache Bookkeeper 4.0 release date

> > > > commons-beanutils-core > > commons-beanutils > > > > > > commons-cli > > commons-cli > > > > > > jna > > net.java.dev.jna > > >

PoweredBy page

Hello bookkeepers, I just added Twitter's use cases in the PoweredBy page. If you are using BookKeeper in your system, please add your use case in the PoweredBy page too :) - Sijie

Re: Apache Bookkeeper 4.0 release date

I think a bug fix release is much safer for you rather than cutting 4.4.0, since there is much more newer features in 4.4.0. On Wed, Sep 9, 2015 at 10:59 AM, Sijie Guo <si...@apache.org> wrote: > Hello Enrico: > > We could port this change to 4.3 branch and cut a bug fix release

Re: Apache Bookkeeper 4.0 release date

Hello Enrico: We could port this change to 4.3 branch and cut a bug fix release 4.3.1. Are u running 4.3.x or 4.2.x branch? - Sijie On Wed, Sep 9, 2015 at 4:02 AM, Enrico Olivelli - Diennea < enrico.olive...@diennea.com> wrote: > (sorry for cross-posting) > > > > Due to that bug in our

Messaging User Group Meetup on 9/21

Hello bookkeepers, We are organizing a cross-community meetup around messaging technologies on 9/21. Come and join us if you happen to be in Bay Area. https://www.eventbrite.com/e/messaging-user-group-meetup-tickets-18190307700 - Sijie

Re: Messaging User Group Meetup on 9/21

We just fixed the registration form now. Come and join us on 9/21. - Sijie On Tue, Aug 25, 2015 at 10:46 AM, Sijie Guo si...@apache.org wrote: Hello bookkeepers, We are organizing a cross-community meetup around messaging technologies on 9/21. Come and join us if you happen to be in Bay