[jira] [Resolved] (BOOKKEEPER-1104) BookieInitializationTest.testWithDiskFullAndAbilityToCreateNewIndexFile testcase is unreliable
[
https://issues.apache.org/jira/browse/BOOKKEEPER-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jia Zhai resolved BOOKKEEPER-1104.
--
Resolution: Fixed
Issue resolved by merging pull request 310
[https://github.com/apache/bookkeeper/pull/310]
{noformat}
commit 0ccbc0e9c3e6a82d5e44b8347cc6567e043f253f
Author: Samuel Just
AuthorDate: Fri Jul 28 12:39:03 2017 +0800
Commit: jiazhai
CommitDate: Fri Jul 28 12:39:03 2017 +0800
BOOKKEEPER-1104: Fix testWithDiskFullAndAbilityToCreateNewIndexFile
The usage threshhold was chosen to be very close to the actual disk
usage at test time. This made the test unnecessarily fragile in the
case that there other things concurrently using the disk. Since we
aren't really testing that here, simply set the threshhold to be really
low.
Signed-off-by: Samuel Just
Author: Samuel Just
Reviewers: Jia Zhai , Matteo Merli
This closes #310 from athanatos/forupstream/BOOKKEEPER-1104
{noformat}
> BookieInitializationTest.testWithDiskFullAndAbilityToCreateNewIndexFile
> testcase is unreliable
> --
>
> Key: BOOKKEEPER-1104
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1104
> Project: Bookkeeper
> Issue Type: Bug
>Reporter: Samuel Just
>Assignee: Samuel Just
>Priority: Minor
> Fix For: 4.5.0
>
>
> The bug is that the test sets the full threshhold really close the actual
> usage (though just below), and if the disk is otherwise in use, the usage can
> fall below that threshhold during the test. Instead, just set it to .001
> since that's not what we're testing here anyway.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (BOOKKEEPER-1101) BookKeeper website menus not working under https
[ https://issues.apache.org/jira/browse/BOOKKEEPER-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-1101: - Attachment: BOOKKEEPER-1101.patch change http://code.jquery.com/jquery.js";> into > BookKeeper website menus not working under https > > > Key: BOOKKEEPER-1101 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1101 > Project: Bookkeeper > Issue Type: Bug > Components: Documentation >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: BOOKKEEPER-1101.patch > > > the 'Documentation', 'Get Involved' and 'Project Info' drop down menus > on https://bookkeeper.apache.org site are not working. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (BOOKKEEPER-1101) BookKeeper website menus not working under https
Jia Zhai created BOOKKEEPER-1101: Summary: BookKeeper website menus not working under https Key: BOOKKEEPER-1101 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1101 Project: Bookkeeper Issue Type: Bug Components: Documentation Reporter: Jia Zhai Assignee: Jia Zhai the 'Documentation', 'Get Involved' and 'Project Info' drop down menus on https://bookkeeper.apache.org site are not working. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (BOOKKEEPER-772) Reorder read sequnce
[ https://issues.apache.org/jira/browse/BOOKKEEPER-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai resolved BOOKKEEPER-772. - Resolution: Fixed https://github.com/apache/bookkeeper/pull/224 merged it. > Reorder read sequnce > - > > Key: BOOKKEEPER-772 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-772 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-client >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.5.0 > > > We should reorder the read sequence base on location, bookie availability, > for latency consideration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030442#comment-16030442 ] Jia Zhai commented on BOOKKEEPER-974: - Hi [~Caio], FYI. Some of the use case is running bookkeeper docker image on DCOS or K8S, And normally, bookkeeper should be a stateful service, which at least need the data of bookies be kept. In DCOS or K8S, there is feature "Persistent Volume" provided to achieve stateful, normally, if one bookie got down, DCOS/K8S will *automatically* start a new bookie docker instance, and try it best to run it on the same node, and mounted it with the "persistent volume" that it used at last run. > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (BOOKKEEPER-1081) Would like to provide a docker file for bookkeeper
[ https://issues.apache.org/jira/browse/BOOKKEEPER-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030382#comment-16030382 ] Jia Zhai edited comment on BOOKKEEPER-1081 at 5/30/17 11:50 PM: Hi [~Caio], Thanks. That is great. We could do it through BOOKKEEPER-974, since that contains more info. was (Author: zhaijia): Hi [~caiok], Thanks. That is great. We could do it through BOOKKEEPER-974, since that contains more info. > Would like to provide a docker file for bookkeeper > -- > > Key: BOOKKEEPER-1081 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1081 > Project: Bookkeeper > Issue Type: Wish > Components: build >Reporter: Jia Zhai >Assignee: Jia Zhai > > Would like to provide a docker file for bookkeeper -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (BOOKKEEPER-1081) Would like to provide a docker file for bookkeeper
[ https://issues.apache.org/jira/browse/BOOKKEEPER-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030382#comment-16030382 ] Jia Zhai edited comment on BOOKKEEPER-1081 at 5/30/17 11:50 PM: Hi [~caiok], Thanks. That is great. We could do it through BOOKKEEPER-974, since that contains more info. was (Author: zhaijia): Hi [~Caiok], Thanks. That is great. We could do it through BOOKKEEPER-974, since that contains more info. > Would like to provide a docker file for bookkeeper > -- > > Key: BOOKKEEPER-1081 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1081 > Project: Bookkeeper > Issue Type: Wish > Components: build >Reporter: Jia Zhai >Assignee: Jia Zhai > > Would like to provide a docker file for bookkeeper -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BOOKKEEPER-1081) Would like to provide a docker file for bookkeeper
[ https://issues.apache.org/jira/browse/BOOKKEEPER-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030382#comment-16030382 ] Jia Zhai commented on BOOKKEEPER-1081: -- Hi [~Caiok], Thanks. That is great. We could do it through BOOKKEEPER-974, since that contains more info. > Would like to provide a docker file for bookkeeper > -- > > Key: BOOKKEEPER-1081 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1081 > Project: Bookkeeper > Issue Type: Wish > Components: build >Reporter: Jia Zhai >Assignee: Jia Zhai > > Would like to provide a docker file for bookkeeper -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BOOKKEEPER-1081) Would like to provide a docker file for bookkeeper
[ https://issues.apache.org/jira/browse/BOOKKEEPER-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028009#comment-16028009 ] Jia Zhai commented on BOOKKEEPER-1081: -- Oh Excuse me [~eolivelli] and [~Caio], I did not notice [~Caio]'s work, until you related the issues together. [~Caio] and [~eolivelli] How could we collaborates together to make these issues done? Seems there is a lot of valuable work in [~Caio] 's [work](https://github.com/caiok/bookkeeper-docker) already. > Would like to provide a docker file for bookkeeper > -- > > Key: BOOKKEEPER-1081 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1081 > Project: Bookkeeper > Issue Type: Wish > Components: build >Reporter: Jia Zhai >Assignee: Jia Zhai > > Would like to provide a docker file for bookkeeper -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (BOOKKEEPER-1081) Would like to provide a docker file for bookkeeper
Jia Zhai created BOOKKEEPER-1081: Summary: Would like to provide a docker file for bookkeeper Key: BOOKKEEPER-1081 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1081 Project: Bookkeeper Issue Type: Wish Components: build Reporter: Jia Zhai Assignee: Jia Zhai Would like to provide a docker file for bookkeeper -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (BOOKKEEPER-982) Open a Ticket for "Who is using Bookkeeper"
[ https://issues.apache.org/jira/browse/BOOKKEEPER-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai reassigned BOOKKEEPER-982: --- Assignee: Jia Zhai > Open a Ticket for "Who is using Bookkeeper" > --- > > Key: BOOKKEEPER-982 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-982 > Project: Bookkeeper > Issue Type: Bug >Reporter: Jia Zhai >Assignee: Jia Zhai > > As discussed, would like to create an always open issue "who is using > Bookkeeper" on github. > Here is some example of other project: > [https://github.com/alibaba/RocketMQ/issues/1] > [https://github.com/mgechev/angular-seed/issues/855] > [https://github.com/labstack/echo/issues/295] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (BOOKKEEPER-934) Relax durability
[ https://issues.apache.org/jira/browse/BOOKKEEPER-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai reassigned BOOKKEEPER-934: --- Assignee: (was: Jia Zhai) > Relax durability > > > Key: BOOKKEEPER-934 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-934 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai > > I am thinking adding a new flag to bookkeeper#addEntry(..., Boolean sync). So > the application can control whether to sync or not for individual entries. > - On the write protocol, adding a flag to indicate whether this write should > sync to disk or not. > - On the bookie side, if the addEntry request is sync, going through original > pipeline. If the addEntry disables sync,complete the add callbacks after > writing to the journal file and before flushing journal. > - Those add entries (disabled syncs) will be flushed to disks with subsequent > sync add entries. > There is already a discussion in mail thread, here this ticket could gather > ideas, and provide the discussion materials -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731341#comment-15731341 ] Jia Zhai commented on BOOKKEEPER-974: - Regarding the "official" tag, for our bookkeeper, we need pull request to docker-github: docker-library/official-images[https://github.com/docker-library/official-images] docker-library/docs[https://github.com/docker-library/docs] Then the Official Repositories team, with help from community contributors, formally review each proposal and provide feedback to the author. Here is a more detailed introduction of it: [https://docs.docker.com/docker-hub/official_repos/] > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731310#comment-15731310 ] Jia Zhai edited comment on BOOKKEEPER-974 at 12/8/16 6:48 AM: -- Yes, agree. We could change the password before the first push, if we will use it. was (Author: zhaijia): Yes, agree. We could change the password before the first push. > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731310#comment-15731310 ] Jia Zhai commented on BOOKKEEPER-974: - Yes, agree. We could change the password before the first push. > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-982) Open a Ticket for "Who is using Bookkeeper"
Jia Zhai created BOOKKEEPER-982: --- Summary: Open a Ticket for "Who is using Bookkeeper" Key: BOOKKEEPER-982 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-982 Project: Bookkeeper Issue Type: Bug Reporter: Jia Zhai As discussed, would like to create an always open issue "who is using Bookkeeper" on github. Here is some example of other project: [https://github.com/alibaba/RocketMQ/issues/1] [https://github.com/mgechev/angular-seed/issues/855] [https://github.com/labstack/echo/issues/295] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-974: Assignee: (was: Jia Zhai) > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728472#comment-15728472 ] Jia Zhai edited comment on BOOKKEEPER-974 at 12/7/16 11:06 AM: --- Yes, for zookeeper the fist docker image is marked as *"official"* in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] Hi, Francesco, May I assign this issue to you? And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper, you may use it directly, if you like. was (Author: zhaijia): Yes, for zookeeper the fist docker image is marked as *"official"* in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] Hi, Francesco, May assign this issue to you? And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper, you may use it directly, if you like. > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728472#comment-15728472 ] Jia Zhai edited comment on BOOKKEEPER-974 at 12/7/16 11:06 AM: --- Yes, for zookeeper the fist docker image is marked as *"official"* in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] Hi, Francesco, May assign this issue to you? And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper, you may use it directly, if you like. was (Author: zhaijia): Yes, for zookeeper the fist docker image is marked as "official" in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] Hi, Francesco, May assign this issue to you? And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper, you may use it directly, if you like. > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728472#comment-15728472 ] Jia Zhai edited comment on BOOKKEEPER-974 at 12/7/16 11:05 AM: --- Yes, for zookeeper the fist docker image is marked as "official" in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] Hi, Francesco, May assign this issue to you? And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper, you may use it directly, if you like. was (Author: zhaijia): Yes, for zookeeper the fist docker image is marked as "official" in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper. Hi, Francesco, May assign this issue to you? > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
[ https://issues.apache.org/jira/browse/BOOKKEEPER-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15728472#comment-15728472 ] Jia Zhai commented on BOOKKEEPER-974: - Yes, for zookeeper the fist docker image is marked as "official" in this list: [https://hub.docker.com/search/?isAutomated=0=0=1=0=zookeeper=0] And for bookkeeper, There is a already a docker account user/psw: bookkeeper/bookkeeper. Hi, Francesco, May assign this issue to you? > make pushing a docker image as part of the release procedure > > > Key: BOOKKEEPER-974 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-971) update bk codahale stats provider version
[ https://issues.apache.org/jira/browse/BOOKKEEPER-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-971: Description: update bk stats provider: from codahale to yammer. Currently io.dropwizard.metrics 3.1.0 is used most widely. will change version to this version, and run the test. And would like to change CodahaleMetricsProvider.getMetrics() to public, since this would be used outside package. was: update bk stats provider: from codahale to yammer. Currently io.dropwizard.metrics 3.1.0 is used most widely. will change version to this version, and run the test > update bk codahale stats provider version > - > > Key: BOOKKEEPER-971 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-971 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > update bk stats provider: from codahale to yammer. > Currently io.dropwizard.metrics 3.1.0 is used most widely. will change > version to this version, and run the test. > And would like to change CodahaleMetricsProvider.getMetrics() to public, > since this would be used outside package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-971) update bk codahale stats provider version
[ https://issues.apache.org/jira/browse/BOOKKEEPER-971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-971: Summary: update bk codahale stats provider version (was: update bk stats provider: from codahale to yammer) > update bk codahale stats provider version > - > > Key: BOOKKEEPER-971 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-971 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > update bk stats provider: from codahale to yammer. > Currently io.dropwizard.metrics 3.1.0 is used most. will change version to > this version, and -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-975) Jenkins failed while test BookieClientTest.testWriteGaps
Jia Zhai created BOOKKEEPER-975:
---
Summary: Jenkins failed while test BookieClientTest.testWriteGaps
Key: BOOKKEEPER-975
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-975
Project: Bookkeeper
Issue Type: Bug
Reporter: Jia Zhai
Here is error reporting:
```
2016-11-23 09:03:24,919 - ERROR -
[BookieWriteThread-13645-orderedsafeexecutor-0-0:WriteEntryProcessorV3@125] -
Unexpected exception while writing 1@1 :
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
at
org.apache.bookkeeper.bookie.SortedLedgerStorage.addEntry(SortedLedgerStorage.java:99)
at
org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:80)
at
org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:1176)
at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:1235)
at
org.apache.bookkeeper.proto.WriteEntryProcessorV3.getAddResponse(WriteEntryProcessorV3.java:109)
at
org.apache.bookkeeper.proto.WriteEntryProcessorV3.safeRun(WriteEntryProcessorV3.java:142)
at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:31)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-23 09:03:24,950 - ERROR -
[BKClientOrderedSafeExecutor-orderedsafeexecutor-1-0:SafeRunnable@33] -
Unexpected throwable caught
java.lang.IndexOutOfBoundsException: Invalid readerIndex: 16 - Maximum is 0
at
org.jboss.netty.buffer.EmptyChannelBuffer.readerIndex(EmptyChannelBuffer.java:50)
at
org.apache.bookkeeper.test.BookieClientTest$1.readEntryComplete(BookieClientTest.java:112)
at
org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion$1.readEntryComplete(PerChannelBookieClient.java:930)
at
org.apache.bookkeeper.proto.PerChannelBookieClient.handleReadResponse(PerChannelBookieClient.java:868)
at
org.apache.bookkeeper.proto.PerChannelBookieClient$7.safeRun(PerChannelBookieClient.java:793)
at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:31)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-23 09:03:25,088 - WARN -
[GarbageCollectorThread:ScanAndCompareGarbageCollector@153] - Exception when
iterating over the metadata {}
java.lang.NullPointerException
at
org.apache.bookkeeper.util.ZkUtils.getChildrenInSingleNode(ZkUtils.java:207)
at
org.apache.bookkeeper.util.ZkUtils.getChildrenInSingleNode(ZkUtils.java:169)
at
org.apache.bookkeeper.meta.FlatLedgerManager$1.preload(FlatLedgerManager.java:110)
at
org.apache.bookkeeper.meta.FlatLedgerManager$1.hasNext(FlatLedgerManager.java:120)
at
org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector.gc(ScanAndCompareGarbageCollector.java:104)
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.doGcLedgers(GarbageCollectorThread.java:371)
at
org.apache.bookkeeper.bookie.GarbageCollectorThread.run(GarbageCollectorThread.java:329)
2016-11-23 09:03:25,944 - INFO - [main:BookieServer@167] - Shutting down
BookieServer
2016-11-23 09:03:25,945 - INFO - [main:BookieNettyServer@127] - Shutting down
BookieNettyServer
2016-11-23 09:03:25,950 - INFO - [New I/O worker
#10:PerChannelBookieClient@701] - Disconnected from bookie channel [id:
0x7b660837, /127.0.0.1:48812 :> /127.0.0.1:13645]
2016-11-23 09:03:25,971 - INFO - [main:Bookie@1096] - Shutting down
Bookie-13645 with exitCode 0
2016-11-23 09:03:25,972 - INFO - [main:Journal@970] - Shutting down Journal
2016-11-23 09:03:25,973 - ERROR -
[ForceWriteThread:Journal$ForceWriteThread@433] - ForceWrite thread interrupted
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.bookkeeper.bookie.Journal$ForceWriteThread.run(Journal.java:393)
2016-11-23 09:03:25,974 - WARN -
[jira] [Created] (BOOKKEEPER-974) make pushing a docker image as part of the release procedure
Jia Zhai created BOOKKEEPER-974: --- Summary: make pushing a docker image as part of the release procedure Key: BOOKKEEPER-974 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-974 Project: Bookkeeper Issue Type: Improvement Reporter: Jia Zhai Assignee: Jia Zhai make pushing a docker image as part of the release procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-973) Provide an DCOS Universe package for bookkeeper
Jia Zhai created BOOKKEEPER-973: --- Summary: Provide an DCOS Universe package for bookkeeper Key: BOOKKEEPER-973 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-973 Project: Bookkeeper Issue Type: Improvement Reporter: Jia Zhai -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-972) Provide an DCOS Universe package for bookkeeper
Jia Zhai created BOOKKEEPER-972: --- Summary: Provide an DCOS Universe package for bookkeeper Key: BOOKKEEPER-972 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-972 Project: Bookkeeper Issue Type: Improvement Reporter: Jia Zhai Assignee: Jia Zhai Currently, the PR for this package is merged into main stream: https://github.com/mesosphere/universe And the example PR is also merged into : https://github.com/dcos/examples Maybe in next release of DCOS, we could use the bookkeeper package directly. Left this ticket to tracking the following work left, such as the docker image automate update. Since this version bookkeeper package use a fix version of bookkeeper docker, we need to handle automaticly "update the package as we release new versions" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-971) update bk stats provider: from codahale to yammer
Jia Zhai created BOOKKEEPER-971: --- Summary: update bk stats provider: from codahale to yammer Key: BOOKKEEPER-971 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-971 Project: Bookkeeper Issue Type: Improvement Reporter: Jia Zhai Assignee: Jia Zhai update bk stats provider: from codahale to yammer -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (BOOKKEEPER-968) Entry log flushes happen on log rotation and cause long spikes in IO utilization
[ https://issues.apache.org/jira/browse/BOOKKEEPER-968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670059#comment-15670059 ] Jia Zhai edited comment on BOOKKEEPER-968 at 11/16/16 10:22 AM: [~hustlmsp], should we add items in doc/bookkeeperConfigParams.textile along with this changes? In this link: https://cwiki.apache.org/confluence/display/BOOKKEEPER/Contributing+to+BookKeeper it said: New server configuration settings should be added and documented in bookkeeper-server/conf/bk_server.conf, and also documented in doc/bookkeeperConfigParams.textile. But seems, this textile file contains only the tip of the iceberg, a lot of config parameter is not added in. If needed, we may need a ticket to fix this. was (Author: zhaijia): @sijie , should we add items in doc/bookkeeperConfigParams.textile along with this changes? In this link: https://cwiki.apache.org/confluence/display/BOOKKEEPER/Contributing+to+BookKeeper it said: New server configuration settings should be added and documented in bookkeeper-server/conf/bk_server.conf, and also documented in doc/bookkeeperConfigParams.textile. But seems, this textile file contains only the tip of the iceberg, a lot of config parameter is not added in. If needed, we may need a ticket to fix this. > Entry log flushes happen on log rotation and cause long spikes in IO > utilization > > > Key: BOOKKEEPER-968 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-968 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-server >Affects Versions: 4.5.0 >Reporter: Andrey Yegorov >Assignee: Andrey Yegorov >Priority: Minor > > Caught this issue on the servers with 128G of RAM. This is probably not an > issue on servers/VMs with less RAM. > With current implementation we end up with single entry log flush during log > rotation. > OS tries to flush everything as fast as possible and saturates disk. This > results in long periods of high latency (reads and writes). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-966) change the bookieServer cmdline to make conf-file and option co-exist
[ https://issues.apache.org/jira/browse/BOOKKEEPER-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15650492#comment-15650492 ] Jia Zhai commented on BOOKKEEPER-966: - Thanks a lot for the support. have prepared a pull reqeust for it. > change the bookieServer cmdline to make conf-file and option co-exist > - > > Key: BOOKKEEPER-966 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-966 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > Currently, when using bookieServer cmdline to start a bookie, you will either > give it a cofiguration file by "-c booke.conf"; or add some options like > " " in a fix > sequential. > It may not satisfy some of the requirement. So changed it to be co-exist for > configuration file and options. > By this change, it will first use settings in configuration file; and then > use options to overwrite some of the settings, if there are some options > provided. > Here is an example after this change: > BookieServer -c bookie.conf -z localhost:2181 -m /bookkeeper/ledgers -p 3181 > -j /mnt/journal -l "/mnt/ledger1 /mnt/ledger2 /mnt/ledger3” > Here, in this command: > -z is for “Zookeeper client instance”; > -m is for "Zookeeper ledgers root path for bookies"; > -p is for "bookie service port exported"; > -j is for "bookie journal directory"; > -l is for "bookie ledgers directories". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (BOOKKEEPER-966) change the bookieServer cmdline to make conf-file and option co-exist
[ https://issues.apache.org/jira/browse/BOOKKEEPER-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-966: Comment: was deleted (was: Thanks a lot for the support. have prepared a pull reqeust for it.) > change the bookieServer cmdline to make conf-file and option co-exist > - > > Key: BOOKKEEPER-966 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-966 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > Currently, when using bookieServer cmdline to start a bookie, you will either > give it a cofiguration file by "-c booke.conf"; or add some options like > " " in a fix > sequential. > It may not satisfy some of the requirement. So changed it to be co-exist for > configuration file and options. > By this change, it will first use settings in configuration file; and then > use options to overwrite some of the settings, if there are some options > provided. > Here is an example after this change: > BookieServer -c bookie.conf -z localhost:2181 -m /bookkeeper/ledgers -p 3181 > -j /mnt/journal -l "/mnt/ledger1 /mnt/ledger2 /mnt/ledger3” > Here, in this command: > -z is for “Zookeeper client instance”; > -m is for "Zookeeper ledgers root path for bookies"; > -p is for "bookie service port exported"; > -j is for "bookie journal directory"; > -l is for "bookie ledgers directories". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-966) change the bookieServer cmdline to make conf-file and option co-exist
[ https://issues.apache.org/jira/browse/BOOKKEEPER-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15650491#comment-15650491 ] Jia Zhai commented on BOOKKEEPER-966: - Thanks a lot for the support. have prepared a pull reqeust for it. > change the bookieServer cmdline to make conf-file and option co-exist > - > > Key: BOOKKEEPER-966 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-966 > Project: Bookkeeper > Issue Type: Improvement >Reporter: Jia Zhai >Assignee: Jia Zhai > > Currently, when using bookieServer cmdline to start a bookie, you will either > give it a cofiguration file by "-c booke.conf"; or add some options like > " " in a fix > sequential. > It may not satisfy some of the requirement. So changed it to be co-exist for > configuration file and options. > By this change, it will first use settings in configuration file; and then > use options to overwrite some of the settings, if there are some options > provided. > Here is an example after this change: > BookieServer -c bookie.conf -z localhost:2181 -m /bookkeeper/ledgers -p 3181 > -j /mnt/journal -l "/mnt/ledger1 /mnt/ledger2 /mnt/ledger3” > Here, in this command: > -z is for “Zookeeper client instance”; > -m is for "Zookeeper ledgers root path for bookies"; > -p is for "bookie service port exported"; > -j is for "bookie journal directory"; > -l is for "bookie ledgers directories". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-966) change the bookieServer cmdline to make conf-file and option co-exist
Jia Zhai created BOOKKEEPER-966: --- Summary: change the bookieServer cmdline to make conf-file and option co-exist Key: BOOKKEEPER-966 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-966 Project: Bookkeeper Issue Type: Improvement Reporter: Jia Zhai Assignee: Jia Zhai Currently, when using bookieServer cmdline to start a bookie, you will either give it a cofiguration file by "-c booke.conf"; or add some options like " " in a fix sequential. It may not satisfy some of the requirement. So changed it to be co-exist for configuration file and options. By this change, it will first use settings in configuration file; and then use options to overwrite some of the settings, if there are some options provided. Here is an example after this change: BookieServer -c bookie.conf -z localhost:2181 -m /bookkeeper/ledgers -p 3181 -j /mnt/journal -l "/mnt/ledger1 /mnt/ledger2 /mnt/ledger3” Here, in this command: -z is for “Zookeeper client instance”; -m is for "Zookeeper ledgers root path for bookies"; -p is for "bookie service port exported"; -j is for "bookie journal directory"; -l is for "bookie ledgers directories". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (BOOKKEEPER-876) meet some thread related issue in pre-commit test
[ https://issues.apache.org/jira/browse/BOOKKEEPER-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai resolved BOOKKEEPER-876. - Resolution: Cannot Reproduce seems not find the same issue for a while. would like to close it. Thanks. > meet some thread related issue in pre-commit test > - > > Key: BOOKKEEPER-876 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-876 > Project: Bookkeeper > Issue Type: Bug >Reporter: Jia Zhai >Assignee: Jia Zhai > > Recently when handling ticket 865 and 866, there is some thread related > issues caused pre-commit test fail. > example could be found here: > https://builds.apache.org/job/bookkeeper-trunk-precommit-build/959. > currently, most of test case could pass, and one test case could always fail: > org.apache.bookkeeper.benchmark.TestBenchmark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-906) Upgrade Zookeeper dependency to 3.4.8
[ https://issues.apache.org/jira/browse/BOOKKEEPER-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361924#comment-15361924 ] Jia Zhai commented on BOOKKEEPER-906: - +1 > Upgrade Zookeeper dependency to 3.4.8 > - > > Key: BOOKKEEPER-906 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-906 > Project: Bookkeeper > Issue Type: Wish > Components: bookkeeper-client, bookkeeper-server >Affects Versions: 4.4.0 >Reporter: Enrico Olivelli >Assignee: Enrico Olivelli >Priority: Minor > > As the 4.4.0 version is going to be released very soon maybe it would be > better to include latest version of ZooKeeper -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-934) Relax durability
Jia Zhai created BOOKKEEPER-934: --- Summary: Relax durability Key: BOOKKEEPER-934 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-934 Project: Bookkeeper Issue Type: Improvement Reporter: Jia Zhai Assignee: Jia Zhai I am thinking adding a new flag to bookkeeper#addEntry(..., Boolean sync). So the application can control whether to sync or not for individual entries. - On the write protocol, adding a flag to indicate whether this write should sync to disk or not. - On the bookie side, if the addEntry request is sync, going through original pipeline. If the addEntry disables sync,complete the add callbacks after writing to the journal file and before flushing journal. - Those add entries (disabled syncs) will be flushed to disks with subsequent sync add entries. There is already a discussion in mail thread, here this ticket could gather ideas, and provide the discussion materials -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-926) Compacted entries are not properly synced before updating index
[
https://issues.apache.org/jira/browse/BOOKKEEPER-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268072#comment-15268072
]
Jia Zhai commented on BOOKKEEPER-926:
-
Got it now, it happens when new entry log created. Thanks a lot for the
explaination.
> Compacted entries are not properly synced before updating index
> ---
>
> Key: BOOKKEEPER-926
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-926
> Project: Bookkeeper
> Issue Type: Bug
>Affects Versions: 4.3.1
>Reporter: Matteo Merli
>Assignee: Matteo Merli
> Fix For: 4.4.0
>
>
> I have identified a couple of issues in Bookie compaction code.
> h4. Compacted entries are not properly synced when index is updated
> When compacting, we read "active" entries from an entry log and we re-append
> to current entry log. After compacting a number of entries, by default 100K,
> or at the very end, we need to update the index pointing to the new entry log
> and offset.
> Before updating the index, we need to wait for this entries to be flushed and
> fsynced, otherwise a bookie crash might leave the index updated, pointing to
> an invalid offset.
> The current code that is supposed to wait until flushed is:
> {code:java}
> // GarbageCollectorThread.java:178
> EntryLocation lastOffset = offsets.get(offsets.size()-1);
> long lastOffsetLogId = EntryLogger.logIdForOffset(lastOffset.location);
> while (lastOffsetLogId < entryLogger.getLeastUnflushedLogId() && running) {
> synchronized (flushLock) {
> flushLock.wait(1000);
> }
> lastOffset = offsets.get(offsets.size()-1);
> lastOffsetLogId = EntryLogger.logIdForOffset(lastOffset.location);
> }
> // update the index
> {code}
> The condition {{lastOffsetLogId < entryLogger.getLeastUnflushedLogId()}} is
> wrong, because if the last compacted entry was written in an earlier entry
> log than the least unflushed log, it means that the entries are already
> flushed and thus we don't need to wait.
> In the normal case what happens is that {{lastOffsetLogId} is actually the
> current entryLog and it's equal to {{entryLogger.getLeastUnflushedLogId()}},
> so we don't wait. But, in this case the entries appended to the current
> entrylog are not flushed nor synced, hence the problem.
> h4. Exception during index flush
> Having an exception when updating the index, combined with the above issue,
> makes the bookie GC to stop indefinitely.
> What happens is that the offset list is not cleared, and at the next bookie
> GC iteration it will find the old compacted entries in that list, for which
> now the entryLogId is less than the current log id, and that makes the while
> loop to never exit.
> Another problem is that, any IOException during the index flush, will make
> the GC thread to bail out and it will not remove even the entry logs that
> were compacted and flushed. Next time, these entry logs will be compacted
> again.
> h4. Proposed solution
> I think the best solution is to trigger the {{entryLogger.flush()}} form the
> bookie GC thread before updating the indexes. That would simplify the code
> and I don't see any disadvantages in doing that.
> Another improvement would be to delete compacted entry logs individually, as
> soon as the compacted data is flush, without waiting the end of the whole
> compaction process.
> The advantages are :
> * If compaction stop halfway, at least we don't have to re-compact what we
> just compacted
> * Compaction won't require significant space overhead. Today a major
> compaction can end up reappending a large amount of data and then deleting
> all the entry logs at the very end, requiring twice the size of the active
> data set to be stored on disk at some point in time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-911) Fix TestReplicationWorker test failures
[ https://issues.apache.org/jira/browse/BOOKKEEPER-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244095#comment-15244095 ] Jia Zhai commented on BOOKKEEPER-911: - Manually test this patch on my machine, and it works, here is the result: without patch: --- T E S T S --- Running org.apache.bookkeeper.replication.TestReplicationWorker Tests run: 27, Failures: 3, Errors: 3, Skipped: 0, Time elapsed: 128.776 sec <<< FAILURE! Results : Failed tests: testRWZKSessionLost[0](org.apache.bookkeeper.replication.TestReplicationWorker): Replication worker should have shut down testRWZKSessionLost[1](org.apache.bookkeeper.replication.TestReplicationWorker): Replication worker should have shut down testRWZKSessionLost[2](org.apache.bookkeeper.replication.TestReplicationWorker): Replication worker should have shut down Tests in error: testRWShutdownOnLocalBookieReadonlyTransition[0](org.apache.bookkeeper.replication.TestReplicationWorker): test timed out after 2 milliseconds testRWShutdownOnLocalBookieReadonlyTransition[1](org.apache.bookkeeper.replication.TestReplicationWorker): test timed out after 2 milliseconds testRWShutdownOnLocalBookieReadonlyTransition[2](org.apache.bookkeeper.replication.TestReplicationWorker): test timed out after 2 milliseconds Tests run: 27, Failures: 3, Errors: 3, Skipped: 0 with patch: --- --- T E S T S --- Running org.apache.bookkeeper.replication.TestReplicationWorker Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.998 sec Results : Tests run: 27, Failures: 0, Errors: 0, Skipped: 0 > Fix TestReplicationWorker test failures > --- > > Key: BOOKKEEPER-911 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-911 > Project: Bookkeeper > Issue Type: Bug > Components: bookkeeper-auto-recovery >Affects Versions: 4.3.2 >Reporter: Arun M. Krishnakumar >Assignee: Matteo Merli > Labels: tests > Fix For: 4.4.0 > > > Currently we have the following test failures in master branch: > Results : > Failed tests: > testRWZKSessionLost[0](org.apache.bookkeeper.replication.TestReplicationWorker): > Replication worker should have shut down > > testRWZKSessionLost[1](org.apache.bookkeeper.replication.TestReplicationWorker): > Replication worker should have shut down > > testRWZKSessionLost[2](org.apache.bookkeeper.replication.TestReplicationWorker): > Replication worker should have shut down > Tests in error: > > testRWShutdownOnLocalBookieReadonlyTransition[0](org.apache.bookkeeper.replication.TestReplicationWorker): > test timed out after 2 milliseconds > > testRWShutdownOnLocalBookieReadonlyTransition[1](org.apache.bookkeeper.replication.TestReplicationWorker): > test timed out after 2 milliseconds > > testRWShutdownOnLocalBookieReadonlyTransition[2](org.apache.bookkeeper.replication.TestReplicationWorker): > test timed out after 2 milliseconds > Tests run: 654, Failures: 3, Errors: 3, Skipped: 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-870) Change the default value for bookie settings.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234376#comment-15234376 ] Jia Zhai commented on BOOKKEEPER-870: - Seems 3 new test errors in the jenkins pre-checkin test? . testBookieShutdownIfReadOnlyModeNotEnabled(org.apache.bookkeeper.test.ReadOnlyBookieTest) . testSyncThreadDisksFull(org.apache.bookkeeper.bookie.TestSyncThread) . testSyncThreadShutdownOnError(org.apache.bookkeeper.bookie.TestSyncThread) > Change the default value for bookie settings. > - > > Key: BOOKKEEPER-870 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-870 > Project: Bookkeeper > Issue Type: Documentation >Reporter: Sijie Guo >Assignee: Sijie Guo > Fix For: 4.4.0 > > Attachments: BOOKKEEPER-870.patch > > > It would be good to tune the default settings for bookies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-865) provide a simple way to compress the content in each Entry of EntryLogger
[ https://issues.apache.org/jira/browse/BOOKKEEPER-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964992#comment-14964992 ] Jia Zhai commented on BOOKKEEPER-865: - Yes. That is the next step. This compressing way, which handle each single log entry, also saves a lot of space if the content of each entry is big. > provide a simple way to compress the content in each Entry of EntryLogger > - > > Key: BOOKKEEPER-865 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-865 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-server >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: BOOKKEEPER-865.patch > > > This issue is an enhancement, it tries to provide a simple way to compress > the content in each Entry of EntryLogger. > By this enhancement, it will save a lot of disk space for the logs, under the > condition that contents is large in Entry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-876) meet some thread related issue in pre-commit test
[ https://issues.apache.org/jira/browse/BOOKKEEPER-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960344#comment-14960344 ] Jia Zhai commented on BOOKKEEPER-876: - Thanks [~hustlmsp]] for the help, keep more builds history will help a lot. > meet some thread related issue in pre-commit test > - > > Key: BOOKKEEPER-876 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-876 > Project: Bookkeeper > Issue Type: Bug >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: narrow_down.patch > > > Recently when handling ticket 865 and 866, there is some thread related > issues caused pre-commit test fail. > example could be found here: > https://builds.apache.org/job/bookkeeper-trunk-precommit-build/959. > After doing some investigate and narrowed this down, this may related to > change of ticket 862. I would like to investigate this a little more to > confrim this, so temp assign this to my self. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-866) Fix compile issue when Updating junit to latest release version( 4.12) in the test of Bookkeeper-server.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-866: Attachment: BOOKKEEPER-866-2.patch Regarding the conflicts in above comment, temp remove timeout parameter in 2 test cases. > Fix compile issue when Updating junit to latest release version( 4.12) in the > test of Bookkeeper-server. > - > > Key: BOOKKEEPER-866 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-866 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-server >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: BOOKKEEPER-866-2.patch, BOOKKEEPER-866.patch > > > When write test cases for BOOKKEEPER-865, It appears that current version of > junit could not support some new features well, such as Parameterized test. > Then we try to update junit to latest release version, but found " > junit.framework.Assert in junit.framework has been deprecated" . > So using this new ticket to trace this to make the objective more clear. > The fix is simple, > replace > import junit.framework.Assert; > to > import org.junit.Assert; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-876) meet some thread related issue in pre-commit test
Jia Zhai created BOOKKEEPER-876: --- Summary: meet some thread related issue in pre-commit test Key: BOOKKEEPER-876 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-876 Project: Bookkeeper Issue Type: Bug Reporter: Jia Zhai Assignee: Jia Zhai Recently when handling ticket 865 and 866, there is some thread related issues caused pre-commit test fail. example could be found here: https://builds.apache.org/job/bookkeeper-trunk-precommit-build/959. After doing some investigate and narrowed this down, this may related to change of ticket 862. I would like to investigate this a little more to confrim this, so temp assign this to my self. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-866) Fix compile issue when Updating junit to latest release version( 4.12) in the test of Bookkeeper-server.
[
https://issues.apache.org/jira/browse/BOOKKEEPER-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909756#comment-14909756
]
Jia Zhai commented on BOOKKEEPER-866:
-
For the first error, seem code in Junit is not working very well with
ZooKeeperUtil.sleepServer
{code}
FailOnTimeout.java
@Override
public void evaluate() throws Throwable {
CallableStatement callable = new CallableStatement();
FutureTask task = new FutureTask(callable);
threadGroup = new ThreadGroup("FailOnTimeoutGroup");
Thread thread = new Thread(threadGroup, task, "Time-limited test");
< ==
thread.setDaemon(true);
thread.start();
callable.awaitStarted();
Throwable throwable = getResult(task, thread);
if (throwable != null) {
throw throwable;
}
}
{code}
code of sleepServer()
{code}
public void sleepServer(final int seconds, final CountDownLatch l)
throws InterruptedException, IOException {
Thread[] allthreads = new Thread[Thread.activeCount()];
Thread.enumerate(allthreads);
< == Here could only get thread "Time-limited test", which
created by above code of junit.
for (final Thread t : allthreads) {
if (t.getName().contains("SyncThread:0")) {
Thread sleeper = new Thread() {
@SuppressWarnings("deprecation")
public void run() {
try {
t.suspend();
l.countDown();
Thread.sleep(seconds*1000);
t.resume();
} catch (Exception e) {
LOG.error("Error suspending thread", e);
}
}
};
sleeper.start();
return;
}
}
throw new IOException("ZooKeeper thread not found");< ===
so will not find wanted thread, and got this exception.
}
{code}
> Fix compile issue when Updating junit to latest release version( 4.12) in the
> test of Bookkeeper-server.
> -
>
> Key: BOOKKEEPER-866
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-866
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-server
>Reporter: Jia Zhai
>Assignee: Jia Zhai
> Attachments: BOOKKEEPER-866.patch
>
>
> When write test cases for BOOKKEEPER-865, It appears that current version of
> junit could not support some new features well, such as Parameterized test.
> Then we try to update junit to latest release version, but found "
> junit.framework.Assert in junit.framework has been deprecated" .
> So using this new ticket to trace this to make the objective more clear.
> The fix is simple,
> replace
> import junit.framework.Assert;
> to
> import org.junit.Assert;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-866) Fix compile issue when Updating junit to latest release version( 4.12) in the test of Bookkeeper-server.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909326#comment-14909326 ] Jia Zhai commented on BOOKKEEPER-866: - org.apache.bookkeeper.client.BookKeeperTest.testConstructionZkDelay[0] org.apache.bookkeeper.client.BookKeeperTest.testConstructionNotConnectedExplicitZk[0] org.apache.bookkeeper.client.BookKeeperTest.testConstructionZkDelay[1] org.apache.bookkeeper.client.BookKeeperTest.testConstructionNotConnectedExplicitZk[1] These 4 tests failed at here: java.io.IOException: ZooKeeper thread not found at org.apache.bookkeeper.test.ZooKeeperUtil.sleepServer(ZooKeeperUtil.java:134) org.apache.bookkeeper.zookeeper.TestZooKeeperClient.testReconnectAfterExipred this test failed at here: java.lang.NullPointerException at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:148) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:763) at org.apache.bookkeeper.zookeeper.TestZooKeeperClient.testReconnectAfterExipred(TestZooKeeperClient.java:165) suspect this is related to zookeeper or some changes in Junit. Will study more on them. > Fix compile issue when Updating junit to latest release version( 4.12) in the > test of Bookkeeper-server. > - > > Key: BOOKKEEPER-866 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-866 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-server >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: BOOKKEEPER-866.patch > > > When write test cases for BOOKKEEPER-865, It appears that current version of > junit could not support some new features well, such as Parameterized test. > Then we try to update junit to latest release version, but found " > junit.framework.Assert in junit.framework has been deprecated" . > So using this new ticket to trace this to make the objective more clear. > The fix is simple, > replace > import junit.framework.Assert; > to > import org.junit.Assert; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-866) Fix compile issue when Updating junit to latest release version( 4.12) in the test of Bookkeeper-server.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908923#comment-14908923 ] Jia Zhai commented on BOOKKEEPER-866: - The failed test case looks related to zookeeper, will investigate this. > Fix compile issue when Updating junit to latest release version( 4.12) in the > test of Bookkeeper-server. > - > > Key: BOOKKEEPER-866 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-866 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-server >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: BOOKKEEPER-866.patch > > > When write test cases for BOOKKEEPER-865, It appears that current version of > junit could not support some new features well, such as Parameterized test. > Then we try to update junit to latest release version, but found " > junit.framework.Assert in junit.framework has been deprecated" . > So using this new ticket to trace this to make the objective more clear. > The fix is simple, > replace > import junit.framework.Assert; > to > import org.junit.Assert; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907760#comment-14907760
]
Jia Zhai commented on BOOKKEEPER-826:
-
Hi [~hustlmsp], thanks a lot for this. The fix looks good, that is indeed a
more suitable place to call sendSuccessfulCallback.
> PendingAddOp is ignoring ack response after meet ack quorum constraint
> ---
>
> Key: BOOKKEEPER-826
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-client
>Reporter: Charles X
>Assignee: Jia Zhai
> Attachments: BOOKKEEPER-826.diff, BOOKKEEPER-826.patch
>
>
> PendingAddOp is set to completed when it meets ack quorum.
> {code}
> if (ackSet.addBookieAndCheck(bookieIndex) && !completed) {
> completed = true;
> LOG.debug("Complete (lid:{}, eid:{}).", ledgerId, entryId);
> // when completed an entry, try to send success add callbacks in
> order
> lh.sendAddSuccessCallbacks();
> }
> {code}
> responses are ignored after completed flag is set.
> {code}
>if (completed) {
> // I am already finished, ignore incoming responses.
> // otherwise, we might hit the following error handling logic,
> which might cause bad things.
> return;
> }
> {code}
> It is not a correctness problem, but it would introduce performance issue
> during ensemble change. A callback (could be acknowledge before ensemble
> change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-866) Fix compile issue when Updating junit to latest release version( 4.12) in the test of Bookkeeper-server.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900115#comment-14900115 ] Jia Zhai commented on BOOKKEEPER-866: - It is strange that this change involved some test error: Test Result (失败) org.apache.bookkeeper.client.BookKeeperTest.testConstructionZkDelay[0] org.apache.bookkeeper.client.BookKeeperTest.testConstructionNotConnectedExplicitZk[0] org.apache.bookkeeper.client.BookKeeperTest.testConstructionZkDelay[1] org.apache.bookkeeper.client.BookKeeperTest.testConstructionNotConnectedExplicitZk[1] org.apache.bookkeeper.zookeeper.TestZooKeeperClient.testReconnectAfterExipred > Fix compile issue when Updating junit to latest release version( 4.12) in the > test of Bookkeeper-server. > - > > Key: BOOKKEEPER-866 > URL: https://issues.apache.org/jira/browse/BOOKKEEPER-866 > Project: Bookkeeper > Issue Type: Improvement > Components: bookkeeper-server >Reporter: Jia Zhai >Assignee: Jia Zhai > Attachments: BOOKKEEPER-866.patch > > > When write test cases for BOOKKEEPER-865, It appears that current version of > junit could not support some new features well, such as Parameterized test. > Then we try to update junit to latest release version, but found " > junit.framework.Assert in junit.framework has been deprecated" . > So using this new ticket to trace this to make the objective more clear. > The fix is simple, > replace > import junit.framework.Assert; > to > import org.junit.Assert; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744670#comment-14744670
]
Jia Zhai commented on BOOKKEEPER-826:
-
Thanks, that is indeed a more suitable place to call sendSuccessfulCallback
> PendingAddOp is ignoring ack response after meet ack quorum constraint
> ---
>
> Key: BOOKKEEPER-826
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-client
>Reporter: Charles X
>Assignee: Jia Zhai
> Attachments: BOOKKEEPER-826.diff, BOOKKEEPER-826.patch
>
>
> PendingAddOp is set to completed when it meets ack quorum.
> {code}
> if (ackSet.addBookieAndCheck(bookieIndex) && !completed) {
> completed = true;
> LOG.debug("Complete (lid:{}, eid:{}).", ledgerId, entryId);
> // when completed an entry, try to send success add callbacks in
> order
> lh.sendAddSuccessCallbacks();
> }
> {code}
> responses are ignored after completed flag is set.
> {code}
>if (completed) {
> // I am already finished, ignore incoming responses.
> // otherwise, we might hit the following error handling logic,
> which might cause bad things.
> return;
> }
> {code}
> It is not a correctness problem, but it would introduce performance issue
> during ensemble change. A callback (could be acknowledge before ensemble
> change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744671#comment-14744671
]
Jia Zhai commented on BOOKKEEPER-826:
-
Thanks, that is indeed a more suitable place to call sendSuccessfulCallback
> PendingAddOp is ignoring ack response after meet ack quorum constraint
> ---
>
> Key: BOOKKEEPER-826
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
> Project: Bookkeeper
> Issue Type: Improvement
> Components: bookkeeper-client
>Reporter: Charles X
>Assignee: Jia Zhai
> Attachments: BOOKKEEPER-826.diff, BOOKKEEPER-826.patch
>
>
> PendingAddOp is set to completed when it meets ack quorum.
> {code}
> if (ackSet.addBookieAndCheck(bookieIndex) && !completed) {
> completed = true;
> LOG.debug("Complete (lid:{}, eid:{}).", ledgerId, entryId);
> // when completed an entry, try to send success add callbacks in
> order
> lh.sendAddSuccessCallbacks();
> }
> {code}
> responses are ignored after completed flag is set.
> {code}
>if (completed) {
> // I am already finished, ignore incoming responses.
> // otherwise, we might hit the following error handling logic,
> which might cause bad things.
> return;
> }
> {code}
> It is not a correctness problem, but it would introduce performance issue
> during ensemble change. A callback (could be acknowledge before ensemble
> change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698652#comment-14698652
]
Jia Zhai commented on BOOKKEEPER-826:
-
Seems we need try to submit all completed pendingAddOps when ensemble change
happens.
PendingAddOp is ignoring ack response after meet ack quorum constraint
---
Key: BOOKKEEPER-826
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
Project: Bookkeeper
Issue Type: Improvement
Components: bookkeeper-client
Reporter: Charles X
Assignee: Jia Zhai
PendingAddOp is set to completed when it meets ack quorum.
{code}
if (ackSet.addBookieAndCheck(bookieIndex) !completed) {
completed = true;
LOG.debug(Complete (lid:{}, eid:{})., ledgerId, entryId);
// when completed an entry, try to send success add callbacks in
order
lh.sendAddSuccessCallbacks();
}
{code}
responses are ignored after completed flag is set.
{code}
if (completed) {
// I am already finished, ignore incoming responses.
// otherwise, we might hit the following error handling logic,
which might cause bad things.
return;
}
{code}
It is not a correctness problem, but it would introduce performance issue
during ensemble change. A callback (could be acknowledge before ensemble
change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698640#comment-14698640
]
Jia Zhai commented on BOOKKEEPER-826:
-
I would like to contribute to this ticket.
PendingAddOp is ignoring ack response after meet ack quorum constraint
---
Key: BOOKKEEPER-826
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
Project: Bookkeeper
Issue Type: Improvement
Components: bookkeeper-client
Reporter: Charles X
Assignee: Jia Zhai
PendingAddOp is set to completed when it meets ack quorum.
{code}
if (ackSet.addBookieAndCheck(bookieIndex) !completed) {
completed = true;
LOG.debug(Complete (lid:{}, eid:{})., ledgerId, entryId);
// when completed an entry, try to send success add callbacks in
order
lh.sendAddSuccessCallbacks();
}
{code}
responses are ignored after completed flag is set.
{code}
if (completed) {
// I am already finished, ignore incoming responses.
// otherwise, we might hit the following error handling logic,
which might cause bad things.
return;
}
{code}
It is not a correctness problem, but it would introduce performance issue
during ensemble change. A callback (could be acknowledge before ensemble
change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jia Zhai updated BOOKKEEPER-826:
Attachment: BOOKKEEPER-826.patch
PendingAddOp is ignoring ack response after meet ack quorum constraint
---
Key: BOOKKEEPER-826
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
Project: Bookkeeper
Issue Type: Improvement
Components: bookkeeper-client
Reporter: Charles X
Assignee: Jia Zhai
Attachments: BOOKKEEPER-826.patch
PendingAddOp is set to completed when it meets ack quorum.
{code}
if (ackSet.addBookieAndCheck(bookieIndex) !completed) {
completed = true;
LOG.debug(Complete (lid:{}, eid:{})., ledgerId, entryId);
// when completed an entry, try to send success add callbacks in
order
lh.sendAddSuccessCallbacks();
}
{code}
responses are ignored after completed flag is set.
{code}
if (completed) {
// I am already finished, ignore incoming responses.
// otherwise, we might hit the following error handling logic,
which might cause bad things.
return;
}
{code}
It is not a correctness problem, but it would introduce performance issue
during ensemble change. A callback (could be acknowledge before ensemble
change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (BOOKKEEPER-826) PendingAddOp is ignoring ack response after meet ack quorum constraint
[
https://issues.apache.org/jira/browse/BOOKKEEPER-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jia Zhai reassigned BOOKKEEPER-826:
---
Assignee: Jia Zhai (was: Charles X)
PendingAddOp is ignoring ack response after meet ack quorum constraint
---
Key: BOOKKEEPER-826
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-826
Project: Bookkeeper
Issue Type: Improvement
Components: bookkeeper-client
Reporter: Charles X
Assignee: Jia Zhai
PendingAddOp is set to completed when it meets ack quorum.
{code}
if (ackSet.addBookieAndCheck(bookieIndex) !completed) {
completed = true;
LOG.debug(Complete (lid:{}, eid:{})., ledgerId, entryId);
// when completed an entry, try to send success add callbacks in
order
lh.sendAddSuccessCallbacks();
}
{code}
responses are ignored after completed flag is set.
{code}
if (completed) {
// I am already finished, ignore incoming responses.
// otherwise, we might hit the following error handling logic,
which might cause bad things.
return;
}
{code}
It is not a correctness problem, but it would introduce performance issue
during ensemble change. A callback (could be acknowledge before ensemble
change) has to be delayed to ensemble change completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-823) Clean up temp files created by hedwig tests
[ https://issues.apache.org/jira/browse/BOOKKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-823: Attachment: BOOKKEEPER-823-2.patch Thanks for Jiannan's comments. updated the patch. Clean up temp files created by hedwig tests --- Key: BOOKKEEPER-823 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-823 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Jia Zhai Fix For: 4.4.0 Attachments: BOOKKEEPER-823-2.patch, BOOKKEEPER-823.patch BOOKKEEPER-814 cleaned up temp files from the bookkeeper tests. This jira is to do the same for hedwig tests (it creates a couple of zookeeper directories). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-823) Clean up temp files created by hedwig tests
[ https://issues.apache.org/jira/browse/BOOKKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-823: Attachment: BOOKKEEPER-823.patch Clean up temp files created by hedwig tests --- Key: BOOKKEEPER-823 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-823 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Jia Zhai Fix For: 4.4.0 Attachments: BOOKKEEPER-823.patch BOOKKEEPER-814 cleaned up temp files from the bookkeeper tests. This jira is to do the same for hedwig tests (it creates a couple of zookeeper directories). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-823) Clean up temp files created by hedwig tests
[ https://issues.apache.org/jira/browse/BOOKKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578203#comment-14578203 ] Jia Zhai commented on BOOKKEEPER-823: - Since this is related with BOOKKEEPER-814, I would like to provide this fix. Tested Hedwig using mvn test, only found TestPubSubServerStartup left dirs uncleared. Clean up temp files created by hedwig tests --- Key: BOOKKEEPER-823 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-823 Project: Bookkeeper Issue Type: Bug Reporter: Ivan Kelly Assignee: Jia Zhai Fix For: 4.4.0 Attachments: BOOKKEEPER-823.patch BOOKKEEPER-814 cleaned up temp files from the bookkeeper tests. This jira is to do the same for hedwig tests (it creates a couple of zookeeper directories). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-847) ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo
[ https://issues.apache.org/jira/browse/BOOKKEEPER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386101#comment-14386101 ] Jia Zhai commented on BOOKKEEPER-847: - Hi Rakesh R, Thanks a lot for your comments. Will update the patch soon. ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo -- Key: BOOKKEEPER-847 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-847 Project: Bookkeeper Issue Type: Bug Reporter: Jia Zhai Assignee: Jia Zhai Attachments: BOOKKEEPER-847.patch ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo, because update ensemble info might happen after re-read ledger metadata, so the ensemble might already change. if ensemble is already changed, skip replacing the bookie doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-847) ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo
[ https://issues.apache.org/jira/browse/BOOKKEEPER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-847: Attachment: BOOKKEEPER-847-2.patch ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo -- Key: BOOKKEEPER-847 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-847 Project: Bookkeeper Issue Type: Bug Reporter: Jia Zhai Assignee: Jia Zhai Attachments: BOOKKEEPER-847-2.patch, BOOKKEEPER-847.patch ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo, because update ensemble info might happen after re-read ledger metadata, so the ensemble might already change. if ensemble is already changed, skip replacing the bookie doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-847) ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo
[ https://issues.apache.org/jira/browse/BOOKKEEPER-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-847: Attachment: BOOKKEEPER-847.patch Thanks for Rakesh and Sijie's help and quick response during this weekend. The patch is straightforward. ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo -- Key: BOOKKEEPER-847 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-847 Project: Bookkeeper Issue Type: Bug Reporter: Jia Zhai Assignee: Jia Zhai Attachments: BOOKKEEPER-847.patch ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo, because update ensemble info might happen after re-read ledger metadata, so the ensemble might already change. if ensemble is already changed, skip replacing the bookie doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-847) ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo
Jia Zhai created BOOKKEEPER-847: --- Summary: ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo Key: BOOKKEEPER-847 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-847 Project: Bookkeeper Issue Type: Bug Reporter: Jia Zhai Assignee: Jia Zhai ArrayIndexOutOfBoundsException in LedgerFragmentReplicator::updateEnsembleInfo, because update ensemble info might happen after re-read ledger metadata, so the ensemble might already change. if ensemble is already changed, skip replacing the bookie doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-836: Attachment: BOOKKEEPER-836-v3.patch changed patch following Sijie's comments. disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.2 Attachments: BOOKKEEPER-836-v2.patch, BOOKKEEPER-836-v3.patch, BOOKKEEPER-836.patch In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361550#comment-14361550 ] Jia Zhai commented on BOOKKEEPER-836: - Got it. It is more clear in this if. Will update it soon. Thanks a lot. disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.2 Attachments: BOOKKEEPER-836-v2.patch, BOOKKEEPER-836.patch In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-836: Attachment: BOOKKEEPER-836-v2.patch Thanks a lot for the comments from Flavio and Sijie. From my understanding, the way progressively release old logs instead of waiting until the end of the process may involve more waiting time in doCompactEntryLogs(), we may need more investigate and test, so not do it here, If possiable we could open another ticket in Jira for this way. The changes in this new patch: 1, Added a new param to choose whether do force GC or suspend GC, when disk is full or almost full. 2, Remove unnecessary synchronized in method suspend and resume GC. disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.2 Attachments: BOOKKEEPER-836-v2.patch, BOOKKEEPER-836.patch In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-838) ForceWriteThread::run() leaks “logFile.close()” when interrupt comes
[
https://issues.apache.org/jira/browse/BOOKKEEPER-838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333146#comment-14333146
]
Jia Zhai commented on BOOKKEEPER-838:
-
I am glad to be helpful. Thanks for taking care of it.
ForceWriteThread::run() leaks “logFile.close()” when interrupt comes
Key: BOOKKEEPER-838
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-838
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Reporter: Jia Zhai
Assignee: Jia Zhai
Fix For: 4.4.0, 4.3.1
Attachments: BOOKKEEPER-838.patch
According to Ivan’s email, I did a check of the build history. Seems recently
failing is with this stack:
java.io.IOException: Unable to delete directory
/tmp/bkTest3561939033223584760.dir/current/0.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1337)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at
org.apache.bookkeeper.test.BookKeeperClusterTestCase.cleanupTempDirs(BookKeeperClusterTestCase.java:186)
at
org.apache.bookkeeper.test.BookKeeperClusterTestCase.tearDown(BookKeeperClusterTestCase.java:114)
This may be caused by an error in ForceWriteThread::run(), which leaked
“logFile.close()” when interrupt comes.
{code}
private class ForceWriteThread {
public void run() {
LOG.info(ForceWrite Thread started);
boolean shouldForceWrite = true;
int numReqInLastForceWrite = 0;
while(running) {
ForceWriteRequest req = null;
try {
…
} catch (IOException ioe) {
LOG.error(I/O exception in ForceWrite thread, ioe);
running = false;
} catch (InterruptedException e) {
LOG.error(ForceWrite thread interrupted, e);
if (null != req) {
req.closeFileIfNecessary(); 2, when
interrupt, “shouldClose” not set properly, so file not close
}
running = false;
}
}
// Regardless of what caused us to exit, we should notify the
// the parent thread as it should either exit or be in the process
// of exiting else we will have write requests hang
threadToNotifyOnEx.interrupt();
}
// shutdown sync thread
void shutdown() throws InterruptedException {
running = false;
this.interrupt(); 1, call interrupt
this.join();
}
}
public void closeFileIfNecessary() {
// Close if shouldClose is set
if (shouldClose) { 3, “shouldClose” is false here.
// We should guard against exceptions so its
// safe to call in catch blocks
try {
logFile.close();
// Call close only once
shouldClose = false;
}
catch (IOException ioe) {
LOG.error(I/O exception while closing file, ioe);
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333150#comment-14333150 ] Jia Zhai commented on BOOKKEEPER-836: - Yes, major compaction is the a problem in current code. disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.2 Attachments: BOOKKEEPER-836.patch In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-838) ForceWriteThread::run() leaks “logFile.close()” when interrupt comes
[
https://issues.apache.org/jira/browse/BOOKKEEPER-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jia Zhai updated BOOKKEEPER-838:
Attachment: BOOKKEEPER-838.patch
ForceWriteThread::run() leaks “logFile.close()” when interrupt comes
Key: BOOKKEEPER-838
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-838
Project: Bookkeeper
Issue Type: Bug
Reporter: Jia Zhai
Assignee: Jia Zhai
Attachments: BOOKKEEPER-838.patch
According to Ivan’s email, I did a check of the build history. Seems recently
failing is with this stack:
java.io.IOException: Unable to delete directory
/tmp/bkTest3561939033223584760.dir/current/0.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1337)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at
org.apache.bookkeeper.test.BookKeeperClusterTestCase.cleanupTempDirs(BookKeeperClusterTestCase.java:186)
at
org.apache.bookkeeper.test.BookKeeperClusterTestCase.tearDown(BookKeeperClusterTestCase.java:114)
This may be caused by an error in ForceWriteThread::run(), which leaked
“logFile.close()” when interrupt comes.
{code}
private class ForceWriteThread {
public void run() {
LOG.info(ForceWrite Thread started);
boolean shouldForceWrite = true;
int numReqInLastForceWrite = 0;
while(running) {
ForceWriteRequest req = null;
try {
…
} catch (IOException ioe) {
LOG.error(I/O exception in ForceWrite thread, ioe);
running = false;
} catch (InterruptedException e) {
LOG.error(ForceWrite thread interrupted, e);
if (null != req) {
req.closeFileIfNecessary(); 2, when
interrupt, “shouldClose” not set properly, so file not close
}
running = false;
}
}
// Regardless of what caused us to exit, we should notify the
// the parent thread as it should either exit or be in the process
// of exiting else we will have write requests hang
threadToNotifyOnEx.interrupt();
}
// shutdown sync thread
void shutdown() throws InterruptedException {
running = false;
this.interrupt(); 1, call interrupt
this.join();
}
}
public void closeFileIfNecessary() {
// Close if shouldClose is set
if (shouldClose) { 3, “shouldClose” is false here.
// We should guard against exceptions so its
// safe to call in catch blocks
try {
logFile.close();
// Call close only once
shouldClose = false;
}
catch (IOException ioe) {
LOG.error(I/O exception while closing file, ioe);
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-838) ForceWriteThread::run() leaks “logFile.close()” when interrupt comes
[
https://issues.apache.org/jira/browse/BOOKKEEPER-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jia Zhai updated BOOKKEEPER-838:
Component/s: bookkeeper-server
ForceWriteThread::run() leaks “logFile.close()” when interrupt comes
Key: BOOKKEEPER-838
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-838
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Reporter: Jia Zhai
Assignee: Jia Zhai
Attachments: BOOKKEEPER-838.patch
According to Ivan’s email, I did a check of the build history. Seems recently
failing is with this stack:
java.io.IOException: Unable to delete directory
/tmp/bkTest3561939033223584760.dir/current/0.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1337)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1910)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1399)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331)
at
org.apache.bookkeeper.test.BookKeeperClusterTestCase.cleanupTempDirs(BookKeeperClusterTestCase.java:186)
at
org.apache.bookkeeper.test.BookKeeperClusterTestCase.tearDown(BookKeeperClusterTestCase.java:114)
This may be caused by an error in ForceWriteThread::run(), which leaked
“logFile.close()” when interrupt comes.
{code}
private class ForceWriteThread {
public void run() {
LOG.info(ForceWrite Thread started);
boolean shouldForceWrite = true;
int numReqInLastForceWrite = 0;
while(running) {
ForceWriteRequest req = null;
try {
…
} catch (IOException ioe) {
LOG.error(I/O exception in ForceWrite thread, ioe);
running = false;
} catch (InterruptedException e) {
LOG.error(ForceWrite thread interrupted, e);
if (null != req) {
req.closeFileIfNecessary(); 2, when
interrupt, “shouldClose” not set properly, so file not close
}
running = false;
}
}
// Regardless of what caused us to exit, we should notify the
// the parent thread as it should either exit or be in the process
// of exiting else we will have write requests hang
threadToNotifyOnEx.interrupt();
}
// shutdown sync thread
void shutdown() throws InterruptedException {
running = false;
this.interrupt(); 1, call interrupt
this.join();
}
}
public void closeFileIfNecessary() {
// Close if shouldClose is set
if (shouldClose) { 3, “shouldClose” is false here.
// We should guard against exceptions so its
// safe to call in catch blocks
try {
logFile.close();
// Call close only once
shouldClose = false;
}
catch (IOException ioe) {
LOG.error(I/O exception while closing file, ioe);
}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-821) Failing to write lastId to ledger directories should not fail startup of bookies
[
https://issues.apache.org/jira/browse/BOOKKEEPER-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321179#comment-14321179
]
Jia Zhai commented on BOOKKEEPER-821:
-
Thanks for taking care of it.
Failing to write lastId to ledger directories should not fail startup of
bookies
Key: BOOKKEEPER-821
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-821
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Affects Versions: 4.3.0
Reporter: Jia Zhai
Assignee: Jia Zhai
Fix For: 4.4.0, 4.3.2
Attachments: BOOKKEEPER-821_no-prefix.patch
Original Estimate: 5h
Remaining Estimate: 5h
At the startup of bookie, the bookie constructor would call setLastLogId()
as this trace:
Bookie() - InterleavedLedgerStorage() - EntryLogger() - Initialize()
- createNewLog() - allocateNewLog() - setLastLogId().
If bw.write() and bw.flush() in setLastLogId() failed, an IOException would
throw and not catch, and cause Bookie constructor fail.
{code}
private void setLastLogId(File dir, long logId) throws IOException {
FileOutputStream fos;
fos = new FileOutputStream(new File(dir, lastId));
BufferedWriter bw = new BufferedWriter(new
OutputStreamWriter(fos, UTF_8));
try { // original: if write fail in this try,
IOException thrown but not catch, and cause bookie startup fail.
bw.write(Long.toHexString(logId) + \n);
bw.flush();
} finally {
try {
bw.close();
} catch (IOException e) {
LOG.error(Could not close lastId file in {},
dir.getPath());
}
}
}
{code}
But failing setLastLogId() could be tolerated, and will not cause any problem
in next time Bookie startup.
Next time, when calling EntryLogger constructor again, in getLastLogId(), if
read ledgerDir.lastId fail, it will walk through all log files to get
LastLogID; If reading an old ledgerDir.lastId, which caused by last failure
of setLastLogId(), and it happened to be the largest logId, then
allocateNewLog() will find the file already exist, and will allocate
newlogfile with bigger ID.
{code}
BufferedLogChannel allocateNewLog() throws IOException {
ListFile list = ledgerDirsManager.getWritableLedgerDirs();
Collections.shuffle(list);
// It would better not to overwrite existing entry log files
File newLogFile = null;
do {
String logFileName = Long.toHexString(++preallocatedLogId) +
.log;
for (File dir : list) {
newLogFile = new File(dir, logFileName);
currentDir = dir;
if (newLogFile.exists()) { === this will handle last
set fail issue, in which LastId update fail, and get a wrong
preallocatedLogId.
LOG.warn(Found existed entry log + newLogFile
+ when trying to create it as a new log.);
newLogFile = null;
break;
}
}
} while (newLogFile == null);
FileChannel channel = new RandomAccessFile(newLogFile,
rw).getChannel();
BufferedLogChannel logChannel = new BufferedLogChannel(channel,
conf.getWriteBufferBytes(), conf.getReadBufferBytes(),
preallocatedLogId);
logChannel.write((ByteBuffer) LOGFILE_HEADER.clear());
for (File f : list) {
setLastLogId(f, preallocatedLogId);
}
LOG.info(Preallocated entry logger {}., preallocatedLogId);
return logChannel;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321178#comment-14321178 ] Jia Zhai commented on BOOKKEEPER-836: - Thanks for taking care of it. disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.2 Attachments: BOOKKEEPER-836.patch In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-836: Attachment: BOOKKEEPER-836.patch attach patch disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.1 Attachments: BOOKKEEPER-836.patch In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-836: Description: In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. was:In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.1 In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. I would like to change old forced garbage collection logic, and suspend major compaction when it reaches warn threshold, suspend minor compaction when it reaches critical threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai reassigned BOOKKEEPER-836: --- Assignee: Jia Zhai disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0 In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
Jia Zhai created BOOKKEEPER-836: --- Summary: disable compaction when disk becomes full, otherwise compaction will fill up disk quickly Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Fix For: 4.4.0 In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (BOOKKEEPER-836) disable compaction when disk becomes full, otherwise compaction will fill up disk quickly
[ https://issues.apache.org/jira/browse/BOOKKEEPER-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai updated BOOKKEEPER-836: Fix Version/s: 4.3.1 disable compaction when disk becomes full, otherwise compaction will fill up disk quickly - Key: BOOKKEEPER-836 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-836 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.2.3 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0, 4.3.1 In doCompactEntryLogs, Entries are added to new logs, while all old logs were not released until the end of handling. So during the process, a lot of space will be used. Need to disable compaction when disk becomes full, otherwise compaction will fill up disk quickly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BOOKKEEPER-834) test case error in test class TestDiskChecker
[
https://issues.apache.org/jira/browse/BOOKKEEPER-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14278820#comment-14278820
]
Jia Zhai commented on BOOKKEEPER-834:
-
Seems in that case, any value that makes
diskSpaceThreshold diskWarnThreshold
is OK.
No need to do * 1.5f or + 0.01f for diskSpaceThreshold.
test case error in test class TestDiskChecker
-
Key: BOOKKEEPER-834
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-834
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Environment: Linux /tmp dir size 2GB
Reporter: Jia Zhai
Assignee: Ivan Kelly
Fix For: 4.4.0
Attachments:
0001-Fix-TestDiskChecker-so-it-can-be-used-on-dev-shm.patch
In test class TestDiskChecker, test case will fail if tmp dir's usableSpace /
totalSpace 0.95, because threshold will be a negative number.
{code}
public void testCheckDiskFull() throws IOException {
File file = File.createTempFile(DiskCheck, test);
long usableSpace = file.getUsableSpace();
long totalSpace = file.getTotalSpace();
float threshold =
(1f - ((float) usableSpace / (float) totalSpace)) - 0.05f;
=== if usableSpace / totalSpace is 0.99, then threshold is -0.04, this
is invalid.
diskChecker.setDiskSpaceThreshold(threshold, threshold);
diskChecker.checkDiskFull(file);
}
{code}
in my running of test:
1
---
2 Test set: org.apache.bookkeeper.util.TestDiskChecker
3
---
4 Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.557 sec
FAILURE!
5 testCheckDiskFull(org.apache.bookkeeper.util.TestDiskChecker) Time
elapsed: 0.066 sec ERROR!
6 java.lang.Exception: Unexpected exception,
expectedorg.apache.bookkeeper.util.DiskChecker$DiskOutOfSpaceException but
wasjava.lang.IllegalArgumentException
7 at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
8 at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
9 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
10 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
11 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
12 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
13 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
14 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
15 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
16 at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
17 at
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
18 at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
19 at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
20 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
21 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
22 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
23 at java.lang.reflect.Method.invoke(Method.java:606)
24 at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
25 at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
26 at
org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:172)
27 at
org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:78)
28 at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:70)
29 Caused by: java.lang.IllegalArgumentException: Disk space threashold
-0.04741608 is not valid. Should be 0 and 1 ==
30 at
org.apache.bookkeeper.util.DiskChecker.validateThreshold(DiskChecker.java:163)
31 at
org.apache.bookkeeper.util.DiskChecker.setDiskSpaceThreshold(DiskChecker.java:157)
32 at
org.apache.bookkeeper.util.TestDiskChecker.testCheckDiskFull(TestDiskChecker.java:51)
33 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
34 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
35 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
36 at java.lang.reflect.Method.invoke(Method.java:606)
37 at
[jira] [Commented] (BOOKKEEPER-827) change throttle in GarbageCollector to use either by entry or by byte
[ https://issues.apache.org/jira/browse/BOOKKEEPER-827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279574#comment-14279574 ] Jia Zhai commented on BOOKKEEPER-827: - Thanks for the help to take care of it. change throttle in GarbageCollector to use either by entry or by byte - Key: BOOKKEEPER-827 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-827 Project: Bookkeeper Issue Type: Improvement Components: bookkeeper-server Affects Versions: 4.3.0 Reporter: Jia Zhai Assignee: Jia Zhai Fix For: 4.4.0 Attachments: BOOKKEEPER-827-v2.patch, BOOKKEEPER-827-v3.patch, BOOKKEEPER-827-v4.patch Current bookie compaction in GarbageCollector has setting: 'compactionRate'. It is throttling and limiting the compaction by entries. But from a bandwidth perspective, it would be good that we could throttle and limit compaction by bytes, which would really reflect the bandwidth of disk. So in this enhancement, we added another by bytes option when doing compaction in GarbageCollector: boolean isThrottleByBytes: true when use by bytes, false when use by entries; int compactionRateByEntries: by entries, number of concurrent entries; int compactionRateByBytes: by bytes, number of bytes of entries before flush. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
