Re: Question regarding force compaction/GC logic in GarbageCollectorThread.java

2017-02-07 Thread Sijie Guo 
On Tue, Feb 7, 2017 at 10:56 PM, Charan Reddy G 
wrote:

> Thanks Sijie, for the clarification. So going by Twitter's code, you intend
> to run force Major/Minor compaction only once when enableForceGC is called.
> Can you please confirm that (which is different from current community code
> behavior.


I don't think it only force compaction once. When force compaction is
enabled, it will go through major compaction. It will then set the
forceGarabageCollection back to false. If after major compaction, the disk
space is freed, we don't need to do force compaction again. If the disks
are still filled up, the disk monitor will set the flag back to true again.


> Not sure if it was bug in the current community code or it was
> intended to be like that.) Also can you please confirm why it is having
> different behavior in current community code?
>

I am not sure about that. I need to go through the details when I am
porting the change back to the community.


>
> Twitter code has diverged quite a bit from community code, any plans of
> merging changes to the community?
>

I am actively working on that. It has been diverged a lot. so It will take
quite a bit effort for me to port them back.

>
> Thanks,
> Charan
>
> On Tue, Feb 7, 2017 at 10:27 PM, Sijie Guo  wrote:
>
> > The forceGarbageCollection was contributed by Twitter before. I
> remembered
> > we wrapped that into a try-finally block. I just checked Twitter's
> branch.
> > We did wrap that in a try-finally block. https://github.com/
> > twitter/bookkeeper/blob/master/bookkeeper-server/src/
> main/java/org/apache/
> > bookkeeper/bookie/GarbageCollectorThread.java#L563
> >
> > I need to merge that back.
> >
> > - Sijie
> >
> > On Tue, Feb 7, 2017 at 10:22 PM, Charan Reddy G  >
> > wrote:
> >
> >> Hey Sijie,
> >>
> >> Thanks for the response, but didn't get which finally block you are
> >> referring to. Are you saying that before executing 'continue' statement
> in
> >> major compaction 'if' block,  forceGarbageCollection should be set to
> >> false, so that force Major/Minor compaction would be run only once when
> >> enableForceGC is called?
> >>
> >> Thanks,
> >> Charan
> >>
> >> On Feb 7, 2017 10:08 PM, "Sijie Guo"  wrote:
> >>
> >>> I think forceGarabageCollection should be set in a final block. Can't
> >>> remember why it wasn't in current master branch.
> >>>
> >>> - Sijie
> >>>
> >>> On Tue, Feb 7, 2017 at 7:01 PM, Charan Reddy G <
> reddychara...@gmail.com>
> >>> wrote:
> >>>
>  Hi,
> 
>  I'm trying to understand the reason behind "continue;" statement in
>  line 352 of GarbageCollectorThread.java (
> https://github.com/apache/boo
>  kkeeper/blob/master/bookkeeper-server/src/main/java/org/apac
>  he/bookkeeper/bookie/GarbageCollectorThread.java). It is
>  understandable that if we have done majorcompaction then minor
> compaction
>  is not required and it can be skipped, but aren't we missing "
>  forceGarbageCollection.set(false);" in line 362? Or is it supposed to
>  be like that?
> 
>  Basically I'm trying to findout when enableForceGC() is called, is it
>  supposed to trigger and do force GC/Compaction just once or untill it
> gets
>  disabled by calling disableForceGC()?
> 
>  Thanks,
>  Charan
> 
> >>>
> >>>
> >
>

Re: Question regarding force compaction/GC logic in GarbageCollectorThread.java

2017-02-07 Thread Sijie Guo 
The forceGarbageCollection was contributed by Twitter before. I remembered
we wrapped that into a try-finally block. I just checked Twitter's branch.
We did wrap that in a try-finally block.
https://github.com/twitter/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L563

I need to merge that back.

- Sijie

On Tue, Feb 7, 2017 at 10:22 PM, Charan Reddy G 
wrote:

> Hey Sijie,
>
> Thanks for the response, but didn't get which finally block you are
> referring to. Are you saying that before executing 'continue' statement in
> major compaction 'if' block,  forceGarbageCollection should be set to
> false, so that force Major/Minor compaction would be run only once when
> enableForceGC is called?
>
> Thanks,
> Charan
>
> On Feb 7, 2017 10:08 PM, "Sijie Guo"  wrote:
>
>> I think forceGarabageCollection should be set in a final block. Can't
>> remember why it wasn't in current master branch.
>>
>> - Sijie
>>
>> On Tue, Feb 7, 2017 at 7:01 PM, Charan Reddy G 
>> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to understand the reason behind "continue;" statement in line
>>> 352 of GarbageCollectorThread.java (https://github.com/apache/boo
>>> kkeeper/blob/master/bookkeeper-server/src/main/java/org/apac
>>> he/bookkeeper/bookie/GarbageCollectorThread.java). It is understandable
>>> that if we have done majorcompaction then minor compaction is not required
>>> and it can be skipped, but aren't we missing "forceGarbageCollection.
>>> set(false);" in line 362? Or is it supposed to be like that?
>>>
>>> Basically I'm trying to findout when enableForceGC() is called, is it
>>> supposed to trigger and do force GC/Compaction just once or untill it gets
>>> disabled by calling disableForceGC()?
>>>
>>> Thanks,
>>> Charan
>>>
>>
>>

[jira] [Commented] (BOOKKEEPER-998) Increased the max entry size to 5MB

2017-02-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857472#comment-15857472
 ] 

ASF GitHub Bot commented on BOOKKEEPER-998:
---

Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/104
  
I think 1MB limitation was coming before we used netty for the network 
transportation. Increasing it to 5MB sounds okay to me. I don't think it breaks 
any backward compatibility issue. I don't have any strong preference here on 
using a system property or configuration setting for it.

 


> Increased the max entry size to 5MB
> ---
>
> Key: BOOKKEEPER-998
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-998
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>
> In practice we have found the Bookkeeper can handle 5MB entries without 
> issue.  We should make the {{PerChannelBookieClient.MAX_FRAME_LENGTH}}, the 
> {{BookieNettyServer.maxMessageSize}} and the 
> {{EntryLogger.MAX_SANE_ENTRY_SIZE}} size all consistent with one another, and 
> ideally 5MB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: bookkeeper-master-git-pullrequest #209

2017-02-07 Thread Apache Jenkins Server 
See 


Changes:

[evans] BOOKKEEPER-998: Increased the max entry size to 5MB

[evans] BOOKEEPER-998: addressed review comments

--
[...truncated 403 lines...]

---
 T E S T S
---
Running org.apache.bookkeeper.proto.TestBackwardCompatCMS42
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.568 sec - in 
org.apache.bookkeeper.proto.TestBackwardCompatCMS42
Running org.apache.bookkeeper.proto.TestDeathwatcher
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.615 sec - in 
org.apache.bookkeeper.proto.TestDeathwatcher
Running org.apache.bookkeeper.proto.NetworkLessBookieTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.331 sec - in 
org.apache.bookkeeper.proto.NetworkLessBookieTest
Running org.apache.bookkeeper.proto.TestBKStats
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.282 sec - in 
org.apache.bookkeeper.proto.TestBKStats
Running org.apache.bookkeeper.proto.TestPerChannelBookieClient
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.664 sec - in 
org.apache.bookkeeper.proto.TestPerChannelBookieClient
Running org.apache.bookkeeper.conf.NoSystemPropertiesConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.172 sec - in 
org.apache.bookkeeper.conf.NoSystemPropertiesConfigurationTest
Running org.apache.bookkeeper.conf.SystemPropertiesConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.21 sec - in 
org.apache.bookkeeper.conf.SystemPropertiesConfigurationTest
Running org.apache.bookkeeper.zookeeper.TestZooKeeperClient
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.072 sec - in 
org.apache.bookkeeper.zookeeper.TestZooKeeperClient
Running org.apache.bookkeeper.bookie.BookieJournalTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.405 sec - in 
org.apache.bookkeeper.bookie.BookieJournalTest
Running org.apache.bookkeeper.bookie.EntryLogTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.058 sec - in 
org.apache.bookkeeper.bookie.EntryLogTest
Running org.apache.bookkeeper.bookie.IndexCorruptionTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.642 sec - in 
org.apache.bookkeeper.bookie.IndexCorruptionTest
Running org.apache.bookkeeper.bookie.IndexPersistenceMgrTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.595 sec - in 
org.apache.bookkeeper.bookie.IndexPersistenceMgrTest
Running org.apache.bookkeeper.bookie.TestSyncThread
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.219 sec - in 
org.apache.bookkeeper.bookie.TestSyncThread
Running org.apache.bookkeeper.bookie.TestLedgerDirsManager
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.734 sec - in 
org.apache.bookkeeper.bookie.TestLedgerDirsManager
Running org.apache.bookkeeper.bookie.BookieShutdownTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.595 sec - in 
org.apache.bookkeeper.bookie.BookieShutdownTest
Running org.apache.bookkeeper.bookie.UpdateCookieCmdTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.393 sec - in 
org.apache.bookkeeper.bookie.UpdateCookieCmdTest
Running org.apache.bookkeeper.bookie.UpgradeTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.586 sec - in 
org.apache.bookkeeper.bookie.UpgradeTest
Running org.apache.bookkeeper.bookie.TestGcOverreplicatedLedger
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.524 sec - in 
org.apache.bookkeeper.bookie.TestGcOverreplicatedLedger
Running org.apache.bookkeeper.bookie.CookieTest
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.783 sec - in 
org.apache.bookkeeper.bookie.CookieTest
Running org.apache.bookkeeper.bookie.BookieThreadTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.247 sec - in 
org.apache.bookkeeper.bookie.BookieThreadTest
Running org.apache.bookkeeper.bookie.BookieInitializationTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.923 sec - in 
org.apache.bookkeeper.bookie.BookieInitializationTest
Running org.apache.bookkeeper.bookie.LedgerCacheTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.495 sec - in 
org.apache.bookkeeper.bookie.LedgerCacheTest
Running org.apache.bookkeeper.bookie.CreateNewLogTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.032 sec - in 
org.apache.bookkeeper.bookie.CreateNewLogTest
Running org.apache.bookkeeper.bookie.CompactionTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.007 sec - 
in org.apache.bookkeeper.bookie.CompactionTest
Running org.apache.bookkeeper.metastore.TestMetaStore
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.32 sec - in

[jira] [Commented] (BOOKKEEPER-998) Increased the max entry size to 5MB

2017-02-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856815#comment-15856815
 ] 

ASF GitHub Bot commented on BOOKKEEPER-998:
---

Github user revans2 commented on the issue:

https://github.com/apache/bookkeeper/pull/104
  
@eolivelli I added in configs for both the server and the client.  Please 
take a look again.


> Increased the max entry size to 5MB
> ---
>
> Key: BOOKKEEPER-998
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-998
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>
> In practice we have found the Bookkeeper can handle 5MB entries without 
> issue.  We should make the {{PerChannelBookieClient.MAX_FRAME_LENGTH}}, the 
> {{BookieNettyServer.maxMessageSize}} and the 
> {{EntryLogger.MAX_SANE_ENTRY_SIZE}} size all consistent with one another, and 
> ideally 5MB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-998) Increased the max entry size to 5MB

2017-02-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856249#comment-15856249
 ] 

ASF GitHub Bot commented on BOOKKEEPER-998:
---

Github user eolivelli commented on the issue:

https://github.com/apache/bookkeeper/pull/104
  
@revans2 
IMHO it would be better to make it configurable as system property or in 
ServerConfiguration it would be even better, so it would be more "testable".
I'm not aware of some system property naming convention, maybe you can 
suggest one and then the discuss is open.

IMHO the max entry size can be related to the max Netty frame size but 
maybe we can use two different limits. For instance on the wire we need headers
@sijie what to you think ?

I think that at Salesforce they make large use of big entries (as they 
store BLOBs on BK), maybe JV as some number to share

For me, I'm starting a new project which will need to store "huge" (more 
than 10 MB) entries and so I will need to set this limit to a greater value 
that 5MB



> Increased the max entry size to 5MB
> ---
>
> Key: BOOKKEEPER-998
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-998
> Project: Bookkeeper
>  Issue Type: Improvement
>  Components: bookkeeper-server
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>
> In practice we have found the Bookkeeper can handle 5MB entries without 
> issue.  We should make the {{PerChannelBookieClient.MAX_FRAME_LENGTH}}, the 
> {{BookieNettyServer.maxMessageSize}} and the 
> {{EntryLogger.MAX_SANE_ENTRY_SIZE}} size all consistent with one another, and 
> ideally 5MB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [discuss] moving the community meeting to Friday

2017-02-07 Thread Bobby Evans 
Yes I would love to join too.  We missed the last meeting invite.  In general 
we can make time most days, especially if it is in the morning.


- Bobby

On Tuesday, February 7, 2017, 5:21:49 AM CST, Enrico Olivelli 
 wrote:Any news ?
This week the meeting will be on Friday ?

@Bobby
It would be great if some of your new team at Yahoo could join the meeting

2017-01-31 7:44 GMT+01:00 Enrico Olivelli :

> For me Friday is ok too. And 8am pt is ideal for me.
>
> I hope that more guus can attend these useful meetings
>
> Enrico
>
> Il mar 31 gen 2017, 02:17 Sijie Guo  ha scritto:
>
>> It was raised in the other email thread. I'd like to pull that out to
>> discuss here. Is anyone good on moving the community meeting to Friday
>> same
>> time (8am pst)?
>>
>> please give us your options and we can try to coordinate here.
>>
>> - Sijie
>>
> --
>
>
> -- Enrico Olivelli
>

Build failed in Jenkins: bookkeeper-master #1651

2017-02-07 Thread Apache Jenkins Server 
See 

--
[...truncated 375 lines...]
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 114 source files to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ bookkeeper-server 
---
[WARNING] The parameter forkMode is deprecated since version 2.14. Use 
forkCount and reuseForks instead.

---
 T E S T S
---
Running org.apache.bookkeeper.conf.SystemPropertiesConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.181 sec - in 
org.apache.bookkeeper.conf.SystemPropertiesConfigurationTest
Running org.apache.bookkeeper.conf.NoSystemPropertiesConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.162 sec - in 
org.apache.bookkeeper.conf.NoSystemPropertiesConfigurationTest
Running org.apache.bookkeeper.test.AsyncLedgerOpsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.296 sec - in 
org.apache.bookkeeper.test.AsyncLedgerOpsTest
Running org.apache.bookkeeper.test.LedgerDeleteTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.559 sec - in 
org.apache.bookkeeper.test.LedgerDeleteTest
Running org.apache.bookkeeper.test.BookieFailureTest
Tests run: 64, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.156 sec - 
in org.apache.bookkeeper.test.BookieFailureTest
Running org.apache.bookkeeper.test.ReadOnlyBookieTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.463 sec - in 
org.apache.bookkeeper.test.ReadOnlyBookieTest
Running org.apache.bookkeeper.test.LedgerCreateDeleteTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.21 sec - in 
org.apache.bookkeeper.test.LedgerCreateDeleteTest
Running org.apache.bookkeeper.test.BookieJournalRollingTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.985 sec - in 
org.apache.bookkeeper.test.BookieJournalRollingTest
Running org.apache.bookkeeper.test.MultipleThreadReadTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.693 sec - in 
org.apache.bookkeeper.test.MultipleThreadReadTest
Running org.apache.bookkeeper.test.BookieZKExpireTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.88 sec - in 
org.apache.bookkeeper.test.BookieZKExpireTest
Running org.apache.bookkeeper.test.TestBackwardCompat
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.503 sec - in 
org.apache.bookkeeper.test.TestBackwardCompat
Running org.apache.bookkeeper.test.BookieClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.872 sec - in 
org.apache.bookkeeper.test.BookieClientTest
Running org.apache.bookkeeper.test.ConditionalSetTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.885 sec - in 
org.apache.bookkeeper.test.ConditionalSetTest
Running org.apache.bookkeeper.test.ConcurrentLedgerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.433 sec - in 
org.apache.bookkeeper.test.ConcurrentLedgerTest
Running org.apache.bookkeeper.test.ForceReadOnlyBookieTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.552 sec - in 
org.apache.bookkeeper.test.ForceReadOnlyBookieTest
Running org.apache.bookkeeper.test.BookieReadWriteTest
Tests run: 104, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.424 sec - 
in org.apache.bookkeeper.test.BookieReadWriteTest
Running org.apache.bookkeeper.test.CloseTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.443 sec - in 
org.apache.bookkeeper.test.CloseTest
Running org.apache.bookkeeper.test.ConfigurationTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.191 sec - in 
org.apache.bookkeeper.test.ConfigurationTest
Running org.apache.bookkeeper.util.TestDiskChecker
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.181 sec - in 
org.apache.bookkeeper.util.TestDiskChecker
Running org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMapTest
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.914 sec - in 
org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMapTest
Running org.apache.bookkeeper.util.collections.ConcurrentOpenHashSetTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.019 sec - in 
org.apache.bookkeeper.util.collections.ConcurrentOpenHashSetTest
Running org.apache.bookkeeper.util.collections.ConcurrentLongHashSetTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.774 sec - in 
org.apache.bookkeeper.util.collections.ConcurrentLongHashSetTest
Running org.apache.bookkeeper.util.collections.ConcurrentOpenHashMapTest
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.294 sec - in

Re: [discuss] moving the community meeting to Friday

2017-02-07 Thread Enrico Olivelli 
Any news ?
This week the meeting will be on Friday ?

@Bobby
It would be great if some of your new team at Yahoo could join the meeting

2017-01-31 7:44 GMT+01:00 Enrico Olivelli :

> For me Friday is ok too. And 8am pt is ideal for me.
>
> I hope that more guus can attend these useful meetings
>
> Enrico
>
> Il mar 31 gen 2017, 02:17 Sijie Guo  ha scritto:
>
>> It was raised in the other email thread. I'd like to pull that out to
>> discuss here. Is anyone good on moving the community meeting to Friday
>> same
>> time (8am pst)?
>>
>> please give us your options and we can try to coordinate here.
>>
>> - Sijie
>>
> --
>
>
> -- Enrico Olivelli
>