[jira] [Commented] (BOOKKEEPER-1092) Ledger Recovery - Add Test Case for Parallel Ledger Recovery

2017-06-22 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059384#comment-16059384
 ] 

Hudson commented on BOOKKEEPER-1092:


FAILURE: Integrated in Jenkins build bookkeeper-master #1792 (See 
[https://builds.apache.org/job/bookkeeper-master/1792/])
BOOKKEEPER-1092: Ledger Recovery (Part 5) - Add Test Case for Parallel (sijie: 
rev 256e7587331eddd51b7c05f27717c9d4dc848500)
* (edit) 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperClientStats.java
* (edit) 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerRecoveryOp.java
* (edit) 
bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/CleanupLedgerManager.java
* (add) 
bookkeeper-server/src/test/java/org/apache/bookkeeper/client/ParallelLedgerRecoveryTest.java
* (edit) 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java
* (edit) 
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookkeeperInternalCallbacks.java
* (edit) 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java


> Ledger Recovery - Add Test Case for Parallel Ledger Recovery
> 
>
> Key: BOOKKEEPER-1092
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1092
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Add test case for parallel ledger recovery



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1092) Ledger Recovery - Add Test Case for Parallel Ledger Recovery

2017-06-21 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058029#comment-16058029
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1092:


Github user asfgit closed the pull request at:

https://github.com/apache/bookkeeper/pull/182


> Ledger Recovery - Add Test Case for Parallel Ledger Recovery
> 
>
> Key: BOOKKEEPER-1092
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1092
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Add test case for parallel ledger recovery



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BOOKKEEPER-1092) Ledger Recovery - Add Test Case for Parallel Ledger Recovery

2017-06-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037517#comment-16037517
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1092:


Github user sijie commented on the issue:

https://github.com/apache/bookkeeper/pull/182
  
@merlimat can you review this as well?


> Ledger Recovery - Add Test Case for Parallel Ledger Recovery
> 
>
> Key: BOOKKEEPER-1092
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1092
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Add test case for parallel ledger recovery



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BOOKKEEPER-1092) Ledger Recovery - Add Test Case for Parallel Ledger Recovery

2017-06-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BOOKKEEPER-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035461#comment-16035461
 ] 

ASF GitHub Bot commented on BOOKKEEPER-1092:


GitHub user sijie opened a pull request:

https://github.com/apache/bookkeeper/pull/182

BOOKKEEPER-1092: Ledger Recovery - Add Test Case for Parallel Ledger 
Recovery

This change is based on #180 (ebf7020 is the change for review)

- Add test case for parallel ledger recovery
- More stats for ledger recovery



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sijie/bookkeeper recovery_improvements_part5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bookkeeper/pull/182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #182


commit 3db0b84570dec166855c70105a5cbf6f2ec14ea2
Author: Sijie Guo 
Date:   2014-01-21T00:42:28Z

bookkeeper recovery improvement (part-1): refactor PendingReadOp

this change is the first part of improving bookkeeper recovery. it is 
basically a refactor change, which:

- abstract an interface for LedgerEntryRequest in PendingReadOp
- rename current implementation to SequenceReadRequest, which read the 
entry in the sequence of quorum.

RB_ID=266137

commit 80ffc6ca86b5e8a63dc3b9baf846ad139d7beea1
Author: Sijie Guo 
Date:   2017-06-01T19:11:40Z

Address conflicts

commit f0fb89cbdfed6c6b0d519a25ea2ea67dcd72834e
Author: Sijie Guo 
Date:   2014-01-21T00:49:52Z

bookkeeper recovery improvement (part-2): add a parallel reading request in 
PendingReadOp

- add a parallel reading request in PendingReadOp
- allow PendingReadOp to configure whether to do parallel reading or not
- add flag in ClientConfiguration to allow configuring whether to do 
parallel reading in LedgerRecoveryOp or not.

RB_ID=266139

commit db3e98ba9250b622db80cac8dd6dbd2761a32381
Author: Sijie Guo 
Date:   2017-06-01T20:00:01Z

Address conflicts

commit 8b8a3c8db86a11a30efb14d04b33bfcd461a6f05
Author: Sijie Guo 
Date:   2014-01-21T00:55:30Z

bookkeeper recovery improvement (part-3): add a ReadEntryListener to 
callback on individual request.

- add read entry listener which allow doing batch read, but callback on 
individual entries in sequence. so in recovery op, we could issue batch reads, 
then on each individual callback do add entry and stop when received 
NoSuchEntry.

RB_ID=266143

commit 455afdad38103db978638a9e3f8774623efbb82a
Author: Sijie Guo 
Date:   2014-01-21T01:00:30Z

bookkeeper recovery improvement (part-4): allow batch reading in ledger 
recovery

- enable batch read in ledger recovery, so we could parallel reading to 
improve recovery time.

RB_ID=266145

commit d9b9d98518d8ee0b1951ad2eee8ce28820a4c652
Author: Sijie Guo 
Date:   2017-06-01T22:04:44Z

Address conflicts in merge

commit 8fdab64666977d7b9fc302024ce690b53c981f8b
Author: Sijie Guo 
Date:   2017-06-02T20:36:03Z

bookkeeper: LAC piggyback at read response

- bookie server changes
  * cache maximum lac in file info
  * provide getLastAddConfirmed & setLastAddConfirmed in ledger storage
  * addEntry will set its lac thru setLastAddConfirmed
  * readEntry will read latest lac and piggyback
- client change
  * check whether the response has lac piggybacked, if there is lac 
update lac in its corresponding ledger handle.

commit ebf702070df21b8106c39fde13d861c1a6c5e8e0
Author: Sijie Guo 
Date:   2014-01-21T01:11:52Z

bookkeeper recovery improvement (part-5): parallel ledger metadata update 
with bookie entries recovery proceedure

currently, the ledger recovery procedure is

- read ledger metadata
- set ledger metadata to IN_RECOVERY state
- recover entries
- close ledger metadata.

as 'set ledger metadata to IN_RECOVERY' could be executed in parallel with 
'recover entries'. this change is to explore this kind of parallelism.

RB_ID=266147




> Ledger Recovery - Add Test Case for Parallel Ledger Recovery
> 
>
> Key: BOOKKEEPER-1092
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-1092
> Project: Bookkeeper
>  Issue Type: Sub-task
>  Components: bookkeeper-client
>Reporter: Sijie Guo
>Assignee: Sijie Guo
> Fix For: 4.5.0
>
>
> Add test case for parallel ledger recovery



--
This message was sent by