[GitHub] sijie commented on issue #1814: Introduce condensed encoding format for availabilityOfEntriesOfLedger

[GitBox]

sijie commented on issue #1814: Introduce condensed encoding format for 
availabilityOfEntriesOfLedger
URL: https://github.com/apache/bookkeeper/pull/1814#issuecomment-439586293
 
 
   run bookkeeper-server bookie tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nicmichael commented on issue #1808: Allow to configure sticky reads

[GitBox]

nicmichael commented on issue #1808: Allow to configure sticky reads
URL: https://github.com/apache/bookkeeper/pull/1808#issuecomment-439558592
 
 
   I actually like @dlg99's suggestion: if each W adjacent entries have the 
same primary bookie, but the next W entries another primary bookie, we'd get W 
times better data locality but still evenly distribute reads across all bookies 
in E. Plus it would work for E>W. Ideally E and W are prime (e.g. E=3, W=7), 
then the primary bookie would be easy to calculate and guaranteed to be evenly 
distributed.
   
   If even distribution is guaranteed, this policy wouldn't even need a flag to 
turn it on/off. It could be the default policy as it wouldn't harm random reads 
(due to even distribution across all bookies in E) but benefit sequential reads 
(at least in the absence of out-of-order writes and entry logs per ledger...).
   
   The benefits of read-ahead anyways depend on the size of the entries and 
file system configuration. Unless entries are tiny, a policy like this that 
benefits from read-ahead of W entries might bring most of the performance gain 
(and a stickiness of the entire ledger to just one bookie might be marginally 
better for sequential reads).
   
   I suppose reordering of reads (e.g. due to slow bookies or number of 
outstanding requests to a bookie, Issue #1489) would still kick in if enabled, 
and rearrange the order if a primary bookie is found to be slow?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1814: Introduce condensed encoding format for availabilityOfEntriesOfLedger

[GitBox]

sijie commented on issue #1814: Introduce condensed encoding format for 
availabilityOfEntriesOfLedger
URL: https://github.com/apache/bookkeeper/pull/1814#issuecomment-439547096
 
 
   run bookkeeper-server bookie tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1812: Allow to configure BK for low latency busy-wait settings

[GitBox]

sijie commented on a change in pull request #1812: Allow to configure BK for 
low latency busy-wait settings
URL: https://github.com/apache/bookkeeper/pull/1812#discussion_r234364898
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java
 ##
 @@ -602,8 +604,7 @@ BookieWatcher getBookieWatcher() {
 return bookieWatcher;
 }
 
-@VisibleForTesting
-OrderedExecutor getMainWorkerPool() {
+public OrderedExecutor getMainWorkerPool() {
 
 Review comment:
   any reason why `@VisibleForTesting` annotation is removed?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1812: Allow to configure BK for low latency busy-wait settings

[GitBox]

sijie commented on a change in pull request #1812: Allow to configure BK for 
low latency busy-wait settings
URL: https://github.com/apache/bookkeeper/pull/1812#discussion_r234364746
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Journal.java
 ##
 @@ -641,6 +652,16 @@ public Journal(int journalIndex, File journalDirectory, 
ServerConfiguration conf
 public Journal(int journalIndex, File journalDirectory, 
ServerConfiguration conf,
 LedgerDirsManager ledgerDirsManager, StatsLogger statsLogger) {
 super("BookieJournal-" + conf.getBookiePort());
+
+if (conf.isBusyWaitEnabled()) {
+// To achieve lower latency, use busy-wait blocking queue 
implementation
+queue = new BlockingMpscQueue<>(16 * 1024);
 
 Review comment:
   can we make this setting configurable?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1812: Allow to configure BK for low latency busy-wait settings

[GitBox]

sijie commented on a change in pull request #1812: Allow to configure BK for 
low latency busy-wait settings
URL: https://github.com/apache/bookkeeper/pull/1812#discussion_r234364632
 
 

 ##
 File path: 
bookkeeper-common/src/main/java/org/apache/bookkeeper/common/util/OrderedExecutor.java
 ##
 @@ -380,6 +400,15 @@ protected OrderedExecutor(String baseName, int 
numThreads, ThreadFactory threadF
 try {
 threads[idx].submit(() -> {
 threadIds[idx] = Thread.currentThread().getId();
+
+if (enableBusyWait) {
+try {
+CpuAffinity.acquireCore();
+} catch (Throwable t) {
+log.warn("Failed to acquire CPU core for thread 
{}", Thread.currentThread().getName(),
 
 Review comment:
   need some clarifications (comments) here: what happen if we failed to 
acquire core.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1812: Allow to configure BK for low latency busy-wait settings

[GitBox]

sijie commented on a change in pull request #1812: Allow to configure BK for 
low latency busy-wait settings
URL: https://github.com/apache/bookkeeper/pull/1812#discussion_r234364502
 
 

 ##
 File path: 
bookkeeper-common/src/main/java/org/apache/bookkeeper/common/util/OrderedExecutor.java
 ##
 @@ -277,7 +287,15 @@ public T call() throws Exception {
 }
 
 protected ThreadPoolExecutor createSingleThreadExecutor(ThreadFactory 
factory) {
-return new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new 
LinkedBlockingQueue<>(), factory);
+BlockingQueue queue;
+if (enableBusyWait) {
+// Use queue with busy-wait polling strategy
+queue = new BlockingMpscQueue<>(1);
 
 Review comment:
   can we make `1` configurable?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1809: Change LedgerManager to use CompletableFuture

[GitBox]

sijie commented on issue #1809: Change LedgerManager to use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#issuecomment-439545892
 
 
   run bookkeeper-server replication tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1809: Change LedgerManager to use CompletableFuture

[GitBox]

sijie commented on issue #1809: Change LedgerManager to use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#issuecomment-439545823
 
 
   run bookkeeper replication tests
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1755: Configure Netty allocator in bookie and client

[GitBox]

sijie commented on issue #1755:  Configure Netty allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#issuecomment-439544955
 
 
   @merlimat can you rebase this pull request to latest master?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #1807: [bookie] Fix sorted ledger storage rotating entry log files too frequent

[GitBox]

sijie closed pull request #1807: [bookie] Fix sorted ledger storage rotating 
entry log files too frequent
URL: https://github.com/apache/bookkeeper/pull/1807
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerBase.java
 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerBase.java
index 701fb7b2e8..f9c6d97cfd 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerBase.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerBase.java
@@ -21,6 +21,9 @@
 
 package org.apache.bookkeeper.bookie;
 
+import static org.apache.bookkeeper.bookie.EntryLogger.UNASSIGNED_LEDGERID;
+
+import com.google.common.annotations.VisibleForTesting;
 import io.netty.buffer.ByteBuf;
 import io.netty.buffer.Unpooled;
 import io.netty.util.concurrent.FastThreadLocal;
@@ -131,7 +134,18 @@ void flushLogChannel(BufferedLogChannel logChannel, 
boolean forceMetadata) throw
  * Creates a new log file. This method should be guarded by a lock,
  * so callers of this method should be in right scope of the lock.
  */
+@VisibleForTesting
 void createNewLog(long ledgerId) throws IOException {
+createNewLog(ledgerId, "");
+}
+
+void createNewLog(long ledgerId, String reason) throws IOException {
+if (ledgerId != UNASSIGNED_LEDGERID) {
+log.info("Creating a new entry log file for ledger '{}' {}", 
ledgerId, reason);
+} else {
+log.info("Creating a new entry log file {}", reason);
+}
+
 BufferedLogChannel logChannel = getCurrentLogForLedger(ledgerId);
 // first tried to create a new log channel. add current log channel to 
ToFlush list only when
 // there is a new log channel. it would prevent that a log channel is 
referenced by both
diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForEntryLogPerLedger.java
 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForEntryLogPerLedger.java
index 39ed60cea5..8093b53b15 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForEntryLogPerLedger.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForEntryLogPerLedger.java
@@ -484,7 +484,7 @@ public boolean commitEntryMemTableFlush() throws 
IOException {
 try {
 if (reachEntryLogLimit(currentLog, 0L)) {
 log.info("Rolling entry logger since it reached size 
limitation for ledger: {}", ledgerId);
-createNewLog(ledgerId);
+createNewLog(ledgerId, "after entry log file is 
rotated");
 }
 } finally {
 lock.unlock();
@@ -640,7 +640,9 @@ BufferedLogChannel getCurrentLogForLedgerForAddEntry(long 
ledgerId, int entrySiz
 if (logChannel != null) {
 logChannel.flushAndForceWriteIfRegularFlush(false);
 }
-createNewLog(ledgerId);
+createNewLog(ledgerId,
+": diskFull = " + diskFull + ", allDisksFull = " + 
allDisksFull
++ ", reachEntryLogLimit = " + reachEntryLogLimit + ", 
logChannel = " + logChannel);
 }
 
 return getCurrentLogForLedger(ledgerId);
diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForSingleEntryLog.java
 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForSingleEntryLog.java
index 3e552d0fca..72c818a30c 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForSingleEntryLog.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogManagerForSingleEntryLog.java
@@ -92,7 +92,7 @@ synchronized BufferedLogChannel 
getCurrentLogForLedgerForAddEntry(long ledgerId,
 boolean rollLog) throws IOException {
 if (null == activeLogChannel) {
 // log channel can be null because the file is deferred to be 
created
-createNewLog(UNASSIGNED_LEDGERID);
+createNewLog(UNASSIGNED_LEDGERID, "because current active log 
channel has not initialized yet");
 }
 
 boolean reachEntryLogLimit = rollLog ? 
reachEntryLogLimit(activeLogChannel, entrySize)
@@ -103,7 +103,8 @@ synchronized BufferedLogChannel 
getCurrentLogForLedgerForAddEntry(long ledgerId,
 if (activeLogChannel != null) {
 activeLogChannel.flushAndForceWriteIfRegularFlush(false);
 }
-

[GitHub] sijie commented on issue #1807: [bookie] Fix sorted ledger storage rotating entry log files too frequent

[GitBox]

sijie commented on issue #1807: [bookie] Fix sorted ledger storage rotating 
entry log files too frequent
URL: https://github.com/apache/bookkeeper/pull/1807#issuecomment-439544616
 
 
   IGNORE IT CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #1816: Fix cpu-affinity module build on jdk10+

[GitBox]

sijie closed pull request #1816: Fix cpu-affinity module build on jdk10+
URL: https://github.com/apache/bookkeeper/pull/1816
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/cpu-affinity/pom.xml b/cpu-affinity/pom.xml
index f58a07991e..1439259f66 100644
--- a/cpu-affinity/pom.xml
+++ b/cpu-affinity/pom.xml
@@ -73,6 +73,7 @@
   
 org.apache.maven.plugins
 maven-assembly-plugin
+${maven-assembly-plugin.version}
 
   
 src/main/assembly/assembly.xml
@@ -104,6 +105,53 @@
   
 
   
+
+  
+  jdk-without-javah
+  
+ [10,)
+  
+  
+
+  
+com.github.maven-nar
+nar-maven-plugin
+${nar-maven-plugin.version}
+true
+
+   
+   
+  default-nar-javah
+  none
+   
+ 
+  
+  
+ org.apache.maven.plugins
+ maven-compiler-plugin
+ ${maven-compiler-plugin.version}
+ 
+1.8
+1.8
+
+  
+  
+  -Xlint:deprecation
+  -Xlint:unchecked
+   
+  -Xpkginfo:always
+  
+  -h
+  
${project.build.directory}/nar/javah-include
+
+ 
+  
+
+
+
 
   mac
   


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1816: Fix cpu-affinity module build on jdk10+

[GitBox]

sijie commented on issue #1816: Fix cpu-affinity module build on jdk10+
URL: https://github.com/apache/bookkeeper/pull/1816#issuecomment-439544299
 
 
   IGNORE IT CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1813: Set default sizes of DbLedgerStorage read and write cache to be propo…

[GitBox]

sijie commented on issue #1813: Set default sizes of DbLedgerStorage read and 
write cache to be propo…
URL: https://github.com/apache/bookkeeper/pull/1813#issuecomment-439543810
 
 
   the reason why it is in blacklist is internal classes are not encouraged to 
use in general, if there are exceptions, those imports should be highlighted 
with these `checkstyle` annotations. so we know these packages are used 
intentionally. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1813: Set default sizes of DbLedgerStorage read and write cache to be propo…

[GitBox]

eolivelli commented on a change in pull request #1813: Set default sizes of 
DbLedgerStorage read and write cache to be propo…
URL: https://github.com/apache/bookkeeper/pull/1813#discussion_r234360476
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
 ##
 @@ -28,6 +28,9 @@
 
 import io.netty.buffer.ByteBuf;
 import io.netty.util.concurrent.DefaultThreadFactory;
+//CHECKSTYLE.OFF: IllegalImport
 
 Review comment:
   Okay


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1813: Set default sizes of DbLedgerStorage read and write cache to be propo…

[GitBox]

merlimat commented on issue #1813: Set default sizes of DbLedgerStorage read 
and write cache to be propo…
URL: https://github.com/apache/bookkeeper/pull/1813#issuecomment-439541827
 
 
   > What do you think?
   Do you know why it was blacklisted?
   
   I have no idea why it was blacklisted. I guess `io.netty.util.internal` 
could be regarded as non-public API package. Maybe it's because there are also 
some shaded classes under that package.
   
   In my opinion, the override here is fine. We should only using things from 
`io.netty.util.internal` if there's a good reason for it and no other way 
around.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1813: Set default sizes of DbLedgerStorage read and write cache to be propo…

[GitBox]

merlimat commented on a change in pull request #1813: Set default sizes of 
DbLedgerStorage read and write cache to be propo…
URL: https://github.com/apache/bookkeeper/pull/1813#discussion_r234294705
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
 ##
 @@ -28,6 +28,9 @@
 
 import io.netty.buffer.ByteBuf;
 import io.netty.util.concurrent.DefaultThreadFactory;
+//CHECKSTYLE.OFF: IllegalImport
 
 Review comment:
   It was marked here: 
   
   
https://github.com/apache/bookkeeper/blob/28dee8464764b0edceeb31ce24424f0947b5789b/buildtools/src/main/resources/bookkeeper/checkstyle.xml#L104-L106


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1813: Set default sizes of DbLedgerStorage read and write cache to be propo…

[GitBox]

eolivelli commented on a change in pull request #1813: Set default sizes of 
DbLedgerStorage read and write cache to be propo…
URL: https://github.com/apache/bookkeeper/pull/1813#discussion_r234292332
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
 ##
 @@ -28,6 +28,9 @@
 
 import io.netty.buffer.ByteBuf;
 import io.netty.util.concurrent.DefaultThreadFactory;
+//CHECKSTYLE.OFF: IllegalImport
 
 Review comment:
   Sorry, why is it illegal?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1813: Set default sizes of DbLedgerStorage read and write cache to be propo…

[GitBox]

merlimat commented on a change in pull request #1813: Set default sizes of 
DbLedgerStorage read and write cache to be propo…
URL: https://github.com/apache/bookkeeper/pull/1813#discussion_r234287351
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/DbLedgerStorage.java
 ##
 @@ -28,6 +28,9 @@
 
 import io.netty.buffer.ByteBuf;
 import io.netty.util.concurrent.DefaultThreadFactory;
+//CHECKSTYLE.OFF: IllegalImport
 
 Review comment:
   @eolivelli This is because `io.netty.util.internal.PlatformDependent` is 
triggering the illegal import error on checkstyle. The comment is to turn the 
check off for this import.
   
   I'm using the `PlatformDependent.maxDirectMemory()` because it's most 
reliable way to get that parameter across platforms and JVMs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #1809: Change LedgerManager to use CompletableFuture

[GitBox]

ivankelly commented on a change in pull request #1809: Change LedgerManager to 
use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#discussion_r234236380
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java
 ##
 @@ -439,12 +439,22 @@ public BKLedgerIdOverflowException() {
  * Extract an exception code from an BKException, or use a default if it's 
another type.
  */
 public static int getExceptionCode(Throwable t, int defaultCode) {
-if (t instanceof BKException) {
+if (t == null) {
+return BKException.Code.OK;
+} else if (t instanceof BKException) {
 return ((BKException) t).getCode();
 } else if (t.getCause() != null) {
 
 Review comment:
   You'll get CompletionException wrapping, if the future you put your handler 
is not the future on which completeExceptionally is called.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build became unstable: bookkeeper_codecoverage #271

[Apache Jenkins Server]

See 

Jenkins build is still unstable: bookkeeper_release_branch_48_java8 #93

[Apache Jenkins Server]

See 

[GitHub] ivankelly commented on a change in pull request #1809: Change LedgerManager to use CompletableFuture

[GitBox]

ivankelly commented on a change in pull request #1809: Change LedgerManager to 
use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#discussion_r234229005
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java
 ##
 @@ -491,26 +489,24 @@ public void asyncGetLedgersContainBookies(final 
Set bookies
 bkc.getLedgerManager().asyncProcessLedgers(new Processor() {
 @Override
 public void process(final Long lid, final 
AsyncCallback.VoidCallback cb) {
-bkc.getLedgerManager().readLedgerMetadata(lid, new 
GenericCallback>() {
-@Override
-public void operationComplete(int rc, 
Versioned metadata) {
-if (BKException.Code.NoSuchLedgerExistsException == 
rc) {
-// the ledger was deleted during this iteration.
-cb.processResult(BKException.Code.OK, null, null);
-return;
-} else if (BKException.Code.OK != rc) {
-cb.processResult(rc, null, null);
-return;
-}
-Set bookiesInLedger = 
metadata.getValue().getBookiesInThisLedger();
-Sets.SetView intersection =
+bkc.getLedgerManager().readLedgerMetadata(lid)
+.whenComplete((metadata, exception) -> {
+if (exception instanceof 
BKException.BKNoSuchLedgerExistsException) {
 
 Review comment:
   No, shouldn't be, but it's dependent on how readLedgerMetadata is 
implemented, so will change to getCode


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #1809: Change LedgerManager to use CompletableFuture

[GitBox]

ivankelly commented on a change in pull request #1809: Change LedgerManager to 
use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#discussion_r234228770
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java
 ##
 @@ -439,12 +439,22 @@ public BKLedgerIdOverflowException() {
  * Extract an exception code from an BKException, or use a default if it's 
another type.
  */
 public static int getExceptionCode(Throwable t, int defaultCode) {
-if (t instanceof BKException) {
+if (t == null) {
 
 Review comment:
   Will add to javadoc


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #1809: Change LedgerManager to use CompletableFuture

[GitBox]

ivankelly commented on a change in pull request #1809: Change LedgerManager to 
use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#discussion_r234228416
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieShell.java
 ##
 @@ -1024,41 +1022,45 @@ public int runCmd(CommandLine cmdLine) throws 
Exception {
 final CountDownLatch processDone = new CountDownLatch(1);
 
 Processor ledgerProcessor = new Processor() {
-@Override
-public void process(Long ledgerId, VoidCallback cb) {
-if (!printMeta && (bookieAddress == null)) {
-printLedgerMetadata(ledgerId, null, false);
-cb.processResult(BKException.Code.OK, null, 
null);
-} else {
-GenericCallback> 
gencb = (rc, ledgerMetadata) -> {
-if (rc == BKException.Code.OK) {
-if ((bookieAddress == null)
-|| 
BookKeeperAdmin.areEntriesOfLedgerStoredInTheBookie(
-ledgerId, bookieAddress, 
ledgerMetadata.getValue())) {
-/*
- * the print method has to be in
- * synchronized scope, otherwise
- * output of printLedgerMetadata
- * could interleave since this
- * callback for different
- * ledgers can happen in
- * different threads.
- */
-synchronized (BookieShell.this) {
-printLedgerMetadata(ledgerId, 
ledgerMetadata.getValue(), printMeta);
-}
-}
-} else if (rc == 
BKException.Code.NoSuchLedgerExistsException) {
-rc = BKException.Code.OK;
-} else {
-LOG.error("Unable to read the ledger: 
" + ledgerId + " information");
-}
-cb.processResult(rc, null, null);
-};
-ledgerManager.readLedgerMetadata(ledgerId, 
gencb);
+@Override
+public void process(Long ledgerId, VoidCallback 
cb) {
+if (!printMeta && (bookieAddress == null)) {
+printLedgerMetadata(ledgerId, null, false);
+cb.processResult(BKException.Code.OK, 
null, null);
+} else {
+
ledgerManager.readLedgerMetadata(ledgerId).whenComplete(
+(metadata, exception) -> {
 
 Review comment:
   in this case, it would be a BKException, though when exceptions are wrapped 
and when they are not is not straightforward. 
   
   If I have an future X and I call completeExceptionally on it, all handlers 
for that exception will get the raw exception. Any futures which have been 
created when installing the handlers will get the wrapped exception. 
   
   In any case, I'm going remove all the instanceof and just pull the error 
codes out


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build is still unstable: bookkeeper_release_branch_48_java9 #93

[Apache Jenkins Server]

See 

[GitHub] ivankelly commented on a change in pull request #1809: Change LedgerManager to use CompletableFuture

[GitBox]

ivankelly commented on a change in pull request #1809: Change LedgerManager to 
use CompletableFuture
URL: https://github.com/apache/bookkeeper/pull/1809#discussion_r234209071
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/CleanupLedgerManager.java
 ##
 @@ -108,64 +115,69 @@ private GenericCallback removeCallback(GenericCallback 
callback) {
 return callbacks.remove(callback);
 }
 
+private void recordPromise(CompletableFuture promise) {
 
 Review comment:
   the underlying implementations don't handle it though. for them to handle 
it, each would have to have this recording logic. so it's cleaner to have it 
abstracted somewhere else.
   
   The reason I did it in this patch is so this patch doesn't remove 
functionality. Currently, the cleanup lm ensures that if the lm is closed, then 
all outstanding requests are completed. If we don't implement this here, then 
this functionality is lost.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Build failed in Jenkins: bookkeeper_release_branch_48_integrationtests #93

[Apache Jenkins Server]

See 


--
[...truncated 1.42 MB...]
2018-11-16\T\13:18:13.926 [INFO] 
2018-11-16\T\13:18:13.926 [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ old-cookie-new-cluster ---
2018-11-16\T\13:18:13.927 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-11-16\T\13:18:13.927 [INFO] Copying 1 resource
2018-11-16\T\13:18:13.927 [INFO] Copying 3 resources
2018-11-16\T\13:18:13.928 [INFO] 
2018-11-16\T\13:18:13.928 [INFO] --- maven-compiler-plugin:3.7.0:testCompile 
(default-testCompile) @ old-cookie-new-cluster ---
2018-11-16\T\13:18:13.935 [INFO] Changes detected - recompiling the module!
2018-11-16\T\13:18:13.937 [INFO] Using Groovy-Eclipse compiler to compile both 
Java and Groovy files
2018-11-16\T\13:18:14.252 [INFO] 
2018-11-16\T\13:18:14.252 [INFO] --- maven-surefire-plugin:2.8.1:test 
(default-test) @ old-cookie-new-cluster ---
2018-11-16\T\13:18:14.256 [INFO] Surefire report directory: 


---
 T E S T S
---
Running 
org.apache.bookkeeper.tests.backwardcompat.TestCompatUpgradeOldServerInClusterWithCookies
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.113 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

2018-11-16\T\13:19:56.661 [INFO] 
2018-11-16\T\13:19:56.661 [INFO] 

2018-11-16\T\13:19:56.661 [INFO] Building Apache BookKeeper :: Tests :: 
Backward Compatibility :: Test old clients working on current server 
4.8.1-SNAPSHOT
2018-11-16\T\13:19:56.661 [INFO] 

2018-11-16\T\13:19:56.692 [INFO] 
2018-11-16\T\13:19:56.692 [INFO] --- 
groovy-eclipse-compiler:2.9.2-04:add-groovy-build-paths 
(default-add-groovy-build-paths) @ backward-compat-current-server-old-clients 
---
2018-11-16\T\13:19:56.693 [INFO] Adding /src/main/groovy to the list of source 
folders
2018-11-16\T\13:19:56.693 [INFO] Adding /src/test/groovy to the list of test 
source folders
2018-11-16\T\13:19:56.693 [INFO] 
2018-11-16\T\13:19:56.693 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ backward-compat-current-server-old-clients ---
2018-11-16\T\13:19:58.577 [INFO] 
2018-11-16\T\13:19:58.578 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ backward-compat-current-server-old-clients ---
2018-11-16\T\13:19:58.580 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-11-16\T\13:19:58.580 [INFO] skip non existing resourceDirectory 

2018-11-16\T\13:19:58.580 [INFO] Copying 3 resources
2018-11-16\T\13:19:58.581 [INFO] 
2018-11-16\T\13:19:58.581 [INFO] --- maven-compiler-plugin:3.7.0:compile 
(default-compile) @ backward-compat-current-server-old-clients ---
2018-11-16\T\13:19:58.583 [INFO] No sources to compile
2018-11-16\T\13:19:58.583 [INFO] 
2018-11-16\T\13:19:58.583 [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ backward-compat-current-server-old-clients ---
2018-11-16\T\13:19:58.584 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-11-16\T\13:19:58.584 [INFO] Copying 1 resource
2018-11-16\T\13:19:58.585 [INFO] Copying 3 resources
2018-11-16\T\13:19:58.585 [INFO] 
2018-11-16\T\13:19:58.585 [INFO] --- maven-compiler-plugin:3.7.0:testCompile 
(default-testCompile) @ backward-compat-current-server-old-clients ---
2018-11-16\T\13:19:58.591 [INFO] Changes detected - recompiling the module!
2018-11-16\T\13:19:58.592 [INFO] Using Groovy-Eclipse compiler to compile both 
Java and Groovy files
2018-11-16\T\13:19:58.917 [INFO] 
2018-11-16\T\13:19:58.917 [INFO] --- maven-surefire-plugin:2.8.1:test 
(default-test) @ backward-compat-current-server-old-clients ---
2018-11-16\T\13:19:58.920 [INFO] Surefire report directory: 


---
 T E S T S
---
Running org.apache.bookkeeper.tests.backwardcompat.TestCompatOldClients
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 362.495 sec

Results :

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

2018-11-16\T\13:26:39.456 [INFO] 
2018-11-16\T\13:26:39.457 [INFO] 

2018-11-16\T\13:26:39.457 [INFO] Building Apache 

Build failed in Jenkins: bookkeeper_release_nightly_snapshot #196

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H33 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/bookkeeper.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/bookkeeper.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/bookkeeper.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5997963d5b68d6fc6d341352fb40562eb92dab2b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5997963d5b68d6fc6d341352fb40562eb92dab2b
Commit message: "Added BlockingQueue implementation based on JCtools"
 > git rev-list --no-walk 5997963d5b68d6fc6d341352fb40562eb92dab2b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
FATAL: Command "git clean -fdx" returned status code 1:
stdout: 
stderr: warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/SOURCES.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/namespace_packages.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/PKG-INFO
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/top_level.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/requires.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/dependency_links.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/not-zip-safe
warning: failed to remove stream/clients/python/build/lib/bookkeeper/types.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/stream_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/storage_pb2_grpc.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/common_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_rpc_pb2_grpc.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_store_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_rpc_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/cluster_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/storage_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/namespace.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/exceptions.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/client.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/namespaces.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/__init__.py
warning: failed to remove stream/clients/python/build/lib/bookkeeper/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/exceptions.py
warning: failed to remove stream/clients/python/build/lib/bookkeeper/kv/table.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/client.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/futures.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/timeout.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/exceptions.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/constants.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/polling.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/_helpers.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/base.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/method.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/grpc_helpers.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/datetime_helpers.py
warning: failed to remove 

Build failed in Jenkins: bookkeeper_release_branch_47_integrationtests #208

[Apache Jenkins Server]

See 


--
[...truncated 12.22 KB...]
message+  1120 1  1120   Jul 25  0.0  0.0 dbus-daemon --system --fork
root  1132 1  1132   Jul 25  0.0  0.0 /lib/systemd/systemd --system
root  1158 1  1158   Jul 25  0.0  0.0 /lib/systemd/systemd-logind
root  1207 1  1207   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty4
root  1211 1  1211   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty5
root  1221 1  1221   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty2
root  1222 1  1222   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty3
root  1224 1  1224   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty6
daemon1255 1  1255   Jul 25  0.0  0.0 atd
root  1256 1  1256   Jul 25  0.0  0.0 cron
root  1289  1132  1289   Jul 25  0.0  0.3 /lib/systemd/systemd-journald
root  1290 2 0   Jul 25  0.0  0.0 [kauditd]
root  1311 1  1311   Jul 25  0.0  0.0 /usr/sbin/irqbalance
root  1493 2 0   Jul 25  0.0  0.0 [kworker/15:1H]
root  1703 1  1703   Oct 04  0.0  0.0 /usr/bin/ruby /usr/bin/puppet 
agent
root  1801 1  1801   Jul 25  0.0  0.0 /usr/lib/postfix/master
postfix   1823  1801  1801   Jul 25  0.0  0.0 qmgr -l -t fifo -u
ntp   1915 1  1915   Jul 25  0.0  0.0 /usr/sbin/ntpd -p 
/var/run/ntpd.pid -g -u 105:116
root  2012 1  2011   Jul 25  0.0  0.0 /usr/bin/python 
/usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p 
/var/run/fail2ban/fail2ban.pid
root  2072 1  2072   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty1
root  2073 1  2073   Jul 25  0.0  0.0 /sbin/getty -L ttyS0 9600 vt102
root  2285 2 0   Nov 10  0.0  0.0 [kworker/8:1]
root  2378 2 0   Jul 25  0.0  0.0 [kworker/10:1H]
nslcd 3312 1  3312   Jul 25  0.1  0.0 /usr/sbin/nslcd
root  3322 2 0   Nov 14  0.0  0.0 [kworker/4:0]
root  3617 1  3617   Oct 31  0.0  0.0 sshd: jenkins [priv]
jenkins   3651  3617  3617   Oct 31  0.0  0.0 sshd: jenkins@notty 
jenkins   3686  3651  3686   Oct 31  0.0  0.0 bash -c cd 
"/home/jenkins/jenkins-slave" && /home/jenkins/tools/java/jdk1.8.0_131/bin/java 
-Xmx2g -Xms512m -jar slave.jar
jenkins   3687  3686  3686   Oct 31  1.2  3.1 
/home/jenkins/tools/java/jdk1.8.0_131/bin/java -Xmx2g -Xms512m -jar slave.jar
root  3763 2 0   Nov 14  0.0  0.0 [kworker/5:0]
jenkins   4416 1 21137   Aug 21  0.0  0.0 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefirebooter123717442526029744.jar
 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefire4123165900408045001tmp
 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefire_06109285983847161674tmp
jenkins   4455  4416 21137   Aug 21  0.7  0.5 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx2048m -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/dependency/org.apache.sling.launchpad-8.jar
 -p 45800 -Dsling.run.modes=author,notshared
root  4580 2 0   Nov 14  0.0  0.0 [kworker/0:0]
root  4927 1  1120   Jul 25  0.0  0.0 
/usr/lib/x86_64-linux-gnu/systemd-shim
root  4977 2 0   Jul 25  0.0  0.0 [kworker/13:1H]
root  5140 2 0   Nov 14  0.0  0.0 [kworker/6:0]
root  5661 2 0   Jul 25  0.0  0.0 [kworker/11:1H]
root  5693 2 0   Nov 14  0.0  0.0 [kworker/10:2]
root  6009 2 0   Jul 25  0.0  0.0 [kworker/9:1H]
jenkins   7335  4416 21137   Aug 21  104  3.3 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx2048m -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/dependency/org.apache.sling.launchpad-8.jar
 -p 41776 -Dsling.run.modes=publish,notshared
root  8603 2 0   Nov 15  0.0  0.0 [kworker/2:2]
root  8647 2 0   Nov 15  0.0  0.0 [kworker/7:1]
root  8650 2 0   Nov 15  0.0  0.0 [kworker/14:2]
root  9287 2 0   Nov 15  0.0  0.0 [kworker/12:0]
root  9342 2 0   Nov 15  0.0  0.0 [kworker/9:0]
root  9628 2 0 23:31:48  0.0  0.0 [kworker/u48:0]
root  9950 2 0   Nov 15  0.0  0.0 [kworker/15:1]
root 10522 2 0   Nov 15  0.0  0.0 [kworker/3:0]
root 12304 2 0   Nov 14  0.0  0.0 [kworker/12:2]
postfix  12373  1801  1801 11:07:23  0.0  0.0 pickup -l -t fifo -u
root 12685 2 0   Nov 14  0.0  0.0 [kworker/10:0]
root 12793 2 0 19:55:24  0.0  0.0 [kworker/5:1]
root 13233 2 0 19:55:36  0.0  0.0 [kworker/14:1]
root 13234 2 0 19:55:36  0.0  0.0 [kworker/15:0]
root 13270 2 0 19:55:36  0.0  0.0 [kworker/6:2]
root 13274 2 0 19:55:36  0.0  0.0 

Build failed in Jenkins: bookkeeper_postcommit_master_python #38

[Apache Jenkins Server]

See 


--
[...truncated 209.05 KB...]
jenkins  25409 25407 22909 12:31:09  0.0  0.0 bash 
.test-infra/scripts/pre-docker-tests.sh
jenkins  25412 25409 22909 12:31:09  0.0  0.0 ps -eo 
euser,pid,ppid,pgid,start,pcpu,pmem,cmd
root 25923   975 25923   Nov 14  0.1  0.0 docker-containerd -l 
unix:///var/run/docker/libcontainerd/docker-containerd.sock 
--metrics-interval=0 --start-timeout 2m --state-dir 
/var/run/docker/libcontainerd/containerd --shim docker-containerd-shim 
--runtime docker-runc
snmp 26216 1 26215   Oct 16  0.1  0.0 /usr/sbin/snmpd -Lsd -Lf 
/dev/null -u snmp -g snmp -p /var/run/snmpd.pid
root 27037 2 0   Aug 12  0.0  0.0 [xfsalloc]
root 27040 2 0   Aug 12  0.0  0.0 [xfs_mru_cache]
root 27043 2 0   Aug 12  0.0  0.0 [jfsIO]
root 27044 2 0   Aug 12  0.0  0.0 [jfsCommit]
root 27045 2 0   Aug 12  0.0  0.0 [jfsCommit]
root 27046 2 0   Aug 12  0.0  0.0 [jfsCommit]
root 27047 2 0   Aug 12  0.0  0.0 [jfsCommit]
root 27048 2 0   Aug 12  0.0  0.0 [jfsSync]
root 27068 2 0   Aug 12  0.0  0.0 [bioset]
jenkins  27586 1 13683   Sep 25  0.0  5.4 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx3072m 
-Duser.timezone=UTC -jar 
/home/jenkins/jenkins-slave/workspace/Tika-trunk/tika-dl/target/surefire/surefirebooter150889610477356659.jar
 
/home/jenkins/jenkins-slave/workspace/Tika-trunk/tika-dl/target/surefire/surefire7948970542352839423tmp
 
/home/jenkins/jenkins-slave/workspace/Tika-trunk/tika-dl/target/surefire/surefire_35530043180101180472tmp
root 30575 2 0 00:31:19  0.0  0.0 [kworker/3:1]
root 31688   975   975   Sep 18  0.0  0.0 /usr/bin/docker-proxy -proto tcp 
-host-ip 0.0.0.0 -host-port 33539 -container-ip 172.19.0.13 -container-port 9160
root 31702   975   975   Sep 18  0.0  0.0 /usr/bin/docker-proxy -proto tcp 
-host-ip 0.0.0.0 -host-port 33540 -container-ip 172.19.0.13 -container-port 9042
root 31715   975   975   Sep 18  0.0  0.0 /usr/bin/docker-proxy -proto tcp 
-host-ip 0.0.0.0 -host-port 33541 -container-ip 172.19.0.13 -container-port 7199
root 31729   975   975   Sep 18  0.0  0.0 /usr/bin/docker-proxy -proto tcp 
-host-ip 0.0.0.0 -host-port 33542 -container-ip 172.19.0.13 -container-port 7001
root 31742   975   975   Sep 18  0.0  0.0 /usr/bin/docker-proxy -proto tcp 
-host-ip 0.0.0.0 -host-port 33543 -container-ip 172.19.0.13 -container-port 7000
root 32735 1 32734   Oct 30  0.0  0.3 python2.7 loggy.py --daemonize 
--user=root --group=root
+ docker system prune -f
Deleted Images:
deleted: sha256:bbd7f431502618e6f6c4d51bd974a3a80e6106a19b7cbbbeaee175030b4b16cd
deleted: sha256:aa2d7a93506838f3c3db3c1e9c8fae06511ada8c32a2b99b26ca533f53dd04d7
deleted: sha256:ca1d814d673864016a056a77f8c89cdd4349e67f442d1329ff2a4a58f20ac84a
deleted: sha256:14b0efd7103726f88c8c94ed25985f151290db766cb4ef8b73a7a9a1909a4897
deleted: sha256:d358bee34213dbe484a1dfba685bc48cd29644721283ecaae33ad2494bc77179
deleted: sha256:07aa60aacb82dc9e4479d3b1d4390c3c7864365ac2e4b4b078b0178cf6996002
deleted: sha256:df2d48cd893aa3a2c8d62cc398902cda8215fa1b19494db1e9268ef95aa68ba0

Total reclaimed space: 159.4MB
+ docker network prune -f --filter 'name=testnetwork_*'
+ echo 25464
+ docker system events
+ docker pull quay.io/coreos/etcd:v3.3
v3.3: Pulling from coreos/etcd
2bc1c5ee8605: Already exists
075a2fcf78bc: Pulling fs layer
35e5ff0bd5f1: Pulling fs layer
ec90809a24af: Pulling fs layer
27b2768170ac: Pulling fs layer
52ad13cdb1de: Pulling fs layer
27b2768170ac: Waiting
52ad13cdb1de: Waiting
ec90809a24af: Verifying Checksum
ec90809a24af: Download complete
27b2768170ac: Download complete
52ad13cdb1de: Verifying Checksum
52ad13cdb1de: Download complete
35e5ff0bd5f1: Verifying Checksum
35e5ff0bd5f1: Download complete
075a2fcf78bc: Verifying Checksum
075a2fcf78bc: Download complete
075a2fcf78bc: Pull complete
35e5ff0bd5f1: Pull complete
ec90809a24af: Pull complete
27b2768170ac: Pull complete
52ad13cdb1de: Pull complete
Digest: sha256:00304fb914dbc8205275bc7630f376171344753a17ec3c81f0b424fa8f841d66
Status: Downloaded newer image for quay.io/coreos/etcd:v3.3
Process leaked file descriptors. See 
https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for 
more information
[bookkeeper_postcommit_master_python] $ /bin/bash -xe 
/tmp/jenkins3063499801067602717.sh
+ docker pull python:3.7
3.7: Pulling from library/python
bc9ab73e5b14: Already exists
193a6306c92a: Already exists
e5c3f8c317dc: Already exists
a587a86c9dcb: Already exists
72744d0a318b: Already exists
3493e487c18d: Already exists
91f279e0491a: Pulling fs layer
39b42a4c244a: Pulling fs layer
4a962b343a0b: Pulling fs layer
39b42a4c244a: Verifying Checksum
39b42a4c244a: Download complete
4a962b343a0b: Verifying Checksum
4a962b343a0b: Download complete
91f279e0491a: