[GitHub] sijie commented on a change in pull request #1798: [tools] package bookkeeper tools into a separated distribution

[GitBox]

sijie commented on a change in pull request #1798: [tools] package bookkeeper 
tools into a separated distribution
URL: https://github.com/apache/bookkeeper/pull/1798#discussion_r232166037
 
 

 ##
 File path: bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
 ##
 @@ -0,0 +1,418 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other modifications
+  represent, as a whole, an original work of authorship. For the purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those patent claims licensable
+  by such Contributor that are necessarily infringed by their
+  Contribution(s) alone or by combination of their Contribution(s)
+  with the Work to which such Contribution(s) was submitted. If You
+  institute patent litigation against any enti

[GitHub] sijie commented on issue #1798: [tools] package bookkeeper tools into a separated distribution

[GitBox]

sijie commented on issue #1798: [tools] package bookkeeper tools into a 
separated distribution
URL: https://github.com/apache/bookkeeper/pull/1798#issuecomment-437279549
 
 
   > Does the script which checks licenses cover this new package?
   
   travis checks this package.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1798: [tools] package bookkeeper tools into a separated distribution

[GitBox]

merlimat commented on a change in pull request #1798: [tools] package 
bookkeeper tools into a separated distribution
URL: https://github.com/apache/bookkeeper/pull/1798#discussion_r232159782
 
 

 ##
 File path: bookkeeper-dist/src/main/resources/LICENSE-bkctl.bin.txt
 ##
 @@ -0,0 +1,418 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other modifications
+  represent, as a whole, an original work of authorship. For the purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those patent claims licensable
+  by such Contributor that are necessarily infringed by their
+  Contribution(s) alone or by combination of their Contribution(s)
+  with the Work to which such Contribution(s) was submitted. If You
+  institute patent litigation against any e

[GitHub] sijie opened a new pull request #1798: [tools] package bookkeeper tools into a separated distribution

[GitBox]

sijie opened a new pull request #1798: [tools] package bookkeeper tools into a 
separated distribution
URL: https://github.com/apache/bookkeeper/pull/1798
 
 
   
   Descriptions of the changes in this PR:
   
   *Motivation*
   
   Make the tools available without pulling in server dependencies
   
   
   > ---
   > In order to uphold a high standard for quality for code contributions, 
Apache BookKeeper runs various precommit
   > checks for pull requests. A pull request can only be merged when it passes 
precommit checks. However running all
   > the precommit checks can take a long time, some trivial changes don't need 
to run all the precommit checks. You
   > can check following list to skip the tests that don't need to run for your 
pull request. Leave them unchecked if
   > you are not sure, committers will help you:
   >
   > - [ ] [skip bookkeeper-server bookie tests]: skip testing 
`org.apache.bookkeeper.bookie` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server client tests]: skip testing 
`org.apache.bookkeeper.client` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server replication tests]: skip testing 
`org.apache.bookkeeper.replication` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server tls tests]: skip testing 
`org.apache.bookkeeper.tls` in bookkeeper-server module.
   > - [ ] [skip bookkeeper-server remaining tests]: skip testing all other 
tests in bookkeeper-server module.
   > - [ ] [skip integration tests]: skip docker based integration tests. if 
you make java code changes, you shouldn't skip integration tests.
   > - [ ] [skip build java8]: skip build on java8. *ONLY* skip this when 
*ONLY* changing files under documentation under `site`.
   > - [ ] [skip build java9]: skip build on java9. *ONLY* skip this when 
*ONLY* changing files under documentation under `site`.
   > ---
   
   > ---
   > Be sure to do all of the following to help us incorporate your contribution
   > quickly and easily:
   >
   > If this PR is a BookKeeper Proposal (BP):
   >
   > - [ ] Make sure the PR title is formatted like:
   > `: Description of bookkeeper proposal`
   > `e.g. BP-1: 64 bits ledger is support`
   > - [ ] Attach the master issue link in the description of this PR.
   > - [ ] Attach the google doc link if the BP is written in Google Doc.
   >
   > Otherwise:
   > 
   > - [ ] Make sure the PR title is formatted like:
   > `: Description of pull request`
   > `e.g. Issue 123: Description ...`
   > - [ ] Make sure tests pass via `mvn clean apache-rat:check install 
spotbugs:check`.
   > - [ ] Replace `` in the title with the actual Issue number.
   > 
   > ---
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #1794: [tools] add cookie related commands

[GitBox]

sijie closed pull request #1794: [tools] add cookie related commands
URL: https://github.com/apache/bookkeeper/pull/1794
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieException.java
 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieException.java
index 3d84148d1c..83002ce252 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieException.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieException.java
@@ -63,6 +63,8 @@ public static BookieException create(int code) {
 return new DiskPartitionDuplicationException();
 case Code.CookieNotFoundException:
 return new CookieNotFoundException();
+case Code.CookieExistsException:
+return new CookieExistException();
 case Code.MetadataStoreException:
 return new MetadataStoreException();
 case Code.UnknownBookieIdException:
@@ -88,6 +90,7 @@ public static BookieException create(int code) {
 int MetadataStoreException = -106;
 int UnknownBookieIdException = -107;
 int OperationRejectedException = -108;
+int CookieExistsException = -109;
 }
 
 public int getCode() {
@@ -118,6 +121,9 @@ public String getMessage(int code) {
 case Code.CookieNotFoundException:
 err = "Cookie not found";
 break;
+case Code.CookieExistsException:
+err = "Cookie already exists";
+break;
 case Code.MetadataStoreException:
 err = "Error performing metadata operations";
 break;
@@ -231,6 +237,23 @@ public CookieNotFoundException(Throwable cause) {
 }
 }
 
+/**
+ * Signal that cookie already exists when creating a new cookie.
+ */
+public static class CookieExistException extends BookieException {
+public CookieExistException() {
+this("");
+}
+
+public CookieExistException(String reason) {
+super(Code.CookieExistsException, reason);
+}
+
+public CookieExistException(Throwable cause) {
+super(Code.CookieExistsException, cause);
+}
+}
+
 /**
  * Signals that an exception occurs on upgrading a bookie.
  */
diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieShell.java 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieShell.java
index 18ce8c4392..e0e79c2de1 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieShell.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/BookieShell.java
@@ -92,6 +92,7 @@
 import org.apache.bookkeeper.client.LedgerHandle;
 import org.apache.bookkeeper.client.LedgerMetadata;
 import org.apache.bookkeeper.client.UpdateLedgerOp;
+import org.apache.bookkeeper.common.annotation.InterfaceAudience.Private;
 import org.apache.bookkeeper.common.util.OrderedExecutor;
 import org.apache.bookkeeper.conf.ClientConfiguration;
 import org.apache.bookkeeper.conf.ServerConfiguration;
@@ -115,6 +116,11 @@
 import org.apache.bookkeeper.tools.cli.commands.bookie.LastMarkCommand;
 import org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand;
 import org.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand;
+import org.apache.bookkeeper.tools.cli.commands.cookie.CreateCookieCommand;
+import org.apache.bookkeeper.tools.cli.commands.cookie.DeleteCookieCommand;
+import org.apache.bookkeeper.tools.cli.commands.cookie.GenerateCookieCommand;
+import org.apache.bookkeeper.tools.cli.commands.cookie.GetCookieCommand;
+import org.apache.bookkeeper.tools.cli.commands.cookie.UpdateCookieCommand;
 import org.apache.bookkeeper.tools.framework.CliFlags;
 import org.apache.bookkeeper.util.BookKeeperConstants;
 import org.apache.bookkeeper.util.DiskChecker;
@@ -191,6 +197,14 @@
 static final String CMD_CONVERT_TO_INTERLEAVED_STORAGE = 
"convert-to-interleaved-storage";
 static final String CMD_REBUILD_DB_LEDGER_LOCATIONS_INDEX = 
"rebuild-db-ledger-locations-index";
 static final String CMD_REGENERATE_INTERLEAVED_STORAGE_INDEX_FILE = 
"regenerate-interleaved-storage-index-file";
+
+// cookie commands
+static final String CMD_CREATE_COOKIE = "cookie_create";
+static final String CMD_DELETE_COOKIE = "cookie_delete";
+static final String CMD_UPDATE_COOKIE = "cookie_update";
+static final String CMD_GET_COOKIE = "cookie_get";
+static final String CMD_GENERATE_COOKIE = "cookie_generate";
+
 static final String CMD_HELP = "help";
 
 final ServerConfiguration bkConf = new ServerConfiguration();
@@ -214,9 +228,15 @@ public BookieShel

[GitHub] sijie commented on issue #1794: [tools] add cookie related commands

[GitBox]

sijie commented on issue #1794: [tools] add cookie related commands
URL: https://github.com/apache/bookkeeper/pull/1794#issuecomment-437257198
 
 
   IGNORE IT CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1794: [tools] add cookie related commands

[GitBox]

sijie commented on issue #1794: [tools] add cookie related commands
URL: https://github.com/apache/bookkeeper/pull/1794#issuecomment-437232463
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

merlimat commented on a change in pull request #1682: Added BlockingQueue 
implementation based on JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#discussion_r232114716
 
 

 ##
 File path: 
bookkeeper-common/src/main/java/org/apache/bookkeeper/common/collections/BusyWait.java
 ##
 @@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.common.collections;
+
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+
+import lombok.experimental.UtilityClass;
+
+/**
+ * Utility class to use "Thread.onSpinWait()" when available.
+ */
+@UtilityClass
+public class BusyWait {
+
+/**
+ * If available (Java 9+), use intrinsic {@link Thread#onSpinWait} which 
will
+ * reduce CPU consumption during the wait, otherwise fallback to regular
+ * spinning.
+ */
+public static void onSpinWait() {
+if (ON_SPIN_WAIT != null) {
+try {
+ON_SPIN_WAIT.invokeExact();
+} catch (Throwable t) {
+// Ignore
+}
+}
+}
+
+private static final MethodHandle ON_SPIN_WAIT;
+
+static {
+MethodHandle handle = null;
+try {
+handle = MethodHandles.lookup().findStatic(Thread.class, 
"onSpinWait", MethodType.methodType(void.class));
+} catch (Throwable t) {
+// Ignore
+}
 
 Review comment:
   This is not strictly critical for performance, it's just a way to consume 
less CPU power. 
   
   Since it's not available in Java8, I'd rather keep this at debug level to 
avoid printing the warning every time (eg: even if busywait is not used).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

merlimat commented on a change in pull request #1682: Added BlockingQueue 
implementation based on JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#discussion_r232113059
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDescriptorImpl.java
 ##
 @@ -116,7 +117,7 @@ ByteBuf getExplicitLac() {
  * @param journal log the fence entry in the Journal
  * @return A future which will be satisfied when add entry to journal 
complete
  */
-private SettableFuture logFenceEntryInJournal(Journal journal) {
+private SettableFuture logFenceEntryInJournal(Journal journal) 
throws InterruptedException {
 
 Review comment:
   Good point. will fix


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

eolivelli commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232079643
 
 

 ##
 File path: 
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/CpuAffinityImpl.java
 ##
 @@ -0,0 +1,210 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.bookkeeper.common.util.affinity.impl;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Implementation of CPU Affinity functionality.
+ */
+@UtilityClass
+@Slf4j
+public class CpuAffinityImpl {
+
+private static final boolean SUPPORTED = checkSupported();
+
+// Id of CPU cores acquired by this process
+private static final SortedSet acquiredProcessors = new 
TreeSet<>();
+
+// Ids of processors that were isolated by Linux at boot time. This is the 
set
+// of processors that can acquired by this or other processes.
+private static SortedSet isolatedProcessors = null;
+
+private static ProcessorsInfo processorsInfo = null;
+
+public static synchronized void acquireCore() {
+if (!SUPPORTED) {
+throw new RuntimeException("CPU Affinity not supported in current 
environment");
+}
+
+if (!CpuAffinityJni.isRoot()) {
+throw new RuntimeException("CPU Affinity can only be set if the 
process is running as root");
 
 Review comment:
   Okay


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

eolivelli commented on a change in pull request #1682: Added BlockingQueue 
implementation based on JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#discussion_r232077650
 
 

 ##
 File path: 
bookkeeper-common/src/main/java/org/apache/bookkeeper/common/collections/BusyWait.java
 ##
 @@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.bookkeeper.common.collections;
+
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+
+import lombok.experimental.UtilityClass;
+
+/**
+ * Utility class to use "Thread.onSpinWait()" when available.
+ */
+@UtilityClass
+public class BusyWait {
+
+/**
+ * If available (Java 9+), use intrinsic {@link Thread#onSpinWait} which 
will
+ * reduce CPU consumption during the wait, otherwise fallback to regular
+ * spinning.
+ */
+public static void onSpinWait() {
+if (ON_SPIN_WAIT != null) {
+try {
+ON_SPIN_WAIT.invokeExact();
+} catch (Throwable t) {
+// Ignore
+}
+}
+}
+
+private static final MethodHandle ON_SPIN_WAIT;
+
+static {
+MethodHandle handle = null;
+try {
+handle = MethodHandles.lookup().findStatic(Thread.class, 
"onSpinWait", MethodType.methodType(void.class));
+} catch (Throwable t) {
+// Ignore
+}
 
 Review comment:
   We should log this error, so that it is clear that we are not able to 
leverage this feature


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

eolivelli commented on a change in pull request #1682: Added BlockingQueue 
implementation based on JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#discussion_r232078181
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDescriptorImpl.java
 ##
 @@ -116,7 +117,7 @@ ByteBuf getExplicitLac() {
  * @param journal log the fence entry in the Journal
  * @return A future which will be satisfied when add entry to journal 
complete
  */
-private SettableFuture logFenceEntryInJournal(Journal journal) {
+private SettableFuture logFenceEntryInJournal(Journal journal) 
throws InterruptedException {
 
 Review comment:
   This method returns a Future, I wonder if it is okay to throw the 
Interrupted Exception or to return a failed Future


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on issue #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

eolivelli commented on issue #1682: Added BlockingQueue implementation based on 
JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#issuecomment-437172041
 
 
   Yes I noticed that Netty has its own copy of jctools queue.
   
   I am fine with this idea.
   Thank you for working on this topic


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1795: Release notes for 4.8.1

[GitBox]

eolivelli commented on a change in pull request #1795: Release notes for 4.8.1
URL: https://github.com/apache/bookkeeper/pull/1795#discussion_r232076409
 
 

 ##
 File path: site/releases.md
 ##
 @@ -64,6 +64,10 @@ Client Guide | API docs
 
 ## News
 
+### [date] Release {{ site.latest_version }} available
 
 Review comment:
   Yes, as usual the day of the release.
   Thank you for the reminder


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli commented on a change in pull request #1795: Release notes for 4.8.1

[GitBox]

eolivelli commented on a change in pull request #1795: Release notes for 4.8.1
URL: https://github.com/apache/bookkeeper/pull/1795#discussion_r232076286
 
 

 ##
 File path: site/_config.yml
 ##
 @@ -8,6 +8,7 @@ destination: local-generated
 twitter_url: https://twitter.com/asfbookkeeper
 
 versions:
+- "4.8.1"
 
 Review comment:
   Sure.
   The a script choose this position.
   Good catch


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on issue #1641: Added module to enable CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#issuecomment-437148304
 
 
   @ivankelly @eolivelli @sijie Please take another look.
   
   > However, I think we need an overarching document for these "performance" 
improvements, with benchmark results, detailed instructions to show how to 
reproduce the scenario in which performance changes, and the overall goals and 
approach for the perf improvements.
   
   This is just the underlying implementation. I will work on the docs and 
instructions when the whole busy-wait change set is ready.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

merlimat commented on issue #1682: Added BlockingQueue implementation based on 
JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#issuecomment-437144006
 
 
   run integration tests
   run bookkeeper-server replication tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232046564
 
 

 ##
 File path: 
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/NativeUtils.java
 ##
 @@ -0,0 +1,90 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.bookkeeper.common.util.affinity.impl;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+import lombok.experimental.UtilityClass;
+
+/**
+ * Utility class to load jni library from inside a JAR.
+ */
+@UtilityClass
+public class NativeUtils {
+/**
+ * loads given library from the this jar. ie: this jar contains: 
/lib/pulsar-checksum.jnilib
+ *
+ * @param path
+ *: absolute path of the library in the jar 
+ *if this jar contains: /lib/pulsar-checksum.jnilib then 
provide the same absolute path as input
+ * @throws Exception
+ */
+public static void loadLibraryFromJar(String path) throws Exception {
 
 Review comment:
   Yes, this code is similar to what we had imported with Circe-checksum, 
though I was trying to avoid interdependencies, dependencies on bk-common from 
this one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dlg99 commented on issue #1782: Issue #1781: Add traces/zipkin support to simplify troubleshooting of slow requests

[GitBox]

dlg99 commented on issue #1782: Issue #1781: Add traces/zipkin support to 
simplify troubleshooting of slow requests
URL: https://github.com/apache/bookkeeper/pull/1782#issuecomment-437131788
 
 
   I think I have an idea what to do, it won't be exactly interceptors fw but 
rather an extension of current metrics. I'll do the POC as time allows and 
we'll look at it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1793: [tools] improve bkctl help message

[GitBox]

sijie commented on issue #1793: [tools] improve bkctl help message
URL: https://github.com/apache/bookkeeper/pull/1793#issuecomment-437109635
 
 
   run bookkeeper-server remaining tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1795: Release notes for 4.8.1

[GitBox]

sijie commented on a change in pull request #1795: Release notes for 4.8.1
URL: https://github.com/apache/bookkeeper/pull/1795#discussion_r232012899
 
 

 ##
 File path: site/_config.yml
 ##
 @@ -8,6 +8,7 @@ destination: local-generated
 twitter_url: https://twitter.com/asfbookkeeper
 
 versions:
+- "4.8.1"
 
 Review comment:
   can you move this down after `# [next_version_placehoder]`?
   
   I think the script requires some fixes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #1793: [tools] improve bkctl help message

[GitBox]

sijie commented on issue #1793: [tools] improve bkctl help message
URL: https://github.com/apache/bookkeeper/pull/1793#issuecomment-437109214
 
 
   run pr validation
   run integration tests
   run bookkeeper remaining tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1795: Release notes for 4.8.1

[GitBox]

sijie commented on a change in pull request #1795: Release notes for 4.8.1
URL: https://github.com/apache/bookkeeper/pull/1795#discussion_r232013073
 
 

 ##
 File path: site/releases.md
 ##
 @@ -64,6 +64,10 @@ Client Guide | API docs
 
 ## News
 
+### [date] Release {{ site.latest_version }} available
 
 Review comment:
   need to update this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232012381
 
 

 ##
 File path: cpu-affinity/src/main/affinity/cpp/affinity_jni.c
 ##
 @@ -0,0 +1,98 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+// Use different error code to differentiate non-implemented error
+static const int NOT_IMPLEMENTED = -2;
 
 Review comment:
   Yup, will reformat


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232012326
 
 

 ##
 File path: cpu-affinity/pom.xml
 ##
 @@ -0,0 +1,181 @@
+
+http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";
+  xmlns="http://maven.apache.org/POM/4.0.0";
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
+  4.0.0
+  
+org.apache.bookkeeper
+bookkeeper
+4.9.0-SNAPSHOT
+..
+  
+
+  cpu-affinity
+  nar
+  Apache BookKeeper :: CPU Affinity Library
+  CPU Affinity Library
+
+  
+dynamic
+  
+
+  
+
+  com.google.guava
+  guava
+
+
+  org.apache.commons
+  commons-lang3
+
+  
+
+  
+
+  
+org.apache.maven.plugins
+maven-compiler-plugin
+
+  1.8
+  1.8
+  
+   
+   
+-Xlint:deprecation
+-Xlint:unchecked
+   
+-Xpkginfo:always
+  
+
+  
+  
+com.github.maven-nar
+nar-maven-plugin
+${nar-maven-plugin.version}
+true
+  
+  
+org.apache.maven.plugins
+maven-assembly-plugin
+
+  
+src/main/assembly/assembly.xml
+  
+  false
+  posix
+
+
+  
+make-assembly
+package
+
+  single
+
+  
+
+  
+  
+org.apache.rat
+apache-rat-plugin
+
+  
+
+**/src/test/resources/proc_cpuinfo.txt
+  
+
+  
+
+  
+
+  
+
+  mac
+  
+
+  Mac OS X
+
+  
+  
+
+  
+com.github.maven-nar
+nar-maven-plugin
+${nar-maven-plugin.version}
+true
+
+  ${nar.runtime}
+  cpu-affinity
+  
+
+  jni
+  
org.apache.bookkeeper.utils.affinity
+
+  
+  
+${nar.cpp.optionSet}
+false
+false
+full
+  
+
+  
+
+  
+
+
+
+  Linux
 
 Review comment:
   This pom was copied from the existing circe-checksum one. The profile name 
is not that important since it's automatically enabled based on the current OS.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232008861
 
 

 ##
 File path: 
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/CpuAffinityImpl.java
 ##
 @@ -0,0 +1,210 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.bookkeeper.common.util.affinity.impl;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Implementation of CPU Affinity functionality.
+ */
+@UtilityClass
+@Slf4j
+public class CpuAffinityImpl {
+
+private static final boolean SUPPORTED = checkSupported();
+
+// Id of CPU cores acquired by this process
+private static final SortedSet acquiredProcessors = new 
TreeSet<>();
+
+// Ids of processors that were isolated by Linux at boot time. This is the 
set
+// of processors that can acquired by this or other processes.
+private static SortedSet isolatedProcessors = null;
+
+private static ProcessorsInfo processorsInfo = null;
+
+public static synchronized void acquireCore() {
+if (!SUPPORTED) {
+throw new RuntimeException("CPU Affinity not supported in current 
environment");
+}
+
+if (!CpuAffinityJni.isRoot()) {
+throw new RuntimeException("CPU Affinity can only be set if the 
process is running as root");
+}
+
+try {
+int cpu = pickAvailableCpu();
+CpuAffinityJni.setAffinity(cpu);
+
+log.info("Thread {} has successfully acquired ownership of cpu 
{}", Thread.currentThread().getName(), cpu);
+} catch (IOException e) {
+throw new RuntimeException("Failed to acquire CPU core: " + 
e.getMessage());
+}
+}
+
+private static final String LOCK_FILE_PREFIX = 
Paths.get(System.getProperty("java.io.tmpdir"), "cpu-lock-")
+.toString();
+
+/**
+ * Other than the cores acquired by this process, there might be other 
processes on the same host trying to acquire
+ * the available cores.
+ *
+ * We use file-locks to ensure that other processes are aware of which 
CPUs are taken and that these locks are
+ * automatically released if the process crashes.
+ */
+private static synchronized int pickAvailableCpu() throws IOException {
+if (isolatedProcessors == null) {
+isolatedProcessors = IsolatedProcessors.get();
+}
+for (int isolatedCpu : isolatedProcessors) {
+if (log.isDebugEnabled()) {
+log.debug("Checking CPU {}", isolatedCpu);
+}
+if (acquiredProcessors.contains(isolatedCpu)) {
+if (log.isDebugEnabled()) {
+log.debug("Ignoring CPU {} since it's already acquired", 
isolatedCpu);
+}
+continue;
+}
+
+if (tryAcquireCpu(isolatedCpu)) {
+if (log.isDebugEnabled()) {
+log.debug("Using CPU {}", isolatedCpu);
+}
+return isolatedCpu;
+}
+}
+
+throw new RuntimeException(
+"There is no available isolated CPU to acquire for thread " + 
Thread.currentThread().getName());
+}
+
+private static boolean tryAcquireCpu(int targetCpu) throws IOException {
+// First, acquire lock on all the cpus that share the same core as 
target cpu
+if (processorsInfo == null) {
+processorsInfo = ProcessorsInfo.parseCpuInfo();
+}
+
+Set cpusToAcquire = 
processorsInfo.getCpusOnSameCore(targetCpu);
+List acquiredCpus = new ArrayList<>();
+
+for (int cpu : cpusToAcquire) {
+Closeable lock = tryAcquireFileLock(cpu);
+if (lock == null) {
+  

[GitHub] merlimat commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232007721
 
 

 ##
 File path: 
cpu-affinity/src/main/java/org/apache/bookkeeper/common/util/affinity/impl/CpuAffinityImpl.java
 ##
 @@ -0,0 +1,210 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+package org.apache.bookkeeper.common.util.affinity.impl;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.RandomAccessFile;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.TreeSet;
+
+import lombok.experimental.UtilityClass;
+import lombok.extern.slf4j.Slf4j;
+
+/**
+ * Implementation of CPU Affinity functionality.
+ */
+@UtilityClass
+@Slf4j
+public class CpuAffinityImpl {
+
+private static final boolean SUPPORTED = checkSupported();
+
+// Id of CPU cores acquired by this process
+private static final SortedSet acquiredProcessors = new 
TreeSet<>();
+
+// Ids of processors that were isolated by Linux at boot time. This is the 
set
+// of processors that can acquired by this or other processes.
+private static SortedSet isolatedProcessors = null;
+
+private static ProcessorsInfo processorsInfo = null;
+
+public static synchronized void acquireCore() {
+if (!SUPPORTED) {
+throw new RuntimeException("CPU Affinity not supported in current 
environment");
+}
+
+if (!CpuAffinityJni.isRoot()) {
+throw new RuntimeException("CPU Affinity can only be set if the 
process is running as root");
 
 Review comment:
   I have not been able to find a way other than being root in order to use 
`sched_setaffinity()`. The check here is just to provide more meaningful error 
message, if we can find a way to configure it we could remove this check.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #1641: Added module to enable CPU affinity

[GitBox]

merlimat commented on a change in pull request #1641: Added module to enable 
CPU affinity
URL: https://github.com/apache/bookkeeper/pull/1641#discussion_r232007192
 
 

 ##
 File path: 
circe-checksum/src/main/java/com/scurrilous/circe/utils/NativeUtils.java
 ##
 @@ -67,13 +67,16 @@ public static void loadLibraryFromJar(String path) throws 
Exception {
 throw new FileNotFoundException("Couldn't find file into jar " + 
path);
 }
 
-OutputStream out = new FileOutputStream(temp);
 try {
-while ((read = input.read(buffer)) != -1) {
-out.write(buffer, 0, read);
+OutputStream out = new FileOutputStream(temp);
 
 Review comment:
   Fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed issue #1791: Read Submission should bypass OSE Threads

[GitBox]

sijie closed issue #1791: Read Submission should bypass OSE Threads
URL: https://github.com/apache/bookkeeper/issues/1791
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #1792: Issue #1791: Read Submission should bypass OSE Threads

[GitBox]

sijie closed pull request #1792: Issue #1791: Read Submission should bypass OSE 
Threads
URL: https://github.com/apache/bookkeeper/pull/1792
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
index 6125a3d1b9..12e00e84fb 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
@@ -840,7 +840,14 @@ public LedgerEntry readLastEntry()
 ws.recycle();
 }
 
-clientCtx.getMainWorkerPool().executeOrdered(ledgerId, op);
+if (isHandleWritable()) {
+// Ledger handle in read/write mode: submit to OSE for ordered 
execution.
+clientCtx.getMainWorkerPool().executeOrdered(ledgerId, op);
+} else {
+// Read-only ledger handle: bypass OSE and execute read 
directly in client thread.
+// This avoids a context-switch to OSE thread and thus reduces 
latency.
+op.run();
+}
 } else {
 
op.future().completeExceptionally(BKException.create(ClientClosedException));
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #1789: Cache InetSocketAddress if hostname is IPAddress

[GitBox]

sijie closed pull request #1789: Cache InetSocketAddress if hostname is 
IPAddress
URL: https://github.com/apache/bookkeeper/pull/1789
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/net/BookieSocketAddress.java
 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/net/BookieSocketAddress.java
index 6d562e3a67..cd0a20cfa4 100644
--- 
a/bookkeeper-server/src/main/java/org/apache/bookkeeper/net/BookieSocketAddress.java
+++ 
b/bookkeeper-server/src/main/java/org/apache/bookkeeper/net/BookieSocketAddress.java
@@ -22,10 +22,12 @@
 
 import static org.apache.bookkeeper.util.BookKeeperConstants.COLON;
 
+import com.google.common.net.InetAddresses;
 import io.netty.channel.local.LocalAddress;
 
 import java.net.InetSocketAddress;
 import java.net.UnknownHostException;
+import java.util.Optional;
 
 /**
  * This is a data wrapper class that is an InetSocketAddress, it would use the 
hostname
@@ -38,11 +40,24 @@
 // Member fields that make up this class.
 private final String hostname;
 private final int port;
+private final Optional socketAddress;
 
 // Constructor that takes in both a port.
 public BookieSocketAddress(String hostname, int port) {
 this.hostname = hostname;
 this.port = port;
+/*
+ * if ipaddress is used for bookieid then lets cache InetSocketAddress
+ * otherwise not cache it. If Hostname is used for bookieid, then it is
+ * ok for node to change its ipaddress. But if ipaddress is used for
+ * bookieid then it is invalid scenario if node's ipaddress changes and
+ * nodes HostName is considered static.
+ */
+if (InetAddresses.isInetAddress(hostname)) {
+socketAddress = Optional.of(new InetSocketAddress(hostname, port));
+} else {
+socketAddress = Optional.empty();
+}
 }
 
 // Constructor from a String "serialized" version of this class.
@@ -57,8 +72,15 @@ public BookieSocketAddress(String addr) throws 
UnknownHostException {
 } catch (NumberFormatException nfe) {
 throw new UnknownHostException(addr);
 }
+if (InetAddresses.isInetAddress(hostname)) {
+socketAddress = Optional.of(new InetSocketAddress(hostname, port));
+} else {
+socketAddress = Optional.empty();
+}
 }
 
+
+
 // Public getters
 public String getHostName() {
 return hostname;
@@ -70,12 +92,15 @@ public int getPort() {
 
 // Method to return an InetSocketAddress for the regular port.
 public InetSocketAddress getSocketAddress() {
-// Return each time a new instance of the InetSocketAddress because 
the hostname
-// gets resolved in its constructor and then cached forever.
-// If we keep using the same InetSocketAddress instance, if bookies 
are advertising
-// hostnames and the IP change, the BK client will keep forever to try 
to connect
-// to the old IP.
-return new InetSocketAddress(hostname, port);
+/*
+ * Return each time a new instance of the InetSocketAddress if hostname
+ * is used as bookieid. If we keep using the same InetSocketAddress
+ * instance, if bookies are advertising hostnames and the IP change, 
the
+ * BK client will keep forever to try to connect to the old IP.
+ */
+return socketAddress.orElseGet(() -> {
+return new InetSocketAddress(hostname, port);
+});
 }
 
 /**
diff --git 
a/bookkeeper-server/src/test/java/org/apache/bookkeeper/net/BookieSocketAddressTest.java
 
b/bookkeeper-server/src/test/java/org/apache/bookkeeper/net/BookieSocketAddressTest.java
new file mode 100644
index 00..1da5643d99
--- /dev/null
+++ 
b/bookkeeper-server/src/test/java/org/apache/bookkeeper/net/BookieSocketAddressTest.java
@@ -0,0 +1,50 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+

[GitHub] sijie commented on issue #1755: Configure Netty allocator in bookie and client

[GitBox]

sijie commented on issue #1755:  Configure Netty allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#issuecomment-437072101
 
 
   run bookkeeper-server tls tests
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1786: BP-36: Stats documentation annotation

[GitBox]

sijie commented on a change in pull request #1786: BP-36: Stats documentation 
annotation
URL: https://github.com/apache/bookkeeper/pull/1786#discussion_r231972932
 
 

 ##
 File path: site/bps/BP-36-stats-documentation-annotation.md
 ##
 @@ -0,0 +1,157 @@
+---
+title: "BP-36: Stats documentation annotation"
+issue: https://github.com/apache/bookkeeper/
+state: "Under Discussion"
+release: "4.9.0"
+---
+
+### Motivation
+
+A common ask from people using bookkeeper is how they can monitor bookies and 
bookkeeper clients, what kind of metrics that bookkeeper exposes
+and what are the important metrics. Currently bookkeeper doesn't provide any 
metrics page for guiding people on monitoring bookkeeper services.
+
+In order to help people on this, we need to provide a few documentation pages 
about metrics. However if we just write such pages, those pages
+can quickly get out-of-dated when code is changed. The proposal here is to 
seek a programming way for generating metrics related pages.
+
+### Public Interfaces
+
+Introduced a `StatsDoc` annotation.
+
+```bash
+/**
+ * Documenting the stats.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Documented
+public @interface StatsDoc {
+
+/**
+ * The name of the category to group stats together.
+ *
+ * @return name of the stats category.
+ */
+String category() default "";
+
+/**
+ * The scope of this stats.
+ *
+ * @return scope of this stats
+ */
+String scope() default "";
+
+/**
+ * The name of this stats
+ *
+ * @return name of this stats
+ */
+String name();
+
+/**
+ * The help message of this stats
+ *
+ * @return help message of this stats
+ */
+String help();
+
+/**
+ * The parent metric name.
+ *
+ * It can used for analyzing the relationships
+ * between the metrics, especially for the latency metrics.
+ *
+ * @return the parent metric name
+ */
+default String parent() { return ""; }
+
+/**
+ * The metric name of an operation that happens
+ * after the operation of this metric.
+ *
+ * similar as {@link #parent()}, it can be used for analyzing
+ * the dependencies between metrics.
+ *
+ * @return the metric name of an operation that happens after the 
operation of this metric.
+ */
+default String happenAfter() { return ""; }
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #1786: BP-36: Stats documentation annotation

[GitBox]

sijie commented on a change in pull request #1786: BP-36: Stats documentation 
annotation
URL: https://github.com/apache/bookkeeper/pull/1786#discussion_r231972864
 
 

 ##
 File path: site/bps/BP-36-stats-documentation-annotation.md
 ##
 @@ -0,0 +1,157 @@
+---
+title: "BP-36: Stats documentation annotation"
+issue: https://github.com/apache/bookkeeper/
+state: "Under Discussion"
+release: "4.9.0"
+---
+
+### Motivation
+
+A common ask from people using bookkeeper is how they can monitor bookies and 
bookkeeper clients, what kind of metrics that bookkeeper exposes
+and what are the important metrics. Currently bookkeeper doesn't provide any 
metrics page for guiding people on monitoring bookkeeper services.
+
+In order to help people on this, we need to provide a few documentation pages 
about metrics. However if we just write such pages, those pages
+can quickly get out-of-dated when code is changed. The proposal here is to 
seek a programming way for generating metrics related pages.
+
+### Public Interfaces
+
+Introduced a `StatsDoc` annotation.
+
+```bash
+/**
+ * Documenting the stats.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Documented
+public @interface StatsDoc {
+
+/**
+ * The name of the category to group stats together.
+ *
+ * @return name of the stats category.
+ */
+String category() default "";
+
+/**
+ * The scope of this stats.
+ *
+ * @return scope of this stats
+ */
+String scope() default "";
+
+/**
+ * The name of this stats
+ *
+ * @return name of this stats
+ */
+String name();
+
+/**
+ * The help message of this stats
+ *
+ * @return help message of this stats
+ */
+String help();
+
+/**
+ * The parent metric name.
+ *
+ * It can used for analyzing the relationships
+ * between the metrics, especially for the latency metrics.
+ *
+ * @return the parent metric name
+ */
+default String parent() { return ""; }
+
+/**
+ * The metric name of an operation that happens
+ * after the operation of this metric.
+ *
+ * similar as {@link #parent()}, it can be used for analyzing
+ * the dependencies between metrics.
+ *
+ * @return the metric name of an operation that happens after the 
operation of this metric.
+ */
+default String happenAfter() { return ""; }
 
 Review comment:
   ah, I was planning to type `happensAfter`. however it becomes `happenAfter` 
:(


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build is back to stable : bookkeeper_release_branch_48_java8 #85

[Apache Jenkins Server]

See 

Jenkins build is still unstable: bookkeeper_release_branch_48_java9 #85

[Apache Jenkins Server]

See 

Jenkins build is back to stable : bookkeeper_codecoverage #263

[Apache Jenkins Server]

See 

[GitHub] ivankelly removed a comment on issue #1797: Move version out of LedgerMetadata

[GitBox]

ivankelly removed a comment on issue #1797: Move version out of LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797#issuecomment-437021530
 
 
   rerun bookie replication tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1797: Move version out of LedgerMetadata

[GitBox]

ivankelly commented on issue #1797: Move version out of LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797#issuecomment-437021810
 
 
   rerun bookkeeper-server replication tests
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on issue #1797: Move version out of LedgerMetadata

[GitBox]

ivankelly commented on issue #1797: Move version out of LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797#issuecomment-437021530
 
 
   rerun bookie replication tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #1797: Move version out of LedgerMetadata

[GitBox]

ivankelly commented on a change in pull request #1797: Move version out of 
LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797#discussion_r231910997
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
 ##
 @@ -190,7 +190,9 @@
 // password, so that the bookie can avoid processing the keys for each 
entry
 this.ledgerKey = DigestManager.generateMasterKey(password);
 distributionSchedule = new RoundRobinDistributionSchedule(
-metadata.getWriteQuorumSize(), metadata.getAckQuorumSize(), 
metadata.getEnsembleSize());
+metadata.getValue().getWriteQuorumSize(),
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build became unstable: bookkeeper_release_branch_46 #210

[Apache Jenkins Server]

See 

Build failed in Jenkins: bookkeeper_release_branch_48_integrationtests #85

[Apache Jenkins Server]

See 


Changes:

[eolivelli] Bump branch-4.8 to 4.8.1-SNAPSHOT During the release process of 
4.8.0

[eolivelli] [maven-release-plugin] prepare release v4.8.1-rc0

[eolivelli] [maven-release-plugin] rollback changes from release preparation of

--
[...truncated 1.38 MB...]
Running 
org.apache.bookkeeper.tests.backwardcompat.TestCompatUpgradeOldServerInClusterWithCookies
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.455 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

2018-11-08\T\13:06:53.382 [INFO] 
2018-11-08\T\13:06:53.383 [INFO] 

2018-11-08\T\13:06:53.383 [INFO] Building Apache BookKeeper :: Tests :: 
Backward Compatibility :: Test old clients working on current server 
4.8.1-SNAPSHOT
2018-11-08\T\13:06:53.383 [INFO] 

2018-11-08\T\13:06:53.415 [INFO] 
2018-11-08\T\13:06:53.416 [INFO] --- 
groovy-eclipse-compiler:2.9.2-04:add-groovy-build-paths 
(default-add-groovy-build-paths) @ backward-compat-current-server-old-clients 
---
2018-11-08\T\13:06:53.416 [INFO] Adding /src/main/groovy to the list of source 
folders
2018-11-08\T\13:06:53.416 [INFO] Adding /src/test/groovy to the list of test 
source folders
2018-11-08\T\13:06:53.416 [INFO] 
2018-11-08\T\13:06:53.416 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ backward-compat-current-server-old-clients ---
2018-11-08\T\13:06:55.298 [INFO] 
2018-11-08\T\13:06:55.298 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ backward-compat-current-server-old-clients ---
2018-11-08\T\13:06:55.299 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-11-08\T\13:06:55.299 [INFO] skip non existing resourceDirectory 

2018-11-08\T\13:06:55.300 [INFO] Copying 3 resources
2018-11-08\T\13:06:55.301 [INFO] 
2018-11-08\T\13:06:55.301 [INFO] --- maven-compiler-plugin:3.7.0:compile 
(default-compile) @ backward-compat-current-server-old-clients ---
2018-11-08\T\13:06:55.303 [INFO] No sources to compile
2018-11-08\T\13:06:55.303 [INFO] 
2018-11-08\T\13:06:55.303 [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ backward-compat-current-server-old-clients ---
2018-11-08\T\13:06:55.304 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-11-08\T\13:06:55.304 [INFO] Copying 1 resource
2018-11-08\T\13:06:55.304 [INFO] Copying 3 resources
2018-11-08\T\13:06:55.305 [INFO] 
2018-11-08\T\13:06:55.305 [INFO] --- maven-compiler-plugin:3.7.0:testCompile 
(default-testCompile) @ backward-compat-current-server-old-clients ---
2018-11-08\T\13:06:55.311 [INFO] Changes detected - recompiling the module!
2018-11-08\T\13:06:55.312 [INFO] Using Groovy-Eclipse compiler to compile both 
Java and Groovy files
2018-11-08\T\13:06:55.638 [INFO] 
2018-11-08\T\13:06:55.638 [INFO] --- maven-surefire-plugin:2.8.1:test 
(default-test) @ backward-compat-current-server-old-clients ---
2018-11-08\T\13:06:55.642 [INFO] Surefire report directory: 


---
 T E S T S
---
Running org.apache.bookkeeper.tests.backwardcompat.TestCompatOldClients
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 354.276 sec

Results :

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

2018-11-08\T\13:13:24.740 [INFO] 
2018-11-08\T\13:13:24.740 [INFO] 

2018-11-08\T\13:13:24.740 [INFO] Building Apache BookKeeper :: Tests :: 
Backward Compatibility :: Test upgrade between yahoo custom version and current 
4.8.1-SNAPSHOT
2018-11-08\T\13:13:24.740 [INFO] 

2018-11-08\T\13:13:24.785 [INFO] 
2018-11-08\T\13:13:24.785 [INFO] --- 
groovy-eclipse-compiler:2.9.2-04:add-groovy-build-paths 
(default-add-groovy-build-paths) @ yahoo-custom-version ---
2018-11-08\T\13:13:24.786 [INFO] Adding /src/main/groovy to the list of source 
folders
2018-11-08\T\13:13:24.786 [INFO] Adding /src/test/groovy to the list of test 
source folders
2018-11-08\T\13:13:24.786 [INFO] 
2018-11-08\T\13:13:24.786 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ yahoo-custom-version ---
2018-11-08\T\13:13:26.609 [INFO] 
2018-11-08\T\13:13:26.609 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ yahoo-custom-version ---
2018-11-08\T\13:13:

[GitHub] eolivelli commented on a change in pull request #1797: Move version out of LedgerMetadata

[GitBox]

eolivelli commented on a change in pull request #1797: Move version out of 
LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797#discussion_r231875315
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
 ##
 @@ -190,7 +190,9 @@
 // password, so that the bookie can avoid processing the keys for each 
entry
 this.ledgerKey = DigestManager.generateMasterKey(password);
 distributionSchedule = new RoundRobinDistributionSchedule(
-metadata.getWriteQuorumSize(), metadata.getAckQuorumSize(), 
metadata.getEnsembleSize());
+metadata.getValue().getWriteQuorumSize(),
 
 Review comment:
   Nit: call getValue() only once?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Build failed in Jenkins: bookkeeper_release_nightly_snapshot #188

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H33 (ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/bookkeeper.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/bookkeeper.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/bookkeeper.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision da3bd81ef8849589d63d6051c4f1dc86981c7436 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f da3bd81ef8849589d63d6051c4f1dc86981c7436
Commit message: "Upgrade to Netty 4.1.31 and use individual dependencies"
 > git rev-list --no-walk da3bd81ef8849589d63d6051c4f1dc86981c7436 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
FATAL: Command "git clean -fdx" returned status code 1:
stdout: 
stderr: warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/SOURCES.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/namespace_packages.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/PKG-INFO
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/top_level.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/requires.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/dependency_links.txt
warning: failed to remove 
stream/clients/python/apache_bookkeeper_client.egg-info/not-zip-safe
warning: failed to remove stream/clients/python/build/lib/bookkeeper/types.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/stream_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/storage_pb2_grpc.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/common_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_rpc_pb2_grpc.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_store_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/kv_rpc_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/cluster_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/proto/storage_pb2.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/namespace.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/exceptions.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/client.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/namespaces.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/admin/__init__.py
warning: failed to remove stream/clients/python/build/lib/bookkeeper/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/exceptions.py
warning: failed to remove stream/clients/python/build/lib/bookkeeper/kv/table.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/client.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/kv/futures.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/timeout.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/exceptions.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/constants.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/polling.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/_helpers.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/__init__.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/future/base.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/method.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/grpc_helpers.py
warning: failed to remove 
stream/clients/python/build/lib/bookkeeper/common/datetime_helpers.py
warning: failed to remove 
stream

Build failed in Jenkins: bookkeeper_release_branch_47_integrationtests #200

[Apache Jenkins Server]

See 


--
[...truncated 10.95 KB...]
root   679 2 0   Jul 25  0.0  0.0 [kworker/14:1H]
root   727 1   726   Jul 25  0.0  0.0 upstart-socket-bridge --daemon
root   753 2 0   Jul 25  0.0  0.0 [kworker/12:1H]
root   754 2 0   Jul 25  0.0  0.0 [kworker/1:1H]
root   762 1   762   Jul 25  0.0  0.0 dhclient -1 -v -pf 
/run/dhclient.em1.pid -lf /var/lib/dhcp/dhclient.em1.leases em1
root   781 2 0   Jul 25  0.0  0.0 [kworker/8:1H]
root   826 2 0   Jul 25  0.0  0.0 [kworker/4:1H]
root   847 2 0   Jul 25  0.0  0.0 [kworker/6:1H]
root   848 2 0   Jul 25  0.0  0.0 [kworker/2:1H]
root   849 2 0   Jul 25  0.0  0.0 [kworker/3:1H]
root   973 1   972   Jul 25  0.0  0.0 upstart-file-bridge --daemon
syslog1056 1  1056   Jul 25  0.0  0.0 rsyslogd
message+  1120 1  1120   Jul 25  0.0  0.0 dbus-daemon --system --fork
root  1132 1  1132   Jul 25  0.0  0.0 /lib/systemd/systemd --system
root  1158 1  1158   Jul 25  0.0  0.0 /lib/systemd/systemd-logind
root  1207 1  1207   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty4
root  1211 1  1211   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty5
root  1221 1  1221   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty2
root  1222 1  1222   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty3
root  1224 1  1224   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty6
daemon1255 1  1255   Jul 25  0.0  0.0 atd
root  1256 1  1256   Jul 25  0.0  0.0 cron
root  1289  1132  1289   Jul 25  0.0  0.7 /lib/systemd/systemd-journald
root  1290 2 0   Jul 25  0.0  0.0 [kauditd]
root  1311 1  1311   Jul 25  0.0  0.0 /usr/sbin/irqbalance
root  1493 2 0   Jul 25  0.0  0.0 [kworker/15:1H]
root  1703 1  1703   Oct 04  0.0  0.0 /usr/bin/ruby /usr/bin/puppet 
agent
root  1801 1  1801   Jul 25  0.0  0.0 /usr/lib/postfix/master
postfix   1823  1801  1801   Jul 25  0.0  0.0 qmgr -l -t fifo -u
ntp   1915 1  1915   Jul 25  0.0  0.0 /usr/sbin/ntpd -p 
/var/run/ntpd.pid -g -u 105:116
root  2012 1  2011   Jul 25  0.0  0.0 /usr/bin/python 
/usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p 
/var/run/fail2ban/fail2ban.pid
root  2072 1  2072   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty1
root  2073 1  2073   Jul 25  0.0  0.0 /sbin/getty -L ttyS0 9600 vt102
root  2378 2 0   Jul 25  0.0  0.0 [kworker/10:1H]
nslcd 3312 1  3312   Jul 25  0.1  0.0 /usr/sbin/nslcd
root  3617 1  3617   Oct 31  0.0  0.0 sshd: jenkins [priv]
jenkins   3651  3617  3617   Oct 31  0.0  0.0 sshd: jenkins@notty 
jenkins   3686  3651  3686   Oct 31  0.0  0.0 bash -c cd 
"/home/jenkins/jenkins-slave" && /home/jenkins/tools/java/jdk1.8.0_131/bin/java 
-Xmx2g -Xms512m -jar slave.jar
jenkins   3687  3686  3686   Oct 31  1.1  3.0 
/home/jenkins/tools/java/jdk1.8.0_131/bin/java -Xmx2g -Xms512m -jar slave.jar
jenkins   4416 1 21137   Aug 21  0.0  0.0 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefirebooter123717442526029744.jar
 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefire4123165900408045001tmp
 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefire_06109285983847161674tmp
jenkins   4455  4416 21137   Aug 21  0.7  0.6 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx2048m -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/dependency/org.apache.sling.launchpad-8.jar
 -p 45800 -Dsling.run.modes=author,notshared
root  4889 2 0 23:17:32  0.0  0.0 [kworker/1:1]
root  4927 1  1120   Jul 25  0.0  0.0 
/usr/lib/x86_64-linux-gnu/systemd-shim
root  4977 2 0   Jul 25  0.0  0.0 [kworker/13:1H]
root  5661 2 0   Jul 25  0.0  0.0 [kworker/11:1H]
root  6009 2 0   Jul 25  0.0  0.0 [kworker/9:1H]
jenkins   7335  4416 21137   Aug 21  104  3.2 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx2048m -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/dependency/org.apache.sling.launchpad-8.jar
 -p 41776 -Dsling.run.modes=publish,notshared
root 10003 2 0 20:34:01  0.0  0.0 [kworker/u49:2]
root 10165 2 0 23:19:18  0.0  0.0 [kworker/14:2]
root 10461 2 0 11:52:29  0.0  0.0 [kworker/u50:0]
root 10510 2 0 23:19:28  0.0  0.0 [kworker/6:2]
root 12056 2 0   Nov 01  0.0  0.0 [kworker/10:1]
root 13405 2 0 00:42:28  0.0  0.0 [kworker/14:1]
root 13416 2 0 00:42:28  0.0  0.0 [kworker/13:1]
root 13423 2 

Build failed in Jenkins: bookkeeper_postcommit_master_python #30

[Apache Jenkins Server]

See 


--
[...truncated 144.53 KB...]
dd-agent 12050 1 12050   Nov 05  0.0  0.1 
/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/bin/supervisord -c 
/etc/dd-agent/supervisor.conf --pidfile 
/opt/datadog-agent/run/datadog-supervisord.pid
dd-agent 12052 12050 12052   Nov 05  1.0  0.1 
/opt/datadog-agent/bin/process-agent
dd-agent 12053 12050 12053   Nov 05  0.0  0.0 /opt/datadog-agent/bin/trace-agent
dd-agent 12054 12050 12054   Nov 05  0.1  0.2 
/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/ddagent.py
dd-agent 12062 12050 12062   Nov 05  0.3  0.2 
/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/agent.py 
foreground --use-local-forwarder
jenkins  13395 1  3587   Oct 26  0.2  1.5 
/home/jenkins/tools/java/latest1.8/bin/java -Xmx512m -XX:PermSize=128m 
-XX:MaxPermSize=128m -classpath 
reef/local/*:reef/global/*:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/examples/beam/target/test-classes:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/examples/beam/target/nemo-examples-beam-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/compiler/frontend/beam/target/nemo-compiler-frontend-beam-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/common/target/nemo-common-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/reef-io/0.16.0/reef-io-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/reef-webserver/0.16.0/reef-webserver-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/reef-common/0.16.0/reef-common-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/reef-annotations/0.16.0/reef-annotations-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/reef-utils/0.16.0/reef-utils-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/wake/0.16.0/wake-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/cglib/cglib/3.1/cglib-3.1.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/ow2/asm/asm/4.2/asm-4.2.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-all/4.1.16.Final/netty-all-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/reef/tang/0.16.0/tang-0.16.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/client/target/nemo-client-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/runtime/driver/target/nemo-driver-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/compiler/backend/target/nemo-compiler-backend-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/compiler/optimizer/target/nemo-compiler-optimizer-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Nemo-master-builder/runtime/common/target/nemo-runtime-common-0.1-SNAPSHOT.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/grpc/grpc-netty/1.7.0/grpc-netty-1.7.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/grpc/grpc-core/1.7.0/grpc-core-1.7.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/grpc/grpc-context/1.7.0/grpc-context-1.7.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/com/google/instrumentation/instrumentation-api/0.4.3/instrumentation-api-0.4.3.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/opencensus/opencensus-api/0.6.0/opencensus-api-0.6.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-codec-http2/4.1.16.Final/netty-codec-http2-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-codec-http/4.1.16.Final/netty-codec-http-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-codec/4.1.16.Final/netty-codec-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-handler/4.1.16.Final/netty-handler-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-buffer/4.1.16.Final/netty-buffer-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-common/4.1.16.Final/netty-common-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-handler-proxy/4.1.16.Final/netty-handler-proxy-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-transport/4.1.16.Final/netty-transport-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-resolver/4.1.16.Final/netty-resolver-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/netty/netty-codec-socks/4.1.16.Final/netty-codec-socks-4.1.16.Final.jar:/home/jenkins/jenkins-slave/maven-repositories/0/io/grpc/grpc-protobuf/1.7.0/grpc-protobuf-1.7.0.jar:/home/jenkins/jenkins-slave/maven-repositories/0/com/google/protobu

[GitHub] ivankelly commented on issue #1797: Move version out of LedgerMetadata

[GitBox]

ivankelly commented on issue #1797: Move version out of LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797#issuecomment-436964726
 
 
   I didn't do it in this change, but I want to change LedgerManager to use 
CompletableFutures instead of callbacks, because everywhere that uses it is 
converting to Futures anyhow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #1797: Move version out of LedgerMetadata

[GitBox]

ivankelly opened a new pull request #1797: Move version out of LedgerMetadata
URL: https://github.com/apache/bookkeeper/pull/1797
 
 
   The version is a property of the metadata store and not of the
   LedgerMetadata object itself. Putting it in the LedgerMetadata forces
   that LedgerMetadata to be mutable, as it needs to be updated before
   being sent to the LedgerManager.
   
   This change moves version out of LedgerMetadata and modifies the
   LedgerManager to use Versioned.
   
   This is a large change, but almost purely mechanical.
   
   TestHttpService has been modified to remove a preexisting flake in a
   test which had to be modified for this change.
   
   Master issue: #281
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #1682: Added BlockingQueue implementation based on JCtools

[GitBox]

merlimat commented on issue #1682: Added BlockingQueue implementation based on 
JCtools
URL: https://github.com/apache/bookkeeper/pull/1682#issuecomment-436907358
 
 
> @merlimat any perf/measurements that you can share?
   
   @jvrao 
   Not isolated to this implementation. It's tricky to properly microbenchmark 
a concurrency queue.
   
   This really brings 2 benefits: 
* Avoiding contention on the enqueuing threads (eg: IO threads passing 
items to journal, or submitting tasks the OrderedSafeExecutor)
* Combined with a busy-poll strategy, it will avoid the latency penalty of 
context switch almost completely since the receiving thread will not "sleep" or 
wait for a new item in the queue, but it will be actively polling and never 
descheduled (this when also combining with CpuAffinity, see #1641).
   
   Some test results were shared here: 
https://github.com/netty/netty/pull/8267#issuecomment-419538339 . Though this 
was including Pulsar in the mix, on top of BK, with same kind of optimizations 
applied.
   
   
   > I am curious why you decided to implement blocking queue on top of 
JCtools' MPSC vs using disruptor that implements it.
   
   @dlg99 @eolivelli I was not aware of conversant library. I was looking at 
Lmax disruptor but it forces to change all your code to fit into its model of 
dedicating a thread to it. I wanted to have a regular blocking queue so that 
this can also be directly plugged into the OrderedSafeExecutor (or any JDK 
executor), so that existing code doesn't have to be modified.
   
   JCTool seems to be well maintained, the concurrency code is very 
scrupulously reviewed and is being used in several high profile projects, from 
Netty, Cassandra, Log4j, etc.
   
   > Same curiosity from me. Even Netty has its own MCSP queue
   
   Netty is actually importing JCTools queue :) 
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services