Build failed in Jenkins: bookkeeper_postcommit_master_python #17

2018-10-26 Thread Apache Jenkins Server
See 


--
[...truncated 15.48 KB...]
root  7076 2 0 19:47:34  0.0  0.0 [kworker/8:1]
root  7634 2 0   Oct 25  0.0  0.0 [loop4]
root  8017 2 0 17:50:22  0.0  0.0 [kworker/6:1]
root  8452 2 0   Oct 24  0.0  0.0 [kworker/21:1H]
root  8573 2 0 17:51:08  0.0  0.0 [kworker/2:1]
root  8652 2 0 17:51:10  0.0  0.0 [kworker/0:0]
root  8804 2 0 17:51:12  0.0  0.0 [kworker/10:0]
dd-agent  8811 1  8811 17:51:12  0.0  0.0 
/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/bin/supervisord -c 
/etc/dd-agent/supervisor.conf
dd-agent  8861  8811  8861 17:51:13  1.2  0.0 
/opt/datadog-agent/bin/process-agent
dd-agent  8862  8811  8862 17:51:13  0.0  0.0 /opt/datadog-agent/bin/trace-agent
dd-agent  8863  8811  8863 17:51:13  0.1  0.0 
/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/ddagent.py
dd-agent  8871  8811  8871 17:51:13  0.3  0.0 
/opt/datadog-agent/embedded/bin/python /opt/datadog-agent/agent/agent.py 
foreground --use-local-forwarder
root  8881 2 0 05:19:30  0.0  0.0 [kworker/u65:2]
root  8886 2 0   Oct 25  0.0  0.0 [kworker/9:0]
root  9197 2 0   Oct 25  0.0  0.0 [kworker/1:2]
root 10164 2 0 20:57:01  0.0  0.0 [kworker/4:2]
root 10351 2 0   Oct 24  0.0  0.0 [kworker/17:1H]
root 10550 2 0   Oct 24  0.0  0.0 [kworker/20:1H]
root 10552 2 0   Oct 24  0.0  0.0 [kworker/22:1H]
root 11594 2 0 12:19:28  0.0  0.0 [kworker/u65:0]
root 12231 2 0 12:31:55  0.0  0.0 [kworker/0:2]
root 13483 2 0   Oct 24  0.0  0.0 [kworker/18:1H]
root 13565 2 0   Oct 25  0.0  0.0 [kworker/1:0]
root 13805 2 0   Oct 25  0.0  0.0 [kworker/2:2]
root 14363 2 0   Oct 25  0.0  0.0 [kworker/12:0]
root 14543 2 0   Oct 24  0.0  0.0 [kworker/13:1H]
root 14681 2 0   Oct 25  0.0  0.0 [kworker/8:0]
root 14971 2 0   Oct 25  0.0  0.0 [kworker/23:1]
root 15185 2 0   Oct 25  0.0  0.0 [kworker/22:1]
root 15216 1 15216   Oct 25  0.0  0.0 /lib/systemd/systemd-udevd
root 15218 2 0   Oct 25  0.0  0.0 [kworker/21:0]
root 15749 2 0 06:33:09  0.0  0.0 [kworker/4:1]
root 15911 2 0   Oct 25  0.0  0.0 [kworker/23:2]
root 15914 2 0   Oct 25  0.0  0.0 [kworker/7:2]
root 16096 2 0   Oct 25  0.0  0.0 [kworker/17:2]
root 16271 2 0   Oct 25  0.0  0.0 [kworker/16:0]
root 16287 2 0   Oct 25  0.0  0.0 [kworker/15:1]
root 16851 2 0   Oct 25  0.0  0.0 [kworker/13:1]
root 17059 2 0   Oct 25  0.0  0.0 [kworker/14:2]
root 17114 2 0   Oct 24  0.0  0.0 [loop0]
root 17241 2 0   Oct 24  0.0  0.0 [kworker/22:2]
root 17242 1 17242   Oct 24  0.0  0.0 /usr/lib/snapd/snapd
root 17477 2 0   Oct 24  0.0  0.0 [loop1]
root 17676 2 0   Oct 24  0.0  0.0 [kworker/3:2]
root 17706 2 0   Oct 25  0.0  0.0 [kworker/16:1]
root 17847 29511 17847   Oct 24  0.0  0.0 sshd: jenkins [priv]
jenkins  17863 1 17863   Oct 24  0.0  0.0 /lib/systemd/systemd --user
jenkins  17868 17863 17863   Oct 24  0.0  0.0 (sd-pam)
jenkins  17878 17847 17847   Oct 24  0.1  0.0 sshd: jenkins@notty
jenkins  17883 17878 17883   Oct 24  0.0  0.0 bash -c cd 
"/home/jenkins/jenkins-slave" && /home/jenkins/tools/java/jdk1.8.0_131/bin/java 
-Xmx8g -Xms1g -XX:MaxPermSize=512m -jar slave.jar
jenkins  17884 17883 17883   Oct 24  1.6  3.4 
/home/jenkins/tools/java/jdk1.8.0_131/bin/java -Xmx8g -Xms1g 
-XX:MaxPermSize=512m -jar slave.jar
root 19275 1 19274   Oct 24  0.0  0.0 python2.7 loggy.py --daemonize 
--user=root --group=root
root 19514 2 0   Oct 24  0.0  0.0 [loop2]
root 19657 2 0   Oct 25  0.0  0.0 [kworker/20:0]
root 19742 2 0   Oct 25  0.0  0.0 [kworker/5:0]
root 19878 2 0 16:47:03  0.0  0.0 [kworker/5:1]
root 20981 2 0   Oct 25  0.0  0.0 [kworker/3:1]
root 22240 1 22240   Oct 24  0.0  0.0 /lib/systemd/systemd-logind
ntp  22372 1 22372   Oct 24  0.0  0.0 /usr/sbin/ntpd -p 
/var/run/ntpd.pid -g -u 113:120
root 22379 2 0   Oct 24  0.0  0.0 [kworker/13:2]
root 22407 2 0   Oct 25  0.0  0.0 [kworker/19:1]
root 23375 2 0   Oct 24  0.0  0.0 [kworker/23:1H]
root 24885 2 0   Oct 25  0.0  0.0 [kworker/21:1]
jenkins  25184 17884 17883 12:31:11  0.0  0.0 /bin/bash -xe 
/tmp/jenkins1961588196358159697.sh
jenkins  25186 25184 17883 12:31:11  0.0  0.0 bash 
.test-infra/scripts/pre-docker-tests.sh
jenkins  25189 25186 17883 12:31:11  0.0  0.0 ps -eo 
euser,pid,ppid,pgid,start,pcpu,pmem,cmd
root 25339 2 0   Oct 25  0.0  0.0 [kworker/15:0]
root 25567 2 0   Oct 24  0.0  0.0 [loop3]

Jenkins build became unstable: bookkeeper_release_branch_48_java8 #72

2018-10-26 Thread Apache Jenkins Server
See 




Build failed in Jenkins: bookkeeper_release_branch_48_integrationtests #72

2018-10-26 Thread Apache Jenkins Server
See 


--
[...truncated 1.32 MB...]
---
 T E S T S
---
Running org.apache.bookkeeper.tests.backwardcompat.TestCompatRecoveryNoPassword
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.384 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

2018-10-26\T\13:05:33.648 [INFO] 
2018-10-26\T\13:05:33.648 [INFO] 

2018-10-26\T\13:05:33.648 [INFO] Building Apache BookKeeper :: Tests :: 
Backward Compatibility :: Test upgrade 4.1.0 to current in cluster with cookies 
4.8.0-SNAPSHOT
2018-10-26\T\13:05:33.648 [INFO] 

2018-10-26\T\13:05:33.686 [INFO] 
2018-10-26\T\13:05:33.686 [INFO] --- 
groovy-eclipse-compiler:2.9.2-04:add-groovy-build-paths 
(default-add-groovy-build-paths) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:33.687 [INFO] Adding /src/main/groovy to the list of source 
folders
2018-10-26\T\13:05:33.687 [INFO] Adding /src/test/groovy to the list of test 
source folders
2018-10-26\T\13:05:33.687 [INFO] 
2018-10-26\T\13:05:33.687 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:35.581 [INFO] 
2018-10-26\T\13:05:35.581 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:35.584 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-10-26\T\13:05:35.584 [INFO] skip non existing resourceDirectory 

2018-10-26\T\13:05:35.584 [INFO] Copying 3 resources
2018-10-26\T\13:05:35.585 [INFO] 
2018-10-26\T\13:05:35.585 [INFO] --- maven-compiler-plugin:3.7.0:compile 
(default-compile) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:35.587 [INFO] No sources to compile
2018-10-26\T\13:05:35.587 [INFO] 
2018-10-26\T\13:05:35.587 [INFO] --- maven-resources-plugin:2.7:testResources 
(default-testResources) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:35.588 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-10-26\T\13:05:35.588 [INFO] Copying 1 resource
2018-10-26\T\13:05:35.589 [INFO] Copying 3 resources
2018-10-26\T\13:05:35.589 [INFO] 
2018-10-26\T\13:05:35.590 [INFO] --- maven-compiler-plugin:3.7.0:testCompile 
(default-testCompile) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:35.595 [INFO] Changes detected - recompiling the module!
2018-10-26\T\13:05:35.596 [INFO] Using Groovy-Eclipse compiler to compile both 
Java and Groovy files
2018-10-26\T\13:05:35.879 [INFO] 
2018-10-26\T\13:05:35.879 [INFO] --- maven-surefire-plugin:2.8.1:test 
(default-test) @ old-cookie-new-cluster ---
2018-10-26\T\13:05:35.884 [INFO] Surefire report directory: 


---
 T E S T S
---
Running 
org.apache.bookkeeper.tests.backwardcompat.TestCompatUpgradeOldServerInClusterWithCookies
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.93 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

2018-10-26\T\13:07:02.610 [INFO] 
2018-10-26\T\13:07:02.611 [INFO] 

2018-10-26\T\13:07:02.611 [INFO] Building Apache BookKeeper :: Tests :: 
Backward Compatibility :: Test old clients working on current server 
4.8.0-SNAPSHOT
2018-10-26\T\13:07:02.611 [INFO] 

2018-10-26\T\13:07:02.661 [INFO] 
2018-10-26\T\13:07:02.661 [INFO] --- 
groovy-eclipse-compiler:2.9.2-04:add-groovy-build-paths 
(default-add-groovy-build-paths) @ backward-compat-current-server-old-clients 
---
2018-10-26\T\13:07:02.662 [INFO] Adding /src/main/groovy to the list of source 
folders
2018-10-26\T\13:07:02.662 [INFO] Adding /src/test/groovy to the list of test 
source folders
2018-10-26\T\13:07:02.662 [INFO] 
2018-10-26\T\13:07:02.662 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ backward-compat-current-server-old-clients ---
2018-10-26\T\13:07:04.525 [INFO] 
2018-10-26\T\13:07:04.525 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ backward-compat-current-server-old-clients ---
2018-10-26\T\13:07:04.527 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2018-10-26\T\13:07:04.527 [INFO] skip non existing resourceDirectory 

Jenkins build became unstable: bookkeeper_postcommit_master_java8 #299

2018-10-26 Thread Apache Jenkins Server
See 




Build failed in Jenkins: bookkeeper_release_branch_47_integrationtests #187

2018-10-26 Thread Apache Jenkins Server
See 


--
[...truncated 10.64 KB...]
root   570 2 0   Jul 25  0.0  0.0 [edac-poller]
root   625 2 0   Jul 25  0.0  0.0 [kipmi0]
root   636 2 0   Jul 25  0.0  0.0 [kworker/5:1H]
root   637 2 0   Jul 25  0.0  0.0 [kworker/0:1H]
root   639 2 0   Jul 25  0.0  0.0 [kvm-irqfd-clean]
root   679 2 0   Jul 25  0.0  0.0 [kworker/14:1H]
root   727 1   726   Jul 25  0.0  0.0 upstart-socket-bridge --daemon
root   753 2 0   Jul 25  0.0  0.0 [kworker/12:1H]
root   754 2 0   Jul 25  0.0  0.0 [kworker/1:1H]
root   762 1   762   Jul 25  0.0  0.0 dhclient -1 -v -pf 
/run/dhclient.em1.pid -lf /var/lib/dhcp/dhclient.em1.leases em1
root   781 2 0   Jul 25  0.0  0.0 [kworker/8:1H]
root   826 2 0   Jul 25  0.0  0.0 [kworker/4:1H]
root   847 2 0   Jul 25  0.0  0.0 [kworker/6:1H]
root   848 2 0   Jul 25  0.0  0.0 [kworker/2:1H]
root   849 2 0   Jul 25  0.0  0.0 [kworker/3:1H]
root   936 2 0   Oct 23  0.0  0.0 [kworker/11:2]
root   973 1   972   Jul 25  0.0  0.0 upstart-file-bridge --daemon
syslog1056 1  1056   Jul 25  0.0  0.0 rsyslogd
message+  1120 1  1120   Jul 25  0.0  0.0 dbus-daemon --system --fork
root  1132 1  1132   Jul 25  0.0  0.0 /lib/systemd/systemd --system
root  1158 1  1158   Jul 25  0.0  0.0 /lib/systemd/systemd-logind
root  1194 2 0   Oct 24  0.0  0.0 [kworker/u49:0]
root  1207 1  1207   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty4
root  1211 1  1211   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty5
root  1221 1  1221   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty2
root  1222 1  1222   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty3
root  1224 1  1224   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty6
root  1240 1  1240   Jul 25  0.0  0.0 /usr/sbin/sshd -D
daemon1255 1  1255   Jul 25  0.0  0.0 atd
root  1256 1  1256   Jul 25  0.0  0.0 cron
root  1289  1132  1289   Jul 25  0.0  0.0 /lib/systemd/systemd-journald
root  1290 2 0   Jul 25  0.0  0.0 [kauditd]
root  1311 1  1311   Jul 25  0.0  0.0 /usr/sbin/irqbalance
root  1493 2 0   Jul 25  0.0  0.0 [kworker/15:1H]
root  1527 2 0   Oct 24  0.0  0.0 [kworker/14:0]
root  1703 1  1703   Oct 04  0.0  0.0 /usr/bin/ruby /usr/bin/puppet 
agent
root  1801 1  1801   Jul 25  0.0  0.0 /usr/lib/postfix/master
postfix   1823  1801  1801   Jul 25  0.0  0.0 qmgr -l -t fifo -u
ntp   1915 1  1915   Jul 25  0.0  0.0 /usr/sbin/ntpd -p 
/var/run/ntpd.pid -g -u 105:116
root  2012 1  2011   Jul 25  0.0  0.0 /usr/bin/python 
/usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p 
/var/run/fail2ban/fail2ban.pid
root  2072 1  2072   Jul 25  0.0  0.0 /sbin/getty -8 38400 tty1
root  2073 1  2073   Jul 25  0.0  0.0 /sbin/getty -L ttyS0 9600 vt102
root  2124 2 0   Oct 24  0.0  0.0 [kworker/11:0]
root  2378 2 0   Jul 25  0.0  0.0 [kworker/10:1H]
nslcd 3312 1  3312   Jul 25  0.1  0.0 /usr/sbin/nslcd
root  4213 2 0   Oct 25  0.0  0.0 [kworker/2:1]
root  4226 2 0   Oct 25  0.0  0.0 [kworker/13:2]
jenkins   4416 1 21137   Aug 21  0.0  0.0 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefirebooter123717442526029744.jar
 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefire4123165900408045001tmp
 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/surefire/surefire_06109285983847161674tmp
jenkins   4455  4416 21137   Aug 21  0.7  0.8 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx2048m -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/dependency/org.apache.sling.launchpad-8.jar
 -p 45800 -Dsling.run.modes=author,notshared
root  4927 1  1120   Jul 25  0.0  0.0 
/usr/lib/x86_64-linux-gnu/systemd-shim
root  4977 2 0   Jul 25  0.0  0.0 [kworker/13:1H]
root  5661 2 0   Jul 25  0.0  0.0 [kworker/11:1H]
root  6009 2 0   Jul 25  0.0  0.0 [kworker/9:1H]
root  6433 2 0   Oct 23  0.0  0.0 [kworker/7:0]
root  7150 2 0   Oct 23  0.0  0.0 [kworker/4:0]
root  7157 2 0   Oct 23  0.0  0.0 [kworker/2:2]
jenkins   7335  4416 21137   Aug 21  104  3.5 
/usr/local/asfpackages/java/jdk1.8.0_172/jre/bin/java -Xmx2048m -jar 
/home/jenkins/jenkins-slave/workspace/sling-org-apache-sling-distribution-it-1.8/target/dependency/org.apache.sling.launchpad-8.jar
 -p 41776 -Dsling.run.modes=publish,notshared
root  8526 2 0 

[GitHub] david-streamlio opened a new issue #1763: Example application link broken

2018-10-26 Thread GitBox
david-streamlio opened a new issue #1763: Example application link broken
URL: https://github.com/apache/bookkeeper/issues/1763
 
 
   Documentation issue:
   
   The "Example Application" link on the 
https://bookkeeper.apache.org/docs/latest/api/ledger-api/ page is broken.  
   
   The Link url is 
https://bookkeeper.apache.org/docs/latest/api/example-application, but when you 
click on it, you get a 404 "The requested URL 
/docs/latest/api/example-application was not found on this server." error 
instead.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
sijie commented on issue #1762: Don't cache Bookie hostname DNS resolution 
forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-433592520
 
 
   IGNORE IT CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #1760: Kill LedgerMetadata#isConflictWith

2018-10-26 Thread GitBox
sijie commented on issue #1760: Kill LedgerMetadata#isConflictWith
URL: https://github.com/apache/bookkeeper/pull/1760#issuecomment-433592578
 
 
   IGNORE IT CI


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] sijie commented on issue #1759: Remove LedgerMetadata#getEnsembles in favour of #getAllEnsembles

2018-10-26 Thread GitBox
sijie commented on issue #1759: Remove LedgerMetadata#getEnsembles in favour of 
#getAllEnsembles
URL: https://github.com/apache/bookkeeper/pull/1759#issuecomment-433592598
 
 
   run bookkeeper-server replication tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dlg99 edited a comment on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
dlg99 edited a comment on issue #1762: Don't cache Bookie hostname DNS 
resolution forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-433547606
 
 
   LGTM but you also need to consider effects of 
   ```-Dnetworkaddress.cache.ttl``` jvm option.
   
   ```text
   networkaddress.cache.ttl
   Specified in java.security to indicate the caching policy for successful 
name lookups from the name service.. 
   The value is specified as integer to indicate the number of seconds to cache 
the successful lookup.
   
   A value of -1 indicates "cache forever". 
   
   The default behavior is to cache forever when a security manager is 
installed, and to cache for an implementation specific period of time, when a 
security manager is not installed.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dlg99 edited a comment on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
dlg99 edited a comment on issue #1762: Don't cache Bookie hostname DNS 
resolution forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-433547606
 
 
   LGTM but you also need to consider effects of 
   ```-Dnetworkaddress.cache.ttl``` jvm option.
   
   ```text
   networkaddress.cache.ttl
   Specified in java.security to indicate the caching policy for successful 
name lookups from the name service.. 
   The value is specified as integer to indicate the number of seconds to cache 
the successful lookup.
   
   A value of -1 indicates "cache forever". 
   
   The default behavior is to cache forever when a security manager is 
installed, 
   and to cache for an implementation specific period of time, when a security 
manager is not installed.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] dlg99 commented on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
dlg99 commented on issue #1762: Don't cache Bookie hostname DNS resolution 
forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-433547606
 
 
   LGTM but you also need to consider effects of 
   ```-Dnetworkaddress.cache.ttl``` jvm option.
   
   ```text
   networkaddress.cache.ttl
   Specified in java.security to indicate the caching policy for successful 
name lookups from the name service.. The value is specified as integer to 
indicate the number of seconds to cache the successful lookup.
   A value of -1 indicates "cache forever". The default behavior is to cache 
forever when a security manager is installed, and to cache for an 
implementation specific period of time, when a security manager is not 
installed.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat opened a new pull request #1764: [website] Fix broken link to ledger-api#example-application

2018-10-26 Thread GitBox
merlimat opened a new pull request #1764: [website] Fix broken link to 
ledger-api#example-application
URL: https://github.com/apache/bookkeeper/pull/1764
 
 
   Link should be to a section in same page


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
merlimat commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228687243
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
 ##
 @@ -416,14 +419,7 @@ protected ChannelFuture connect() {
 bootstrap.channel(NioSocketChannel.class);
 }
 
-ByteBufAllocator allocator;
-if (this.conf.isNettyUsePooledBuffers()) {
 
 Review comment:
   Sure


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
merlimat commented on issue #1762: Don't cache Bookie hostname DNS resolution 
forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-433566878
 
 
   run integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
merlimat commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228686668
 
 

 ##
 File path: bookkeeper-server/pom.xml
 ##
 @@ -37,6 +37,18 @@
 
   
 
+
+  org.apache.bookkeeper
+  bookkeeper-common-allocator
+  ${project.parent.version}
+  
+
 
 Review comment:
   Good point, will fix that. In general we should also refactor netty 
dependency to not use netty-all, but that's a completely different story


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
merlimat commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228659329
 
 

 ##
 File path: conf/bk_server.conf
 ##
 @@ -956,3 +956,53 @@ storage.serve.readonly.tables=false
 
 # the cluster controller schedule interval, in milliseconds. default is 30 
seconds.
 storage.cluster.controller.schedule.interval.ms=3
+
+
+#
+## Netty Allocator Settings
+#
+
+# Define the memory pooling policy.
+# Available options are:
+#   - PooledDirect: Use Direct memory for all buffers and pool the memory.
+#   Direct memory will avoid the overhead of JVM GC and most
+#   memory copies when reading and writing to socket channel.
+#   Pooling will add memory space overhead due to the fact that
+#   there will be fragmentation in the allocator and that 
threads
+#   will keep a portion of memory as thread-local to avoid
+#   contention when possible.
+#   - UnpooledHeap: Allocate memory from JVM heap without any pooling.
+#   This option has the least overhead in terms of memory usage
+#   since the memory will be automatically reclaimed by the
+#   JVM GC but might impose a performance penalty at high
+#   throughput.
+# Default is: PooledDirect
+# allocatorPoolingPolicy=PooledDirect
+  
+# Controls the amount of concurrency for the memory pool.
+# Default is to have a number of allocator arenas equals to 2 * CPUS.
 
 Review comment:
   That's current Netty default


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
merlimat commented on issue #1762: Don't cache Bookie hostname DNS resolution 
forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-433536461
 
 
   @eolivelli Added comment


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on issue #1762: Don't cache Bookie hostname DNS resolution forever

2018-10-26 Thread GitBox
merlimat commented on issue #1762: Don't cache Bookie hostname DNS resolution 
forever
URL: https://github.com/apache/bookkeeper/pull/1762#issuecomment-43389
 
 
   @dlg99 `-Dnetworkaddress.cache.ttl` cannot be passed as system property from 
CLI but rather included in `java.security` property file. 
   
   In any case, the current behavior is caching the DNS resolution forever in 
the BK client process. That comes before reaching the JVM DNS cache or the OS 
DNS cache. By creating new instances of `InetSocketAddress` we will rely on the 
JVM cache, that depends on the `networkaddress.cache.ttl` setting. 
   
   (BTW: I think the JVM cache is a *bad* thing on its own, but that's a 
different story... :) )


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
merlimat commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228687202
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java
 ##
 @@ -369,7 +391,30 @@ public BookKeeper(ClientConfiguration conf, ZooKeeper zk)
  */
 public BookKeeper(ClientConfiguration conf, ZooKeeper zk, EventLoopGroup 
eventLoopGroup)
 throws IOException, InterruptedException, BKException {
-this(conf, validateZooKeeper(zk), 
validateEventLoopGroup(eventLoopGroup), NullStatsLogger.INSTANCE,
+this(conf, validateZooKeeper(zk), 
validateEventLoopGroup(eventLoopGroup), null);
+}
+
+/**
+ * Create a bookkeeper client but use the passed in zookeeper client and
+ * client event loop group instead of instantiating those.
+ *
+ * @param conf
+ *  Client Configuration Object
+ *  {@link ClientConfiguration}
+ * @param zk
+ *  Zookeeper client instance connected to the zookeeper with which
+ *  the bookies have registered. The ZooKeeper client must be 
connected
+ *  before it is passed to BookKeeper. Otherwise a KeeperException 
is thrown.
+ * @param eventLoopGroup
+ *  An event loop group that will be used to create connections to 
the bookies
+ * @throws IOException
+ * @throws InterruptedException
+ * @throws BKException in the event of a bookkeeper connection error
+ */
+public BookKeeper(ClientConfiguration conf, ZooKeeper zk, EventLoopGroup 
eventLoopGroup,
 
 Review comment:
   Ok, I was doing this since this is the constructor we use in Pulsar :) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] merlimat commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
merlimat commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228687050
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -634,6 +639,21 @@ public Bookie(ServerConfiguration conf, StatsLogger 
statsLogger)
 this.indexDirsManager = createIndexDirsManager(conf, diskChecker, 
statsLogger.scope(LD_INDEX_SCOPE),
this.ledgerDirsManager);
 
+this.allocator = ByteBufAllocatorBuilder.create()
+.poolingPolicy(conf.getAllocatorPoolingPolicy())
+.poolingConcurrency(conf.getAllocatorPoolingConcurrency())
+.outOfMemoryPolicy(conf.getAllocatorOutOfMemoryPolicy())
+.outOfMemoryListener((ex) -> {
+try {
+LOG.error("Unable to allocate memory, exiting bookie", 
ex);
+shutdown(-1);
+} finally {
+Runtime.getRuntime().halt(-1);
 
 Review comment:
   How should be pass it? 
   
   What I wanted to achieve here is: 
* Try to shutdown bookie gracefully (mainly to close ZK session such that 
new ledgers won't choose this bookie)
* No matter what, bring the process down after that attempt. This would 
protect from a thread hanging on in background to keep JVM alive. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228458464
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -1224,8 +1245,8 @@ public void recoveryAddEntry(ByteBuf entry, 
WriteCallback cb, Object ctx, byte[]
 }
 }
 
-static ByteBuf createExplicitLACEntry(long ledgerId, ByteBuf explicitLac) {
-ByteBuf bb = PooledByteBufAllocator.DEFAULT.directBuffer(8 + 8 + 4 + 
explicitLac.capacity());
+private ByteBuf createExplicitLACEntry(long ledgerId, ByteBuf explicitLac) 
{
 
 Review comment:
   Unrelated to this patch, we really shouldn't have serialization code in 
Bookie


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228462059
 
 

 ##
 File path: conf/bk_server.conf
 ##
 @@ -956,3 +956,53 @@ storage.serve.readonly.tables=false
 
 # the cluster controller schedule interval, in milliseconds. default is 30 
seconds.
 storage.cluster.controller.schedule.interval.ms=3
+
+
+#
+## Netty Allocator Settings
+#
+
+# Define the memory pooling policy.
+# Available options are:
+#   - PooledDirect: Use Direct memory for all buffers and pool the memory.
+#   Direct memory will avoid the overhead of JVM GC and most
+#   memory copies when reading and writing to socket channel.
+#   Pooling will add memory space overhead due to the fact that
+#   there will be fragmentation in the allocator and that 
threads
+#   will keep a portion of memory as thread-local to avoid
+#   contention when possible.
+#   - UnpooledHeap: Allocate memory from JVM heap without any pooling.
+#   This option has the least overhead in terms of memory usage
+#   since the memory will be automatically reclaimed by the
+#   JVM GC but might impose a performance penalty at high
+#   throughput.
+# Default is: PooledDirect
+# allocatorPoolingPolicy=PooledDirect
+  
+# Controls the amount of concurrency for the memory pool.
+# Default is to have a number of allocator arenas equals to 2 * CPUS.
 
 Review comment:
   on a 48 core machine, that's 96 pools. Seems a bit high.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228460372
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
 ##
 @@ -416,14 +419,7 @@ protected ChannelFuture connect() {
 bootstrap.channel(NioSocketChannel.class);
 }
 
-ByteBufAllocator allocator;
-if (this.conf.isNettyUsePooledBuffers()) {
 
 Review comment:
   isNettyUsePooledBuffers should be marked as deprecated now, and the 
deprecation note pointing to the new method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228449352
 
 

 ##
 File path: bookkeeper-server/pom.xml
 ##
 @@ -37,6 +37,18 @@
 
   
 
+
+  org.apache.bookkeeper
+  bookkeeper-common-allocator
+  ${project.parent.version}
+  
+
 
 Review comment:
   Is everything that pulls in the allocator going to have to do this? It may 
be worth marking netty-buffer as provided in the allocator pom


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228458464
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -1224,8 +1245,8 @@ public void recoveryAddEntry(ByteBuf entry, 
WriteCallback cb, Object ctx, byte[]
 }
 }
 
-static ByteBuf createExplicitLACEntry(long ledgerId, ByteBuf explicitLac) {
-ByteBuf bb = PooledByteBufAllocator.DEFAULT.directBuffer(8 + 8 + 4 + 
explicitLac.capacity());
+private ByteBuf createExplicitLACEntry(long ledgerId, ByteBuf explicitLac) 
{
 
 Review comment:
   Unrelated to this patch, but we really shouldn't have serialization code in 
Bookie


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228459284
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java
 ##
 @@ -369,7 +391,30 @@ public BookKeeper(ClientConfiguration conf, ZooKeeper zk)
  */
 public BookKeeper(ClientConfiguration conf, ZooKeeper zk, EventLoopGroup 
eventLoopGroup)
 throws IOException, InterruptedException, BKException {
-this(conf, validateZooKeeper(zk), 
validateEventLoopGroup(eventLoopGroup), NullStatsLogger.INSTANCE,
+this(conf, validateZooKeeper(zk), 
validateEventLoopGroup(eventLoopGroup), null);
+}
+
+/**
+ * Create a bookkeeper client but use the passed in zookeeper client and
+ * client event loop group instead of instantiating those.
+ *
+ * @param conf
+ *  Client Configuration Object
+ *  {@link ClientConfiguration}
+ * @param zk
+ *  Zookeeper client instance connected to the zookeeper with which
+ *  the bookies have registered. The ZooKeeper client must be 
connected
+ *  before it is passed to BookKeeper. Otherwise a KeeperException 
is thrown.
+ * @param eventLoopGroup
+ *  An event loop group that will be used to create connections to 
the bookies
+ * @throws IOException
+ * @throws InterruptedException
+ * @throws BKException in the event of a bookkeeper connection error
+ */
+public BookKeeper(ClientConfiguration conf, ZooKeeper zk, EventLoopGroup 
eventLoopGroup,
 
 Review comment:
   Please don't create another public constructor for BookKeeper. The client 
should be constructed through the builder.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228458207
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -634,6 +639,21 @@ public Bookie(ServerConfiguration conf, StatsLogger 
statsLogger)
 this.indexDirsManager = createIndexDirsManager(conf, diskChecker, 
statsLogger.scope(LD_INDEX_SCOPE),
this.ledgerDirsManager);
 
+this.allocator = ByteBufAllocatorBuilder.create()
+.poolingPolicy(conf.getAllocatorPoolingPolicy())
+.poolingConcurrency(conf.getAllocatorPoolingConcurrency())
+.outOfMemoryPolicy(conf.getAllocatorOutOfMemoryPolicy())
+.outOfMemoryListener((ex) -> {
+try {
+LOG.error("Unable to allocate memory, exiting bookie", 
ex);
+shutdown(-1);
+} finally {
+Runtime.getRuntime().halt(-1);
 
 Review comment:
   Don't call this directly here.  Bookie is used in all tests, and halt 
getting called in the middle of a test is a pain to track down (especially 
since in OOM scenario, the LOG may fail). 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on a change in pull request #1755: Configure Netty allocator in bookie and client

2018-10-26 Thread GitBox
ivankelly commented on a change in pull request #1755:  Configure Netty 
allocator in bookie and client 
URL: https://github.com/apache/bookkeeper/pull/1755#discussion_r228460816
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/checksum/CRC32CDigestManager.java
 ##
 @@ -38,10 +39,10 @@ protected MutableInt initialValue() throws Exception {
 }
 };
 
-public CRC32CDigestManager(long ledgerId, boolean useV2Protocol) {
-super(ledgerId, useV2Protocol);
+public CRC32CDigestManager(long ledgerId, boolean useV2Protocol, 
ByteBufAllocator allocator) {
+super(ledgerId, useV2Protocol, allocator);
 if (!Sse42Crc32C.isSupported()) {
-log.error("Sse42Crc32C is not supported, will use less slower 
CRC32C implementation.");
+log.error("Sse42Crc32C is not supported, will use slower CRC32C 
implementation.");
 
 Review comment:
   If you're correcting the grammar, you should at least include the indefinite 
article ;)
   
   Let's see if this suggestion thing works...
   
   ```suggestion
   log.error("Sse42Crc32C is not supported, will use a slower 
CRC32C implementation.");
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] ivankelly commented on issue #1760: Kill LedgerMetadata#isConflictWith

2018-10-26 Thread GitBox
ivankelly commented on issue #1760: Kill LedgerMetadata#isConflictWith
URL: https://github.com/apache/bookkeeper/pull/1760#issuecomment-433366561
 
 
   rerun integration tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services