2018-03-09 15:25:15 UTC - Nicolas Ha: I am trying to set up a CI environment
using CircleCI 2, the command I use to run tests fails with this:
```
Caused by: java.io.IOException: Failed to load RocksDB JNI library
at
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>
(KeyValueStorageRocksDB.java:81)
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>
(KeyValueStorageRocksDB.java:73)
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB$1.newKeyValueStorage
(KeyValueStorageRocksDB.java:44)
org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>
(LedgerMetadataIndex.java:51)
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize
(DbLedgerStorage.java:142)
org.apache.bookkeeper.bookie.Bookie.<init> (Bookie.java:516)
org.apache.bookkeeper.proto.BookieServer.newBookie (BookieServer.java:313)
org.apache.bookkeeper.proto.BookieServer.<init>
(BookieServer.java:116)
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies
(LocalBookkeeperEnsemble.java:179)
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone
(LocalBookkeeperEnsemble.java:231)
sun.reflect.NativeMethodAccessorImpl.invoke0
(NativeMethodAccessorImpl.java:-2)
sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke (Method.java:498)
clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
clojure.lang.Reflector.invokeNoArgInstanceMember (Reflector.java:313)
backend.components.pulsar_broker.PulsarBroker/fn (pulsar_broker.clj:109)
backend.components.pulsar_broker.PulsarBroker.start (pulsar_broker.clj:101)
```
Is this a know issue with CI providers? Do I have to package a special
dependency for some environments?
----
2018-03-09 17:25:14 UTC - Matteo Merli: @Nicolas Ha What kind of environment
the CI build is running on?
----
2018-03-09 17:27:09 UTC - Nicolas Ha: not sure about the details - I was trying
out circle CI 2.0 <https://circleci.com/docs/2.0/>
I know 1.0 works fine, but as far as I know 2.0 is nearly a rewrite to support
a container/docker-based workflow
----
2018-03-09 17:27:23 UTC - Nicolas Ha: I can SSH into the box though, so I will
try learning more
----
2018-03-09 17:27:54 UTC - Matteo Merli: Ok, also is there any other logged
exception that explains why it fails to load the library?
----
2018-03-09 17:28:33 UTC - Matteo Merli: @david-jin You can tune down the memory
a lot more. In any case, if you have a single VM for all the components, you
should consider using the pulsar standalone, rather than the individual services
----
2018-03-09 17:29:14 UTC - Matteo Merli: With standalone, you have 1 single JVM
process that contains everything (broker, bookie, zookeeper).
----
2018-03-09 17:30:11 UTC - Matteo Merli: I’ve seen it failing on Windows… but it
should work in virtually every linux/mac distribution, since the jni library is
statically compiled
----
2018-03-09 17:30:28 UTC - Nicolas Ha: Oh there is actually! I missed that:
```
Caused by: java.lang.UnsatisfiedLinkError:
/tmp/librocksdbjni4356631003914935156.so: Error loading shared library
ld-linux-x86-64.so.2: No such file or directory (needed by
/tmp/librocksdbjni4356631003914935156.so)
at java.lang.ClassLoader$NativeLibrary.load (ClassLoader.java:-2)
java.lang.ClassLoader.loadLibrary0 (ClassLoader.java:1941)
java.lang.ClassLoader.loadLibrary (ClassLoader.java:1824)
java.lang.Runtime.load0 (Runtime.java:809)
java.lang.System.load (System.java:1086)
org.rocksdb.NativeLibraryLoader.loadLibraryFromJar
(NativeLibraryLoader.java:78)
org.rocksdb.NativeLibraryLoader.loadLibrary (NativeLibraryLoader.java:56)
org.rocksdb.RocksDB.loadLibrary (RocksDB.java:64)
org.rocksdb.RocksDB.<clinit> (RocksDB.java:35)
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>
(KeyValueStorageRocksDB.java:79)
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>
(KeyValueStorageRocksDB.java:73)
org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB$1.newKeyValueStorage
(KeyValueStorageRocksDB.java:44)
org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>
(LedgerMetadataIndex.java:51)
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize
(DbLedgerStorage.java:142)
org.apache.bookkeeper.bookie.Bookie.<init> (Bookie.java:516)
org.apache.bookkeeper.proto.BookieServer.newBookie (BookieServer.java:313)
org.apache.bookkeeper.proto.BookieServer.<init>
(BookieServer.java:116)
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.runBookies
(LocalBookkeeperEnsemble.java:179)
org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble.startStandalone
(LocalBookkeeperEnsemble.java:231)
```
----
2018-03-09 17:31:26 UTC - Nicolas Ha: (using
`org.apache.pulsar/pulsar-broker-shaded "1.21.0-incubating"`)
----
2018-03-09 17:31:40 UTC - Matteo Merli: Ok, is that based on Alpine linux ?
----
2018-03-09 17:32:27 UTC - Nicolas Ha: It is based on
`clojure:boot-2.7.2-alpine` (so I guess yes)
----
2018-03-09 17:33:01 UTC - Matteo Merli: ok, that’s the issue, Alpine doesn’t
have the regular linux glibc
----
2018-03-09 17:33:27 UTC - Matteo Merli: can you switch the image to one based
on debian/redhat ?
----
2018-03-09 17:33:51 UTC - Nicolas Ha: I sure can :slightly_smiling_face: which
one is it? (not super familiar with the current state of docker)
----
2018-03-09 17:34:17 UTC - Nicolas Ha: I think all I have to do is edit this one
<https://github.com/Quantisan/docker-clojure/blob/master/alpine/boot/Dockerfile>
and change the “FROM”
----
2018-03-09 17:34:21 UTC - Matteo Merli: just try to remove the “alpine” tag:
`clojure:boot-2.7.2`
----
2018-03-09 17:34:53 UTC - Nicolas Ha: Oh right! :smile:
----
2018-03-09 17:39:59 UTC - Nicolas Ha: it worked! onto the next error
:slightly_smiling_face: Thanks!
----
2018-03-09 17:42:21 UTC - Matteo Merli: :slightly_smiling_face:
----
2018-03-10 01:11:31 UTC - David Kjerrumgaard: @David Kjerrumgaard has joined
the channel
----
2018-03-10 01:36:46 UTC - david-jin: but， i want to experiment with the
geo-replication. if i just use standalone mode, can i deploy a instance? and
geo-replication?
----
2018-03-10 02:03:07 UTC - david-jin: Is there any advice to deploy an instance
using standalone mode ?
----
2018-03-10 02:07:05 UTC - david-jin: So, if I want to deploy an instance. I
should first deploy 3 global zookeeper in 3 machines, then deploy 3 standalone
pulsar? Is that right?
----
2018-03-10 02:08:25 UTC - david-jin: thus, every machine have 2 JVMs?
----
2018-03-10 02:10:15 UTC - Matteo Merli: For this kind of scenario, with just
one machine in each region, I would suggest to not bother deploying the global
zookeeper. Just do the standalone in each region and set the configuration
manually in each of them
----
2018-03-10 02:11:31 UTC - Matteo Merli: For example you have to use the CLI
tool to create the "cluster" and pass a name and the URL for that cluster (the
IP of the standalone service...)
----
2018-03-10 02:12:13 UTC - Matteo Merli: You have to do that for each region so
that everyone knows about the others and where they sre
----
2018-03-10 02:12:21 UTC - david-jin: but, how to do the geo-replication
manually?
----
2018-03-10 02:13:01 UTC - Matteo Merli: After that, you can create a global
namespace and set it to do replication between regions a, b and c
----
2018-03-10 02:13:36 UTC - Matteo Merli: Since there is no global zookeeper, you
have to repeat the operation for each region
----
2018-03-10 02:13:57 UTC - Matteo Merli: At that point everything is done
----
2018-03-10 02:14:35 UTC - david-jin: In my opion, pulsar's advantage over other
MQ, the geo-replication is important.
----
2018-03-10 02:15:39 UTC - Matteo Merli: Yes, the point is that you get
geo-replication of data without need for global zookeeper
----
2018-03-10 02:16:03 UTC - Matteo Merli: That is just to keep the configuration
consistent across region
----
2018-03-10 02:16:25 UTC - Matteo Merli: Which is necessary if you have many
namespaces and tenants
----
2018-03-10 02:16:59 UTC - Matteo Merli: But otherwise you can get away without
it -- but still have geo replicationw
----
2018-03-10 02:17:53 UTC - Matteo Merli: Let me try to get a simple example for
replicating between 2 standalone servers
----
2018-03-10 02:30:06 UTC - david-jin: yes, could you give me an example? thank
you very much.
----
2018-03-10 02:44:36 UTC - Matteo Merli: @david-jin Here’s the quick example to
setup the replication between Pulsar standalone processes:
<https://gist.github.com/merlimat/bfdd3c66bfa7387fe6e3bfa3bb263051>
----
2018-03-10 02:44:45 UTC - Matteo Merli: @Matteo Merli shared a file:
<https://apache-pulsar.slack.com/files/U680ZCXA5/F9M31L1PS/standalonereplication.md|StandaloneReplication.md>
----
2018-03-10 03:17:36 UTC - david-jin: thanks very much
----
2018-03-10 03:49:23 UTC - Matteo Merli: no problem, let me know if it works for
you
----
