2018-08-15 13:16:12 UTC - Shalin: I keep getting this error
`KeeperException$NodeExistsException: KeeperErrorCode = NodeExists` while
trying to run `initialize_cluster_metadata` on the zookeeper.
----
2018-08-15 14:16:46 UTC - Shalin: Looks like this is the reason
`java.net.UnknownHostException:<zookeeper_ipv4_address:2181> : invalid
IPv6 address`
----
2018-08-15 14:17:17 UTC - Shalin: Any clue why it is being identified as an
IPv6 address?
----
2018-08-15 14:22:24 UTC - Mate Varga: Hi.
We need to implement an event log -- first for audit purposes. In some cases we
will have to query that log - it'd possibly store a few millions of rows per
day and we'd need to store stuff for many years.
We'll use Pulsar as the messaging tool, but would you actually consider Pulsar
as an event store?
----
2018-08-15 14:29:41 UTC - William Fry: Full stack trace here:
```
Aug 15 14:23:06 ip-10-0-1-77 pulsar: 14:23:06.724 [main] ERROR
org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Failed to create zookeeper
client to 10.0.1.210:2181:2181,10.0.1.171:2181:2181,10.0.1.215:2181:2181
Aug 15 14:23:06 ip-10-0-1-77 pulsar: java.net.UnknownHostException:
10.0.1.210:2181: invalid IPv6 address
```
----
2018-08-15 14:37:05 UTC - Matteo Merli: @Mate Varga in 2.1 release we have
introduced the concept of "tiered storage". That will make it possible to push
older data into 2nd tier storage, like S3. This is the biggest enabler, from a
cost and manageability perspective to use Pulsar as an event store
----
2018-08-15 14:38:08 UTC - Sijie Guo: You have two 2181 per host.
----
2018-08-15 14:38:51 UTC - Shalin: Could you clarify that a bit more?
----
2018-08-15 14:39:02 UTC - Sijie Guo: It should be
10.0.1.210:2181,10.0.1.171:2181,10.0.1.215:2181
----
2018-08-15 14:39:30 UTC - Mate Varga: thanks. I'm not too worried about the
storage requirements for now, I am mostly interested in querying
----
2018-08-15 14:39:57 UTC - Mate Varga: so what's going to happen if there's e.g.
an audit_events topic, and I'd like to find all the events with e.g.
correlation_id=1234
----
2018-08-15 14:40:21 UTC - Mate Varga: e.g. with Kafka this can (in theory) be
done with KSQL (afaik), but not sure about Pulsar
----
2018-08-15 14:40:22 UTC - William Fry: Now the issue is different and much more
concise:
```
Exception in thread "main"
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /managed-ledgers
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:139)
```
----
2018-08-15 14:40:50 UTC - William Fry: Looks the same as this:
<https://github.com/apache/incubator-pulsar/issues/2037>
----
2018-08-15 14:43:26 UTC - William Fry: How does the `/managed-ledger` node
already exist? How can we remove it if it does already exist // or ignore and
continue with cluster initialization?
----
2018-08-15 14:48:12 UTC - William Fry: For context, this is triggered when
running `bin/pulsar initialize-cluster-metadata --cluster local --zookeeper
localhost:2181 --configuration-store localhost:2181 --web-service-url
<http://10.0.1.77:8080/> --broker-service-url <pulsar://10.0.1.77:6650/>`
----
2018-08-15 14:54:14 UTC - William Fry: I was able to fix this by following the
thread here:
<https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1533234641000563>
----
2018-08-15 14:54:32 UTC - William Fry: But I still don’t quite understand how
it happened
----
2018-08-15 14:54:58 UTC - William Fry: Using the `zookeeper-shell` and then
running `rmr` on existing allowed us to initialize the cluster matadata without
any trouble
----
2018-08-15 14:55:06 UTC - William Fry: But why was it there in the first place?
----
2018-08-15 14:55:17 UTC - William Fry: cc: @Josh West @Sijie Guo @Matteo Merli
----
2018-08-15 15:25:23 UTC - Matteo Merli: with KSQL, you can do a query on
streaming data, though not on the data stored in past
----
2018-08-15 15:26:34 UTC - Matteo Merli: Please take a look at this proposal,
<https://github.com/apache/incubator-pulsar/wiki/PIP-19:-Pulsar-SQL>
This is being actively worked on and scheduled for 2.2 release very soon.
----
2018-08-15 15:39:35 UTC - Matteo Merli: ..buiding..
----
2018-08-15 16:54:17 UTC - Mate Varga: brilliant
----
2018-08-15 16:54:30 UTC - Mate Varga: I hope Pulsar's going to be as good as
these features are promising :slightly_smiling_face:
ok_hand : Matteo Merli
----
2018-08-15 16:57:05 UTC - Mate Varga: What's Pulsar's upgrade story now (and in
the future)? Is it likely to be easy to upgrade between minor versions?
----
2018-08-15 16:57:48 UTC - Matteo Merli: It should always be easy to upgrade
across minor and major versions
----
2018-08-15 16:58:32 UTC - Matteo Merli: even 1.22 to 2.x, the only big changes
were few scripts renamed or log4j 1.x to log4j2
----
2018-08-15 16:58:53 UTC - Mate Varga: can newer versions read data persisted by
older versions?
----
2018-08-15 16:58:56 UTC - Matteo Merli: it’s always possible to do upgrade
& dowgrade across versions
----
2018-08-15 16:59:00 UTC - Mate Varga: oh. cool.
----
2018-08-15 17:10:50 UTC - Matteo Merli: @Poule Available now
heart_eyes : Poule
----
2018-08-15 17:31:08 UTC - Sijie Guo: @William Fry
> Exception in thread “main”
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /managed-ledgers
you already initialized cluster metadata, so when you tried to initialize it
again, it will throw this NodeExists exception.
the command you found about rmr is kind of removing the metadata, so you can
reinitialize the metadata again.
> But why was it there in the first place?
you mean why it is
`10.0.1.210:2181:2181,10.0.1.171:2181:2181,10.0.1.215:2181:2181`?
can you describe the sequence of your deployment ?
----
2018-08-15 21:29:48 UTC - Poule: thanks
----
