2019-05-22 11:06:38 UTC - Alexandre DUVAL: Hi, if I have acknowledge a message
that is the last one of a reading, all the previous messages of this reading
are ack too?
----
2019-05-22 11:10:34 UTC - Shivji Kumar Jha: @Alexandre DUVAL may not be, for
that behavior you have to send cumulative ack
----
2019-05-22 11:10:54 UTC - Alexandre DUVAL: Okay :slightly_smiling_face:, thanks.
----
2019-05-22 11:20:31 UTC - Shivji Kumar Jha: Hi, Is there a java util method
somewhere in pulsar client that returns the
org.apache.pulsar.client.api.Authentication instance given:
private String authPluginClassName;
private String authParams;
?
In a way that it works for all implementations of
org.apache.pulsar.client.api.Authentication…
----
2019-05-22 11:23:17 UTC - Shivji Kumar Jha: I was going to write one but then I
think there should be a util somewhere and i dont want to duplicate it if it is
there already!
----
2019-05-22 11:31:21 UTC - Shivji Kumar Jha: @jia zhai ^
----
2019-05-22 11:32:32 UTC - Venugopal Doddi: @Venugopal Doddi has joined the
channel
----
2019-05-22 11:34:20 UTC - jia zhai: @Shivji Kumar Jha AuthenticationFactory for
server, and AuthenticationUtil for client
----
2019-05-22 11:34:21 UTC - Venugopal Doddi: @Sijie Guo I am following you on the
<http://streaml.io|streaml.io> and i am currently working on a POC for pulsar
in my organization. I am looking for some help on the geo-replication setup. Is
there something you can help
----
2019-05-22 11:34:56 UTC - Venugopal Doddi: I am getting trouble in setting the
--allowed-clusters parameter
----
2019-05-22 11:35:38 UTC - Venugopal Doddi: i am trying this in two datacenters
and i have clusters names test and test-ttc created in both
----
2019-05-22 11:35:57 UTC - Venugopal Doddi: i was not sure how the connection or
the handshake i can establish between both of them..
----
2019-05-22 11:47:16 UTC - jia zhai: @Venugopal Doddi It is auto established
once the topic under a configured namespace is created
----
2019-05-22 12:05:08 UTC - jia zhai: As in this pic, there are 2 cluster:
beijing, shanghai; And a namespace that configured to replicated between
beijing and shanghai.
once the topic is created, there is a replicator been created to handle the
connection.
----
2019-05-22 13:00:37 UTC - Venugopal Doddi: @jia zhai thanks for the picture..
but how will i make the connection or the communication between both the
clusters.. in my example.. there are two clusters.. test-ttc and test-tte both
are in different data centers altogether.
----
2019-05-22 13:00:54 UTC - Venugopal Doddi: when i am running this command
```pulsar-admin properties create my-tenant \
--admin-roles my-admin-role \
--allowed-clusters us-west,us-east,us-cent```
----
2019-05-22 13:01:06 UTC - Venugopal Doddi: in place of the us-west, cent, east
what is that i have to give..
----
2019-05-22 13:01:47 UTC - Venugopal Doddi: and i am running this command on
one of the cluster in ttc..
----
2019-05-22 13:02:06 UTC - Venugopal Doddi: m y question is how will i make the
cluster in ttc and tte talk to each other.
----
2019-05-22 13:03:10 UTC - jia zhai: right it gives the cluster name: test-ttc,
test-tte,
----
2019-05-22 13:04:39 UTC - jia zhai: Here is the docker-compose that I used to
create 3 clusters：
<https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5>
----
2019-05-22 13:05:43 UTC - jia zhai: and here is the command to create tenants
and namespace:
```
bin/pulsar-admin tenants create my-tenant --allowed-clusters
beijing,shanghai,guangzhou
bin/pulsar-admin namespaces create my-tenant/my-namespace --clusters
beijing,shanghai,guangzhou
```
----
2019-05-22 13:14:25 UTC - Venugopal Doddi: Right.. it do created cluster ttc
and tte. when i run the above command by passing the --allowed-clusters
followed by test-ttc,test-tte and when i run the above command on one of the
ttc bookkeeper/broker node i am getting error like ```Invalid cluster name
test-ttc``` and exiting
----
2019-05-22 13:14:31 UTC - Venugopal Doddi: so this is the issue i am facing
----
2019-05-22 13:15:21 UTC - Venugopal Doddi: when i list the clusters on one of
the server located in ttc datacenter, i am getting `test-ttc` similary when i
run the list on tte server i am getting `test-tte`
----
2019-05-22 13:15:31 UTC - Jim Lambert: @Jim Lambert has joined the channel
----
2019-05-22 13:16:00 UTC - Venugopal Doddi: @jia zhai ^^
----
2019-05-22 13:16:12 UTC - jia zhai: there maybe some config issue
----
2019-05-22 13:16:17 UTC - jia zhai: wait for a minute. I am writing a tutorial
to use above docker-compose file.
----
2019-05-22 13:16:31 UTC - jia zhai: Then you could use docker to verify 3
clusters replciation
----
2019-05-22 13:16:41 UTC - Venugopal Doddi: yes.. thats what i am trying to
figure out.. but individually the clusters are succesfully producing and
consuming.
----
2019-05-22 13:17:35 UTC - Venugopal Doddi: also the initialize-cluster-metadata
----
2019-05-22 13:17:35 UTC - Venugopal Doddi: is successfully applied
----
2019-05-22 13:20:41 UTC - jia zhai: maybe the config in metadata store?
----
2019-05-22 13:22:24 UTC - Venugopal Doddi: how can i verify that?
----
2019-05-22 13:26:28 UTC - Sijie Guo: @Venugopal Doddi do you deploy a
configuration store (global zookeeper)?
----
2019-05-22 13:26:40 UTC - jia zhai: Here is the tutorial:
<https://gist.github.com/jiazhai/e1f0e828754b73ad7792b34d2449690c>
based on the docker-compose here:
<https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5>
Please make sure configure you docker to have as many cpu and memory as possible
----
2019-05-22 13:26:49 UTC - Sijie Guo: Do you mind sharing the commands sequence
you used for creating clusters.
----
2019-05-22 13:45:46 UTC - Venugopal Doddi: yes.. surely i will share..
----
2019-05-22 13:45:49 UTC - Venugopal Doddi: one moment
----
2019-05-22 13:46:01 UTC - Venugopal Doddi: but i am not using docker yet
----
2019-05-22 13:46:09 UTC - Venugopal Doddi: its a stand alone cluster on
openstack
----
2019-05-22 13:50:29 UTC - Venugopal Doddi: after the bookkeeper and broker
services are started. and able to produce and consume messages.. for
geo-replication i am following this..
----
2019-05-22 13:50:38 UTC - Venugopal Doddi:
<https://pulsar.apache.org/docs/en/administration-geo/>
----
2019-05-22 13:51:33 UTC - Venugopal Doddi: ```pulsar-admin properties create
venu-tenant \
--admin-roles venu-admin-role \
--allowed-cluster test-ttc, test-tte```
----
2019-05-22 13:54:08 UTC - Venugopal Doddi: when i list the clusters.. i am
getting only one.. and thats obvious as this is on ttc servers.
----
2019-05-22 13:54:10 UTC - Venugopal Doddi: ```[root@pulsar-bk1-ttc bin]#
./pulsar-admin clusters list
test-ttc```
----
2019-05-22 13:55:31 UTC - Venugopal Doddi: ```[root@pulsar-bk1-ttc bin]#
./pulsar-admin tenants list
public
tenant1
venu-tenant```
----
2019-05-22 13:57:36 UTC - Venugopal Doddi: this is the error...
----
2019-05-22 13:57:39 UTC - Venugopal Doddi: ```[root@pulsar-bk1-ttc bin]#
./pulsar-admin namespaces set-clusters venu-tenant/global/full-mesh --clusters
test-ttc,test
Invalid cluster id: test
Reason: Invalid cluster id: test```
----
2019-05-22 13:57:53 UTC - Venugopal Doddi: @jia zhai ^^
----
2019-05-22 13:59:53 UTC - Venugopal Doddi: no.. i didnt deployed the
global-zookeeper @Sijie Guo
----
2019-05-22 14:00:38 UTC - Venugopal Doddi: thats a must for the geo-replication?
----
2019-05-22 14:00:54 UTC - Venugopal Doddi: i have the zookeeper cluster of 3
nodes for each data center
----
2019-05-22 14:03:29 UTC - Venugopal Doddi: and configuration-store IP is same
as one of the zookeeper node
----
2019-05-22 14:04:00 UTC - Sijie Guo: @Venugopal Doddi are the two clusters
pointing to the same zookeeper cluster?
----
2019-05-22 14:05:37 UTC - Sijie Guo: @Venugopal Doddi :
it is not must. it is just a convenient way for propagating the the replication
config.
If you don’t have a configuration store, you can try following steps:
a) in test-ttc cluster, run `bin/pulsar-admin clusters create` to create
`test-tte` cluster, so in test-ttc cluster, it knows how to connect to
`test-tte`.
b) in test-tte cluster, run `bin/pulsar-admin clusters create` to create
`test-ttc` cluster, so in test-ttc cluster, it knows how to connect to
`test-tte`.
if the two clusters don’t share the same configuration store, you have to
configure at both clusters to let each cluster know the other cluster.
----
2019-05-22 14:28:14 UTC - Venugopal Doddi: no
----
2019-05-22 14:28:29 UTC - Venugopal Doddi: they are not point to a same
zookeeper cluster
----
2019-05-22 14:28:43 UTC - Venugopal Doddi: they are pointing to both different
----
2019-05-22 14:29:58 UTC - Venugopal Doddi: ```if the two clusters don’t share
the same configuration store, you have to configure at both clusters to let
each cluster know the other cluster.``` i didnt get this statement.. is there a
command or a file i can change with
----
2019-05-22 14:29:59 UTC - Venugopal Doddi: ?
----
2019-05-22 14:30:41 UTC - Venugopal Doddi: @Sijie Guo
----
2019-05-22 14:46:28 UTC - jia zhai: @Venugopal Doddi you could reference the
docker-compose file.
line 28 is for the configuration store, and line 52 is do the init for each
cluster.
<https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5#file-docker-compose-yml-L52>
A configuration store was also called global zookeeper.
----
2019-05-22 14:46:59 UTC - Sijie Guo: @jia zhai he doesn’t have the global
configuration store. you have to change your example
----
2019-05-22 14:47:21 UTC - jia zhai: :joy:
----
2019-05-22 14:47:30 UTC - Sijie Guo: @Venugopal Doddi let me try to change
@jia zhai’s example and show you how to do it
----
2019-05-22 14:54:57 UTC - Venugopal Doddi: sure
----
2019-05-22 14:56:36 UTC - Venugopal Doddi: @Sijie Guo but i am running the
cluster in standalone openstack server and not on container.
----
2019-05-22 14:57:22 UTC - Sijie Guo: @Venugopal Doddi understand that. It is
just to show you what commands you need to run
----
2019-05-22 14:59:59 UTC - Venugopal Doddi: ok
----
2019-05-22 15:03:34 UTC - Venugopal Doddi: also trying to check.. if we have to
implement the apart form the above cluster create commands, both the ttc and
tte clusters should point to a single zookeeper cluster only...
----
2019-05-22 15:03:48 UTC - Venugopal Doddi: @Sijie Guo
----
2019-05-22 15:05:09 UTC - Sijie Guo: @Venugopal Doddi :
if you can point to one single configuration store, it will make things easier.
if you can’t do that, you have to create clusters in each cluster to make them
learn the other cluster.
----
2019-05-22 15:23:15 UTC - Venugopal Doddi: i will try the second part.. but
again.. what i was not convinced is.. even i create the same clusters in both
the Datacenters.. how will they both know each other and replicate the data on
the same topic that created in both the clusters..
----
2019-05-22 15:35:51 UTC - Sijie Guo: when you create the cluster, you need to
provide the brokerServiceUrl and webServiceUrl of that cluster. basically when
you create a cluster test-tte in cluster test-ttc, you are telling test-ttc how
to access test-tte.
so when you create a namespace with the list of replicated clusters, it will
resolve the cluster location from the cluster information you just created.
----
2019-05-22 16:33:36 UTC - Sijie Guo: @Venugopal Doddi: I created a tutorial of
geo-replication without a global configuration store.
<https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930>
----
2019-05-22 16:34:21 UTC - Sijie Guo:
<https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930#2-create-clusters>
and
<https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930#2-create-tenants-and-namespace>
these two sections are the keys to set up geo-replication between clusters
without a global configuration store.
----
2019-05-22 17:32:14 UTC - Devin G. Bost: Any chance someone has a
higher-resolution image of the Pulsar logo? I've got this one, but it looks
really grainy when I scale it up in a PowerPoint.
----
2019-05-22 17:34:03 UTC - Matteo Merli: There’s an SVG:
<https://github.com/apache/pulsar/blob/7d75fd28b9fff830697afc3505e30c80369b27f5/site2/website/static/img/pulsar.svg>
----
2019-05-22 17:54:33 UTC - Devin G. Bost: Thanks!
----
2019-05-22 19:06:12 UTC - Sree Vaddi: Please join our meetup group:
<https://www.meetup.com/SF-Bay-Area-Apache-Pulsar-Meetup>
+1 : Sijie Guo, lan
----
2019-05-22 21:24:45 UTC - Matteo Merli: Can you share the complete stack trace?
----
2019-05-22 21:25:11 UTC - Matteo Merli: Or any way to reproduce this?
----
2019-05-22 21:25:40 UTC - Matteo Merli: Would that be just having a project
with pulsar-client and jackson-databind together?
----
2019-05-23 05:03:49 UTC - divyasree: @divyasree has joined the channel
----
2019-05-23 06:52:01 UTC - Venugopal Doddi: this helped. Thanks Alot. @Sijie Guo
----
2019-05-23 06:54:46 UTC - Venugopal Doddi: One more question.. when the
consumer is not opened on one data center with some error in the consumer
command and we didnt checked that as assumed the consumer is good, where as the
producer successfully produced 10 messages on the other data center., Later
realised the issue in the consumer command and corrected,. again produced 10
messages on the other data center. this time i consumed 10 messages
successfully, but how will i know the previous message where are they? and why
they are not consumed.
----
2019-05-23 07:10:34 UTC - Romain Castagnet: I just created :
<https://github.com/apache/pulsar/issues/4337>
----
2019-05-23 07:15:29 UTC - Sijie Guo: @Venugopal Doddi if the subscription
doesn’t exist, the subscription will be created with MessageId.latest.
----
2019-05-23 07:16:04 UTC - Sijie Guo: you can create a consumer using
`withSubscriptionInitialPosition(SubscriptionInitialPosition.earliest)`.
----
2019-05-23 07:44:51 UTC - Venugopal Doddi: that a conf setting in some file?
----
2019-05-23 07:45:04 UTC - Venugopal Doddi: or the command parameter?
----
2019-05-23 07:47:18 UTC - Sijie Guo: this is on the consumer builder. I am not
sure if the `public-client` command line tool has the option to control it.
----
2019-05-23 08:00:28 UTC - Venugopal Doddi: ok
----
2019-05-23 08:41:07 UTC - Lewey: @Lewey has joined the channel
----
2019-05-23 08:53:56 UTC - Ulf Lilleengen: @Ulf Lilleengen has joined the channel
----
