2019-07-16 09:38:09 UTC - Rui Fu: @Rui Fu has joined the channel
----
2019-07-16 09:44:14 UTC - Vladimir Shchur: Hi, guys! Can someone clear up to me
what is the difference between Close and Unsubscribe for Consumer? From code it
looks like the only difference is that Close calls failPendingReceive while
Unsubscribe doesn't, am I correct?
----
2019-07-16 09:50:59 UTC - Sijie Guo: close doesn’t delete subscription.
unsuscribe will delete the subscription
----
2019-07-16 10:06:42 UTC - Vladimir Shchur: Sorry, what do you mean by "delete
subscription"? Are you saying for example if there is an exclusive subscription
mode and consumer was closed, no one can take it over? What is a purpose of
close then, when should client call it?
----
2019-07-16 10:20:02 UTC - Sijie Guo: close is usually called when your
application is shutting down. the you still need the subscription to consume
from last position.
unsuscribe is called when your application doesn’t need the subscription
anymore.
----
2019-07-16 10:24:08 UTC - Vladimir Shchur: I see, thank you!
----
2019-07-16 12:00:45 UTC - vikash: @jia zhai Thanks for your reply now i
able to start standalone server with authentoication enabled,i have
used below property
+1 : jia zhai
----
2019-07-16 12:01:24 UTC - jia zhai: welcome
----
2019-07-16 12:01:25 UTC - vikash: but when i am trying to authenticate
with Websocket Cleint written in c# request.Headers.Add("Authorization",
"Bearer
eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ2aXN1ci11c2VyIn0.Y78f_iXB9vyslrrpfEvlyk_poSowClb4HtksUCU5wAA");
----
2019-07-16 12:01:35 UTC - vikash: i have always failed authentocate
----
2019-07-16 12:02:17 UTC - vikash: Pulsar Dashboard also not working even
i have used like this
----
2019-07-16 12:02:18 UTC - vikash: docker run -d -p 80:80 -e
SERVICE_URL=<http://192.168.1.248:8080> -e
JWT_TOKEN=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ2aXN1ci11c2VyIn0.Y78f_iXB9vyslrrpfEvlyk_poSowClb4HtksUCU5wAA
pulsar
----
2019-07-16 12:02:25 UTC - vikash: may i know what i missed
----
2019-07-16 12:05:21 UTC - vikash: only this log i get
----
2019-07-16 12:05:21 UTC - vikash: va-v2.4.0" 19
17:18:35.062 [main] INFO org.apache.pulsar.PulsarStandalone - This operation
requires super-user access
17:18:35.078 [pulsar-web-57-8] INFO org.eclipse.jetty.server.RequestLog -
192.168.1.248 - - [16/Jul/2019:17:18:35 +0530] "GET /admin/v2/tenants HTTP/1.1"
401 54 "-" "Pulsar-Java-v2.4.0" 10
17:18:35.079 [main] INFO org.apache.pulsar.PulsarStandalone - This operation
requires super-user access
17:18:35.151 [pulsar-io-50-2] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/functions/persistent/coordinate-participants] Rewind from 2414:0 to
2414:0
17:18:42.037 [SessionTracker] INFO org.apache.zookeeper.server.ZooKeeperServer
- Expiring session 0x100012b5193000b, timeout of 30000ms exceeded
17:18:42.037 [ProcessThread(sid:0 cport:2181):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Processed session
termination for sessionid: 0x100012b5193000b
17:18:55.347 [pulsar-web-57-3] INFO org.eclipse.jetty.server.RequestLog -
172.17.0.2 - - [16/Jul/2019:17:18:55 +0530] "GET /admin/v2/clusters HTTP/1.1"
200 14 "-" "python-requests/2.22.0" 11
17:18:55.380 [pulsar-web-57-7] INFO org.eclipse.jetty.server.RequestLog -
172.17.0.2 - - [16/Jul/2019:17:18:55 +0530] "GET /admin/v2/clusters/standalone
HTTP/1.1" 401 54 "-" "python-requests/2.22.0" 5
17:19:04.967 [pulsar-web-57-1] INFO org.eclipse.jetty.server.RequestLog -
192.168.1.248 - - [16/Jul/2019:17:19:04 +0530] "GET
/admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 823 "-"
"Pulsar-Java-v2.4.0" 17
----
2019-07-16 12:49:17 UTC - vikash: Actually its solved now i have missed
superUserRoles= Parameter ,now able to authenticate
+1 : jia zhai
----
2019-07-16 13:52:03 UTC - ishara: Hello I am using pulsar standalone via the
docker image and I am trying to modify the default configuration file. At the
moment i have this in my docker-compose file : "command: /bin/bash -c
"/conf/apply-config-from-env.py /conf/standalone.conf
&&bin/pulsar standalone"" and I have my file linked to the conf
in the docker with volumes. But when I start my standalone my changes are not
applied at all ! Did anyone encounter this problem ?
----
2019-07-16 15:36:43 UTC - Grant Wu: Any idea what might cause this exception?
```
Unexpected character ('-' (code 45)): Expected space separating root-level
values
at [Source: 2019-07-16 15:15:37.349 INFO ConnectionPool:72 | Created
connection for <pulsar://pulsar-broker.petuum-system:6650;> line: 1, column: 6]
```
----
2019-07-16 17:11:50 UTC - Matteo Merli: Looks like some JSON parsing issue.
What service URL are you using? http:// or pulsar://
----
2019-07-16 17:12:05 UTC - Grant Wu: We’re using the Pulsar Python client
----
2019-07-16 17:12:11 UTC - Grant Wu: I think I may have a lead on this, I’ll let
you know later
----
2019-07-16 19:09:13 UTC - Devin G. Bost: When brokers are restarted, do they
re-download Jars for Pulsar functions when those functions were deployed with
URLs for the Jars? We've noticed that if the Jars no longer exist in our Jar
repo (due to them getting cleaned-up), our functions will start up in a dead
state and need to be updated with a valid Jar URL.
----
2019-07-16 20:13:29 UTC - Grant Wu: Never mind, this is unrelated to Pulsar,
this is a stupid Apache Livy issue that’s been around for a while that blocks
Pulsar usage within Spark jobs in Livy
----
2019-07-16 22:10:36 UTC - Grant Wu: In case you’re curious at all,
<https://issues.apache.org/jira/browse/LIVY-322?focusedCommentId=16886493&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16886493>
----
2019-07-16 22:18:48 UTC - Matteo Merli: Oh I see. I think we had a discussion
some time back on routing the C++ logs back to python (as we do for Go)
----
2019-07-16 22:19:54 UTC - Grant Wu: Yes, I may pursue implementing that as a
workaround
----
2019-07-16 22:20:01 UTC - Grant Wu: But this is really nonsense from Livy
----
2019-07-17 07:38:06 UTC - Olivier Chicha: Hello,
If I use a key_shared subscription,
let say: At the begining I have only one consumer on my key_shared
subscription: consumer 1.
- Consumer1 receive message 1 (with hash 0.6)
- Message2 with hash 0.7 arrives on the broker but has not been yet sent to
Consumer1
- Consumer2 is added to the subscription and gets the range 0.5..1
- Message3 arrive with hash 0.8
What will happend?
- Will the broker wait for Message1 to be acknowledged before sending message2
and Message 3 to consumer 2 ?
- Will message 3 be sent to to consumer2 and Message2 to consumer1 ?
In fact on our side we would like to be sure that consumer 2 will not receive
message2 or Message3 before Message1 has been effectively acknowledged, so that
we are sure that the message are effectively processed in the right order event
if the consumer are not on the same process.
Is this something that is granted?
----
2019-07-17 07:45:31 UTC - Yong Zhang: I think it doesn’ t re-download Jars
after brokers restarted. It’s downloaded when you create a function.
----
2019-07-17 07:55:51 UTC - Penghui Li: @Olivier Chicha key_shared subscription
working principle can refer to the following :
```
1.Each consumer serves a fixed range of hash value
2.The whole range of hash value could be covered by all the consumers.
3.Once a consumer is removed, the left consumers could still serve the whole
range.
Initializing with a fixed hash range, by default 2 << 5.
First consumer added, hash range looks like:
0 -> 65536(consumer-1)
Second consumer added, will find a biggest range to split:
0 -> 32768(consumer-2) -> 65536(consumer-1)
While a consumer removed, The range for this consumer will be taken over
by other consumer, consumer-2 removed:
0 -> 65536(consumer-1)
In this approach use skip list map to maintain the hash range and consumers.
Select consumer will return the ceiling key of message key hashcode % range
size.
```
----
2019-07-17 08:09:22 UTC - Olivier Chicha: Thanks @Penghui Li for your answer.
This is effectivelly what I understood, but, as far as I can understand, does
not answer to my question.
Here is what could be another version of my question (maybe simplier and
clearer).
For two messages with the same key,
am I granted that the first message will be aknowledged before the second
message is received by the new consumer (in case the key hash is allocated to a
new consumer before the second message is received by the broker)?
----
2019-07-17 08:16:11 UTC - Penghui Li: if key hash of that message take over by
the new consumer, the following messages with that key will dispatch to the new
consumer.
----
2019-07-17 08:20:56 UTC - Penghui Li: Currently, the broker does not care
whether the previous message was acknowledged. a new consumer takeover the hash
range, the message will be distributed to this consumer.
----
2019-07-17 08:21:22 UTC - Sijie Guo: @Olivier Chicha You are not guaranteed at
this moment. When a new consumer joins, the hash ring is changed. Broker
doesn’t wait until any all the outstanding messages delivered to the first
consumer acknowledged. It is actually hard to do so given the distributed
environment.
I guess you are looking for a solution for a better control how keys are
distributed to consumers. there is an ongoing efforts introducing sticky
consumer for key_shared subscription. You can specify a key_range to consume
message when you join the key_shared subscription. That might be the thing you
are looking for.
----
2019-07-17 08:47:39 UTC - Olivier Chicha: I looked at it, but it does not
provide a simple solution as well to my issue.
Our case is we have "micro services" that ignore each other and that provide
service by consuming messages from a given topic.
The stickyness grant us the ordering per key, while it allows for load
balancing of the messages across the different services.
Services can be added or removed by the orchestrator.
the key_shared subscription would respond to our case perfectly except when a
new service is added, it is the only case where the message processing ordering
is not granted.
If we use the feature you describe in (which I guess is the
<https://github.com/apache/pulsar/issues/4169>), this requires that we
implement a distributed resource allocation mechanism on top the key_shared
subscription (which is not a small thing)
Note that if you don't grant the ordering of the processing of messages across
consumer for a given key, it might also mean that we could have a race
condition where the new client would receive message1 that the old client did
not succeed to process (crashed) after having processed message2, => the
ordering of the message is then not granted at consumer level. (unless I am
missing something).
it would be really great for us if you could find a way to grant this ordering
without having to implement on our side a distributed resource allocation
mechanism.
Best regards,
----
2019-07-17 08:55:16 UTC - Gurgen Hovhannisyan: @Gurgen Hovhannisyan has joined
the channel
----
2019-07-17 08:58:36 UTC - Gurgen Hovhannisyan: Hi
I have some issues, please help :
working with python 2 and pulsar 2.4.0 version, Ubuntu 18.04.2 LTS (Bionic
Beaver) running in VirtualBox under Windows 10
The following schema doesn't work at all during create_producer, I want to pass
array of maps (list of dictionaries), why it doesn't work ? please ?
class MySchema(Record):
a = String()
b = Long()
# array of dict, json data
data = Array(Map(String()))
this crashes during instance creation, even I didn't reach to send.
producer = client.create_producer(topic='mytopic15',
schema=AvroSchema(MySchema))
----
2019-07-17 08:59:00 UTC - Sijie Guo: @Olivier Chicha I see what you are saying
now. I think it can achieve by providing an option form the consumer. The
option is used for telling the broker to wait until previous outstanding
messages to be acked before reassigning a range. does that work for your use
case?
----
2019-07-17 09:00:59 UTC - Olivier Chicha: Yes That would definitively do the
trick as far as I understand.
----
2019-07-17 09:02:07 UTC - Sijie Guo: ^ @Penghui Li can you help looking into
adding this option?
----
2019-07-17 09:04:11 UTC - Penghui Li: Ok
----
2019-07-17 09:07:24 UTC - Penghui Li: To ensure that messages are not processed
twice by two consumers, if a new consumer connected, we should wait the old
consumer have already ack all message dispatched
----
2019-07-17 09:07:38 UTC - Penghui Li: @Sijie Guo I understand it right?
----
2019-07-17 09:09:28 UTC - Sijie Guo: @Penghui Li correct. I think the idea is
to “delay” splitting the hash ring until a given condition is met. The
condition for the use case described by @Olivier Chicha is wait until the
messages for a given ring are acknowledged and split and reassign the ring.
+1 : Penghui Li
----
2019-07-17 09:09:41 UTC - Penghui Li: If we use sticky consumer, messages are
not assigned to two consumers, can this feature not be applied to this scenario?
----
