2018-11-21 11:47:23 UTC - Ivan Kelly: have you any documentation on using this
method? I sounds like each cert has to be publicly downloadable by the broker
or something
----
2018-11-21 11:48:28 UTC - Ivan Kelly: ah, we don't support that now I don't
think. Once the message leaves the client, it doesn't get modified by the broker
----
2018-11-21 12:45:10 UTC - Ganga Lakshmanasamy: So any suggestions here?
----
2018-11-21 12:46:16 UTC - Ganga Lakshmanasamy: @Ivan Kelly So is it mandatory
that the message to be converted to byte array while sent through producer?
----
2018-11-21 12:47:43 UTC - Ivan Kelly: no, so if you use schema you can pass in
objects
----
2018-11-21 12:48:08 UTC - Ivan Kelly: are you using java ?
----
2018-11-21 12:48:26 UTC - Ganga Lakshmanasamy: ok. I am looking up for some
code samples/ documentation in converting current messagebuilder to typed
builder
----
2018-11-21 12:48:31 UTC - Ganga Lakshmanasamy: yes, I am using java
----
2018-11-21 12:49:43 UTC - Ganga Lakshmanasamy: The main problem is we have a
map object which needs to be sent and parsed back to map or jsonobject to pass
it on to one of our rest api, which is failing during parse.
----
2018-11-21 12:50:36 UTC - Ivan Kelly: so, it's something like
try (Producer<MyObject> p =
client.newProducer(Schema.AVRO(MyObject.class).topic("my-topic").create()) {
p.newMessage().value(new MyObject()).send();
----
2018-11-21 12:51:10 UTC - Ivan Kelly: or, you can skip the builder completely,
and just use p.send(new MyObject());
----
2018-11-21 13:07:12 UTC - Ganga Lakshmanasamy: ok Thanks, let me check this out.
----
2018-11-21 15:12:23 UTC - Ryan Samo: I’m struggling to find the exact example
but this link shows the use of a certificate chain with Jetty. It explains sort
of what I’m talking about. Basically everything relies on the root like normal
but the clients would rely on the intermediate layer allowing you to revoke the
intermediate without changing the root.
<https://support.cloudbees.com/hc/en-us/articles/222829768-How-to-setup-HTTPS-within-Jetty-with-Intermediate-certificates-?mobile_site=true>
----
2018-11-21 15:40:39 UTC - Ivan Kelly: well, how many requests do you think
you'll have per second?
----
2018-11-21 15:40:49 UTC - Ivan Kelly: what size machine is this? sounds massive
----
2018-11-21 16:15:45 UTC - Christophe Bornet: Small question about sync
replication. Suppose I have 1 cluster over 2 datacenters, E=4, Qw=2, Qa=2 and
region aware policy. In nominal mode, Pulsar will distribute the data over the
2 datacenters, is that correct ? Now if one of the DCs crashes, can I assume
that Pulsar will redistribute the fragments over the remaining DC ? Will I lose
messages in the process ? If the failed DC comes back, are the fragment
redistributed over the 2 DCs immediately ?
----
2018-11-21 16:22:10 UTC - Matteo Merli: > In nominal mode, Pulsar will
distribute the data over the 2 datacenters, is that correct ?
That’s correct, each DC is viewed as a rack and the replicas will be placed in
order to maximize the distribution.
> Now if one of the DCs crashes, can I assume that Pulsar will redistribute
the fragments over the remaining DC ?
Are you referring to the existing data or new data published after 1 DC is
offline?
* Existing data : BK auto-recovery will re-establish the configured
replication factor (in this case `Qw=2`) and, since DC1 is down, it will place
the 2 copies in DC2
* New data will have both copies be placed in DC2 immediately
> Will I lose messages in the process ?
No, unless you loose 2 machines (not only they’re offline, but either “gone” or
disk broke down)
> If the failed DC comes back, are the fragment redistributed over the 2 DCs
immediately ?
Yes, existing data from failed DC will be immediately available and new data
will start to be written there again
----
2018-11-21 16:28:56 UTC - Christophe Bornet: > No, unless you loose 2
machines (not only they’re offline, but either “gone” or disk broke down)
You mean loose 2 machines from DC2 in addition to the DC1 failure ? Or the 2
machines from DC1 (I'm trying to imagine, DC1 got nuked)
----
2018-11-21 16:34:11 UTC - Matteo Merli: In normal condition, with `Qw=2`, the 2
replicas will be place 1 in DC1 and the other in DC2.
If DC1 is offline, data will still be there, just unavailable (for some
extended amount of time). If DC1 is nuked.. well we assume the disk is broken.
Now we have 1 copy in DC2.
* If this bookie goes down --> Data will be temporarily unavailable
* If this bookie disk crashes --> Data will be either unavail or lost
(depending on DC1)
BK auto-recovery will start to copy data to have 2 copies again, in this case
by writing to another bookie in DC2. This process will start immediately after
DC1 goes offline (or with a configurable delay) .
----
2018-11-21 16:36:18 UTC - Christophe Bornet: This is just so cool :sunglasses: !
----
2018-11-21 16:37:02 UTC - Christophe Bornet: > new data will start to be
written there again
----
2018-11-21 16:37:39 UTC - Christophe Bornet: you mean that new data is again
distributed automatically on both DCs, right ?
----
2018-11-21 16:38:40 UTC - Matteo Merli: That’s correct, new ledgers will always
consider all the available bookies when forming an ensemble (and then applying
the placement policy).
----
2018-11-21 16:43:07 UTC - Christophe Bornet: Awesome ! You can't do such things
with monolithic partitions if you see what I mean :smile:
----
2018-11-21 16:43:27 UTC - Matteo Merli: :smile:
----
2018-11-21 18:33:46 UTC - Adam Kazberuk: @Adam Kazberuk has joined the channel
----
2018-11-21 19:35:54 UTC - Adam Kazberuk: This is likely a configurational
oversight on my end (still new & ignorant), hoping someone can help me out!
Trying to set up a DeadLetterQueue on a websocket consumer, looks like the
functionality was added fairly recently:
<https://github.com/apache/pulsar/pull/2968>
I've got a consumer connecting and receiving messages (with DLQ params on
connect), and the un-acked messages are re-sent after 10s. However, the
messages never stop being retried.
I see this in the logs for the consumer connection:
`11:04:34.411 [pulsar-web-357] INFO org.eclipse.jetty.server.RequestLog -
127.0.0.1 - - [21/Nov/2018:11:04:34 -0800] "GET
/ws/v2/consumer/persistent/public/default/my-topic/subscription-name?ackTimeoutMillis=10000&maxRedeliverCount=1&deadLetterTopic=dead
HTTP/1.1" 101 0 "-" "-" 85`
with `maxRedeliverCount=1&deadLetterTopic=dead` which appears to be the
correct parameters (I've tried various params, i.e. 'maxRedeliverCount=5'):
<https://github.com/apache/pulsar/pull/2968/files#diff-9de5d3e81a341a0af69fd50e24aea480R316>
This appears on each message retry:
```
11:14:14.420 [pulsar-timer-74-1] WARN
org.apache.pulsar.client.impl.UnAckedMessageTracker -
[ConsumerBase{subscription='subscription-name', consumerName='c2e41',
topic='<persistent://public/default/my-topic>'}] 1 messages have timed-out
11:14:14.420 [broker-topic-workers-OrderedScheduler-7-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/my-topic-subscription-name] Rewind from 659:1 to
659:0
```
----
2018-11-21 19:50:10 UTC - Matteo Merli: @Adam Kazberuk Support for DLQ in
websocket proxy was just added recently
<https://github.com/apache/pulsar/pull/2968> and will be out on 2.3.0
----
2018-11-21 19:54:22 UTC - Adam Kazberuk: @Matteo Merli, solid, thank you!
----
2018-11-21 22:40:39 UTC - Byron: Hi folks, is there a download available for a
pre-built C++ library (with headers) so I can link against it for the Go
library?
----
2018-11-21 22:42:03 UTC - Ali Ahmed: no prebuilt but a brew formulae to
install the C++ lib is present
----
2018-11-21 22:42:43 UTC - Byron: My macOS version isn’t supported anymore so I
would like to build a linux Docker image to build my Go code
----
2018-11-21 22:44:00 UTC - Byron: does the pulsar docker image contain the C++
library? an ideal image would be all stripped down with just that and the
dependencies
----
2018-11-21 22:44:06 UTC - Matteo Merli: @Byron you can use the formula to build
it yourself. Have you tried:
<http://pulsar.apache.org/docs/en/client-libraries-cpp/#macos>
----
2018-11-21 22:45:31 UTC - Byron: I did, but my macOS version is too old 10.11
according to Homebrew
----
2018-11-21 22:45:59 UTC - Byron: Failing on this
```
==> ./bootstrap --prefix=/usr/local/Cellar/cmake/3.12.4 --no-system-libs
--parallel=4 --datadir=/share/cm
==> make
Last 15 lines from /Users/ruthb/Library/Logs/Homebrew/cmake/02.make:
[ 25%] Building C object
Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/simple_decoder.c.o
[ 25%] Building C object
Utilities/cmlibuv/CMakeFiles/cmlibuv.dir/src/unix/proctitle.c.o
[ 25%] Building C object
Source/CursesDialog/form/CMakeFiles/cmForm.dir/fty_ipv4.c.o
[ 26%] Building C object
Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/simple_encoder.c.o
[ 26%] Building C object
Source/CursesDialog/form/CMakeFiles/cmForm.dir/fty_num.c.o
[ 26%] Building C object
Source/CursesDialog/form/CMakeFiles/cmForm.dir/fty_regex.c.o
[ 26%] Linking C static library libcmlibuv.a
[ 26%] Building C object
Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/sparc.c.o
[ 26%] Built target cmlibuv
[ 26%] Building C object
Utilities/cmliblzma/CMakeFiles/cmliblzma.dir/liblzma/simple/x86.c.o
[ 27%] Linking C static library libcmForm.a
[ 27%] Linking C static library libcmliblzma.a
[ 27%] Built target cmForm
[ 27%] Built target cmliblzma
make: *** [all] Error 2
```
----
2018-11-21 22:46:17 UTC - Byron: And then Homebrew outputs this:
```
Do not report this issue to Homebrew/brew or Homebrew/core!
These open issues may also help:
cmake 3.13.0 <https://github.com/Homebrew/homebrew-core/pull/34361>
swi-prolog 7.7.20 cmake based build
<https://github.com/Homebrew/homebrew-core/pull/33706>
Error: You are using macOS 10.11.
We (and Apple) do not provide support for this old version.
```
----
2018-11-21 22:47:16 UTC - Matteo Merli: Got it
----
2018-11-21 22:47:26 UTC - Byron: However I see packages for Linux are available
now: <http://pulsar.apache.org/docs/en/client-libraries-cpp/#linux>
----
2018-11-21 22:47:32 UTC - Ali Ahmed: @Byron just follow the shell commands in
the brew itself
<https://raw.githubusercontent.com/apache/pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb>
----
2018-11-21 22:48:09 UTC - Byron: So I can create a Docker image installing
those packages with header to be able to link the Go library.
----
2018-11-21 22:48:24 UTC - Byron: Thanks @Ali Ahmed I will try it manually to
see if that changes anything
----
2018-11-21 22:48:44 UTC - Ali Ahmed: were are targeting to run osx or linux ?
----
2018-11-21 22:49:29 UTC - Ali Ahmed: the python packages should install the
native c++ libs if you use pip or easy install
----
2018-11-21 22:49:45 UTC - Byron: Oh ok, good to know. I am targeting linux, but
developing on a mac
----
2018-11-21 22:51:08 UTC - Matteo Merli: @Ali Ahmed python are self-contained,
but Go needs the library to be there
----
2018-11-21 22:51:08 UTC - Byron: So that is why I can do a Docker-based
workflow if need be
----
2018-11-21 22:51:24 UTC - Byron: I will install the client-devel debian package?
----
2018-11-21 22:51:27 UTC - Byron: That should be good?
----
2018-11-21 22:51:30 UTC - Matteo Merli: yes
----
2018-11-21 22:51:35 UTC - Byron: :+1:
----
2018-11-21 22:52:52 UTC - Byron: Thank you all
+1 : Matteo Merli
----
2018-11-21 23:00:03 UTC - Byron: @Matteo Merli Sorry one more issue if you are
available
----
2018-11-21 23:00:28 UTC - Byron: Here is my Docker image:
```
FROM golang:1.11
COPY apache-pulsar-client-2.2.0.deb /opt/apache-pulsar-client.deb
COPY apache-pulsar-client-dev-2.2.0.deb /opt/apache-pulsar-client-dev.deb
RUN dpkg -i /opt/apache-pulsar-client.deb && \
dpkg -i /opt/apache-pulsar-client-dev.deb && \
apt-get install -f
ENV CGO_ENABLED=1
RUN go get
<http://github.com/apache/pulsar/pulsar-client-go/pulsar|github.com/apache/pulsar/pulsar-client-go/pulsar>
```
----
2018-11-21 23:00:47 UTC - Byron: Getting this error on build:
```
#
<http://github.com/apache/pulsar/pulsar-client-go/pulsar|github.com/apache/pulsar/pulsar-client-go/pulsar>
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:232:3: could
not determine kind of name for C.pulsar_string_list_free
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:229:31: could
not determine kind of name for C.pulsar_string_list_get
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:226:24: could
not determine kind of name for C.pulsar_string_list_size
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:220:78: could
not determine kind of name for C.pulsar_string_list_t
cgo:
gcc errors for preamble:
In file included from
src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_client.go:24:0:
./c_go_pulsar.h:133:66: error: unknown type name 'pulsar_string_list_t'
void pulsarGetTopicPartitionsCallbackProxy(pulsar_result result,
pulsar_string_list_t* partitions, void *ctx);
^~~~~~~~~~~~~~~~~~~~
./c_go_pulsar.h: In function '_pulsar_client_get_topic_partitions':
./c_go_pulsar.h:137:61: error: 'pulsarGetTopicPartitionsCallbackProxy'
undeclared (first use in this function)
pulsar_client_get_topic_partitions_async(client, topic,
pulsarGetTopicPartitionsCallbackProxy, ctx);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./c_go_pulsar.h:137:61: note: each undeclared identifier is reported only once
for each function it appears in
```
----
2018-11-21 23:01:03 UTC - Matteo Merli: You should get the 2.2.20 tag
----
2018-11-21 23:01:27 UTC - Matteo Merli: master code has already some changes
for 2.3 release (you’d need to compile the master c++ lib)
----
2018-11-21 23:01:35 UTC - Matteo Merli: (2.2.0 tag)
----
2018-11-21 23:01:57 UTC - Byron: For the Go library. Ok
----
2018-11-21 23:02:28 UTC - Matteo Merli: Go tag needs to be <= C++ tag
----
2018-11-21 23:02:38 UTC - Byron: Ok
----
2018-11-21 23:02:40 UTC - Byron: Thanks
----
2018-11-21 23:03:47 UTC - Matteo Merli: (I wonder is there’s a way to give more
meaningful errors there)
----
2018-11-22 03:37:04 UTC - Zheng Dayu: @Zheng Dayu has joined the channel
----
2018-11-22 04:45:45 UTC - Abhishek Mulay: @Abhishek Mulay has joined the channel
----
2018-11-22 08:13:12 UTC - zero: @zero has joined the channel
----
