2018-11-30 16:08:36 UTC - Beast in Black: @Matteo Merli @Ali Ahmed @Sijie Guo
reposting a question I had asked on Wednesday: I'm seeing some broker policy
errors on a namespace (possibly related to partitioned topics, but I'm not
using those) when creating producers (and also consumer) to a nonpersistent
topic in a namespace. I'm using pulsar-2.2 and the 2.1 CPP client.
The cpp client library shows this error when trying to create a subscriber to
the topic:
```
07:06:03.430 [0x7fa4cd197700] ERROR
/myapp/3rdParty/pulsar-client-cpp/lib/ClientConnection.cc(699) -
[100.96.3.9:35232 -> 100.96.3.5:6650] Failed partition-metadata lookup
req_id: 33 error: 1
07:06:03.430 [0x7fa4cd197700] ERROR
/myapp/3rdParty/pulsar-client-cpp/lib/ClientImpl.cc(138) - Error
Checking/Getting Partition Metadata while creating producer on
<non-persistent://diagnostics/local.diagnostics.guestnamespace/diagnostics_nonpersist_stream>
-- 5
```
Broker log (one of the brokers, I have 3) shows this *repeatedly*:
```
22:26:28.416 [ForkJoinPool.commonPool-worker-0] ERROR
org.apache.pulsar.broker.web.PulsarWebResource - Policies not found for
diagnostics/local.diagnostics.guestnamespace namespace
22:26:28.416 [ForkJoinPool.commonPool-worker-0] WARN
org.apache.pulsar.broker.service.ServerCnx - Failed to get Partitioned Metadata
[/100.96.2.13:41676]
<non-persistent://diagnostics/local.diagnostics.guestnamespace/_routing>:
Policies not found for diagnostics/local.diagnostics.guestnamespace namespace
org.apache.pulsar.broker.web.RestException: Policies not found for
diagnostics/local.diagnostics.guestnamespace namespace
at
org.apache.pulsar.broker.web.PulsarWebResource.lambda$4(PulsarWebResource.java:657)
~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
at
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_171]
at
org.apache.pulsar.zookeeper.ZooKeeperDataCache.lambda$0(ZooKeeperDataCache.java:66)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.2.0.jar:2.2.0]
at
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_171]
at
org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$7(ZooKeeperCache.java:349)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.2.0.jar:2.2.0]
at
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_171]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_171]
at
org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$13(ZooKeeperCache.java:334)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.2.0.jar:2.2.0]
at
java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
[?:1.8.0_171]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[?:1.8.0_171]
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[?:1.8.0_171]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[?:1.8.0_171]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[?:1.8.0_171]
```
Any idea how I can fix this `Policies not found` issue? I checked the pulsar
documentation but perhaps I missed something related to this...
----
2018-11-30 16:09:42 UTC - Beast in Black: s/subscriber/producer/
----
2018-11-30 17:31:15 UTC - Julien Plissonneau Duquène: good morning West Coast
I was wondering if there will be Pulsar talks/events planned in Europe next
year, any chance to meet some of you guys in the old world?
----
2018-11-30 17:33:10 UTC - Julien Plissonneau Duquène: btw the calendar on
<http://pulsar.apache.org|pulsar.apache.org> is broken
----
2018-11-30 17:39:57 UTC - Jon Bock: Hi Julien! A number of people, including
from us here at Streamlio, have submitted conference talk proposals for events
in Europe, as we hear back we’ll definitely make sure the accepted ones get on
the calendar (and get the calendar fixed). Let us know if there are any
particular events that you’d suggest we look into submitting for, or if there’s
something you’d like to submit where any of us can help.
And of course if you want to propose a Pulsar talk at a local meetup, let us
know how we can support that!
----
2018-11-30 17:47:06 UTC - Byron: @Matteo Merli I finally upgraded to macOS
10.14 and ran the `brew upgrade` using the raw URL and got an error:
```
% brew upgrade
<https://raw.githubusercontent.com/apache/pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb>
######################################################################## 100.0%
==> Upgrading 1 outdated package:
libpulsar 2.1.0-incubating-SNAPSHOT -> 2.2.0
==> Upgrading libpulsar --with-python3
==> Downloading
<https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=pulsar/pulsar-2.2>
######################################################################## 100.0%
==> cmake . -DBUILD_TESTS=OFF -DLINK_STATIC=ON
-DPYTHON_INCLUDE_DIR=/usr/local/Frameworks/Python.framewor
==> make pulsarShared pulsarStatic
Last 15 lines from /Users/ruthb/Library/Logs/Homebrew/libpulsar/02.make:
"_nghttp2_submit_request", referenced from:
_http2_send in libcurl.a(libcurl_la-http2.o)
"_nghttp2_submit_rst_stream", referenced from:
_Curl_http2_done in libcurl.a(libcurl_la-http2.o)
_on_frame_recv in libcurl.a(libcurl_la-http2.o)
"_nghttp2_submit_settings", referenced from:
_Curl_http2_switched in libcurl.a(libcurl_la-http2.o)
"_nghttp2_version", referenced from:
_Curl_http2_ver in libcurl.a(libcurl_la-http2.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [lib/libpulsar.2.2.0.dylib] Error 1
make[2]: *** [lib/CMakeFiles/pulsarShared.dir/all] Error 2
make[1]: *** [lib/CMakeFiles/pulsarShared.dir/rule] Error 2
make: *** [pulsarShared] Error 2
Do not report this issue to Homebrew/brew or Homebrew/core!
These open issues may also help:
libpulsar 2.2.0 (new formula)
<https://github.com/Homebrew/homebrew-core/pull/33584>
```
----
2018-11-30 17:51:24 UTC - Matteo Merli: @Byron Let me try again with newer
brew. I was able to compile it without problems earlier.
Also, the homebrew CI the build goes through. I was trying to get it included
in the official repo… I don’t know any maintainer to ping
:slightly_smiling_face:
----
2018-11-30 17:52:41 UTC - Byron: Thanks.. yea I unfortunately don’t have any
connections there. Hopefully it will be merged soon
----
2018-11-30 17:53:13 UTC - Matteo Merli: Actually, I had simplified the recipe
there, based on the feedback
----
2018-11-30 17:53:36 UTC - Matteo Merli: Can you try to install
<https://raw.githubusercontent.com/Homebrew/homebrew-core/31505b03c97755db2a11ad70d8ad605f7578b1a8/Formula/libpulsar.rb>
----
2018-11-30 17:55:49 UTC - Julien Plissonneau Duquène: Thanks for the support!
I'm just beginning with Pulsar for now, but I will probably give a talk next
year in my company, then in a local CNCF or DevOps meetup. A lot of IT people
around heard about Pulsar and are interested to know more about it.
+1 : Guangzhong Yao
----
2018-11-30 17:59:03 UTC - Byron: Running..
----
2018-11-30 18:01:37 UTC - Byron: Same error
----
2018-11-30 18:02:23 UTC - Matteo Merli: Gotcha. Do you have any other `libcurl`
installed?
----
2018-11-30 18:02:46 UTC - Matteo Merli: I think we generally pick the version
that comes with macos
----
2018-11-30 18:03:44 UTC - Byron: I had a homebrew version. I purged and
rerunning the pulsar install
----
2018-11-30 18:03:57 UTC - Matteo Merli: Also, can you use `--verbose` ?
----
2018-11-30 18:04:05 UTC - Matteo Merli: That should print the commands it’s
using
----
2018-11-30 18:04:19 UTC - Byron: Yes
----
2018-11-30 18:10:20 UTC - Byron: The install worked
----
2018-11-30 18:10:41 UTC - Byron: So `brew install curl` may break the install
----
2018-11-30 18:11:49 UTC - Byron: Probably should add this to the docs or an FAQ
(or figure out why it breaks). Thanks for the help
----
2018-11-30 18:12:01 UTC - Matteo Merli: :+1:
----
2018-11-30 20:12:39 UTC - Tobias: @Tobias has joined the channel
----
2018-11-30 22:38:24 UTC - Matteo Merli: @Christophe Bornet (Sorry I forgot
about this thread).
> I don’t see how the consumer priorityLevel can bring the feature. To my
understanding, at one point in time one broker is the partition owner and all
reads go to that broker. Reads from consumers of the other DC will result in
cross-dc bandwidth consumption. What would be needed is that these consumer
read from a broker on their dc (so not the partition owner but another
read-only one). (e
In any case, with replication the messages have to be sent to the other DC1. If
you’re consuming 1 copy of the data, the network bandwidth will be the same.
> What would be nice would be to have region aware election of partition
owners. So that we could configure for a given topic or partition on which DC
the owner broker shall be chosen preferably.
That’s not easily achievable in an async-replication mode. You want to isolate
the clusters (as much as possible) to prevent outages to spread to different
regions.
----
2018-11-30 23:13:14 UTC - Christophe Bornet: My question is for sync
replication (only one cluster spanning over multiple regions). It's in this
setup that we have to consider cross-dc consumption
----
2018-11-30 23:17:10 UTC - Christophe Bornet: I can setup Bookies so that they
are placed region aware. Then from what I understand, the brokers will do
speculative reads on the closest bookie. But currently it's not possible to set
region affinity to a partition.
----
2018-11-30 23:28:26 UTC - Matteo Merli: That’s correct. In that case it’s a
single logical cluster, where the data is configured to be stored in the
different regions. Under normal conditions though, the brokers don’t read the
messages from BK, but rather dispatch directly from cache
----
2018-12-01 07:48:06 UTC - Yifan: One quick question, is it true that using
MessageId.earliest will seek to the earliest message including acknowledged
messages?
----
2018-12-01 08:19:33 UTC - Sijie Guo: @Yifan in what context / method?
----
2018-12-01 08:23:41 UTC - Yifan: I am using consumer, I am trying to understand
earliest, latest and the default cursor
----
2018-12-01 08:26:52 UTC - Sijie Guo: yes. when you call consumer#seek() it is
basically reset the subscription to the earliest message (including acked
messages)
----
2018-12-01 08:29:21 UTC - Pavel Vetokhin: Hi! I am trying to deploy Pulsar to
the local microk8s kube node.
But `pulsar-cluster-metadata-init` pod getting the error
`[main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn -
Client session timed out, have not heard from server in 31029ms for sessionid
0x0`.
Is anyone having the same issue or is it just my kube misconfiguration?
----
2018-12-01 08:30:36 UTC - Yifan: make sense @Sijie Guo thank you
----
2018-12-01 08:32:33 UTC - Pavel Vetokhin: I use latest specs from
<https://github.com/apache/pulsar/tree/master/deployment/kubernetes/generic>
And follow the guide
<https://pulsar.apache.org/docs/latest/deployment/Kubernetes/#pulsar-on-a-custom-kubernetes-cluster>
----
