2020-06-16 09:21:45 UTC - dionjansen: Hi Fernando, thanks for the suggestion,
the retention policy on the namespace is both set to infinite for time/ size I
have a feeling the offloader mechanism just got into some weird state because
of the disk space pressure. In the end I just rebuild the cluster from scratch
an moved the persistent ledgers into the new one
+1 : Fernando
----
2020-06-16 09:29:33 UTC - xue:
<https://github.com/apache/pulsar/issues/7270>，test pulsar, bookie oom
----
2020-06-16 09:38:21 UTC - Marcio Martins: Hey Anup, so I did some more testing,
and it seems the issue is related to the order at which the components come up,
if the broker comes up before the bookie, then it will fail forever, until you
restart it, even if the bookie comes up shortly after. I was specting it to
have some errors but then fix it self once the bookies are up. The reason seems
to be related to those DNS errors. If the broker always comes up after all
bookies are up, then there is no DNS error and everything is smooth. So I
suppose there is some caching of failed lookups. Not sure if this is done at
the broker level, or if it's done at a Kubes cluster level...
----
2020-06-16 09:47:49 UTC - Ankur Jain: @Matteo Merli Thanks for the explanation!
This makes it a lot clearer on how writes and reads do not impact each other
significantly. Would broker use read-ahead to get more data over network so
that subsequent reads for catch-up consumer are served from broker's cache?
My last query is bookkeeper setup related. Maybe this is not the right forum,
will raise it BK's slack community.
----
2020-06-16 12:06:32 UTC - Luke Stephenson: are you using the helm template to
deploy the proxy? If so, check out
<https://github.com/apache/pulsar-helm-chart/pull/12/files> which added similar
support for grafana / pulsar manager to have an allow list.
----
2020-06-16 12:49:15 UTC - Raphael Enns: Thanks for the information
:slightly_smiling_face:
----
2020-06-16 14:55:00 UTC - Marcio Martins: I got an cluster with 0 activity, but
one of the bookies has 100% cpu usage - it has been like this for over 1 hour
after I disconnected the producers. The previous activity was a pulsar-perf
with only 5 msg/sec. It seems like all 5 bookies are also running really high
on memory, > 2GB for such a light load. Anyone has any insights into what is
going on?
----
2020-06-16 15:05:32 UTC - Matteo Merli: There’s no support for IP whitelisting
at this in the proxy directly, though it should be very easy to add and it
would be a nice addition.
----
2020-06-16 15:12:08 UTC - Alexander Ursu: Hi, I was wondering if there will be
VODs from the Pulsar Summit over the next 2 days
----
2020-06-16 15:43:35 UTC - Asaf Mesika: Did you at the bookie metrics ?
----
2020-06-16 16:16:06 UTC - Daniel Ciocirlan: can someone help explain more on
the correct setup for client `ioThreads` and `connectionsPerBroker`, we have 6
brokers and a partitioned topic with 12 partitions (so 2 partitions per each
broker). Also we run 3 proxies behind a AWS NLB, so producers/consumers don't
talk directly with brokers.
----
2020-06-16 16:17:14 UTC - Addison Higham: AFAIK, they will be recorded, but no
idea on the release date
----
2020-06-16 16:17:36 UTC - Addison Higham: are you facing issues?
----
2020-06-16 16:20:02 UTC - Daniel Ciocirlan: no, but we are not yet in prod and
i want to catch this before, we used 3 connections per broker and 12 ioThreads
----
2020-06-16 16:20:20 UTC - Matthew Follegot: Thank you David
----
2020-06-16 16:20:57 UTC - Daniel Ciocirlan: i see the default is 1 for both,
but with 1 ioThreads the producer was not able to connect to the partitioned
topic, so we used 12 as we have 12 partitions
----
2020-06-16 16:21:29 UTC - Daniel Ciocirlan: was wondering if there is any
"catch" to this or guide lines :slightly_smiling_face:
----
2020-06-16 16:23:35 UTC - Matteo Merli: > Would broker use read-ahead to
get more data over network so that subsequent reads for catch-up consumer are
served from broker's cache?
For catch-up reads, the read-ahead is mostly done on the Bookie node itself,
this is to exploit the fact that data for same topic is mostly stored
sequentially on the disk.
----
2020-06-16 16:24:06 UTC - Matteo Merli: > My last query is bookkeeper setup
related. Maybe this is not the right forum, will raise it BK's slack community.
Sure, most BK people are here too though :slightly_smiling_face:
----
2020-06-16 16:32:09 UTC - Addison Higham: pulsar client is mostly async
internally, so ioThreads don't need to map one to one. Likewise,
producers/consumers can be multiplexed over a single connection
----
2020-06-16 16:33:22 UTC - Marcio Martins: Yes, everything seemed normal. It
went back to idle after a long time, but still consuming close to 2GB memory
----
2020-06-16 16:36:24 UTC - Daniel Ciocirlan: Well we have the app servers in the
same AZ as the brokers, the timeout is 200 ms, most of the pub actions takes
3-5ms but there are times that p99 is more than 200 ms and tried to increase
the IOThreads and Connections per broker, did not fix this.
----
2020-06-16 16:37:11 UTC - Daniel Ciocirlan: what would be a real scenario that
would require more ioThreads ?
----
2020-06-16 16:39:21 UTC - Daniel Ciocirlan: we are around 1M RPM on around 36
producers geo replicated to 4 regions
----
2020-06-16 16:42:19 UTC - Addison Higham: it does make sense to add more
ioThreads and possibly extra connectionsPerBroker, but I just mean that it
isn't necessary to have one per partition or anything
----
2020-06-16 16:43:23 UTC - Addison Higham: as far as those latencies, that may
well be from the broker. For example, perhaps due to GC pause if you are close
on heap? or could be disk
----
2020-06-16 16:45:04 UTC - Daniel Ciocirlan: heap not an issue the instances
jave huge memory allocated
----
2020-06-16 16:45:30 UTC - Daniel Ciocirlan: so you recommendation is to leave
the default 1/1 in this configuration ?
----
2020-06-16 16:47:23 UTC - Addison Higham: I am not quite familiar with all that
enough to give a recommendation of exact numbers, but just in general to
explain what those things do :slightly_smiling_face:
----
2020-06-16 16:48:35 UTC - Daniel Ciocirlan: :slightly_smiling_face: thanks,
will have more close look on this.
----
2020-06-16 16:48:36 UTC - Addison Higham:
<https://github.com/openmessaging/openmessaging-benchmark/blob/master/driver-pulsar/src/main/java/io/openmessaging/benchmark/driver/pulsar/config/PulsarClientConfig.java>
<- this is what the openBenchmark does
----
2020-06-16 16:48:48 UTC - Addison Higham: 8/8 is what it does
----
2020-06-16 16:49:42 UTC - Daniel Ciocirlan: for sure i need more than 1
iothreads, when the producer / consumer tries to subscribe to a 12 partitioned
traffic it would just block there
----
2020-06-16 16:50:15 UTC - Daniel Ciocirlan: but thanks for your time!
:slightly_smiling_face:
----
2020-06-16 16:51:34 UTC - Rich Adams: I have noticed a similar issue in the
broker. When it first starts up it can go up to 1.7GB with no load. If I
restart it it will only take up about 300MB
----
2020-06-16 16:55:59 UTC - Addison Higham: @Marcio Martins some of what bookies
are doing is background tasks like moving records from journal -> ledgers,
compaction, etc. That does sound like more than I would expect, but the metrics
should be able to tell you more of what was happening
----
2020-06-16 17:13:50 UTC - Albert Xu: @Albert Xu has joined the channel
----
2020-06-16 17:14:18 UTC - Marcio Martins: I had a look at it seems like
everything looks normal - in the JVM metrics it says heap 300MB, DM 400MB, but
K8s reports 2.5GB memory, again, with 0 activity.
----
2020-06-16 17:20:33 UTC - Albert Xu: hello i was trying to use the pulsar
client for python but after i do `pip install pulsar-client==2.5.2` and try to
`import pulsar` in my python repl, I get
```In [2]: import pulsar
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-07c1fdd48b98> in <module>
----> 1 import pulsar
~/.pyenv/versions/test/lib/python3.7/site-packages/pulsar/__init__.py in
<module>
100 """
101
--> 102 import _pulsar
103
104 from _pulsar import Result, CompressionType, ConsumerType,
InitialPosition, PartitionsRoutingMode # noqa: F401
ImportError:
dlopen(/Users/albertxu/.pyenv/versions/test/lib/python3.7/site-packages/_pulsar.cpython-37m-darwin.so,
2): Symbol not found: __Py_tracemalloc_config
Referenced from:
/Users/albertxu/.pyenv/versions/test/lib/python3.7/site-packages/_pulsar.cpython-37m-darwin.so
Expected in: flat namespace
in
/Users/albertxu/.pyenv/versions/test/lib/python3.7/site-packages/_pulsar.cpython-37m-darwin.so```
another person working with me also had the same issue. Is there something I'm
doing wrong?
I've tried using other versions. The only one I've gotten to work locally is
`pip3 install pulsar-client==2.4.1.post1` but no version that i've found seems
to work on AWS lambda instances. In the AWS lambda environment they look like
they all run into the same import error.
----
2020-06-16 17:34:03 UTC - Patrik Kleindl: Have you enabled the state service on
BK or are you using it in Pulsar functions?
High non-heap memory is a side effect of rocksdb
----
2020-06-16 17:34:27 UTC - Marcio Martins: What is the state service?
----
2020-06-16 17:34:34 UTC - Marcio Martins: I am using almost all default settings
----
2020-06-16 17:35:25 UTC - Marcio Martins:
```ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage```
I am using that with all default settings
----
2020-06-16 17:35:56 UTC - Patrik Kleindl: It‘s state or table service. Used for
pulsar functions to manage state.
----
2020-06-16 17:36:15 UTC - Marcio Martins: Functions are disabled
----
2020-06-16 17:40:10 UTC - Marcio Martins: I see, that is why it's not JVM
memory, ok... Can I somehow limit it without impacting performance too much?
What do you recomend?
----
2020-06-16 17:43:21 UTC - Patrik Kleindl: I have no idea, that was just a guess
because we had a similar issue with Rocksdb on Kafka Streams
Try to disable the service at BK if possible and see if it helps
----
2020-06-16 17:44:55 UTC - Marcio Martins: Thanks Patrik! I didn't remember
RocksDB memory is not from the JVM heap, so that helped a lot, and now I know
what to tune.
----
2020-06-16 17:51:01 UTC - Wayne Collins: @Wayne Collins has joined the channel
----
2020-06-16 17:52:19 UTC - Jesse Anderson: @Jesse Anderson has joined the channel
----
2020-06-16 17:56:55 UTC - Julien: Hi there, we have a spark application running
in our k8s cluster along with pulsar. This is a streaming app, constantly
trying to consume messages from a topic using pulsar spark connector. This app
is literally doing a DDOS attack on our pulsar (deployed with the helm chart
with default low requests), making brokers going out of memory after 2/3 hours
of responding to thousands/short time of lastMessageId requests (the topic
being empty all that time).
Is this normal? Is our Pulsar just too weak or there is something to
parametrize in our spark-application?
Have a good one!
----
2020-06-16 18:08:54 UTC - Piyush: @Piyush has joined the channel
----
2020-06-16 18:11:49 UTC - Marcio Martins: So after I tweaked the RocksDB memory
it's still > 2.5G... I have retention on, with 100GB offloaded to S3. Could
it be related to that? Somehow tries to keep some index of the S3 storage in
memory?
----
2020-06-16 18:14:55 UTC - Marcio Martins: Those should not even be known to
bookkeeper, right? Can't think of why else it should need 2.5GB...
----
2020-06-16 18:27:42 UTC - Patrik Kleindl: Don‘t tweak, disable it if you are
not using it. One less variable.
Did you completely restart the BKs?
----
2020-06-16 18:27:55 UTC - Marcio Martins: yes
----
2020-06-16 18:28:21 UTC - Marcio Martins: But can I disable the RocksDB? I
thought that was part of the ledger storage?
----
2020-06-16 18:34:54 UTC - Chris Herzog: @Chris Herzog has joined the channel
----
2020-06-16 18:59:48 UTC - Patrik Kleindl: @Sijie Guo @Penghui Li is it possible
to disable the table service on BK just to rule out rocksdb as a source of
memory consumption?
----
2020-06-16 19:18:07 UTC - Alan Broddle: In a Terraform AWS VM based cluster we
were in the process of mapping the Bookie Storage devices, and we see that the
Bookie is using an AWS i3.xlarge instance type. This instance type by default
creates a 950 GB SSD ephemeral storage device.
I3.xlarge
4 CPU 30.5 GB
1x 950 SSD
Network up to 10 Gigabit
In the instance build process, a nvme0n1 device is created but not
mapped/mounted to the instance.
When we run the “aws/setup-disk.yaml” command in the Ansible setup, we noticed
several issues:
1. The script specifically looks for 2 devices to map the /mt/storage and
/mnt/journal
• Only 1 device is created with the instance build
• Do we create another device? What type?
1. The SSD storage created by an EC2 i3.xlarge instance is ephemeral and is not
retained after a Server ‘stop’.
Can someone please explain a couple basic questions that we have:
1. Why was a i3.xlarge instance set up as the default instance type in the
“aws/terraform.tfvars”?
instance_types = {
“zookeeper” = “t2.large”
“bookie” = “i3.xlarge”
“broker” = “c5.2xlarge”
“proxy” = “c5.2xlarge”
}
1. What type of AWS Instance do people typically use and why?
2. If this is recommended, I assume we would need to add a second device to the
instance to get the Journal and Storage onto different devices.
3. Is there some reason that we should leverage the nvme0n1 device that is set
up by AWS with this instance type that we are not aware of?
Basically, our assumption is that when a Bookie is stopped or rebooted, that we
want the data to be retained. That would mean the default AWS instance would be
a non-ephemeral based storage instance.
----
2020-06-16 19:30:04 UTC - Marcio Martins: I believe the reason to use NVMe
storage is mostly because in high volume production deployments, the
blottleneck is going to the bookie disk i/o. NVMe have very large amount of
iops, but are ephemeral like you mentioned. However the bookie cluster *is*
performing replication - the broker will only consider the message published if
it has been stored to at least 2 bookies (configurable), so as long as you
don't turn off all your bookies at once, you should be covered. The alternative
are EBS volumes, which themselves perform replication, and are durable past
start/stop cycles, but you'd then have "double replication" which would
increase your latency, and be you'd be *very* limited by IOPS on EBS. You could
also resort to provisioned IOPS, but those are also quite expensive.
+1 : Alan Broddle
----
2020-06-16 19:52:19 UTC - Duane Sessions: @Duane Sessions has joined the channel
----
2020-06-16 20:32:49 UTC - Varghese C: @Varghese C has joined the channel
----
2020-06-16 22:52:23 UTC - Sijie Guo: You can disable table service if you don’t
use functions or function state
----
2020-06-16 23:18:42 UTC - Eric Bowden: @Eric Bowden has joined the channel
----
2020-06-17 00:15:04 UTC - Luke Stephenson: Please try to refer to this feature
as "allow list".
----
2020-06-17 00:28:26 UTC - Sijie Guo: did you install the pulsar cpp library?
----
2020-06-17 00:39:31 UTC - Matteo Merli: Yes, I guess that's a more sensible name
----
2020-06-17 04:14:46 UTC - Sijie Guo: @Sijie Guo set the channel topic: - Pulsar
Summit Virtual Conference 2020. Join <#C015MT1DE7N|pulsarsummit> channel for
real-time discussions. Schedule: <https://pulsar-summit.org/schedule/first-day/>
- Pulsar 2.5.2 released
<http://pulsar.apache.org/blog/2020/05/19/Apache-Pulsar-2-5-2/>
----
2020-06-17 04:42:50 UTC - Sankararao Routhu: thanks @Luke Stephenson @Matteo
Merli.We are deploying proxy using aws cloud formation templates. Our allowed
list would be very huge(in thousands) and it should be reflected on proxy
dynamically as and when we add a new ip. Can you please give us some pointers
how to add this feature?
----
2020-06-17 06:21:48 UTC - Asaf Mesika: Isn’t the worst is duplicate task
processing happening in parallel?
----
2020-06-17 06:22:36 UTC - Asaf Mesika: Since they write of the job shouldn’t
worry about concurrency right? (It might run in the same instance or in
different instance, but compete on writing to a DB, or something like that)
----
2020-06-17 07:24:22 UTC - Charvak Patel: @Charvak Patel has joined the channel
----
2020-06-17 08:07:25 UTC - Arushi Sharma: @Arushi Sharma has joined the channel
----
