2019-06-11 09:13:49 UTC - LennonChin: @LennonChin has joined the channel
----
2019-06-11 10:17:28 UTC - Guillaume Braibant: @Guillaume Braibant has joined
the channel
----
2019-06-11 14:08:53 UTC - Alexandre DUVAL: Hi, I have got 3 bookkeepers nodes,
considers the stats of my pulsar topics I have ~500MB of storage size sum, but
all my bookeepers nodes have 15GB used. There is a way to view state,
understand why and clean?
----
2019-06-11 14:43:16 UTC - Alexandre DUVAL: Other thing, I launched offload for
a topic, the topic storage size is 25MB, and it's ongoing for 30 minutes, the
status indicates offload is running. Is it normal?
----
2019-06-11 14:55:39 UTC - Addison Higham: the little bit I have done with
offload had it run quite quickly. but depending on how you have offload
configured and how much you are ingesting, it might be offload from subsequent
runs?
----
2019-06-11 14:55:56 UTC - Addison Higham: not sure about storage size, haven't
dug in to understand how that number is calculated
----
2019-06-11 14:59:43 UTC - Alexandre DUVAL: Hum.. ```kannar@pond ~ % s3cmd du
0 0 objects <s3://pulsar-storage/>
--------
0 Total
``` I think it's stuck in running mode.
----
2019-06-11 15:01:40 UTC - David Kjerrumgaard: @Alexandre DUVAL Are there any
entries in the log file? Maybe it is a permissions issue with the S3 bucket?
----
2019-06-11 15:03:00 UTC - Alexandre DUVAL: @David Kjerrumgaard There is custom
a logfile for offloads or which logfile do you mean?
----
2019-06-11 15:05:02 UTC - David Kjerrumgaard: Any of them really, perhaps in
`pulsar.log` ?
----
2019-06-11 15:07:43 UTC - Alexandre DUVAL: ```yo-pulsar-c1-n7 /pulsar #
bin/pulsar-admin topics offload-status yo/logs/applications-addons-unclassified
Offload is currently running
yo-pulsar-c1-n7 /pulsar # cat logs/pulsar.log
yo-pulsar-c1-n7 /pulsar #
```
----
2019-06-11 15:07:45 UTC - Alexandre DUVAL: nope
----
2019-06-11 15:09:28 UTC - Addison Higham: are you running in docker? the docker
image defaults to writing all logs to stdout for docker logs
----
2019-06-11 15:10:55 UTC - Alexandre DUVAL: Docker is evil, no.
:stuck_out_tongue:
----
2019-06-11 15:10:57 UTC - Alexandre DUVAL: ```14:34:04.282 [pulsar-web-30-6]
INFO org.apache.pulsar.broker.service.persistent.PersistentTopic -
[<persistent://yo/logs/applications-addons-unclassified>] Starting offload
operation at messageId 619:0:-1
14:34:04.284 [pulsar-web-30-6] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[yo/logs/persistent/applications-addons-unclassified] Going to offload ledgers
[558, 577, 583, 589]
14:34:04.285 [pulsar-web-30-6] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[yo/logs/persistent/applications-addons-unclassified] Preparing metadata to
offload ledger 558 with uuid d2801733-f935-4feb-a719-71ea075701ff
14:34:04.298 [pulsar-web-30-6] INFO org.eclipse.jetty.server.RequestLog -
192.168.10.15 - - [11/Jun/2019:14:34:04 +0000] "PUT
/admin/v2/persistent/yo/logs/applications-addons-unclassified/offload HTTP/1.1"
204 0 "-" "Pulsar-Java-v2.4.0-SNAPSHOT" 20
14:34:04.306 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[yo/logs/persistent/applications-addons-unclassified] Metadata prepared for
offload of ledger 558 with uuid d2801733-f935-4feb-a719-71ea075701ff
```
----
2019-06-11 15:11:12 UTC - Alexandre DUVAL: on broker node which manages the
offloading topic, then nothing.
----
2019-06-11 15:13:07 UTC - Alexandre DUVAL: But I'm not using a real s3, I'm
using a ceph cluster s3 like. I think it's related, and maybe the error
management is not good here. I will run a pulsar locally and debug it.
:slightly_smiling_face:
+1 : David Kjerrumgaard
----
2019-06-11 15:21:22 UTC - Chris DiGiovanni: @Alexandre DUVAL We had a few
issues with offloading to ceph and currently running 2.3.0 as we have a a
couple custom patches. One of our the issues that we patched was fixed
upstream. The other is still open.
<https://github.com/apache/bookkeeper/issues/1970>
+1 : Sijie Guo
----
2019-06-11 15:22:06 UTC - Chris DiGiovanni: Anytime the broker's bookkeeper
client hit this error, our offloading would silently stop.
----
2019-06-11 15:23:27 UTC - Alexandre DUVAL: Ok, I'm running a master from begin
of May. Will dive in code :wink:. Thanks.
----
2019-06-11 15:27:23 UTC - Chris DiGiovanni: For reference, the closed fixed
issues: <https://github.com/apache/pulsar/pull/3929>
----
2019-06-11 21:12:02 UTC - Sree Vaddi: We welcome speakers for this recurring
monthly meetup to bring the community together, further.
----
2019-06-11 21:28:12 UTC - Sree Vaddi: .
----
2019-06-11 21:28:27 UTC - Sree Vaddi:
<https://www.meetup.com/SF-Bay-Area-Apache-Pulsar-Meetup/events/zpsdcryzjbwb/>
+1 : Sijie Guo, Alexandre DUVAL, Ryan Samo, Penghui Li, xiaolong.ran
tada : Sijie Guo, Alexandre DUVAL, Penghui Li, xiaolong.ran
bananadance : Sijie Guo, Penghui Li, xiaolong.ran
----
2019-06-12 01:34:48 UTC - Alexandre DUVAL: @Sijie Guo just a bit context about
the DM i sent you. I've got only these logs, then nothing about offload on
broker logs.
----
2019-06-12 02:21:13 UTC - 光头: @光头 has joined the channel
----
2019-06-12 08:13:05 UTC - Brian Doran: Any update on a release date for 2.4.0?
----
