2019-08-18 10:31:40 UTC - Sijie Guo: FYI. <https://github.com/apache/pulsar/pull/4975> the fix for localrun connecting with proxy ---- 2019-08-18 12:47:43 UTC - Martin Svensson: Hi, I'm struggling to get good read-performance when reading from topics with offloaded ledgers. At first, I thought it had something to do with the offloading driver or the storage itself, but after switching from S3 to GCS and seeing the same performance issues, I wonder if it's something else.
Has anyone else had problems with read-performance on offloaded topics? Any ideas on how I can improve it? I'm running pretty vanilla pulsar on gke. ---- 2019-08-18 13:09:40 UTC - nxliu: @nxliu has joined the channel ---- 2019-08-18 13:35:53 UTC - Chris Bartholomew: @Ming Fang does this PIP line up with your thinking for your Raspberry Pi scenario?<https://github.com/apache/pulsar/wiki/PIP-21%3A-Pulsar-Edge-Component> ---- 2019-08-18 13:55:18 UTC - Ming Fang: @Chris Bartholomew Yes the Pulsar Edge looks great. My concern is it’s too elaborate for a limited target audience and therefore will not get done. I’m planning to hack something up with the current Pulsar client and function runtimes ---- 2019-08-18 14:13:32 UTC - Chris Bartholomew: Sound cool. If you want help testing it, let me know. ---- 2019-08-18 14:14:39 UTC - Ming Fang: :+1: ---- 2019-08-18 16:45:55 UTC - Luke Lu: What’s the Pulsar version? Do you have any numbers? Would adjusting offloading threshold help? ---- 2019-08-18 20:16:39 UTC - Martin Svensson: I'm using 2.4.0. I've tried a bunch of different scenarios but I'll give you two examples: • producing 100 msg/s at 0.8Mbit/s for a few hours results in several offloaded ledgers • starting a new subscription with new consumers (Shared) (from earliest) gives me messages at a rate under 200 msg/s (barely catching up) • producing 1000 msg/s for half a minute doesn't trigger any offloading • consuming from the get-go will initially consume msgs at a rate of ~5k msg/s for a few seconds and then assume the producing rate of 1000 msg/s ---- 2019-08-19 01:15:30 UTC - William Xu: @William Xu has joined the channel ---- 2019-08-19 06:02:12 UTC - Luke Lu: Does increasing s3ManagedLedgerOffloadReadBufferSizeInBytes (from the default 1MB) help? @Sijie Guo, anybody/progress working on <https://github.com/apache/pulsar/issues/4025>? ----
