andrasbeni opened a new issue, #18723: URL: https://github.com/apache/pulsar/issues/18723
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version 2.10.2 ### Minimal reproduce step 1. Configure offloading to s3 for `public/default` namespace . 2. Produce data (e.g. `pulsar-perf produce -r 10000 my-topic`) 3. Wait for ledger to be offloaded (take note of ledger id, e.g. :3042) 4. Start performance reader `pulsar-perf read --start-message-id 3042:10000 my-topic` 5. Start another performance reader on the same ledger parallelly (e.g. `pulsar-perf read --start-message-id 3042:1 my-topic`) ### What did you expect to see? I expected to see that the total throughput does not decrease when I started the second reader. ### What did you see instead? Output of the first reader: ``` 2022-12-02T17:48:46,513+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 355251 msg --- 867.685 msg/s -- 6.779 Mbit/s 2022-12-02T17:48:56,785+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 366587 msg --- 1101.922 msg/s -- 8.609 Mbit/s 2022-12-02T17:49:07,099+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 379534 msg --- 1261.065 msg/s -- 9.852 Mbit/s 2022-12-02T17:49:17,254+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 390209 msg --- 1033.806 msg/s -- 8.077 Mbit/s 2022-12-02T17:49:27,509+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 397502 msg --- 717.947 msg/s -- 5.609 Mbit/s 2022-12-02T17:49:37,791+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 397502 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:49:47,956+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 399108 msg --- 156.512 msg/s -- 1.223 Mbit/s 2022-12-02T17:49:58,214+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 399108 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:50:08,490+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 400779 msg --- 163.682 msg/s -- 1.279 Mbit/s 2022-12-02T17:50:18,769+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 400779 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:50:28,964+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 402452 msg --- 163.732 msg/s -- 1.279 Mbit/s 2022-12-02T17:50:39,316+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 402452 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:50:49,634+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 402452 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:50:59,839+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 415205 msg --- 1242.467 msg/s -- 9.707 Mbit/s 2022-12-02T17:51:10,048+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 427002 msg --- 1159.108 msg/s -- 9.056 Mbit/s 2022-12-02T17:51:20,297+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 440807 msg --- 1350.310 msg/s -- 10.549 Mbit/s ``` Output of the second reader: ``` 2022-12-02T17:49:34,032+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 6 msg --- 0.600 msg/s -- 0.005 Mbit/s 2022-12-02T17:49:44,328+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 6 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:49:54,587+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 1255 msg --- 120.749 msg/s -- 0.943 Mbit/s 2022-12-02T17:50:04,903+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 1255 msg --- 0.000 msg/s -- 0.000 Mbit/s 2022-12-02T17:50:15,144+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 2658 msg --- 135.650 msg/s -- 1.060 Mbit/s 2022-12-02T17:50:25,407+0100 [main] INFO o.a.p.t.PerformanceReader - Read throughput: 2658 msg --- 0.000 msg/s -- 0.000 Mbit/s ``` The sum of the two throughputs is significantly smaller than the throughput before the second reader is started. ### Anything else? In some cases, the first reader does not recover and keeps reading slowly while the following line is frequently logged by the broker: ` WARN org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreBackedReadHandleImpl - There hasn't enough data to read, current available data has -129613847 bytes, seek to the first entry 136730 to avoid EOF exception ` ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
