merlimat commented on issue #4158: Create more offloader indices per each block uploaded to 2nd tier URL: https://github.com/apache/pulsar/pull/4158#issuecomment-487661431 > In my opinion, the issue here is really that we are calling seek each time in the ReadHandle implementation. I didn't really want to get much in the implementation, just apply the easiest fix. All considered, I think the index overhead is still negligible compared to the data size. > What are the before and after numbers for this? Before: ``` presto> SELECT COUNT(*) FROM pulsar."public/default".tweets; _col0 -------- 255251 (1 row) Query 20190427_062727_00031_cihap, FINISHED, 4 nodes Splits: 33 total, 33 done (100.00%) 1:48 [255K rows, 68.4MB] [2.35K rows/s, 646KB/s] ``` After: ``` presto> SELECT COUNT(*) FROM pulsar."public/default".tweets; _col0 --------- 7810072 (1 row) Query 20190429_165451_00016_cihap, FINISHED, 4 nodes Splits: 33 total, 33 done (100.00%) 0:13 [7.81M rows, 2.22GB] [598K rows/s, 174MB/s] ``` > What is the benchmark setup? 4 nodes reading from S3
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
