Jayer23 opened a new issue, #229:
URL: https://github.com/apache/pulsar-client-python/issues/229
When using pulsar-client v3.5.0 and pulsar-client v2.10.2 to send the same
batch of data (about 100MB), 3.5.0 takes about 3.5 times longer than 2.10.2.
### Core Code
```shell
async def consume_pulsar(q: asyncio.Queue, url: str, size: int):
try:
cli, pro = None, None
for info in pulsar_infos:
cli = pulsar.Client(
io_threads=4,
service_url=info.service_url)
pro = cli.create_producer(
info.topic,
schema=pulsar.schema.StringSchema(),
compression_type=pulsar.CompressionType.LZ4,
batching_enabled=True,
batching_max_allowed_size_in_bytes=1048576,
max_pending_messages=10000,
max_pending_messages_across_partitions=50000,
batching_max_publish_delay_ms=10,
block_if_queue_full=True
)
while True:
lines = await q.get()
if lines is None:
break
for line in lines:
pro.send_async(line, callback=send_callback)
pro.flush()
cli.close()
except Exception:
raise
```
### 3.5.0
### 2.10.2
The above is the running time statistics of the line_profiler tool. The main
time consumption is `pro.send_async(line, callback=send_callback)`, which
accounts for more than 97%. The pulsar-client v3.5.0 takes about 127s, and the
pulsar-client v2.10.2 version takes about 35.6s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
