tenthe opened a new issue, #5034:
URL: https://github.com/apache/streampipes/issues/5034
### Body
Raise the throughput of writing events into datasets through the REST bulk
write, the CSV import and the pipeline sink by removing avoidable per-request
and per-event overhead in the Influx write path.
## Description
Writing events into a dataset is limited by three sources of overhead:
- The REST bulk write endpoint POST /api/v4/datalake/measurements/{name}
creates a new InfluxDB client for every request, pings it and lists databases,
and the CSV import writes through the same path in 5,000-row batches. Together
with the sleep on close this capped the endpoint at about 8,000 events per
second regardless of batch size and the CSV import near 4,000 rows per second.
- Points are handed to the client's asynchronous batch queue, so a request
returns before its data is durable.
- For every field of every event, InfluxNameSanitizer streams over the list
of about 76 InfluxQL reserved keywords with a case-insensitive comparison, and
the event field is removed and re-added even when its name does not change.
### StreamPipes Committer
I acknowledge that I am a maintainer/committer of the Apache StreamPipes
project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
