echauchot commented on pull request #17097: URL: https://github.com/apache/beam/pull/17097#issuecomment-1070534414
> > b) For anyone using stateful batching before, no performance change would be present. In terms of _Elasticsearch_ performance, state-based batching is highly preferred in my experience. I have been able to improve throughput 100x by using state-based over bundle-based batching. In terms of Beam performance on a runner, I don't have concrete numbers describing the performance delta. I do know that a production workload I manage processes millions of documents (GBs of data) per hour on a single vCPU using stateful batching. In my experience, the process of indexing data into ES has always been heavily IO bound. > > Makes sense. Thanks for clarification. Agree but state support in streaming mode does not work with spark RDD/DataStream runner. So in some cases, still needs to do in memory batching. And state is still unimplemented for Spark Structured Streaming runner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
