I'm not that great at Java at the moment, but it appears that the single threaded nature of the elasticsearch sink has trouble keeping up with ~5k events/second at 2k batch size. It looks like the HDFS sink has the ability to run multiple threads that write to the HDFS. I can get some performance increase by adding multiple ElasticSearch sinks to simulate parallelism, but it would be great for the sink itself to support multiple threads.
Is there a sink example that should be used as a guide towards getting the same features in the elasticsearch sink? Thanks, Allan
