Peter Schulz created FLINK-33857:
------------------------------------
Summary: Expose BulkProcessor.flush via RequestIndexer to allow
custom flush logic
Key: FLINK-33857
URL: https://issues.apache.org/jira/browse/FLINK-33857
Project: Flink
Issue Type: Improvement
Components: Connectors / ElasticSearch
Affects Versions: elasticsearch-3.0.1
Reporter: Peter Schulz
There's a bug in elasticsearch that may lead to underestimated bulk request
sizes. We are hit by this bug and therefore, I would like to propose a simple
improvement to work around it:
{{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES'
{{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under
the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the
server when any of the configurable limits is hit (there's another bug:
flushing will happen _after_ the limit has exceeded).
(+) *Proposal:* Expose {{BulkProcessor.flush()}} via
{{RequestIndexer.flush()}}. This way we can easily implement logic to decide
when to flush inside the emitter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
