[
https://issues.apache.org/jira/browse/FLINK-33857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Schulz updated FLINK-33857:
---------------------------------
Description:
There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in
elasticsearch that may lead to underestimated bulk request sizes. We are hit by
this bug and therefore, I would like to propose a simple improvement to work
around it:
{{ElasticsearchWriter}} uses {{RequestIndexer}} as a facade for ES'
{{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under
the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the
server when any of the configurable limits is hit (there's another bug:
flushing will happen _after_ the limit has exceeded).
(+) *Proposal:* Expose {{BulkProcessor.flush()}} via
{{RequestIndexer.flush()}}. This way we can easily implement logic to decide
when to flush, inside the emitter.
I created a [pull
request|https://github.com/apache/flink-connector-elasticsearch/pull/85].
Feedback is highly welcome. :)
was:
There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in
elasticsearch that may lead to underestimated bulk request sizes. We are hit by
this bug and therefore, I would like to propose a simple improvement to work
around it:
{{ElasticsearchSinkWriter}} uses {{RequestIndexer}} as a facade for ES'
{{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated. Under
the hood, the {{BulkProcessor}} takes care of flushing bulk requests to the
server when any of the configurable limits is hit (there's another bug:
flushing will happen _after_ the limit has exceeded).
(+) *Proposal:* Expose {{BulkProcessor.flush()}} via
{{RequestIndexer.flush()}}. This way we can easily implement logic to decide
when to flush, inside the emitter.
I created a [pull
request|https://github.com/apache/flink-connector-elasticsearch/pull/85].
Feedback is highly welcome. :)
> Expose BulkProcessor.flush via RequestIndexer to allow custom flush logic
> -------------------------------------------------------------------------
>
> Key: FLINK-33857
> URL: https://issues.apache.org/jira/browse/FLINK-33857
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / ElasticSearch
> Affects Versions: elasticsearch-3.0.1
> Reporter: Peter Schulz
> Priority: Major
> Labels: pull-request-available
>
> There's [a bug|https://github.com/elastic/elasticsearch/issues/103406] in
> elasticsearch that may lead to underestimated bulk request sizes. We are hit
> by this bug and therefore, I would like to propose a simple improvement to
> work around it:
> {{ElasticsearchWriter}} uses {{RequestIndexer}} as a facade for ES'
> {{BulkProcessor}}. As of now, only the {{add(…)}} methods are delegated.
> Under the hood, the {{BulkProcessor}} takes care of flushing bulk requests to
> the server when any of the configurable limits is hit (there's another bug:
> flushing will happen _after_ the limit has exceeded).
> (+) *Proposal:* Expose {{BulkProcessor.flush()}} via
> {{RequestIndexer.flush()}}. This way we can easily implement logic to decide
> when to flush, inside the emitter.
> I created a [pull
> request|https://github.com/apache/flink-connector-elasticsearch/pull/85].
> Feedback is highly welcome. :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
