[
https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125806#comment-17125806
]
Etienne Chauchot commented on FLINK-17961:
------------------------------------------
[~chesnay] ESÂ source can definitely mask the overall complexity to the user. As
an example in Apache Beam ([available
here|https://github.com/apache/beam/blob/e1963c11f9a853564d62f83993dec08ed8a9321f/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L156])
what we do is we use sliced scroll to split the input collection for parallel
reading and apply it to the user ES query or to a default _select * from index_
when there is no provided query. Thus, the user API remains simple with
_ESIO.read().from(index).withQuery(query)._ My worries here are more related to
streaming and failover capabilities raised by Aljoscha. Even though ES is a
main source (not an enrichment one IMO) it does not meet some Flink
expectancies (cf comments above). So the question is reduced to: is it worth
investing some time to make an ES source still? Regarding the thread on an ES
table source, I'll read it and comment if I have anything useful to say.
> Create an Elasticsearch source
> ------------------------------
>
> Key: FLINK-17961
> URL: https://issues.apache.org/jira/browse/FLINK-17961
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / ElasticSearch
> Reporter: Etienne Chauchot
> Priority: Minor
>
> There is only an Elasticsearch sink available. There are opensource github
> repos such as [this
> one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also
> the apache bahir project does not provide an Elasticsearch source connector
> for flink either. IMHO I think the project would benefit from having an
> bundled source connector for ES alongside with the available sink connector.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
