[
https://issues.apache.org/jira/browse/BEAM-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074543#comment-16074543
]
ASF GitHub Bot commented on BEAM-2541:
--------------------------------------
GitHub user echauchot opened a pull request:
https://github.com/apache/beam/pull/3493
[BEAM-2541] Check Elasticsearch backend version when the pipeline is run
not when it is constructed
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [X] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [X] Make sure tests pass via `mvn clean verify`.
- [X] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [X] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf).
---
R: @jbonofre
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/echauchot/beam
BEAM-2541-move-ES-version-check-to-validate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3493.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3493
----
commit f64156c4df7de97767adf70b2f02310727d9e902
Author: Etienne Chauchot <[email protected]>
Date: 2017-07-05T09:35:59Z
[BEAM-2541] Check Elasticsearch backend version when the pipeline is run
not when it is constructed
----
> ElasticsearchIO breaks for non locally accessible DB
> ----------------------------------------------------
>
> Key: BEAM-2541
> URL: https://issues.apache.org/jira/browse/BEAM-2541
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-extensions
> Reporter: Rik Nauta
> Assignee: Etienne Chauchot
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> We have a ES (2.4) database running in production that is not accessible from
> dev-machines because of VPS firewall rules. However, we do create the Beam
> graph locally on our machines and submit it to Google Dataflow for execution.
> However because of this check (line 213)
> {code:java}
> private static void checkVersion(ConnectionConfiguration
> connectionConfiguration)
> throws IOException {
> RestClient restClient = connectionConfiguration.createClient();
> Response response = restClient.performRequest("GET", "", new
> BasicHeader("", ""));
> JsonNode jsonNode = parseResponse(response);
> String version = jsonNode.path("version").path("number").asText();
> boolean version2x = version.startsWith("2.");
> restClient.close();
> checkArgument(
> version2x,
> "ConnectionConfiguration.create(addresses, index, type): "
> + "the Elasticsearch version to connect to is different of 2.x.
> "
> + "This version of the ElasticsearchIO is only compatible with
> Elasticsearch v2.x");
> }
> {code}
> the creation of the graph fails because it can't complete the restClient
> request from our dev machines. This check should probably not happen this
> early since there is no guarantee that the database is reachable when
> building the graph
> {code:java}
> exception in thread "main" java.net.ConnectException
> at
> org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
> at
> org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
> at
> org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
> at
> org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
> at
> org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
