[jira] [Commented] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624078#comment-16624078
]
ASF GitHub Bot commented on METRON-1508:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1185
> In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
>
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When spinning up the Ubuntu 14 development environment, the indexing topology
> fails to write to Elasticsearch. This does not appear to be caused by
> resource constraints or OOM conditions. The indexing topology reports the
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the
> configured nodes are available:
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
> ~[stormjar.jar:?]
> at
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623632#comment-16623632
]
ASF GitHub Bot commented on METRON-1508:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1185
+1 by inspection
> In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
>
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When spinning up the Ubuntu 14 development environment, the indexing topology
> fails to write to Elasticsearch. This does not appear to be caused by
> resource constraints or OOM conditions. The indexing topology reports the
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the
> configured nodes are available:
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
> ~[stormjar.jar:?]
> at
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603618#comment-16603618
]
ASF GitHub Bot commented on METRON-1508:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1185
METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
When spinning up the Ubuntu 14 development environment, the indexing
topology fails to write to Elasticsearch. The indexing topology reports the
following error.
```
2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3
3] [ERROR]
org.elasticsearch.client.transport.NoNodeAvailableException: None of the
configured nodes are available:
[{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
at
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
~[stormjar.jar:?]
at
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
~[stormjar.jar:?]
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
~[stormjar.jar:?]
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
~[stormjar.jar:?]
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
~[stormjar.jar:?]
at
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
~[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
[stormjar.jar:?]
at
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at
org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
```
In Ubuntu, Elasticsearch is not setup to bind to `node1:9300`. When the
index topology spins-up and attempts to connect, it fails with this exception.
Simply binding to `localhost:9300` fixes this problem in the Ubuntu development
environment.
I extracted the `es_hosts` variable from `single_node_vm.yml` in a way that
allows each development environment (Ubuntu and CentOS) to define this value
independently. This allows CentOS to continue to use `node1:9300` as it always
has and Ubuntu can use `localhost:9300` as it needs to.
This change only impacts the way the development environments are deployed.
This does not impact any of the production deployment tooling.
## Testing
1. Spin-up the CentOS development environment. Ensure that alerts are
visible within the Alerts UI and that the Metron Service Check passes.
```
cd metron-deployment/development/centos6
vagrant up
```
1. Spin-up the Ubuntu development environment. Ensure that alerts are
visible within the Alerts UI and that the Metron Service Check passes.
```
cd metron-deployment/development/ubuntu14
vagrant up
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summ
