elasticsearch component config issues
We have been using the camel-elasticsearch component in transport client configuration for some time, however we were seeing NodeClosedExceptions when the node with which it was communicating was restarted so switched to the node client to take advantage of the automatic failover. We configured the client with data=false as we didn't want it to store data and things worked quite nicely. However we also use Marvel to collect cluster stats and after a cluster restart we found that the node client in our Camel application had become the master node and no stats were being collected by Marvel on the other nodes in the cluster. When our application restarted one of the regular nodes became master and Marvel started working again. This leads me to question whether an application node client should be eligible to be a master node or not. I have also seen some plugins that only run on the master node in a cluster, and if the master node is a node client that would stop them working, wouldn't it? Perhaps this could be configurable. I'd much rather use the transport client as it feels a better fit for our application's purpose, however the lack of failover support in the component is unacceptable. I'd like to be able to specify a list of hosts for the component to use (maybe in the ActiveMQ style of failover(host1:9200,host2:9200,host3:9200)) or enable sniffing so that the transport client could discover the rest of the cluster if it can connect once. All things considered, it would be cumbersome to specify all these things in the URI on a route, and I wonder if it would be better to have a more flexible way of configuring the component in code or XML (again, like ActiveMQ :-)) Apologies if that is a bit rambling. I think Camel and elasticsearch are fantastic and have been spoiled so far with how easy it is integrating things that maybe I expect things to be too simple now! Regards, Alex. -- View this message in context: http://camel.465427.n5.nabble.com/elasticsearch-component-config-issues-tp5761369.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: elasticsearch component config issues
It doesn’t make sense that camel-elasticsearch endpoint became a master node. In camel we just setup the Client parameter here[1], please feel free to add more option on it if you need. I think we can also introduce addresses option to add other host address for the transport client to use, please feel free to add a JIRA for it. [1]https://github.com/apache/camel/blob/master/components/camel-elasticsearch/src/main/java/org/apache/camel/component/elasticsearch/ElasticsearchEndpoint.java#L82-L86 -- Willem Jiang Red Hat, Inc. Web: http://www.redhat.com Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On January 6, 2015 at 8:40:56 PM, batwad (a...@oh.is.it) wrote: We have been using the camel-elasticsearch component in transport client configuration for some time, however we were seeing NodeClosedExceptions when the node with which it was communicating was restarted so switched to the node client to take advantage of the automatic failover. We configured the client with data=false as we didn't want it to store data and things worked quite nicely. However we also use Marvel to collect cluster stats and after a cluster restart we found that the node client in our Camel application had become the master node and no stats were being collected by Marvel on the other nodes in the cluster. When our application restarted one of the regular nodes became master and Marvel started working again. This leads me to question whether an application node client should be eligible to be a master node or not. I have also seen some plugins that only run on the master node in a cluster, and if the master node is a node client that would stop them working, wouldn't it? Perhaps this could be configurable. I'd much rather use the transport client as it feels a better fit for our application's purpose, however the lack of failover support in the component is unacceptable. I'd like to be able to specify a list of hosts for the component to use (maybe in the ActiveMQ style of failover(host1:9200,host2:9200,host3:9200)) or enable sniffing so that the transport client could discover the rest of the cluster if it can connect once. All things considered, it would be cumbersome to specify all these things in the URI on a route, and I wonder if it would be better to have a more flexible way of configuring the component in code or XML (again, like ActiveMQ :-)) Apologies if that is a bit rambling. I think Camel and elasticsearch are fantastic and have been spoiled so far with how easy it is integrating things that maybe I expect things to be too simple now! Regards, Alex. -- View this message in context: http://camel.465427.n5.nabble.com/elasticsearch-component-config-issues-tp5761369.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: elasticsearch component config issues
Hi Willem, Thanks for your reply. I should have added that we are using Camel 2.12 and I see the code you've highlighted is tagged as 2.14, so it's possible you've fixed a bug already. I notice the 2.14 code does enable sniffing, which the 2.12 code does not. However it looks like node.client is only set when using the transport client (i.e.: when you specify an IP address). When using the node client (i.e.: cluster name only, no IP) no settings are given so it doesn't look like the node will be started in client mode. Maybe that could be why our application node became master. Regards, Alex. -- View this message in context: http://camel.465427.n5.nabble.com/elasticsearch-component-config-issues-tp5761369p5761396.html Sent from the Camel - Users mailing list archive at Nabble.com.
