Shane, What's JVM version is ES running under, as Mark asked? That exception usually indicates that you're running two different JVMs in the cluster, which unfortunately is not supported due to how Java serializes exceptions.
Ross On Wednesday, 7 January 2015 10:22:46 UTC+11, shane adams wrote: > > Hi Mark > > Thank you for your response. > > Yes both have the same ES version [ same AMI with ES pre-installed]. I > can telnet between the two servers on 9300, 9200, 9400. Running netstat > shows the two machines are actively connected on port 9300. > > The original machine which becomes the master machine has around 2.8G with > of indexes. The secondary machine I've tried with a complete copy of the > index or starting with an empty index. > > Cheers, > Shane > > > On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom <[email protected] > <javascript:>> wrote: > >> Are both running the same ES and java versions? >> >> Can you telnet between the data and master nodes on 9300? >> >> On 7 January 2015 at 09:56, <[email protected] <javascript:>> wrote: >> >>> Howdy - I cannot get two ec2 servers to connect to one another as a >>> cluster. The servers are successfully discovering themselves via the >>> supplied AWS credentials with proper permissions, however I the non-master >>> server continually connects to and fails when joining the master. I have >>> verified the ports are open for both TCP and UDP on both servers in the >>> 9200-9400 port range. >>> >>> The specific error, found in the log below is: >>> >>> [2015-01-06 22:52:25,954][TRACE][discovery.ec2 ] [es0] failed to send >>> join request to master >>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] >>> org.elasticsearch.transport.RemoteTransportException: >>> [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] Caused by: >>> org.elasticsearch.transport.NotSerializableTransportException: >>> [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ >>> 172.31.29.66:9300]] connect_timeout[30s]; connection timed out: / >>> 172.31.29.66:9300; >>> >>> I'm stuck after several hours, hoping someone can point me in the >>> correct direction. >>> >>> Thanks! >>> >>> Shane >>> >>> >>> *Log excerpt:* >>> [2015-01-06 22:51:52,618][TRACE][discovery.ec2 ] [es0] starting to ping >>> >>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] building dynamic >>> unicast discovery nodes... >>> >>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] adding >>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>> 172.31.29.66:9300] >>> >>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] adding >>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>> 172.31.23.251:9300] >>> >>> [2015-01-06 22:51:52,758][DEBUG][discovery.ec2 ] [es0] using dynamic >>> discovery nodes >>> [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], >>> [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] >>> >>> [2015-01-06 22:51:54,348][TRACE][discovery.ec2 ] [es0] building dynamic >>> unicast discovery nodes... >>> >>> [2015-01-06 22:51:54,348][TRACE][discovery.ec2 ] [es0] adding >>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>> 172.31.29.66:9300] >>> >>> [2015-01-06 22:51:54,349][TRACE][discovery.ec2 ] [es0] adding >>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>> 172.31.23.251:9300] >>> >>> [2015-01-06 22:51:54,349][DEBUG][discovery.ec2 ] [es0] using dynamic >>> discovery nodes >>> [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], >>> [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] >>> >>> [2015-01-06 22:51:55,913][TRACE][discovery.ec2 ] [es0] building dynamic >>> unicast discovery nodes... >>> >>> [2015-01-06 22:51:55,913][TRACE][discovery.ec2 ] [es0] adding >>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>> 172.31.29.66:9300] >>> >>> [2015-01-06 22:51:55,914][TRACE][discovery.ec2 ] [es0] adding >>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>> 172.31.23.251:9300] >>> >>> [2015-01-06 22:51:55,914][DEBUG][discovery.ec2 ] [es0] using dynamic >>> discovery nodes >>> [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/172.31.29.66:9300]], >>> [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/172.31.23.251:9300]]] >>> >>> [2015-01-06 22:51:55,922][TRACE][discovery.ec2 ] [es0] full ping >>> responses: >>> >>> --> ping_response{node >>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>> >>> id[51], master >>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>> >>> hasJoinedOnce [true], cluster_name[elasticsearch]} >>> >>> [2015-01-06 22:51:55,922][DEBUG][discovery.ec2 ] [es0] filtered ping >>> responses: (filter_client[true], filter_data[false]) >>> >>> --> ping_response{node >>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>> >>> id[51], master >>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>> >>> hasJoinedOnce [true], cluster_name[elasticsearch]} >>> >>> [2015-01-06 22:51:55,922][TRACE][discovery.ec2 ] [es0] joining master >>> [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]] >>> >>> >>> [2015-01-06 22:52:25,954][TRACE][discovery.ec2 ] [es0] failed to send >>> join request to master >>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]] >>> >>> >>> >>> org.elasticsearch.transport.RemoteTransportException: >>> [es1][inet[/172.31.23.251:9300]][internal:discovery/zen/join] >>> >>> Caused by: >>> org.elasticsearch.transport.NotSerializableTransportException: >>> [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ >>> 172.31.29.66:9300]] connect_timeout[30s]; connection timed out: / >>> 172.31.29.66:9300; >>> >>> >>> *And here is the config file I am using on both servers:* >>> cluster.name: elasticsearch >>> plugin: >>> mandatory: cloud-aws >>> >>> cloud.aws.region: us-west-2 >>> cloud: >>> aws: >>> access_key: AAA >>> secret_key: BBB >>> discovery: >>> type: ec2 >>> discovery.ec2.tag.stack: staging >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
