Sorry new born at home and I'm tired, missed the JVM version request. I verified it's the same [same AMI image I created for both instances so I expected this to be the case]. java version "1.8.0_25"
I've tried reconfiguring to use unicast and I get a similar error but I need to retest to confirm 100%. Shane On Tue, Jan 6, 2015 at 7:43 PM, Ross Simpson <[email protected]> wrote: > Shane, > > What's JVM version is ES running under, as Mark asked? That exception > usually indicates that you're running two different JVMs in the cluster, > which unfortunately is not supported due to how Java serializes exceptions. > > Ross > > > On Wednesday, 7 January 2015 10:22:46 UTC+11, shane adams wrote: >> >> Hi Mark >> >> Thank you for your response. >> >> Yes both have the same ES version [ same AMI with ES pre-installed]. I >> can telnet between the two servers on 9300, 9200, 9400. Running netstat >> shows the two machines are actively connected on port 9300. >> >> The original machine which becomes the master machine has around 2.8G >> with of indexes. The secondary machine I've tried with a complete copy of >> the index or starting with an empty index. >> >> Cheers, >> Shane >> >> >> On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom <[email protected]> wrote: >> >>> Are both running the same ES and java versions? >>> >>> Can you telnet between the data and master nodes on 9300? >>> >>> On 7 January 2015 at 09:56, <[email protected]> wrote: >>> >>>> Howdy - I cannot get two ec2 servers to connect to one another as a >>>> cluster. The servers are successfully discovering themselves via the >>>> supplied AWS credentials with proper permissions, however I the non-master >>>> server continually connects to and fails when joining the master. I have >>>> verified the ports are open for both TCP and UDP on both servers in the >>>> 9200-9400 port range. >>>> >>>> The specific error, found in the log below is: >>>> >>>> [2015-01-06 22:52:25,954][TRACE][discovery.ec2 ] [es0] failed to send >>>> join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]] org. >>>> elasticsearch.transport.RemoteTransportException: [es1][inet[/ >>>> 172.31.23.251:9300]][internal:discovery/zen/join] Caused by: >>>> org.elasticsearch.transport.NotSerializableTransportException: >>>> [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ >>>> 172.31.29.66:9300]] connect_timeout[30s]; connection timed out: / >>>> 172.31.29.66:9300; >>>> >>>> I'm stuck after several hours, hoping someone can point me in the >>>> correct direction. >>>> >>>> Thanks! >>>> >>>> Shane >>>> >>>> >>>> *Log excerpt:* >>>> [2015-01-06 22:51:52,618][TRACE][discovery.ec2 ] [es0] starting to >>>> ping >>>> >>>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] building >>>> dynamic unicast discovery nodes... >>>> >>>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] adding >>>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>>> 172.31.29.66:9300] >>>> >>>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] adding >>>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>>> 172.31.23.251:9300] >>>> >>>> [2015-01-06 22:51:52,758][DEBUG][discovery.ec2 ] [es0] using dynamic >>>> discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ >>>> 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ >>>> 172.31.23.251:9300]]] >>>> >>>> [2015-01-06 22:51:54,348][TRACE][discovery.ec2 ] [es0] building >>>> dynamic unicast discovery nodes... >>>> >>>> [2015-01-06 22:51:54,348][TRACE][discovery.ec2 ] [es0] adding >>>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>>> 172.31.29.66:9300] >>>> >>>> [2015-01-06 22:51:54,349][TRACE][discovery.ec2 ] [es0] adding >>>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>>> 172.31.23.251:9300] >>>> >>>> [2015-01-06 22:51:54,349][DEBUG][discovery.ec2 ] [es0] using dynamic >>>> discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ >>>> 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ >>>> 172.31.23.251:9300]]] >>>> >>>> [2015-01-06 22:51:55,913][TRACE][discovery.ec2 ] [es0] building >>>> dynamic unicast discovery nodes... >>>> >>>> [2015-01-06 22:51:55,913][TRACE][discovery.ec2 ] [es0] adding >>>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>>> 172.31.29.66:9300] >>>> >>>> [2015-01-06 22:51:55,914][TRACE][discovery.ec2 ] [es0] adding >>>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>>> 172.31.23.251:9300] >>>> >>>> [2015-01-06 22:51:55,914][DEBUG][discovery.ec2 ] [es0] using dynamic >>>> discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ >>>> 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ >>>> 172.31.23.251:9300]]] >>>> >>>> [2015-01-06 22:51:55,922][TRACE][discovery.ec2 ] [es0] full ping >>>> responses: >>>> >>>> --> ping_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master >>>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>>> hasJoinedOnce [true], cluster_name[elasticsearch]} >>>> >>>> [2015-01-06 22:51:55,922][DEBUG][discovery.ec2 ] [es0] filtered ping >>>> responses: (filter_client[true], filter_data[false]) >>>> >>>> --> ping_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master >>>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>>> hasJoinedOnce [true], cluster_name[elasticsearch]} >>>> >>>> [2015-01-06 22:51:55,922][TRACE][discovery.ec2 ] [es0] joining master >>>> [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172. >>>> 31.23.251:9300]] >>>> >>>> [2015-01-06 22:52:25,954][TRACE][discovery.ec2 ] [es0] failed to send >>>> join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]] >>>> >>>> org.elasticsearch.transport.RemoteTransportException: [es1][inet[/ >>>> 172.31.23.251:9300]][internal:discovery/zen/join] >>>> >>>> Caused by: org.elasticsearch.transport.NotSerializableTransportException: >>>> [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ >>>> 172.31.29.66:9300]] connect_timeout[30s]; connection timed out: / >>>> 172.31.29.66:9300; >>>> >>>> >>>> *And here is the config file I am using on both servers:* >>>> cluster.name: elasticsearch >>>> plugin: >>>> mandatory: cloud-aws >>>> >>>> cloud.aws.region: us-west-2 >>>> cloud: >>>> aws: >>>> access_key: AAA >>>> secret_key: BBB >>>> discovery: >>>> type: ec2 >>>> discovery.ec2.tag.stack: staging >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7% >>>> 40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>> topic/elasticsearch/TpFqBy5GdXA/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac% >>> 3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG8XaFWh5wGXyjriL4FW_zD2s%2BZiJQHNqRKaf-rufVQVKQQJ_Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
