Figured it out. One of the servers was not a member of the proper security group and port 9300 was closed. I had misconfigured ES to use 9200 for cluster transport and that's why it was giving an odd error, attempting a non-HTTP protocol against the REST API. Sorry to bug anyone.
On Wed, Jan 7, 2015 at 9:55 AM, shane adams <[email protected]> wrote: > Sorry new born at home and I'm tired, missed the JVM version request. I > verified it's the same [same AMI image I created for both instances so I > expected this to be the case]. java version "1.8.0_25" > > I've tried reconfiguring to use unicast and I get a similar error but I > need to retest to confirm 100%. > > Shane > > On Tue, Jan 6, 2015 at 7:43 PM, Ross Simpson <[email protected]> wrote: > >> Shane, >> >> What's JVM version is ES running under, as Mark asked? That exception >> usually indicates that you're running two different JVMs in the cluster, >> which unfortunately is not supported due to how Java serializes exceptions. >> >> Ross >> >> >> On Wednesday, 7 January 2015 10:22:46 UTC+11, shane adams wrote: >>> >>> Hi Mark >>> >>> Thank you for your response. >>> >>> Yes both have the same ES version [ same AMI with ES pre-installed]. I >>> can telnet between the two servers on 9300, 9200, 9400. Running netstat >>> shows the two machines are actively connected on port 9300. >>> >>> The original machine which becomes the master machine has around 2.8G >>> with of indexes. The secondary machine I've tried with a complete copy of >>> the index or starting with an empty index. >>> >>> Cheers, >>> Shane >>> >>> >>> On Tue, Jan 6, 2015 at 3:15 PM, Mark Walkom <[email protected]> wrote: >>> >>>> Are both running the same ES and java versions? >>>> >>>> Can you telnet between the data and master nodes on 9300? >>>> >>>> On 7 January 2015 at 09:56, <[email protected]> wrote: >>>> >>>>> Howdy - I cannot get two ec2 servers to connect to one another as a >>>>> cluster. The servers are successfully discovering themselves via the >>>>> supplied AWS credentials with proper permissions, however I the non-master >>>>> server continually connects to and fails when joining the master. I have >>>>> verified the ports are open for both TCP and UDP on both servers in the >>>>> 9200-9400 port range. >>>>> >>>>> The specific error, found in the log below is: >>>>> >>>>> [2015-01-06 22:52:25,954][TRACE][discovery.ec2 ] [es0] failed to send >>>>> join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]] org. >>>>> elasticsearch.transport.RemoteTransportException: [es1][inet[/ >>>>> 172.31.23.251:9300]][internal:discovery/zen/join] Caused by: >>>>> org.elasticsearch.transport.NotSerializableTransportException: >>>>> [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ >>>>> 172.31.29.66:9300]] connect_timeout[30s]; connection timed out: / >>>>> 172.31.29.66:9300; >>>>> >>>>> I'm stuck after several hours, hoping someone can point me in the >>>>> correct direction. >>>>> >>>>> Thanks! >>>>> >>>>> Shane >>>>> >>>>> >>>>> *Log excerpt:* >>>>> [2015-01-06 22:51:52,618][TRACE][discovery.ec2 ] [es0] starting to >>>>> ping >>>>> >>>>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] building >>>>> dynamic unicast discovery nodes... >>>>> >>>>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] adding >>>>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>>>> 172.31.29.66:9300] >>>>> >>>>> [2015-01-06 22:51:52,758][TRACE][discovery.ec2 ] [es0] adding >>>>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>>>> 172.31.23.251:9300] >>>>> >>>>> [2015-01-06 22:51:52,758][DEBUG][discovery.ec2 ] [es0] using dynamic >>>>> discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ >>>>> 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ >>>>> 172.31.23.251:9300]]] >>>>> >>>>> [2015-01-06 22:51:54,348][TRACE][discovery.ec2 ] [es0] building >>>>> dynamic unicast discovery nodes... >>>>> >>>>> [2015-01-06 22:51:54,348][TRACE][discovery.ec2 ] [es0] adding >>>>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>>>> 172.31.29.66:9300] >>>>> >>>>> [2015-01-06 22:51:54,349][TRACE][discovery.ec2 ] [es0] adding >>>>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>>>> 172.31.23.251:9300] >>>>> >>>>> [2015-01-06 22:51:54,349][DEBUG][discovery.ec2 ] [es0] using dynamic >>>>> discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ >>>>> 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ >>>>> 172.31.23.251:9300]]] >>>>> >>>>> [2015-01-06 22:51:55,913][TRACE][discovery.ec2 ] [es0] building >>>>> dynamic unicast discovery nodes... >>>>> >>>>> [2015-01-06 22:51:55,913][TRACE][discovery.ec2 ] [es0] adding >>>>> i-dd78fdd1, address 172.31.29.66, transport_address inet[/ >>>>> 172.31.29.66:9300] >>>>> >>>>> [2015-01-06 22:51:55,914][TRACE][discovery.ec2 ] [es0] adding >>>>> i-a220ea54, address 172.31.23.251, transport_address inet[/ >>>>> 172.31.23.251:9300] >>>>> >>>>> [2015-01-06 22:51:55,914][DEBUG][discovery.ec2 ] [es0] using dynamic >>>>> discovery nodes [[#cloud-i-dd78fdd1-0][ip-172-31-29-66][inet[/ >>>>> 172.31.29.66:9300]], [#cloud-i-a220ea54-0][ip-172-31-29-66][inet[/ >>>>> 172.31.23.251:9300]]] >>>>> >>>>> [2015-01-06 22:51:55,922][TRACE][discovery.ec2 ] [es0] full ping >>>>> responses: >>>>> >>>>> --> ping_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master >>>>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>>>> hasJoinedOnce [true], cluster_name[elasticsearch]} >>>>> >>>>> [2015-01-06 22:51:55,922][DEBUG][discovery.ec2 ] [es0] filtered ping >>>>> responses: (filter_client[true], filter_data[false]) >>>>> >>>>> --> ping_response{node [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]], id[51], master >>>>> [[es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172.31.23.251:9300]]], >>>>> hasJoinedOnce [true], cluster_name[elasticsearch]} >>>>> >>>>> [2015-01-06 22:51:55,922][TRACE][discovery.ec2 ] [es0] joining master >>>>> [es1][VNcvo8b3TSqObDo7nEwkAg][ip-172-31-23-251][inet[/172. >>>>> 31.23.251:9300]] >>>>> >>>>> [2015-01-06 22:52:25,954][TRACE][discovery.ec2 ] [es0] failed to send >>>>> join request to master [[es1][VNcvo8b3TSqObDo7nEwkAg] >>>>> [ip-172-31-23-251][inet[/172.31.23.251:9300]]] >>>>> >>>>> org.elasticsearch.transport.RemoteTransportException: [es1][inet[/ >>>>> 172.31.23.251:9300]][internal:discovery/zen/join] >>>>> >>>>> Caused by: org.elasticsearch.transport.NotSerializableTransportException: >>>>> [org.elasticsearch.transport.ConnectTransportException] [es0][inet[/ >>>>> 172.31.29.66:9300]] connect_timeout[30s]; connection timed out: / >>>>> 172.31.29.66:9300; >>>>> >>>>> >>>>> *And here is the config file I am using on both servers:* >>>>> cluster.name: elasticsearch >>>>> plugin: >>>>> mandatory: cloud-aws >>>>> >>>>> cloud.aws.region: us-west-2 >>>>> cloud: >>>>> aws: >>>>> access_key: AAA >>>>> secret_key: BBB >>>>> discovery: >>>>> type: ec2 >>>>> discovery.ec2.tag.stack: staging >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7% >>>>> 40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/ee6d6225-f383-4e6b-b4d7-464ae82414e7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>> topic/elasticsearch/TpFqBy5GdXA/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac% >>>> 3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-n4aXvneytPc2JQ2HSfaac%3D4BJ9pak0X1aT3vm9ZTdXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/TpFqBy5GdXA/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/f6db3827-b544-43c3-872c-8c433698cc5e%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAG8XaFWWf5LWc66Sw1S5x2chRpMQ2TTZ4r0DbbQaA-LPUxSEFA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
