[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252692#comment-15252692
]
Anshum Gupta edited comment on SOLR-8599 at 4/21/16 8:48 PM:
-
I see this in the CHANGE log for 6.0 but there's no Fix version. I guess this
was just a mistake. Please correct me if I'm missing something here.
I'm also back porting this for 5.5.1.
was (Author: anshumg):
I see this in the CHANGE log for 6.0 but there's no Fix version. I guess this
was just a mistake. Please correct me if I'm missing something here.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189585#comment-15189585
]
Keith Laban edited comment on SOLR-8599 at 3/10/16 6:45 PM:
uploaded a new patch against master addressing Uwes concerns. I see that this
was already merged in master, but that seems to be the only branch its
currently on so hopefully its ok to put this out there.
Instead of scheduling a thread to change the address I was able to mock the
DefaultConnectionStrategy to throw an exception in the reconnect call the first
time its called to verify that it is being retried. With this change I also
changed the server address to be final again and removed the public setter
method seeing how they were no longer needed
Note: The new patch doesn't have the changes which introduced the retry logic
in {{ConnectionManager}} since that part is already merged in on the master
branch and branch_6x/branch_6_0. I can put in a separate patch containing both
for backport to branch_5x if needed
was (Author: k317h):
uploaded a new patch against master addressing Uwes concerns. I see that this
was already merged in master, but that seems to be the only branch its
currently on so hopefully its ok to put this out there.
Instead of scheduling a thread to change the address I was able to mock the
DefaultConnectionStrategy to throw an exception in the reconnect call the first
time its called to verify that it is being retried. With this change I also
changed the server address to be final again and removed the public setter
method seeing how they were no longer needed
Note: The new patch doesn't have the changes which introduced the retry logic
in {{ConnectionManager}} since that part is already merged in on the master
branch. I can put in a separate patch containing both for backport to branch_6x
and branch_5x
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189585#comment-15189585
]
Keith Laban edited comment on SOLR-8599 at 3/10/16 5:27 PM:
uploaded a new patch against master addressing Uwes concerns. I see that this
was already merged in master, but that seems to be the only branch its
currently on so hopefully its ok to put this out there.
Instead of scheduling a thread to change the address I was able to mock the
DefaultConnectionStrategy to throw an exception in the reconnect call the first
time its called to verify that it is being retried. With this change I also
changed the server address to be final again and removed the public setter
method seeing how they were no longer needed
Note: The new patch doesn't have the changes which introduced the retry logic
in {{ConnectionManager}} since that part is already merged in on the master
branch. I can put in a separate patch containing both for backport to branch_6x
and branch_5x
was (Author: k317h):
uploaded a new patch against master addressing Uwes concerns. I see that this
was already merged in master, but that seems to be the only branch its
currently on so hopefully its ok to put this out there.
Instead of scheduling a thread to change the address I was able to mock the
DefaultConnectionStrategy to throw an exception in the reconnect call the first
time its called to verify that it is being retried. With this change I also
changed the server address to be final again and removed the public setter
method seeing how they were no longer needed
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158022#comment-15158022
]
Dennis Gove edited comment on SOLR-8599 at 2/23/16 12:35 AM:
-
New patch with slightly updated test. This now causes an exception due to an
invalid url instead of a DNS failure. It exposes the same issue being fixed and
the test shows that the patch does in fact resolve it. This patch is what was
committed in 2c0a5e30364d83dc82383075a5f7c65200022494.
was (Author: dpgove):
New patch with slightly updated test. This now causes an exception due to an
invalid url instead of a DNS failure. It exposes the same issue being fixed and
the test shows that the patch does in fact resolve it.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157834#comment-15157834
]
Uwe Schindler edited comment on SOLR-8599 at 2/22/16 10:32 PM:
---
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
FYI, {{ff01::}} is the address range for local Multicast addresses, but only
{{::1}} (all local nodes) and {{::2}} (all routers) are allowed in IPv6
networks. {{ff01::114}} is plain invalid and refused by host's TCP stack for
using in TCP connections. So it is guaranteed to fail.
was (Author: thetaphi):
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
FYI, {{ff01::}} is the address range for Multicast addresses, but only {{::1}}
(all local nodes) and {{::2}} (all routers) are allowed in IPv6 networks.
{{ff01::114}} is plain invalid and refused by host's TCP stack for using in TCP
connections. So it is guaranteed to fail.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157834#comment-15157834
]
Uwe Schindler edited comment on SOLR-8599 at 2/22/16 10:32 PM:
---
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
FYI, {{ff01::}} is the address range for local Multicast addresses, but only
{{ff01::1}} (all local nodes) and {{ff01::2}} (all routers) are allowed in IPv6
networks. {{ff01::114}} is plain invalid and refused by host's TCP stack for
using in TCP connections. So it is guaranteed to fail.
was (Author: thetaphi):
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
FYI, {{ff01::}} is the address range for local Multicast addresses, but only
{{::1}} (all local nodes) and {{::2}} (all routers) are allowed in IPv6
networks. {{ff01::114}} is plain invalid and refused by host's TCP stack for
using in TCP connections. So it is guaranteed to fail.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157834#comment-15157834
]
Uwe Schindler edited comment on SOLR-8599 at 2/22/16 10:31 PM:
---
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
FYI, {{ff01::}} is the address range for Multicast addresses, but only {{::1}}
(all local nodes) and {{::2}} (all routers) are allowed in IPv6 networks.
{{ff01::114}} is plain invalid and refused by host's TCP stack for using in TCP
connections. So it is guaranteed to fail.
was (Author: thetaphi):
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157834#comment-15157834
]
Uwe Schindler edited comment on SOLR-8599 at 2/22/16 10:27 PM:
---
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr tests. For similar reasons we
also do not use "localhost" as hostname, but instead use 127.0.0.1 as IP
address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
was (Author: thetaphi):
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr. For similar reasons we also
do not use "localhost" as hostname, but instead use 127.0.0.1 as IP address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141360#comment-15141360
]
Dennis Gove edited comment on SOLR-8599 at 2/10/16 6:16 PM:
This fails the forbidden api precommit check.
{code}
[forbidden-apis] Forbidden method invocation:
java.util.concurrent.Executors#newSingleThreadScheduledExecutor() [Spawns
threads with vague names; use a custom thread factory (Lucene's
NamedThreadFactory, Solr's DefaultSolrThreadFactory) and name threads so that
you can tell (by its name) which executor it is associated with]
[forbidden-apis] in org.apache.solr.cloud.ConnectionManagerTest
(ConnectionManagerTest.java:119)
{code}
was (Author: dpgove):
This fais the forbidden api precommit check.
{code}
[forbidden-apis] Forbidden method invocation:
java.util.concurrent.Executors#newSingleThreadScheduledExecutor() [Spawns
threads with vague names; use a custom thread factory (Lucene's
NamedThreadFactory, Solr's DefaultSolrThreadFactory) and name threads so that
you can tell (by its name) which executor it is associated with]
[forbidden-apis] in org.apache.solr.cloud.ConnectionManagerTest
(ConnectionManagerTest.java:119)
{code}
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
