[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787153#comment-16787153
]
Christine Poerschke commented on SOLR-8599:
---
bq. ... maybe fix it in another Jira ...
SOLR-13254 has a fix.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
>Priority: Major
> Fix For: 5.5.1, 6.0
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236750#comment-16236750
]
Varun Thacker commented on SOLR-8599:
-
Hi Dennis,
In ConnectionManager do we intend to wait for 1 or 5 seconds? Just wanted to
clarify and maybe fix it in another Jira
{code}
log.info("Could not connect due to error, sleeping for 5s and trying
agian");
waitSleep(1000);
{code}
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
>Priority: Major
> Fix For: 5.5.1, 6.0
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252955#comment-15252955
]
ASF subversion and git services commented on SOLR-8599:
---
Commit 8d24d72ab64d435a5e6bdca11b5e79c22f0057ef in lucene-solr's branch
refs/heads/branch_6_0 from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8d24d72 ]
SOLR-8599: Improved the tests for this issue to avoid changing a variable to
non-final
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252953#comment-15252953
]
ASF subversion and git services commented on SOLR-8599:
---
Commit 78176e23bcac5c6e4accd8989dc931ec6cedb188 in lucene-solr's branch
refs/heads/branch_6x from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=78176e2 ]
SOLR-8599: Improved the tests for this issue to avoid changing a variable to
non-final
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252868#comment-15252868
]
ASF subversion and git services commented on SOLR-8599:
---
Commit 983abb1ca14f7ee42678a03f9d754af8e05e8288 in lucene-solr's branch
refs/heads/branch_5_5 from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=983abb1 ]
SOLR-8599: Improved the tests for this issue to avoid changing a variable to
non-final
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252867#comment-15252867
]
ASF subversion and git services commented on SOLR-8599:
---
Commit 853e1b99b10ccce4029fd77ba88df17dbc77ce3d in lucene-solr's branch
refs/heads/branch_5_5 from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=853e1b9 ]
SOLR-8599: After a failed connection during construction of SolrZkClient
attempt to retry until a connection can be made
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252863#comment-15252863
]
Anshum Gupta commented on SOLR-8599:
Thanks Keith. We need to track these better :)
I'll commit the other one to 6x too. I got both of them to 5x and I'm about to
commit these to 5.5.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252815#comment-15252815
]
ASF subversion and git services commented on SOLR-8599:
---
Commit 20e2caba9615e19f84fbcc59a950fb197385592e in lucene-solr's branch
refs/heads/branch_5x from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=20e2cab ]
SOLR-8599: Improved the tests for this issue to avoid changing a variable to
non-final
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252814#comment-15252814
]
ASF subversion and git services commented on SOLR-8599:
---
Commit d9875832f4798e5f732f4ae5627c7b306ccafa9c in lucene-solr's branch
refs/heads/branch_5x from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d987583 ]
SOLR-8599: After a failed connection during construction of SolrZkClient
attempt to retry until a connection can be made
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252720#comment-15252720
]
Keith Laban commented on SOLR-8599:
---
[~anshumg] There were two separate commits for this ticket, but they didn't all
land on 6x or 6.0 but are both on master:
{code}
commit e3b785a906d6f93e04f2cb45c436516158af0425
Author: Dennis Gove
Date: Sun Mar 20 11:13:56 2016 -0400
SOLR-8599: Improved the tests for this issue to avoid changing a variable
to non-final
commit 2c0a5e30364d83dc82383075a5f7c65200022494
Author: Dennis Gove
Date: Wed Feb 10 15:02:18 2016 -0500
SOLR-8599: After a failed connection during construction of SolrZkClient
attempt to retry until a connection can be made
{code}
however only the first commit found its way to 6x and 6.0 so please port that
second commit and remember to port both for 5.5.1, thanks
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Fix For: master, 6.0, 5.5.1
>
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203380#comment-15203380
]
ASF subversion and git services commented on SOLR-8599:
---
Commit e3b785a906d6f93e04f2cb45c436516158af0425 in lucene-solr's branch
refs/heads/master from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e3b785a ]
SOLR-8599: Improved the tests for this issue to avoid changing a variable to
non-final
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch,
> SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158520#comment-15158520
]
Uwe Schindler commented on SOLR-8599:
-
Ah, the whole test is marked as slow.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158511#comment-15158511
]
Uwe Schindler commented on SOLR-8599:
-
2nd: Do we really need to wait 5 !!! seconds in this test? This is horrible for
people running the test! At least you should mark it with @Slow so it only runs
nightly.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158506#comment-15158506
]
Uwe Schindler commented on SOLR-8599:
-
Hi, I would go a completely other route to test this:
- make the zkAddress final again. The change here will not work correctly in
Multi-CPU environments. Also it is always a bad idea to "suddenly" change
accessibility of fields
- instead of trying to connect to a wronmg address, we should emulate what is
really the issue: the same address does not resolve or work. The correct way to
"emulate" this failure is to intercept the connection to zookeeper. All socket
connections in Java are handled through a SocketFactory. During the test you
can exchange the default SocketFactory by a custom one that throws IOException,
if the connect address is the one in the test and delegate otherwise (to not
break other stuff like HTTP connections,...). After a few seconds restore the
original socket factory (and also in finally!!!). By this we emulate the real
failure and we dont need to use "bad"/"invalid" adresses. You can also throw
the real execption like {{java.net.UnknownHostException}}.
Beware: This may not work, if Zookeeper caches the socketfactory internally,
but that is unlikely.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158473#comment-15158473
]
Uwe Schindler commented on SOLR-8599:
-
In addition there is a bug in the test: It sets the "correct" server address in
another thread but there is no synchronization. This may work many times but
fails easily, especially on highly parallel CPU architectures! Making the
zkServerAddress volatile would be a fix, but as its for testing only, it is not
a good idea.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158459#comment-15158459
]
Uwe Schindler commented on SOLR-8599:
-
Hi,
why not make the setter of the zkAddress package private? Test is in same
package, so it does not need to be fully public!
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158017#comment-15158017
]
Dennis Gove commented on SOLR-8599:
---
Thanks. I'll keep that in mind going forward - and I agree, depending on a DNS
failure was ripe for issues down the road. As the test was changed to instead
rely on an invalid url I think what's in there will be good. New attachment
forthcoming.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
>Assignee: Dennis Gove
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158011#comment-15158011
]
ASF subversion and git services commented on SOLR-8599:
---
Commit 2c0a5e30364d83dc82383075a5f7c65200022494 in lucene-solr's branch
refs/heads/master from [~dpgove]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2c0a5e3 ]
SOLR-8599: After a failed connection during construction of SolrZkClient
attempt to retry until a connection can be made
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157834#comment-15157834
]
Uwe Schindler commented on SOLR-8599:
-
Hi,
please don't write tests that rely on DNS failures. Every local provider does
something else, but not all return a correct "not found". Unless we have a
"mock DNS server", we cannot use any DNS in Solr. For similar reasons we also
do not use "localhost" as hostname, but instead use 127.0.0.1 as IP address.
To have some broken address that always fails to connect to, use an IPv6
address that is standardized to be routed nowhere and already refused by the
Hosts's IPv6 stack. According to spec one possible address is:
{{http://[ff01::114]:anyport/}}
As example look at SolrExceptionTest.java
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157810#comment-15157810
]
Mark Miller commented on SOLR-8599:
---
Yeah, we try to avoid that in tests due to dns hijacking. I think there is an
ip6 address that works for this? [~thetaphi], am I remembering right?
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157644#comment-15157644
]
Dennis Gove commented on SOLR-8599:
---
I have somewhat of an interesting situation at hand here.
As part of this patch a test is added to ConnectionManagerTest which forces a
DNS failure on the zookeeper connection by attempting to connect to
"BADADDRESS" and then fixing it after 5 seconds. This shows that the change
Keith put in ConnectionManager will continually try to make a connection until
it can. It's a good test and it exercises the bug and fix perfectly.
However, the test depends on my ISP. I've run the test under 5 scenarios and
only 3 of them pass.
1. Connected to my corporate network
In this scenario the test passes perfectly as it should.
2. Connected to no network (ie, wifi card turned off)
In this scenario the test passes perfectly as it should.
3. Connected to my home network backed by Verizon FIOS
In this scenario the test hangs and upon further investigation I found that it
is in an "infinite" loop in ConnectionManager::waitForConnected. This appears
to be an infinite loop because while there is a timeout the timeout is
Long.MAX_VALUE. The problem here is that the loop waits until it is either
connected or closed. Neither of those conditions are ever hit. But why? We're
trying to hit http://BADADDRESS and clearly that is a DNS lookup failure. Oh no
no no, not according to Verizon. See, Verizon instead says "Oh, you must've
typed something in wrong so instead of returning to you a DNS failure let me
return to you a redirect to a search page - you clearly want this search page".
It appears that because of this redirection a connection is never made nor is
it ever closed. Hence, loop forever.
4. Connected to my personal wifi hotspot backed by T-Mobile
Same issue as seen with Verizon FIOS, though a T-Mobile specific search page.
5. Connected to a hotspot through my iPhone backed by Verizon Wireless
In this scenario the test passes perfectly as it should.
Note that this difference is *only* seen when a DNS lookup failure is in play.
If I change the bad address to "http://BADADDRESS; then it fails instead
because "//BADADDRESSIS" is said to be an invalid path string. Technically this
is testing a slightly different case but I'm comfortable calling it the same
test because the issue being corrected is a failure to make a connection during
the construction of SolrZooKeeper and a malformed url fails just the same.
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch, SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread]
[jira] [Commented] (SOLR-8599) Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent state
[
https://issues.apache.org/jira/browse/SOLR-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141360#comment-15141360
]
Dennis Gove commented on SOLR-8599:
---
This fais the forbidden api precommit check.
{code}
[forbidden-apis] Forbidden method invocation:
java.util.concurrent.Executors#newSingleThreadScheduledExecutor() [Spawns
threads with vague names; use a custom thread factory (Lucene's
NamedThreadFactory, Solr's DefaultSolrThreadFactory) and name threads so that
you can tell (by its name) which executor it is associated with]
[forbidden-apis] in org.apache.solr.cloud.ConnectionManagerTest
(ConnectionManagerTest.java:119)
{code}
> Errors in construction of SolrZooKeeper cause Solr to go into an inconsistent
> state
> ---
>
> Key: SOLR-8599
> URL: https://issues.apache.org/jira/browse/SOLR-8599
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Reporter: Keith Laban
> Attachments: SOLR-8599.patch
>
>
> We originally saw this happen due to a DNS exception (see stack trace below).
> Although any exception thrown in the constructor of SolrZooKeeper or the
> parent class, ZooKeeper, will cause DefaultConnectionStrategy to fail to
> update the zookeeper client. Once it gets into this state, it will not try to
> connect again until the process is restarted. The node itself will also
> respond successfully to query requests, but not to update requests.
> Two things should be address here:
> 1) Fix the error handling and issue some number of retries
> 2) If we are stuck in a state like this stop responding to all requests
> {code}
> 2016-01-23 13:49:20.222 ERROR ConnectionManager [main-EventThread] -
> :java.net.UnknownHostException: HOSTNAME: unknown error
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
> at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at
> org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> at org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:41)
> at
> org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:53)
> at
> org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-23 13:49:20.222 INFO ConnectionManager [main-EventThread] -
> Connected:false
> 2016-01-23 13:49:20.222 INFO ClientCnxn [main-EventThread] - EventThread shut
> down
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
