[
https://issues.apache.org/jira/browse/STORM-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heather McCartney updated STORM-2736:
-------------------------------------
Description:
Sometimes, after our topologies have been running for a while, Zookeeper does
not respond within an appropriate time and we see
{code}
2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal
lost leadership.
2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal
gained leadership, checking if it has all the topology code locally.
2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all active
topology found localy.
{code}
That's fine, and we probably need to allocate more resources. But after a new
leader is chosen, we then see:
{code}
o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
{code}
over and over.
I can't figure out yet how to cause the conditions that lead to Zookeeper
becoming unresponsive, but it is possible to reproduce the {{BlobStoreUtils}}
error by restarting Zookeeper.
The problem, I think, is that the loop
[here|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175]
never executes because the {{NimbusInfos}} list is empty. If I add a check
similar to
[this|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244]
for a node which exists but has no children, the error goes away.
was:
Sometimes, after our topologies have been running for a while, Zookeeper does
not respond within an appropriate time and we see
{code}
2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal
lost leadership.
2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal
gained leadership, checking if it has all the topology code locally.
2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all active
topology found localy.
{code}
That's fine, and we probably need to allocate more resources. But after a new
leader is chosen, we then see:
{code}
o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
{code}
over and over.
I can't figure out yet how to cause the conditions that lead to Zookeeper
becoming unresponsive, but it is possible to reproduce the BlobStoreUtils error
by restarting Zookeeper.
The problem, I think, is that the loop
[here|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175]
never executes because the {{NimbusInfos}} list is empty. If I add a check
similar to
[this|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244]
for a node which exists but has no children, the error goes away.
> o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key
> -----------------------------------------------------------------
>
> Key: STORM-2736
> URL: https://issues.apache.org/jira/browse/STORM-2736
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 1.1.1
> Reporter: Heather McCartney
> Assignee: Heather McCartney
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Sometimes, after our topologies have been running for a while, Zookeeper does
> not respond within an appropriate time and we see
> {code}
> 2017-08-16 10:18:38.859 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal
> lost leadership.
> 2017-08-16 10:21:31.144 o.a.s.zookeeper [INFO] ip-10-181-20-70.ec2.internal
> gained leadership, checking if it has all the topology code locally.
> 2017-08-16 10:21:46.201 o.a.s.zookeeper [INFO] Accepting leadership, all
> active topology found localy.
> {code}
> That's fine, and we probably need to allocate more resources. But after a new
> leader is chosen, we then see:
> {code}
> o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key<key>
> {code}
> over and over.
> I can't figure out yet how to cause the conditions that lead to Zookeeper
> becoming unresponsive, but it is possible to reproduce the {{BlobStoreUtils}}
> error by restarting Zookeeper.
> The problem, I think, is that the loop
> [here|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L175]
> never executes because the {{NimbusInfos}} list is empty. If I add a check
> similar to
> [this|https://github.com/apache/storm/blob/v1.1.1/storm-core/src/jvm/org/apache/storm/blobstore/BlobStoreUtils.java#L244]
> for a node which exists but has no children, the error goes away.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
