[
https://issues.apache.org/jira/browse/ZOOKEEPER-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Satyawan Parab updated ZOOKEEPER-4400:
--------------------------------------------
Description:
We have three(3) node zookeeper cluster running as a pod on Kubernetes cluster,
Zookeeper version is 3.6.2,we are implementing graceful termination in our
zookeeper development through
*terminationGracePeriodSeconds:*
*refer link:
[https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/*]
When we delete pods then kuberentes sends SIGTERM signal and wait for graceful
termination(*terminationGracePeriodSeconds=30)* time before forceful deletion,
then service should catch this signal(sigterm) and should shutdown gracefully
within 30 seconds.
But zookeeper doesn't seems to be handling this sigterm and not terminating
gracefully terminated without graceful.
Trying deleting the pod
{code:java}
kubectl delete pod/test-zk-0 -n test
{code}
{code:java}
kubectl log -f pod/test-zk-0 -n test
2021-10-22T07:54:57.655+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):QuorumPeer@868]
- Peer state changed: following - synchronization
2021-10-22T07:54:57.661+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):CommitProcessor@476]
- Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1
2021-10-22T07:54:57.662+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):CommitProcessor@438]
- Configuring CommitProcessor with 2 worker threads.
2021-10-22T07:54:57.666+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):RequestThrottler@74]
- zookeeper.request_throttler.shutdownTimeout = 10000
2021-10-22T07:54:57.680+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):QuorumPeer@863]
- Peer state changed: following - broadcast{code}
zookeeper not terminating gracefully when we delete the pod not logs of
zookeeper shutdown coming and immediately pod get deleted.
Expectation: Zookeeper must shutdown gracefully when we delete the pod and its
evidence should be coming in logs
was:
We have three(3) node zookeeper cluster running as a pod on Kubernetes cluster,
Zookeeper version is 3.6.2,we are implementing graceful termination in our
zookeeper development through
*terminationGracePeriodSeconds:*
*refer link:
[https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/*]
When we delete pods then kuberentes sends SIGTERM signal and wait for graceful
termination(*terminationGracePeriodSeconds=30)* time before forceful deletion,
then service should catch this signal(sigterm) and should shutdown gracefully
within 30 seconds.
But zookeeper doesn't seems to be handling this sigterm and not terminating
gracefully terminated without graceful.
Trying deleting the pod
{code:java}
kubectl delete pod/test-zk-0 -n test
{code}
{code:java}
kubectl log -f pod/eric-data-coordinator-zk-0 -n zparsag
2021-10-22T07:54:57.655+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):QuorumPeer@868]
- Peer state changed: following - synchronization
2021-10-22T07:54:57.661+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):CommitProcessor@476]
- Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1
2021-10-22T07:54:57.662+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):CommitProcessor@438]
- Configuring CommitProcessor with 2 worker threads.
2021-10-22T07:54:57.666+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):RequestThrottler@74]
- zookeeper.request_throttler.shutdownTimeout = 10000
2021-10-22T07:54:57.680+0000 [myid:] - INFO
[QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):QuorumPeer@863]
- Peer state changed: following - broadcast{code}
zookeeper not terminating gracefully when we delete the pod not logs of
zookeeper shutdown coming and immediately pod get deleted.
Expectation: Zookeeper must shutdown gracefully when we delete the pod and its
evidence should be coming in logs
> Zookeeper not getting Graceful Termination
> ------------------------------------------
>
> Key: ZOOKEEPER-4400
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4400
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.6.2
> Environment: Kuberentes v1.20
> Reporter: Sagar Satyawan Parab
> Priority: Major
>
> We have three(3) node zookeeper cluster running as a pod on Kubernetes
> cluster,
> Zookeeper version is 3.6.2,we are implementing graceful termination in our
> zookeeper development through
> *terminationGracePeriodSeconds:*
> *refer link:
> [https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/*]
> When we delete pods then kuberentes sends SIGTERM signal and wait for
> graceful termination(*terminationGracePeriodSeconds=30)* time before forceful
> deletion, then service should catch this signal(sigterm) and should shutdown
> gracefully within 30 seconds.
> But zookeeper doesn't seems to be handling this sigterm and not
> terminating gracefully terminated without graceful.
> Trying deleting the pod
>
> {code:java}
> kubectl delete pod/test-zk-0 -n test
> {code}
>
> {code:java}
> kubectl log -f pod/test-zk-0 -n test
> 2021-10-22T07:54:57.655+0000 [myid:] - INFO
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):QuorumPeer@868]
> - Peer state changed: following - synchronization
> 2021-10-22T07:54:57.661+0000 [myid:] - INFO
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):CommitProcessor@476]
> - Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1
> 2021-10-22T07:54:57.662+0000 [myid:] - INFO
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):CommitProcessor@438]
> - Configuring CommitProcessor with 2 worker threads.
> 2021-10-22T07:54:57.666+0000 [myid:] - INFO
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):RequestThrottler@74]
> - zookeeper.request_throttler.shutdownTimeout = 10000
> 2021-10-22T07:54:57.680+0000 [myid:] - INFO
> [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=[0:0:0:0:0:0:0:0]:2281):QuorumPeer@863]
> - Peer state changed: following - broadcast{code}
> zookeeper not terminating gracefully when we delete the pod not logs of
> zookeeper shutdown coming and immediately pod get deleted.
>
> Expectation: Zookeeper must shutdown gracefully when we delete the pod and
> its evidence should be coming in logs
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
