Re: Broker won't exit...

[Skip Montanaro]

Skip:Can you pastebin the stack trace of the stuck broker ?


Will try to later, when the system is quiescent. I forgot to save the
gstack output yesterday.

S

Re: Broker won't exit...

[Jan Filipiak]

HI

brokers still try todo a gracefull shutdown I supose?
It would only shut down if it is not the leader of any partition anymore.

Can you verify: there are other brokers alive that took over leadership?
and the broker in question stepped down as a leader for all partitions?

Best Jan



On 10.01.2018 12:57, Ted Yu wrote:

Skip:Can you pastebin the stack trace of the stuck broker ?
Thanks
 Original message From: Skip Montanaro 
<skip.montan...@gmail.com> Date: 1/10/18  3:52 AM  (GMT-08:00) To: 
users@kafka.apache.org Subject: Re: Broker won't exit...
Did you stop the broker before stoping zookeeper?


Yes. My stop script executes the server stop scripts in reverse order from
my start script. Should I have stuck in a couple second sleep between
stopping the brokers and stopping zookeeper?

I was actually running two brokers. The one my stop script stopped first
exited properly.

Skip

Re: Broker won't exit...

[Svante Karlsson]

if you really want all the brokers to die, try

change server.properties

controlled.shutdown.enable=false

I had a similar problem on dev laptop with a single broker. It refused to
die on system shutdowns (or took a very long time).

2018-01-10 12:57 GMT+01:00 Ted Yu <yuzhih...@gmail.com>:

> Skip:Can you pastebin the stack trace of the stuck broker ?
> Thanks
>  Original message From: Skip Montanaro <
> skip.montan...@gmail.com> Date: 1/10/18  3:52 AM  (GMT-08:00) To:
> users@kafka.apache.org Subject: Re: Broker won't exit...
> Did you stop the broker before stoping zookeeper?
>
>
> Yes. My stop script executes the server stop scripts in reverse order from
> my start script. Should I have stuck in a couple second sleep between
> stopping the brokers and stopping zookeeper?
>
> I was actually running two brokers. The one my stop script stopped first
> exited properly.
>
> Skip
>

Re: Broker won't exit...

[Ted Yu]

Skip:Can you pastebin the stack trace of the stuck broker ?
Thanks
 Original message From: Skip Montanaro 
<skip.montan...@gmail.com> Date: 1/10/18  3:52 AM  (GMT-08:00) To: 
users@kafka.apache.org Subject: Re: Broker won't exit... 
Did you stop the broker before stoping zookeeper?


Yes. My stop script executes the server stop scripts in reverse order from
my start script. Should I have stuck in a couple second sleep between
stopping the brokers and stopping zookeeper?

I was actually running two brokers. The one my stop script stopped first
exited properly.

Skip

Re: Broker won't exit...

[Skip Montanaro]

Did you stop the broker before stoping zookeeper?


Yes. My stop script executes the server stop scripts in reverse order from
my start script. Should I have stuck in a couple second sleep between
stopping the brokers and stopping zookeeper?

I was actually running two brokers. The one my stop script stopped first
exited properly.

Skip

Re: Broker won't exit...

[Skip Montanaro]

Have you tried a normal kill (sigterm) against the java process?


Yes. Nothing worked until kill -9.

Skip

Re: Broker won't exit...

[Damian Guy]

Did you stop the broker before stoping zookeeper?

On Wed, 10 Jan 2018 at 10:38 Ted Yu  wrote:

> I think that is the default signal.
> From the script:
>
> SIGNAL=${SIGNAL:-TERM}
>
> FYI
>
> On Wed, Jan 10, 2018 at 2:35 AM, Sam Pegler <
> sam.peg...@infectiousmedia.com>
> wrote:
>
> > Have you tried a normal kill (sigterm) against the java process?
> >
> > __
> >
> > Sam Pegler
> >
> > PRODUCTION ENGINEER
> >
> > T. +44(0) 07 562 867 486
> >
> > 
> > 3-7 Herbal Hill / London / EC1R 5EJ
> > www.infectiousmedia.com
> >
> > This email and any attachments are confidential and may also be
> privileged.
> > If you
> > are not the intended recipient, please notify the sender immediately, and
> > do not
> > disclose the contents to another person, use it for any purpose, or
> store,
> > or copy
> > the information in any medium. Please also destroy and delete the message
> > from
> > your computer.
> >
> >
> > On 9 January 2018 at 22:44, Skip Montanaro 
> > wrote:
> >
> > > I only discovered the kafka-server-stop.sh script a couple days ago. I
> > > can't seem to make it do its thing (the corresponding zookeeper stop
> > > script seems to work just fine). All consumers have been stopped. Lsof
> > > still shows the Kafka broker process listening on its port. The last
> > > connection left the CLOSE_WAIT state several minutes ago. Gstack shows
> > > 169 threads, most in pthread_cond_wait(), a handful in other wait-like
> > > functions (sem_wait, pthread_join, pthread_cond_timedwait, poll,
> > > epoll_wait). I'm running 2.11-1.0.0 on a Red Hat 6 server.
> > >
> > > What does it take to get a broker to exit (short of kill -9)?
> > >
> > > Thx,
> > >
> > > Skip Montanaro
> > >
> >
>

Re: Broker won't exit...

[Ted Yu]

I think that is the default signal.
>From the script:

SIGNAL=${SIGNAL:-TERM}

FYI

On Wed, Jan 10, 2018 at 2:35 AM, Sam Pegler 
wrote:

> Have you tried a normal kill (sigterm) against the java process?
>
> __
>
> Sam Pegler
>
> PRODUCTION ENGINEER
>
> T. +44(0) 07 562 867 486
>
> 
> 3-7 Herbal Hill / London / EC1R 5EJ
> www.infectiousmedia.com
>
> This email and any attachments are confidential and may also be privileged.
> If you
> are not the intended recipient, please notify the sender immediately, and
> do not
> disclose the contents to another person, use it for any purpose, or store,
> or copy
> the information in any medium. Please also destroy and delete the message
> from
> your computer.
>
>
> On 9 January 2018 at 22:44, Skip Montanaro 
> wrote:
>
> > I only discovered the kafka-server-stop.sh script a couple days ago. I
> > can't seem to make it do its thing (the corresponding zookeeper stop
> > script seems to work just fine). All consumers have been stopped. Lsof
> > still shows the Kafka broker process listening on its port. The last
> > connection left the CLOSE_WAIT state several minutes ago. Gstack shows
> > 169 threads, most in pthread_cond_wait(), a handful in other wait-like
> > functions (sem_wait, pthread_join, pthread_cond_timedwait, poll,
> > epoll_wait). I'm running 2.11-1.0.0 on a Red Hat 6 server.
> >
> > What does it take to get a broker to exit (short of kill -9)?
> >
> > Thx,
> >
> > Skip Montanaro
> >
>

Re: Broker won't exit...

[Sam Pegler]

Have you tried a normal kill (sigterm) against the java process?

__

Sam Pegler

PRODUCTION ENGINEER

T. +44(0) 07 562 867 486


3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com

This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and
do not
disclose the contents to another person, use it for any purpose, or store,
or copy
the information in any medium. Please also destroy and delete the message
from
your computer.


On 9 January 2018 at 22:44, Skip Montanaro  wrote:

> I only discovered the kafka-server-stop.sh script a couple days ago. I
> can't seem to make it do its thing (the corresponding zookeeper stop
> script seems to work just fine). All consumers have been stopped. Lsof
> still shows the Kafka broker process listening on its port. The last
> connection left the CLOSE_WAIT state several minutes ago. Gstack shows
> 169 threads, most in pthread_cond_wait(), a handful in other wait-like
> functions (sem_wait, pthread_join, pthread_cond_timedwait, poll,
> epoll_wait). I'm running 2.11-1.0.0 on a Red Hat 6 server.
>
> What does it take to get a broker to exit (short of kill -9)?
>
> Thx,
>
> Skip Montanaro
>

Broker won't exit...

[Skip Montanaro]

I only discovered the kafka-server-stop.sh script a couple days ago. I
can't seem to make it do its thing (the corresponding zookeeper stop
script seems to work just fine). All consumers have been stopped. Lsof
still shows the Kafka broker process listening on its port. The last
connection left the CLOSE_WAIT state several minutes ago. Gstack shows
169 threads, most in pthread_cond_wait(), a handful in other wait-like
functions (sem_wait, pthread_join, pthread_cond_timedwait, poll,
epoll_wait). I'm running 2.11-1.0.0 on a Red Hat 6 server.

What does it take to get a broker to exit (short of kill -9)?

Thx,

Skip Montanaro