Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Dou Liyang]

Hi Eric,

At 02/14/2018 01:40 AM, Eric W. Biederman wrote:

Dou Liyang  writes:


Hi Baoquan,

At 02/12/2018 11:08 AM, Eric W. Biederman wrote:

Baoquan He  writes:


This is a regression fix.

Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
calling after disable_IO_APIC(). This introdued a regression. The
root cause is that disable_IO_APIC() not only clears IO_APIC, also
restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
after disable_IO_APIC() will disable LAPIC and ruin the possible
virtual wire mode setting which the code has been trying to do all
along.

The consequence is, in KVM guest kernel always prints warning as below
during kexec/kdump kernel boots up. That happened in setup_local_APIC()
since 'do { xxx } while (queued && max_loops > 0)' loop does not function


I am not sure another thing here

AFAIK, according to the order of SMP machine shutdown, other CPUs will
be stopped firstly, then the last CPU disable its local apic.

  --machine_shutdown
|..
|stop_other_cpus()
|local_shutdown()

So, the pending interrupts exist only in BSP and only be ACKed by
BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
all CPUs, found only BSP had the non-zero value).


We don't know.  In the kexec on panic case we try and shutdown the other
cpus but we have a timeout because we might fail.

Further you have to be careful with the concept of boot cpu.  In the
normal kexec case shutdown on the boot cpu and leave it running.  In the
kexec on panic case we shutdown on an arbitrary cpu.


If it is right, We will do not need check the pending interrupt for each
cpus.


It is also cheap if there are no pending interrupts as there is nothing
to do.



Yes, It's cheap.

But, the local APICs in APs were disabled at that time, not sure if
writing to the end-of-interrupt (EOI) register could cause the local
APIC to clear the ISR successfully.

If the ISR was not cleared, doing that check is useless.

I can't produce any cases that the lapics in APs have pending
interrupts. do you have some suggestions?


BTW, the pending interrupt check code is mixed with the local
APIC setup code, that looks messy. How about extracting the code which
related to the crash interrupt check, put it into a new function and
only invoked it when the CPU is BSP?


Moving it into it's own function makes sense.  Let's not taint the
concept with ``crash''.  We don't know that the only way this will
ever happen is from kexec on panic.   We only know it was easy to
trigger the condition from kexec on panic.



Yes, I see.


There are a lot of cases I can think of that interrupts might fire while
interrupts are disabled because the kernel is booting.  A normal kexec
is also possible.

Throw in MSI interrupts and transitioning from one state to another in
non-legacy apic mode and we might be more likely to get some irqs in
a pending state.

Your patch makes me nervous as it is not just code motion but a much
more substantial change.


As much as I agree that we need to fix the regression in the apic
shutdown code that is causing problems.  What we really need to do
is to completely remove the apic shutdown code from the kexec on panic
code path.  Over the long term that will provide us with a much more


Wow, indeed, and it is related to many hypervisors on x86, For me, still
need more investigations and tests.

Thanks,
dou

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Dou Liyang]

Hi Eric,

At 02/14/2018 01:40 AM, Eric W. Biederman wrote:

Dou Liyang  writes:


Hi Baoquan,

At 02/12/2018 11:08 AM, Eric W. Biederman wrote:

Baoquan He  writes:


This is a regression fix.

Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
calling after disable_IO_APIC(). This introdued a regression. The
root cause is that disable_IO_APIC() not only clears IO_APIC, also
restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
after disable_IO_APIC() will disable LAPIC and ruin the possible
virtual wire mode setting which the code has been trying to do all
along.

The consequence is, in KVM guest kernel always prints warning as below
during kexec/kdump kernel boots up. That happened in setup_local_APIC()
since 'do { xxx } while (queued && max_loops > 0)' loop does not function


I am not sure another thing here

AFAIK, according to the order of SMP machine shutdown, other CPUs will
be stopped firstly, then the last CPU disable its local apic.

  --machine_shutdown
|..
|stop_other_cpus()
|local_shutdown()

So, the pending interrupts exist only in BSP and only be ACKed by
BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
all CPUs, found only BSP had the non-zero value).


We don't know.  In the kexec on panic case we try and shutdown the other
cpus but we have a timeout because we might fail.

Further you have to be careful with the concept of boot cpu.  In the
normal kexec case shutdown on the boot cpu and leave it running.  In the
kexec on panic case we shutdown on an arbitrary cpu.


If it is right, We will do not need check the pending interrupt for each
cpus.


It is also cheap if there are no pending interrupts as there is nothing
to do.



Yes, It's cheap.

But, the local APICs in APs were disabled at that time, not sure if
writing to the end-of-interrupt (EOI) register could cause the local
APIC to clear the ISR successfully.

If the ISR was not cleared, doing that check is useless.

I can't produce any cases that the lapics in APs have pending
interrupts. do you have some suggestions?


BTW, the pending interrupt check code is mixed with the local
APIC setup code, that looks messy. How about extracting the code which
related to the crash interrupt check, put it into a new function and
only invoked it when the CPU is BSP?


Moving it into it's own function makes sense.  Let's not taint the
concept with ``crash''.  We don't know that the only way this will
ever happen is from kexec on panic.   We only know it was easy to
trigger the condition from kexec on panic.



Yes, I see.


There are a lot of cases I can think of that interrupts might fire while
interrupts are disabled because the kernel is booting.  A normal kexec
is also possible.

Throw in MSI interrupts and transitioning from one state to another in
non-legacy apic mode and we might be more likely to get some irqs in
a pending state.

Your patch makes me nervous as it is not just code motion but a much
more substantial change.


As much as I agree that we need to fix the regression in the apic
shutdown code that is causing problems.  What we really need to do
is to completely remove the apic shutdown code from the kexec on panic
code path.  Over the long term that will provide us with a much more


Wow, indeed, and it is related to many hypervisors on x86, For me, still
need more investigations and tests.

Thanks,
dou

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

On 02/13/18 at 11:44am, Eric W. Biederman wrote:
> Baoquan He  writes:
> 
> > Hi Eric,
> >
> > On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
> >> Baoquan He  writes:
> >> 
> >> > This is a regression fix.
> >> >
> >> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> >> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> >> > calling after disable_IO_APIC(). This introdued a regression. The
> >> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> >> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> >> > after disable_IO_APIC() will disable LAPIC and ruin the possible
> >> > virtual wire mode setting which the code has been trying to do all
> >> > along.
> >> > To fix this, just break down disable_IO_APIC(), then call
> >> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> >> > and call restore_boot_irq_mode() to restore boot irq mode before
> >> > reboot or kexec/kdump jump.
> >> 
> >> Two things here.
> >> a) This is missing a fixes tag and a CC stable.
> >> b) What makes your change to the KEXEC_JUMP code path safe?
> >>Have the lapic and ioapic already been shut down?
> >> 
> >> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
> >> either need to be documented in the change long why they are safe
> >> so that this change becomes obviously safe and correct.
> >
> > Re-read the code, I have to admit I didn't check the KEXEC_JUMP code
> > path carefully. 
> >
> > kernel_kexec() {
> > if (kexec_image->preserve_context) {
> > ...
> > freeze_processes();
> > ...
> > disable_nonboot_cpus();
> > ...
> > 
> > else {
> > ...
> > machine_shutdown();
> > ...
> > }
> > machine_kexec(kexec_image);
> > ...
> > }
> >
> >   --machine_shutdown()
> > --native_machine_shutdown()
> >   --disable_IO_APIC()
> >   --lapic_shutdown()
> >
> > machine_kexec() {
> > ...
> > if (image->preserve_context) {
> > disable_IO_APIC();
> > }
> > ...
> > }
> >
> > KEXEC_JUMP code path is different than kexec/kdump, it doesn't call
> > lapic_shutdown() before jump. So commit 522e66464467
> > ("x86/apic: Disable I/O APIC before shutdown of the local APIC") didn't
> > impact it. And here I break down disable_IO_APIC() and change to only
> > call restore_boot_irq_mode() to make a possible danger. I am not an
> > expert on KEXEC_JUMP, and don't know how to test it, so will keep the
> > code implementation consistent as before. For now, I plan to change it
> > as below if you don't object. As you pointed out, I will describe this
> > in patch log. 
> >
> > diff --git a/arch/x86/kernel/machine_kexec_64.c 
> > b/arch/x86/kernel/machine_kexec_64.c
> > index 1f790cf9d38f..cb0c2d0a4c99 100644
> > --- a/arch/x86/kernel/machine_kexec_64.c
> > +++ b/arch/x86/kernel/machine_kexec_64.c
> > @@ -297,7 +297,7 @@ void machine_kexec(struct kimage *image)
> >  * one form or other. kexec jump path also need
> >  * one.
> >  */
> > -   disable_IO_APIC();
> > +   clear_IO_APIC();
> > +   restore_boot_irq_mode();
> >  #endif
> > }
> >
> >
> 
> Let me give a very concrete suggestion:
> Patch 1) Replace "disable_IO_APIC();" with "clear_IO_APIC(); 
> restore_boot_irq_mode();"
> Patch 2) Move restore_boot_irq_mode(); to fix the regression.
> 
> I think that will be a slightly shorter patch sequence than what you are
> dealing with and one that is slightly easier to read.

Sure, will do.

Thanks for reviewing and suggestion.

> 
> We need to sort out KEXEC_JUMP but that is something for another time.

Agree. We ever contacted the author, intel dev, seems they don't
maintain it any more. And very few people are interested and report
the relevant issues. While recently someone is testing KEXEC_JUMP
functionality and updating related doc, will continue tracking it.

Thanks
Baoquan

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

On 02/13/18 at 11:44am, Eric W. Biederman wrote:
> Baoquan He  writes:
> 
> > Hi Eric,
> >
> > On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
> >> Baoquan He  writes:
> >> 
> >> > This is a regression fix.
> >> >
> >> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> >> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> >> > calling after disable_IO_APIC(). This introdued a regression. The
> >> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> >> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> >> > after disable_IO_APIC() will disable LAPIC and ruin the possible
> >> > virtual wire mode setting which the code has been trying to do all
> >> > along.
> >> > To fix this, just break down disable_IO_APIC(), then call
> >> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> >> > and call restore_boot_irq_mode() to restore boot irq mode before
> >> > reboot or kexec/kdump jump.
> >> 
> >> Two things here.
> >> a) This is missing a fixes tag and a CC stable.
> >> b) What makes your change to the KEXEC_JUMP code path safe?
> >>Have the lapic and ioapic already been shut down?
> >> 
> >> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
> >> either need to be documented in the change long why they are safe
> >> so that this change becomes obviously safe and correct.
> >
> > Re-read the code, I have to admit I didn't check the KEXEC_JUMP code
> > path carefully. 
> >
> > kernel_kexec() {
> > if (kexec_image->preserve_context) {
> > ...
> > freeze_processes();
> > ...
> > disable_nonboot_cpus();
> > ...
> > 
> > else {
> > ...
> > machine_shutdown();
> > ...
> > }
> > machine_kexec(kexec_image);
> > ...
> > }
> >
> >   --machine_shutdown()
> > --native_machine_shutdown()
> >   --disable_IO_APIC()
> >   --lapic_shutdown()
> >
> > machine_kexec() {
> > ...
> > if (image->preserve_context) {
> > disable_IO_APIC();
> > }
> > ...
> > }
> >
> > KEXEC_JUMP code path is different than kexec/kdump, it doesn't call
> > lapic_shutdown() before jump. So commit 522e66464467
> > ("x86/apic: Disable I/O APIC before shutdown of the local APIC") didn't
> > impact it. And here I break down disable_IO_APIC() and change to only
> > call restore_boot_irq_mode() to make a possible danger. I am not an
> > expert on KEXEC_JUMP, and don't know how to test it, so will keep the
> > code implementation consistent as before. For now, I plan to change it
> > as below if you don't object. As you pointed out, I will describe this
> > in patch log. 
> >
> > diff --git a/arch/x86/kernel/machine_kexec_64.c 
> > b/arch/x86/kernel/machine_kexec_64.c
> > index 1f790cf9d38f..cb0c2d0a4c99 100644
> > --- a/arch/x86/kernel/machine_kexec_64.c
> > +++ b/arch/x86/kernel/machine_kexec_64.c
> > @@ -297,7 +297,7 @@ void machine_kexec(struct kimage *image)
> >  * one form or other. kexec jump path also need
> >  * one.
> >  */
> > -   disable_IO_APIC();
> > +   clear_IO_APIC();
> > +   restore_boot_irq_mode();
> >  #endif
> > }
> >
> >
> 
> Let me give a very concrete suggestion:
> Patch 1) Replace "disable_IO_APIC();" with "clear_IO_APIC(); 
> restore_boot_irq_mode();"
> Patch 2) Move restore_boot_irq_mode(); to fix the regression.
> 
> I think that will be a slightly shorter patch sequence than what you are
> dealing with and one that is slightly easier to read.

Sure, will do.

Thanks for reviewing and suggestion.

> 
> We need to sort out KEXEC_JUMP but that is something for another time.

Agree. We ever contacted the author, intel dev, seems they don't
maintain it any more. And very few people are interested and report
the relevant issues. While recently someone is testing KEXEC_JUMP
functionality and updating related doc, will continue tracking it.

Thanks
Baoquan

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Eric W. Biederman]

Baoquan He  writes:

> Hi Eric,
>
> On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
>> Baoquan He  writes:
>> 
>> > This is a regression fix.
>> >
>> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
>> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
>> > calling after disable_IO_APIC(). This introdued a regression. The
>> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
>> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
>> > after disable_IO_APIC() will disable LAPIC and ruin the possible
>> > virtual wire mode setting which the code has been trying to do all
>> > along.
>> > To fix this, just break down disable_IO_APIC(), then call
>> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
>> > and call restore_boot_irq_mode() to restore boot irq mode before
>> > reboot or kexec/kdump jump.
>> 
>> Two things here.
>> a) This is missing a fixes tag and a CC stable.
>> b) What makes your change to the KEXEC_JUMP code path safe?
>>Have the lapic and ioapic already been shut down?
>> 
>> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
>> either need to be documented in the change long why they are safe
>> so that this change becomes obviously safe and correct.
>
> Re-read the code, I have to admit I didn't check the KEXEC_JUMP code
> path carefully. 
>
> kernel_kexec() {
>   if (kexec_image->preserve_context) {
>   ...
>   freeze_processes();
>   ...
>   disable_nonboot_cpus();
>   ...
>   
>   else {
>   ...
>   machine_shutdown();
>   ...
>   }
>   machine_kexec(kexec_image);
>   ...
> }
>
>   --machine_shutdown()
> --native_machine_shutdown()
>   --disable_IO_APIC()
>   --lapic_shutdown()
>
> machine_kexec() {
>   ...
>   if (image->preserve_context) {
>   disable_IO_APIC();
>   }
>   ...
> }
>
> KEXEC_JUMP code path is different than kexec/kdump, it doesn't call
> lapic_shutdown() before jump. So commit 522e66464467
> ("x86/apic: Disable I/O APIC before shutdown of the local APIC") didn't
> impact it. And here I break down disable_IO_APIC() and change to only
> call restore_boot_irq_mode() to make a possible danger. I am not an
> expert on KEXEC_JUMP, and don't know how to test it, so will keep the
> code implementation consistent as before. For now, I plan to change it
> as below if you don't object. As you pointed out, I will describe this
> in patch log. 
>
> diff --git a/arch/x86/kernel/machine_kexec_64.c 
> b/arch/x86/kernel/machine_kexec_64.c
> index 1f790cf9d38f..cb0c2d0a4c99 100644
> --- a/arch/x86/kernel/machine_kexec_64.c
> +++ b/arch/x86/kernel/machine_kexec_64.c
> @@ -297,7 +297,7 @@ void machine_kexec(struct kimage *image)
>  * one form or other. kexec jump path also need
>  * one.
>  */
> -   disable_IO_APIC();
> + clear_IO_APIC();
> +   restore_boot_irq_mode();
>  #endif
> }
>
>

Let me give a very concrete suggestion:
Patch 1) Replace "disable_IO_APIC();" with "clear_IO_APIC(); 
restore_boot_irq_mode();"
Patch 2) Move restore_boot_irq_mode(); to fix the regression.

I think that will be a slightly shorter patch sequence than what you are
dealing with and one that is slightly easier to read.

We need to sort out KEXEC_JUMP but that is something for another time.

Eric

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Eric W. Biederman]

Baoquan He  writes:

> Hi Eric,
>
> On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
>> Baoquan He  writes:
>> 
>> > This is a regression fix.
>> >
>> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
>> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
>> > calling after disable_IO_APIC(). This introdued a regression. The
>> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
>> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
>> > after disable_IO_APIC() will disable LAPIC and ruin the possible
>> > virtual wire mode setting which the code has been trying to do all
>> > along.
>> > To fix this, just break down disable_IO_APIC(), then call
>> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
>> > and call restore_boot_irq_mode() to restore boot irq mode before
>> > reboot or kexec/kdump jump.
>> 
>> Two things here.
>> a) This is missing a fixes tag and a CC stable.
>> b) What makes your change to the KEXEC_JUMP code path safe?
>>Have the lapic and ioapic already been shut down?
>> 
>> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
>> either need to be documented in the change long why they are safe
>> so that this change becomes obviously safe and correct.
>
> Re-read the code, I have to admit I didn't check the KEXEC_JUMP code
> path carefully. 
>
> kernel_kexec() {
>   if (kexec_image->preserve_context) {
>   ...
>   freeze_processes();
>   ...
>   disable_nonboot_cpus();
>   ...
>   
>   else {
>   ...
>   machine_shutdown();
>   ...
>   }
>   machine_kexec(kexec_image);
>   ...
> }
>
>   --machine_shutdown()
> --native_machine_shutdown()
>   --disable_IO_APIC()
>   --lapic_shutdown()
>
> machine_kexec() {
>   ...
>   if (image->preserve_context) {
>   disable_IO_APIC();
>   }
>   ...
> }
>
> KEXEC_JUMP code path is different than kexec/kdump, it doesn't call
> lapic_shutdown() before jump. So commit 522e66464467
> ("x86/apic: Disable I/O APIC before shutdown of the local APIC") didn't
> impact it. And here I break down disable_IO_APIC() and change to only
> call restore_boot_irq_mode() to make a possible danger. I am not an
> expert on KEXEC_JUMP, and don't know how to test it, so will keep the
> code implementation consistent as before. For now, I plan to change it
> as below if you don't object. As you pointed out, I will describe this
> in patch log. 
>
> diff --git a/arch/x86/kernel/machine_kexec_64.c 
> b/arch/x86/kernel/machine_kexec_64.c
> index 1f790cf9d38f..cb0c2d0a4c99 100644
> --- a/arch/x86/kernel/machine_kexec_64.c
> +++ b/arch/x86/kernel/machine_kexec_64.c
> @@ -297,7 +297,7 @@ void machine_kexec(struct kimage *image)
>  * one form or other. kexec jump path also need
>  * one.
>  */
> -   disable_IO_APIC();
> + clear_IO_APIC();
> +   restore_boot_irq_mode();
>  #endif
> }
>
>

Let me give a very concrete suggestion:
Patch 1) Replace "disable_IO_APIC();" with "clear_IO_APIC(); 
restore_boot_irq_mode();"
Patch 2) Move restore_boot_irq_mode(); to fix the regression.

I think that will be a slightly shorter patch sequence than what you are
dealing with and one that is slightly easier to read.

We need to sort out KEXEC_JUMP but that is something for another time.

Eric

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Eric W. Biederman]

Dou Liyang  writes:

> Hi Baoquan,
>
> At 02/12/2018 11:08 AM, Eric W. Biederman wrote:
>> Baoquan He  writes:
>>
>>> This is a regression fix.
>>>
>>> Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
>>> I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
>>> calling after disable_IO_APIC(). This introdued a regression. The
>>> root cause is that disable_IO_APIC() not only clears IO_APIC, also
>>> restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
>>> after disable_IO_APIC() will disable LAPIC and ruin the possible
>>> virtual wire mode setting which the code has been trying to do all
>>> along.
>>>
>>> The consequence is, in KVM guest kernel always prints warning as below
>>> during kexec/kdump kernel boots up. That happened in setup_local_APIC()
>>> since 'do { xxx } while (queued && max_loops > 0)' loop does not function
>
> I am not sure another thing here
>
> AFAIK, according to the order of SMP machine shutdown, other CPUs will
> be stopped firstly, then the last CPU disable its local apic.
>
>  --machine_shutdown
>|..
>|stop_other_cpus()
>|local_shutdown()
>
> So, the pending interrupts exist only in BSP and only be ACKed by
> BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
> all CPUs, found only BSP had the non-zero value).

We don't know.  In the kexec on panic case we try and shutdown the other
cpus but we have a timeout because we might fail.

Further you have to be careful with the concept of boot cpu.  In the
normal kexec case shutdown on the boot cpu and leave it running.  In the
kexec on panic case we shutdown on an arbitrary cpu.

> If it is right, We will do not need check the pending interrupt for each
> cpus.

It is also cheap if there are no pending interrupts as there is nothing
to do.

> BTW, the pending interrupt check code is mixed with the local
> APIC setup code, that looks messy. How about extracting the code which
> related to the crash interrupt check, put it into a new function and
> only invoked it when the CPU is BSP?

Moving it into it's own function makes sense.  Let's not taint the
concept with ``crash''.  We don't know that the only way this will
ever happen is from kexec on panic.   We only know it was easy to
trigger the condition from kexec on panic.

There are a lot of cases I can think of that interrupts might fire while
interrupts are disabled because the kernel is booting.  A normal kexec
is also possible.

Throw in MSI interrupts and transitioning from one state to another in
non-legacy apic mode and we might be more likely to get some irqs in
a pending state.

Your patch makes me nervous as it is not just code motion but a much
more substantial change.


As much as I agree that we need to fix the regression in the apic
shutdown code that is causing problems.  What we really need to do
is to completely remove the apic shutdown code from the kexec on panic
code path.  Over the long term that will provide us with a much more
robust path to getting crash dumps and the like.


Eric


> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
> index 3fc259b4dd2d..0350528b320d 100644
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -1408,6 +1408,57 @@ static void lapic_setup_esr(void)
> oldvalue, value);
>  }
>
> +
> +static void clear_crash_pending_intr(void)
> +{
> +   long long max_loops = cpu_khz ? cpu_khz : 100;
> +   unsigned long long tsc = 0, ntsc;
> +   unsigned int value, queued;
> +   int i, j, acked = 0;
> +
> +   if (boot_cpu_has(X86_FEATURE_TSC))
> +   tsc = rdtsc();
> +   /*
> +* After a crash, we no longer service the interrupts and a pending
> +* interrupt from previous kernel might still have ISR bit set.
> +*
> +* Most probably by now CPU has serviced that pending interrupt and
> +* it might not have done the ack_APIC_irq() because it thought,
> +* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
> +* does not clear the ISR bit and cpu thinks it has already serivced
> +* the interrupt. Hence a vector might get locked. It was noticed
> +* for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
> +*/
> +   do {
> +   queued = 0;
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--)
> +   queued |= apic_read(APIC_IRR + i*0x10);
> +
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--) {
> +   value = apic_read(APIC_ISR + i*0x10);
> +   for (j = 31; j >= 0; j--) {
> +   if (value & (1< +   ack_APIC_irq();
> +   acked++;
> +   }
> +   }
> +   }
> + 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Eric W. Biederman]

Dou Liyang  writes:

> Hi Baoquan,
>
> At 02/12/2018 11:08 AM, Eric W. Biederman wrote:
>> Baoquan He  writes:
>>
>>> This is a regression fix.
>>>
>>> Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
>>> I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
>>> calling after disable_IO_APIC(). This introdued a regression. The
>>> root cause is that disable_IO_APIC() not only clears IO_APIC, also
>>> restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
>>> after disable_IO_APIC() will disable LAPIC and ruin the possible
>>> virtual wire mode setting which the code has been trying to do all
>>> along.
>>>
>>> The consequence is, in KVM guest kernel always prints warning as below
>>> during kexec/kdump kernel boots up. That happened in setup_local_APIC()
>>> since 'do { xxx } while (queued && max_loops > 0)' loop does not function
>
> I am not sure another thing here
>
> AFAIK, according to the order of SMP machine shutdown, other CPUs will
> be stopped firstly, then the last CPU disable its local apic.
>
>  --machine_shutdown
>|..
>|stop_other_cpus()
>|local_shutdown()
>
> So, the pending interrupts exist only in BSP and only be ACKed by
> BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
> all CPUs, found only BSP had the non-zero value).

We don't know.  In the kexec on panic case we try and shutdown the other
cpus but we have a timeout because we might fail.

Further you have to be careful with the concept of boot cpu.  In the
normal kexec case shutdown on the boot cpu and leave it running.  In the
kexec on panic case we shutdown on an arbitrary cpu.

> If it is right, We will do not need check the pending interrupt for each
> cpus.

It is also cheap if there are no pending interrupts as there is nothing
to do.

> BTW, the pending interrupt check code is mixed with the local
> APIC setup code, that looks messy. How about extracting the code which
> related to the crash interrupt check, put it into a new function and
> only invoked it when the CPU is BSP?

Moving it into it's own function makes sense.  Let's not taint the
concept with ``crash''.  We don't know that the only way this will
ever happen is from kexec on panic.   We only know it was easy to
trigger the condition from kexec on panic.

There are a lot of cases I can think of that interrupts might fire while
interrupts are disabled because the kernel is booting.  A normal kexec
is also possible.

Throw in MSI interrupts and transitioning from one state to another in
non-legacy apic mode and we might be more likely to get some irqs in
a pending state.

Your patch makes me nervous as it is not just code motion but a much
more substantial change.


As much as I agree that we need to fix the regression in the apic
shutdown code that is causing problems.  What we really need to do
is to completely remove the apic shutdown code from the kexec on panic
code path.  Over the long term that will provide us with a much more
robust path to getting crash dumps and the like.


Eric


> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
> index 3fc259b4dd2d..0350528b320d 100644
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -1408,6 +1408,57 @@ static void lapic_setup_esr(void)
> oldvalue, value);
>  }
>
> +
> +static void clear_crash_pending_intr(void)
> +{
> +   long long max_loops = cpu_khz ? cpu_khz : 100;
> +   unsigned long long tsc = 0, ntsc;
> +   unsigned int value, queued;
> +   int i, j, acked = 0;
> +
> +   if (boot_cpu_has(X86_FEATURE_TSC))
> +   tsc = rdtsc();
> +   /*
> +* After a crash, we no longer service the interrupts and a pending
> +* interrupt from previous kernel might still have ISR bit set.
> +*
> +* Most probably by now CPU has serviced that pending interrupt and
> +* it might not have done the ack_APIC_irq() because it thought,
> +* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
> +* does not clear the ISR bit and cpu thinks it has already serivced
> +* the interrupt. Hence a vector might get locked. It was noticed
> +* for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
> +*/
> +   do {
> +   queued = 0;
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--)
> +   queued |= apic_read(APIC_IRR + i*0x10);
> +
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--) {
> +   value = apic_read(APIC_ISR + i*0x10);
> +   for (j = 31; j >= 0; j--) {
> +   if (value & (1< +   ack_APIC_irq();
> +   acked++;
> +   }
> +   }
> +   }
> +   if (acked > 256) {
> +   

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

Hi Eric,

On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
> Baoquan He  writes:
> 
> > This is a regression fix.
> >
> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> > calling after disable_IO_APIC(). This introdued a regression. The
> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> > after disable_IO_APIC() will disable LAPIC and ruin the possible
> > virtual wire mode setting which the code has been trying to do all
> > along.
> > To fix this, just break down disable_IO_APIC(), then call
> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> > and call restore_boot_irq_mode() to restore boot irq mode before
> > reboot or kexec/kdump jump.
> 
> Two things here.
> a) This is missing a fixes tag and a CC stable.
> b) What makes your change to the KEXEC_JUMP code path safe?
>Have the lapic and ioapic already been shut down?
> 
> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
> either need to be documented in the change long why they are safe
> so that this change becomes obviously safe and correct.

Re-read the code, I have to admit I didn't check the KEXEC_JUMP code
path carefully. 

kernel_kexec() {
if (kexec_image->preserve_context) {
...
freeze_processes();
...
disable_nonboot_cpus();
...

else {
...
machine_shutdown();
...
}
machine_kexec(kexec_image);
...
}

  --machine_shutdown()
--native_machine_shutdown()
  --disable_IO_APIC()
  --lapic_shutdown()

machine_kexec() {
...
if (image->preserve_context) {
disable_IO_APIC();
}
...
}

KEXEC_JUMP code path is different than kexec/kdump, it doesn't call
lapic_shutdown() before jump. So commit 522e66464467
("x86/apic: Disable I/O APIC before shutdown of the local APIC") didn't
impact it. And here I break down disable_IO_APIC() and change to only
call restore_boot_irq_mode() to make a possible danger. I am not an
expert on KEXEC_JUMP, and don't know how to test it, so will keep the
code implementation consistent as before. For now, I plan to change it
as below if you don't object. As you pointed out, I will describe this
in patch log. 

diff --git a/arch/x86/kernel/machine_kexec_64.c 
b/arch/x86/kernel/machine_kexec_64.c
index 1f790cf9d38f..cb0c2d0a4c99 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -297,7 +297,7 @@ void machine_kexec(struct kimage *image)
 * one form or other. kexec jump path also need
 * one.
 */
-   disable_IO_APIC();
+   clear_IO_APIC();
+   restore_boot_irq_mode();
 #endif
}




> 
> Otherwise we risk and trivial and obvious looking change causing another
> regression like changing the order of lapic_shutdown and disable_IOAPIC
> did.
> 
> Eric
> 
> 
> >
> > Signed-off-by: Baoquan He 
> > ---
> >  arch/x86/include/asm/io_apic.h | 1 +
> >  arch/x86/kernel/apic/io_apic.c | 2 +-
> >  arch/x86/kernel/crash.c| 3 ++-
> >  arch/x86/kernel/machine_kexec_32.c | 2 +-
> >  arch/x86/kernel/machine_kexec_64.c | 2 +-
> >  arch/x86/kernel/reboot.c   | 3 ++-
> >  6 files changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> > index 558d1a6a13ad..0fa95bfacb39 100644
> > --- a/arch/x86/include/asm/io_apic.h
> > +++ b/arch/x86/include/asm/io_apic.h
> > @@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int 
> > apic, unsigned int reg)
> >  extern void setup_IO_APIC(void);
> >  extern void enable_IO_APIC(void);
> >  extern void disable_IO_APIC(void);
> > +extern void clear_IO_APIC(void);
> >  extern void restore_boot_irq_mode(void);
> >  extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
> >  extern void print_IO_APICs(void);
> > diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> > index 7b73b6b9b4b6..2d7cd2db77f5 100644
> > --- a/arch/x86/kernel/apic/io_apic.c
> > +++ b/arch/x86/kernel/apic/io_apic.c
> > @@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, 
> > unsigned int pin)
> >mpc_ioapic_id(apic), pin);
> >  }
> >  
> > -static void clear_IO_APIC (void)
> > +void clear_IO_APIC (void)
> >  {
> > int apic, pin;
> >  
> > diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> > index 10e74d4778a1..1f6680427ff0 100644
> > --- a/arch/x86/kernel/crash.c
> > +++ b/arch/x86/kernel/crash.c
> > @@ -199,9 +199,10 @@ void native_machine_crash_shutdown(struct pt_regs 
> > *regs)
> >  #ifdef CONFIG_X86_IO_APIC
> > 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

Hi Eric,

On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
> Baoquan He  writes:
> 
> > This is a regression fix.
> >
> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> > calling after disable_IO_APIC(). This introdued a regression. The
> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> > after disable_IO_APIC() will disable LAPIC and ruin the possible
> > virtual wire mode setting which the code has been trying to do all
> > along.
> > To fix this, just break down disable_IO_APIC(), then call
> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> > and call restore_boot_irq_mode() to restore boot irq mode before
> > reboot or kexec/kdump jump.
> 
> Two things here.
> a) This is missing a fixes tag and a CC stable.
> b) What makes your change to the KEXEC_JUMP code path safe?
>Have the lapic and ioapic already been shut down?
> 
> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
> either need to be documented in the change long why they are safe
> so that this change becomes obviously safe and correct.

Re-read the code, I have to admit I didn't check the KEXEC_JUMP code
path carefully. 

kernel_kexec() {
if (kexec_image->preserve_context) {
...
freeze_processes();
...
disable_nonboot_cpus();
...

else {
...
machine_shutdown();
...
}
machine_kexec(kexec_image);
...
}

  --machine_shutdown()
--native_machine_shutdown()
  --disable_IO_APIC()
  --lapic_shutdown()

machine_kexec() {
...
if (image->preserve_context) {
disable_IO_APIC();
}
...
}

KEXEC_JUMP code path is different than kexec/kdump, it doesn't call
lapic_shutdown() before jump. So commit 522e66464467
("x86/apic: Disable I/O APIC before shutdown of the local APIC") didn't
impact it. And here I break down disable_IO_APIC() and change to only
call restore_boot_irq_mode() to make a possible danger. I am not an
expert on KEXEC_JUMP, and don't know how to test it, so will keep the
code implementation consistent as before. For now, I plan to change it
as below if you don't object. As you pointed out, I will describe this
in patch log. 

diff --git a/arch/x86/kernel/machine_kexec_64.c 
b/arch/x86/kernel/machine_kexec_64.c
index 1f790cf9d38f..cb0c2d0a4c99 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -297,7 +297,7 @@ void machine_kexec(struct kimage *image)
 * one form or other. kexec jump path also need
 * one.
 */
-   disable_IO_APIC();
+   clear_IO_APIC();
+   restore_boot_irq_mode();
 #endif
}




> 
> Otherwise we risk and trivial and obvious looking change causing another
> regression like changing the order of lapic_shutdown and disable_IOAPIC
> did.
> 
> Eric
> 
> 
> >
> > Signed-off-by: Baoquan He 
> > ---
> >  arch/x86/include/asm/io_apic.h | 1 +
> >  arch/x86/kernel/apic/io_apic.c | 2 +-
> >  arch/x86/kernel/crash.c| 3 ++-
> >  arch/x86/kernel/machine_kexec_32.c | 2 +-
> >  arch/x86/kernel/machine_kexec_64.c | 2 +-
> >  arch/x86/kernel/reboot.c   | 3 ++-
> >  6 files changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> > index 558d1a6a13ad..0fa95bfacb39 100644
> > --- a/arch/x86/include/asm/io_apic.h
> > +++ b/arch/x86/include/asm/io_apic.h
> > @@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int 
> > apic, unsigned int reg)
> >  extern void setup_IO_APIC(void);
> >  extern void enable_IO_APIC(void);
> >  extern void disable_IO_APIC(void);
> > +extern void clear_IO_APIC(void);
> >  extern void restore_boot_irq_mode(void);
> >  extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
> >  extern void print_IO_APICs(void);
> > diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> > index 7b73b6b9b4b6..2d7cd2db77f5 100644
> > --- a/arch/x86/kernel/apic/io_apic.c
> > +++ b/arch/x86/kernel/apic/io_apic.c
> > @@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, 
> > unsigned int pin)
> >mpc_ioapic_id(apic), pin);
> >  }
> >  
> > -static void clear_IO_APIC (void)
> > +void clear_IO_APIC (void)
> >  {
> > int apic, pin;
> >  
> > diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> > index 10e74d4778a1..1f6680427ff0 100644
> > --- a/arch/x86/kernel/crash.c
> > +++ b/arch/x86/kernel/crash.c
> > @@ -199,9 +199,10 @@ void native_machine_crash_shutdown(struct pt_regs 
> > *regs)
> >  #ifdef CONFIG_X86_IO_APIC
> > /* Prevent crash_kexec() from 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

On 02/13/18 at 10:43am, Dou Liyang wrote:
> Hi Baoquan,
> 
> At 02/12/2018 11:08 AM, Eric W. Biederman wrote:
> > Baoquan He  writes:
> > 
> > > This is a regression fix.
> > > 
> > > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> > > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> > > calling after disable_IO_APIC(). This introdued a regression. The
> > > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> > > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> > > after disable_IO_APIC() will disable LAPIC and ruin the possible
> > > virtual wire mode setting which the code has been trying to do all
> > > along.
> > > 
> > > The consequence is, in KVM guest kernel always prints warning as below
> > > during kexec/kdump kernel boots up. That happened in setup_local_APIC()
> > > since 'do { xxx } while (queued && max_loops > 0)' loop does not function
> 
> I am not sure another thing here
> 
> AFAIK, according to the order of SMP machine shutdown, other CPUs will
> be stopped firstly, then the last CPU disable its local apic.
> 
>  --machine_shutdown
>|..
>|stop_other_cpus()
>|local_shutdown()
> 
> So, the pending interrupts exist only in BSP and only be ACKed by
> BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
> all CPUs, found only BSP had the non-zero value).
> 
> If it is right, We will do not need check the pending interrupt for each
> cpus.
> 
> BTW, the pending interrupt check code is mixed with the local
> APIC setup code, that looks messy. How about extracting the code which
> related to the crash interrupt check, put it into a new function and
> only invoked it when the CPU is BSP?

Yes, looks a good code cleaning up. Virtual wire irq mode only go
through the boot cpu's lapic.

In fact, or put in apic_bsp_setup() since it's only related to boot
cpu to make it look cleaner if can do.

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 3fc259b4dd2d..cec2d0692ace 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2380,6 +2380,7 @@ void __init apic_bsp_setup(bool upmode)
connect_bsp_APIC();
if (upmode)
apic_bsp_up_setup();
+   clear_crash_pending_intr();
setup_local_APIC();
 
enable_IO_APIC();

> 
> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
> index 3fc259b4dd2d..0350528b320d 100644
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -1408,6 +1408,57 @@ static void lapic_setup_esr(void)
> oldvalue, value);
>  }
> 
> +
> +static void clear_crash_pending_intr(void)
> +{
> +   long long max_loops = cpu_khz ? cpu_khz : 100;
> +   unsigned long long tsc = 0, ntsc;
> +   unsigned int value, queued;
> +   int i, j, acked = 0;
Better do a check here.

  if (disable_apic)
return;
> +
> +   if (boot_cpu_has(X86_FEATURE_TSC))
> +   tsc = rdtsc();
> +   /*
> +* After a crash, we no longer service the interrupts and a pending
> +* interrupt from previous kernel might still have ISR bit set.
> +*
> +* Most probably by now CPU has serviced that pending interrupt and
> +* it might not have done the ack_APIC_irq() because it thought,
> +* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
> +* does not clear the ISR bit and cpu thinks it has already serivced
> +* the interrupt. Hence a vector might get locked. It was noticed
> +* for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
> +*/
> +   do {
> +   queued = 0;
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--)
> +   queued |= apic_read(APIC_IRR + i*0x10);
> +
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--) {
> +   value = apic_read(APIC_ISR + i*0x10);
> +   for (j = 31; j >= 0; j--) {
> +   if (value & (1< +   ack_APIC_irq();
> +   acked++;
> +   }
> +   }
> +   }
> +   if (acked > 256) {
> +   printk(KERN_ERR "LAPIC pending interrupts after %d
> EOI\n",
> +  acked);
> +   break;
> +   }
> +   if (queued) {
> +   if (boot_cpu_has(X86_FEATURE_TSC) && cpu_khz) {
> +   ntsc = rdtsc();
> +   max_loops = (cpu_khz << 10) - (ntsc - tsc);
> +   } else
> +   max_loops--;
> +   }
> +   } while (queued && max_loops > 0);
> +   WARN_ON(max_loops <= 0);
> +}
> +
>  /**
>   * 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

On 02/13/18 at 10:43am, Dou Liyang wrote:
> Hi Baoquan,
> 
> At 02/12/2018 11:08 AM, Eric W. Biederman wrote:
> > Baoquan He  writes:
> > 
> > > This is a regression fix.
> > > 
> > > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> > > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> > > calling after disable_IO_APIC(). This introdued a regression. The
> > > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> > > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> > > after disable_IO_APIC() will disable LAPIC and ruin the possible
> > > virtual wire mode setting which the code has been trying to do all
> > > along.
> > > 
> > > The consequence is, in KVM guest kernel always prints warning as below
> > > during kexec/kdump kernel boots up. That happened in setup_local_APIC()
> > > since 'do { xxx } while (queued && max_loops > 0)' loop does not function
> 
> I am not sure another thing here
> 
> AFAIK, according to the order of SMP machine shutdown, other CPUs will
> be stopped firstly, then the last CPU disable its local apic.
> 
>  --machine_shutdown
>|..
>|stop_other_cpus()
>|local_shutdown()
> 
> So, the pending interrupts exist only in BSP and only be ACKed by
> BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
> all CPUs, found only BSP had the non-zero value).
> 
> If it is right, We will do not need check the pending interrupt for each
> cpus.
> 
> BTW, the pending interrupt check code is mixed with the local
> APIC setup code, that looks messy. How about extracting the code which
> related to the crash interrupt check, put it into a new function and
> only invoked it when the CPU is BSP?

Yes, looks a good code cleaning up. Virtual wire irq mode only go
through the boot cpu's lapic.

In fact, or put in apic_bsp_setup() since it's only related to boot
cpu to make it look cleaner if can do.

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 3fc259b4dd2d..cec2d0692ace 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2380,6 +2380,7 @@ void __init apic_bsp_setup(bool upmode)
connect_bsp_APIC();
if (upmode)
apic_bsp_up_setup();
+   clear_crash_pending_intr();
setup_local_APIC();
 
enable_IO_APIC();

> 
> diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
> index 3fc259b4dd2d..0350528b320d 100644
> --- a/arch/x86/kernel/apic/apic.c
> +++ b/arch/x86/kernel/apic/apic.c
> @@ -1408,6 +1408,57 @@ static void lapic_setup_esr(void)
> oldvalue, value);
>  }
> 
> +
> +static void clear_crash_pending_intr(void)
> +{
> +   long long max_loops = cpu_khz ? cpu_khz : 100;
> +   unsigned long long tsc = 0, ntsc;
> +   unsigned int value, queued;
> +   int i, j, acked = 0;
Better do a check here.

  if (disable_apic)
return;
> +
> +   if (boot_cpu_has(X86_FEATURE_TSC))
> +   tsc = rdtsc();
> +   /*
> +* After a crash, we no longer service the interrupts and a pending
> +* interrupt from previous kernel might still have ISR bit set.
> +*
> +* Most probably by now CPU has serviced that pending interrupt and
> +* it might not have done the ack_APIC_irq() because it thought,
> +* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
> +* does not clear the ISR bit and cpu thinks it has already serivced
> +* the interrupt. Hence a vector might get locked. It was noticed
> +* for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
> +*/
> +   do {
> +   queued = 0;
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--)
> +   queued |= apic_read(APIC_IRR + i*0x10);
> +
> +   for (i = APIC_ISR_NR - 1; i >= 0; i--) {
> +   value = apic_read(APIC_ISR + i*0x10);
> +   for (j = 31; j >= 0; j--) {
> +   if (value & (1< +   ack_APIC_irq();
> +   acked++;
> +   }
> +   }
> +   }
> +   if (acked > 256) {
> +   printk(KERN_ERR "LAPIC pending interrupts after %d
> EOI\n",
> +  acked);
> +   break;
> +   }
> +   if (queued) {
> +   if (boot_cpu_has(X86_FEATURE_TSC) && cpu_khz) {
> +   ntsc = rdtsc();
> +   max_loops = (cpu_khz << 10) - (ntsc - tsc);
> +   } else
> +   max_loops--;
> +   }
> +   } while (queued && max_loops > 0);
> +   WARN_ON(max_loops <= 0);
> +}
> +
>  /**
>   * setup_local_APIC - setup the 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Dou Liyang]

Hi Baoquan,

At 02/12/2018 11:08 AM, Eric W. Biederman wrote:

Baoquan He  writes:


This is a regression fix.

Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
calling after disable_IO_APIC(). This introdued a regression. The
root cause is that disable_IO_APIC() not only clears IO_APIC, also
restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
after disable_IO_APIC() will disable LAPIC and ruin the possible
virtual wire mode setting which the code has been trying to do all
along.

The consequence is, in KVM guest kernel always prints warning as below
during kexec/kdump kernel boots up. That happened in setup_local_APIC()
since 'do { xxx } while (queued && max_loops > 0)' loop does not function


I am not sure another thing here

AFAIK, according to the order of SMP machine shutdown, other CPUs will
be stopped firstly, then the last CPU disable its local apic.

 --machine_shutdown
   |..
   |stop_other_cpus()
   |local_shutdown()

So, the pending interrupts exist only in BSP and only be ACKed by
BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
all CPUs, found only BSP had the non-zero value).

If it is right, We will do not need check the pending interrupt for each
cpus.

BTW, the pending interrupt check code is mixed with the local
APIC setup code, that looks messy. How about extracting the code which
related to the crash interrupt check, put it into a new function and
only invoked it when the CPU is BSP?

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 3fc259b4dd2d..0350528b320d 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1408,6 +1408,57 @@ static void lapic_setup_esr(void)
oldvalue, value);
 }

+
+static void clear_crash_pending_intr(void)
+{
+   long long max_loops = cpu_khz ? cpu_khz : 100;
+   unsigned long long tsc = 0, ntsc;
+   unsigned int value, queued;
+   int i, j, acked = 0;
+
+   if (boot_cpu_has(X86_FEATURE_TSC))
+   tsc = rdtsc();
+   /*
+* After a crash, we no longer service the interrupts and a pending
+* interrupt from previous kernel might still have ISR bit set.
+*
+* Most probably by now CPU has serviced that pending interrupt and
+* it might not have done the ack_APIC_irq() because it thought,
+* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
+* does not clear the ISR bit and cpu thinks it has already serivced
+* the interrupt. Hence a vector might get locked. It was noticed
+* for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
+*/
+   do {
+   queued = 0;
+   for (i = APIC_ISR_NR - 1; i >= 0; i--)
+   queued |= apic_read(APIC_IRR + i*0x10);
+
+   for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+   value = apic_read(APIC_ISR + i*0x10);
+   for (j = 31; j >= 0; j--) {
+   if (value & (1< 256) {
+   printk(KERN_ERR "LAPIC pending interrupts after 
%d EOI\n",

+  acked);
+   break;
+   }
+   if (queued) {
+   if (boot_cpu_has(X86_FEATURE_TSC) && cpu_khz) {
+   ntsc = rdtsc();
+   max_loops = (cpu_khz << 10) - (ntsc - tsc);
+   } else
+   max_loops--;
+   }
+   } while (queued && max_loops > 0);
+   WARN_ON(max_loops <= 0);
+}
+
 /**
  * setup_local_APIC - setup the local APIC
  *
@@ -1417,13 +1468,7 @@ static void lapic_setup_esr(void)
 void setup_local_APIC(void)
 {
int cpu = smp_processor_id();
-   unsigned int value, queued;
-   int i, j, acked = 0;
-   unsigned long long tsc = 0, ntsc;
-   long long max_loops = cpu_khz ? cpu_khz : 100;
-
-   if (boot_cpu_has(X86_FEATURE_TSC))
-   tsc = rdtsc();
-   tsc = rdtsc();
+   unsigned int value;

if (disable_apic) {
disable_ioapic_support();
@@ -1475,45 +1520,8 @@ void setup_local_APIC(void)
value &= ~APIC_TPRI_MASK;
apic_write(APIC_TASKPRI, value);

-   /*
-* After a crash, we no longer service the interrupts and a pending
-* interrupt from previous kernel might still have ISR bit set.
-*
-* Most probably by now CPU has serviced that pending interrupt and
-* it might not have done the ack_APIC_irq() because it thought,
-* interrupt came 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Dou Liyang]

Hi Baoquan,

At 02/12/2018 11:08 AM, Eric W. Biederman wrote:

Baoquan He  writes:


This is a regression fix.

Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
calling after disable_IO_APIC(). This introdued a regression. The
root cause is that disable_IO_APIC() not only clears IO_APIC, also
restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
after disable_IO_APIC() will disable LAPIC and ruin the possible
virtual wire mode setting which the code has been trying to do all
along.

The consequence is, in KVM guest kernel always prints warning as below
during kexec/kdump kernel boots up. That happened in setup_local_APIC()
since 'do { xxx } while (queued && max_loops > 0)' loop does not function


I am not sure another thing here

AFAIK, according to the order of SMP machine shutdown, other CPUs will
be stopped firstly, then the last CPU disable its local apic.

 --machine_shutdown
   |..
   |stop_other_cpus()
   |local_shutdown()

So, the pending interrupts exist only in BSP and only be ACKed by
BSP. is it right?(I validated this by print the value of APIC_IRR/ISR of
all CPUs, found only BSP had the non-zero value).

If it is right, We will do not need check the pending interrupt for each
cpus.

BTW, the pending interrupt check code is mixed with the local
APIC setup code, that looks messy. How about extracting the code which
related to the crash interrupt check, put it into a new function and
only invoked it when the CPU is BSP?

diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 3fc259b4dd2d..0350528b320d 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -1408,6 +1408,57 @@ static void lapic_setup_esr(void)
oldvalue, value);
 }

+
+static void clear_crash_pending_intr(void)
+{
+   long long max_loops = cpu_khz ? cpu_khz : 100;
+   unsigned long long tsc = 0, ntsc;
+   unsigned int value, queued;
+   int i, j, acked = 0;
+
+   if (boot_cpu_has(X86_FEATURE_TSC))
+   tsc = rdtsc();
+   /*
+* After a crash, we no longer service the interrupts and a pending
+* interrupt from previous kernel might still have ISR bit set.
+*
+* Most probably by now CPU has serviced that pending interrupt and
+* it might not have done the ack_APIC_irq() because it thought,
+* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
+* does not clear the ISR bit and cpu thinks it has already serivced
+* the interrupt. Hence a vector might get locked. It was noticed
+* for timer irq (vector 0x31). Issue an extra EOI to clear ISR.
+*/
+   do {
+   queued = 0;
+   for (i = APIC_ISR_NR - 1; i >= 0; i--)
+   queued |= apic_read(APIC_IRR + i*0x10);
+
+   for (i = APIC_ISR_NR - 1; i >= 0; i--) {
+   value = apic_read(APIC_ISR + i*0x10);
+   for (j = 31; j >= 0; j--) {
+   if (value & (1< 256) {
+   printk(KERN_ERR "LAPIC pending interrupts after 
%d EOI\n",

+  acked);
+   break;
+   }
+   if (queued) {
+   if (boot_cpu_has(X86_FEATURE_TSC) && cpu_khz) {
+   ntsc = rdtsc();
+   max_loops = (cpu_khz << 10) - (ntsc - tsc);
+   } else
+   max_loops--;
+   }
+   } while (queued && max_loops > 0);
+   WARN_ON(max_loops <= 0);
+}
+
 /**
  * setup_local_APIC - setup the local APIC
  *
@@ -1417,13 +1468,7 @@ static void lapic_setup_esr(void)
 void setup_local_APIC(void)
 {
int cpu = smp_processor_id();
-   unsigned int value, queued;
-   int i, j, acked = 0;
-   unsigned long long tsc = 0, ntsc;
-   long long max_loops = cpu_khz ? cpu_khz : 100;
-
-   if (boot_cpu_has(X86_FEATURE_TSC))
-   tsc = rdtsc();
-   tsc = rdtsc();
+   unsigned int value;

if (disable_apic) {
disable_ioapic_support();
@@ -1475,45 +1520,8 @@ void setup_local_APIC(void)
value &= ~APIC_TPRI_MASK;
apic_write(APIC_TASKPRI, value);

-   /*
-* After a crash, we no longer service the interrupts and a pending
-* interrupt from previous kernel might still have ISR bit set.
-*
-* Most probably by now CPU has serviced that pending interrupt and
-* it might not have done the ack_APIC_irq() because it thought,
-* interrupt came from i8259 as ExtInt. LAPIC did not get EOI so it
-* does not clear the ISR bit and cpu thinks it has already serivced
-* the interrupt. Hence a vector might get locked. It was noticed
-* for timer irq (vector 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
> Baoquan He  writes:
> 
> > This is a regression fix.
> >
> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> > calling after disable_IO_APIC(). This introdued a regression. The
> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> > after disable_IO_APIC() will disable LAPIC and ruin the possible
> > virtual wire mode setting which the code has been trying to do all
> > along.
> >
> > The consequence is, in KVM guest kernel always prints warning as below
> > during kexec/kdump kernel boots up. That happened in setup_local_APIC()
> > since 'do { xxx } while (queued && max_loops > 0)' loop does not function
> > well any more if pending irq exists in APIC IRR after LAPIC is disabled.
> > And people even saw casual kdump kernel hang once in ~30 attempts during
> > stress testing of kdump on KVM machine.
> >
> > [0.001000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1467 
> > setup_local_APIC+0x228/0x330
> > [0.001000] Modules linked in:
> > [0.001000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc5+ #3
> > [0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
> > 1.10.2-1.fc26 04/01/2014
> > [0.001000] RIP: 0010:setup_local_APIC+0x228/0x330
> > [0.001000] RSP: :b6e03eb8 EFLAGS: 00010286
> > [0.001000] RAX: 009edb4c4d84 RBX:  RCX: 
> > b099d800
> > [0.001000] RDX: 009e RSI:  RDI: 
> > 0810
> > [0.001000] RBP:  R08:  R09: 
> > 0001
> > [0.001000] R10: 98ce6a801c00 R11: 0761076d072f0776 R12: 
> > 0001
> > [0.001000] R13: 00f0 R14: 4000 R15: 
> > c6ff
> > [0.001000] FS:  () GS:98ce6bc0() 
> > knlGS:
> > [0.001000] CS:  0010 DS:  ES:  CR0: 80050033
> > [0.001000] CR2:  CR3: 22209000 CR4: 
> > 000406b0
> > [0.001000] Call Trace:
> > [0.001000]  apic_bsp_setup+0x56/0x74
> > [0.001000]  x86_late_time_init+0x11/0x16
> > [0.001000]  start_kernel+0x3c9/0x486
> > [0.001000]  secondary_startup_64+0xa5/0xb0
> > [0.001000] Code: 00 85 c9 74 2d 0f 31 c1 e1 0a 48 c1 e2 20 41 89 cf 4c 
> > 03 7c 24 08 48 09 d0 49 29 c7 4c 89 3c 24 48 83 3c 24 00 0f 8f 8f fe ff
> > ff <0f> ff e9 10 ff ff ff 48 83 2c 24 01 eb e7 48 83 c4 18 5b 5d 41
> > [0.001000] ---[ end trace b88e71b9a6ebebdd ]---
> > [0.001000] masked ExtINT on CPU#0
> >
> > To fix this, just break down disable_IO_APIC(), then call
> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> > and call restore_boot_irq_mode() to restore boot irq mode before
> > reboot or kexec/kdump jump.
> 
> Two things here.
> a) This is missing a fixes tag and a CC stable.
> b) What makes your change to the KEXEC_JUMP code path safe?
>Have the lapic and ioapic already been shut down?
> 
> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
> either need to be documented in the change long why they are safe
> so that this change becomes obviously safe and correct.
> 
> Otherwise we risk and trivial and obvious looking change causing another
> regression like changing the order of lapic_shutdown and disable_IOAPIC
> did.

Makes sense. Will change as you suggested and repost. Thanks a lot.

Thanks
Baoquan

> > ---
> >  arch/x86/include/asm/io_apic.h | 1 +
> >  arch/x86/kernel/apic/io_apic.c | 2 +-
> >  arch/x86/kernel/crash.c| 3 ++-
> >  arch/x86/kernel/machine_kexec_32.c | 2 +-
> >  arch/x86/kernel/machine_kexec_64.c | 2 +-
> >  arch/x86/kernel/reboot.c   | 3 ++-
> >  6 files changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> > index 558d1a6a13ad..0fa95bfacb39 100644
> > --- a/arch/x86/include/asm/io_apic.h
> > +++ b/arch/x86/include/asm/io_apic.h
> > @@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int 
> > apic, unsigned int reg)
> >  extern void setup_IO_APIC(void);
> >  extern void enable_IO_APIC(void);
> >  extern void disable_IO_APIC(void);
> > +extern void clear_IO_APIC(void);
> >  extern void restore_boot_irq_mode(void);
> >  extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
> >  extern void print_IO_APICs(void);
> > diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> > index 7b73b6b9b4b6..2d7cd2db77f5 100644
> > --- a/arch/x86/kernel/apic/io_apic.c
> > +++ b/arch/x86/kernel/apic/io_apic.c
> > @@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, 
> > unsigned int pin)
> >mpc_ioapic_id(apic), pin);
> >  }
> >  
> > 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

On 02/11/18 at 09:08pm, Eric W. Biederman wrote:
> Baoquan He  writes:
> 
> > This is a regression fix.
> >
> > Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> > I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> > calling after disable_IO_APIC(). This introdued a regression. The
> > root cause is that disable_IO_APIC() not only clears IO_APIC, also
> > restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> > after disable_IO_APIC() will disable LAPIC and ruin the possible
> > virtual wire mode setting which the code has been trying to do all
> > along.
> >
> > The consequence is, in KVM guest kernel always prints warning as below
> > during kexec/kdump kernel boots up. That happened in setup_local_APIC()
> > since 'do { xxx } while (queued && max_loops > 0)' loop does not function
> > well any more if pending irq exists in APIC IRR after LAPIC is disabled.
> > And people even saw casual kdump kernel hang once in ~30 attempts during
> > stress testing of kdump on KVM machine.
> >
> > [0.001000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1467 
> > setup_local_APIC+0x228/0x330
> > [0.001000] Modules linked in:
> > [0.001000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc5+ #3
> > [0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
> > 1.10.2-1.fc26 04/01/2014
> > [0.001000] RIP: 0010:setup_local_APIC+0x228/0x330
> > [0.001000] RSP: :b6e03eb8 EFLAGS: 00010286
> > [0.001000] RAX: 009edb4c4d84 RBX:  RCX: 
> > b099d800
> > [0.001000] RDX: 009e RSI:  RDI: 
> > 0810
> > [0.001000] RBP:  R08:  R09: 
> > 0001
> > [0.001000] R10: 98ce6a801c00 R11: 0761076d072f0776 R12: 
> > 0001
> > [0.001000] R13: 00f0 R14: 4000 R15: 
> > c6ff
> > [0.001000] FS:  () GS:98ce6bc0() 
> > knlGS:
> > [0.001000] CS:  0010 DS:  ES:  CR0: 80050033
> > [0.001000] CR2:  CR3: 22209000 CR4: 
> > 000406b0
> > [0.001000] Call Trace:
> > [0.001000]  apic_bsp_setup+0x56/0x74
> > [0.001000]  x86_late_time_init+0x11/0x16
> > [0.001000]  start_kernel+0x3c9/0x486
> > [0.001000]  secondary_startup_64+0xa5/0xb0
> > [0.001000] Code: 00 85 c9 74 2d 0f 31 c1 e1 0a 48 c1 e2 20 41 89 cf 4c 
> > 03 7c 24 08 48 09 d0 49 29 c7 4c 89 3c 24 48 83 3c 24 00 0f 8f 8f fe ff
> > ff <0f> ff e9 10 ff ff ff 48 83 2c 24 01 eb e7 48 83 c4 18 5b 5d 41
> > [0.001000] ---[ end trace b88e71b9a6ebebdd ]---
> > [0.001000] masked ExtINT on CPU#0
> >
> > To fix this, just break down disable_IO_APIC(), then call
> > clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> > and call restore_boot_irq_mode() to restore boot irq mode before
> > reboot or kexec/kdump jump.
> 
> Two things here.
> a) This is missing a fixes tag and a CC stable.
> b) What makes your change to the KEXEC_JUMP code path safe?
>Have the lapic and ioapic already been shut down?
> 
> The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
> either need to be documented in the change long why they are safe
> so that this change becomes obviously safe and correct.
> 
> Otherwise we risk and trivial and obvious looking change causing another
> regression like changing the order of lapic_shutdown and disable_IOAPIC
> did.

Makes sense. Will change as you suggested and repost. Thanks a lot.

Thanks
Baoquan

> > ---
> >  arch/x86/include/asm/io_apic.h | 1 +
> >  arch/x86/kernel/apic/io_apic.c | 2 +-
> >  arch/x86/kernel/crash.c| 3 ++-
> >  arch/x86/kernel/machine_kexec_32.c | 2 +-
> >  arch/x86/kernel/machine_kexec_64.c | 2 +-
> >  arch/x86/kernel/reboot.c   | 3 ++-
> >  6 files changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> > index 558d1a6a13ad..0fa95bfacb39 100644
> > --- a/arch/x86/include/asm/io_apic.h
> > +++ b/arch/x86/include/asm/io_apic.h
> > @@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int 
> > apic, unsigned int reg)
> >  extern void setup_IO_APIC(void);
> >  extern void enable_IO_APIC(void);
> >  extern void disable_IO_APIC(void);
> > +extern void clear_IO_APIC(void);
> >  extern void restore_boot_irq_mode(void);
> >  extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
> >  extern void print_IO_APICs(void);
> > diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> > index 7b73b6b9b4b6..2d7cd2db77f5 100644
> > --- a/arch/x86/kernel/apic/io_apic.c
> > +++ b/arch/x86/kernel/apic/io_apic.c
> > @@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, 
> > unsigned int pin)
> >mpc_ioapic_id(apic), pin);
> >  }
> >  
> > -static void 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Eric W. Biederman]

Baoquan He  writes:

> This is a regression fix.
>
> Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> calling after disable_IO_APIC(). This introdued a regression. The
> root cause is that disable_IO_APIC() not only clears IO_APIC, also
> restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> after disable_IO_APIC() will disable LAPIC and ruin the possible
> virtual wire mode setting which the code has been trying to do all
> along.
>
> The consequence is, in KVM guest kernel always prints warning as below
> during kexec/kdump kernel boots up. That happened in setup_local_APIC()
> since 'do { xxx } while (queued && max_loops > 0)' loop does not function
> well any more if pending irq exists in APIC IRR after LAPIC is disabled.
> And people even saw casual kdump kernel hang once in ~30 attempts during
> stress testing of kdump on KVM machine.
>
> [0.001000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1467 
> setup_local_APIC+0x228/0x330
> [0.001000] Modules linked in:
> [0.001000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc5+ #3
> [0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
> 1.10.2-1.fc26 04/01/2014
> [0.001000] RIP: 0010:setup_local_APIC+0x228/0x330
> [0.001000] RSP: :b6e03eb8 EFLAGS: 00010286
> [0.001000] RAX: 009edb4c4d84 RBX:  RCX: 
> b099d800
> [0.001000] RDX: 009e RSI:  RDI: 
> 0810
> [0.001000] RBP:  R08:  R09: 
> 0001
> [0.001000] R10: 98ce6a801c00 R11: 0761076d072f0776 R12: 
> 0001
> [0.001000] R13: 00f0 R14: 4000 R15: 
> c6ff
> [0.001000] FS:  () GS:98ce6bc0() 
> knlGS:
> [0.001000] CS:  0010 DS:  ES:  CR0: 80050033
> [0.001000] CR2:  CR3: 22209000 CR4: 
> 000406b0
> [0.001000] Call Trace:
> [0.001000]  apic_bsp_setup+0x56/0x74
> [0.001000]  x86_late_time_init+0x11/0x16
> [0.001000]  start_kernel+0x3c9/0x486
> [0.001000]  secondary_startup_64+0xa5/0xb0
> [0.001000] Code: 00 85 c9 74 2d 0f 31 c1 e1 0a 48 c1 e2 20 41 89 cf 4c 03 
> 7c 24 08 48 09 d0 49 29 c7 4c 89 3c 24 48 83 3c 24 00 0f 8f 8f fe ff
> ff <0f> ff e9 10 ff ff ff 48 83 2c 24 01 eb e7 48 83 c4 18 5b 5d 41
> [0.001000] ---[ end trace b88e71b9a6ebebdd ]---
> [0.001000] masked ExtINT on CPU#0
>
> To fix this, just break down disable_IO_APIC(), then call
> clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> and call restore_boot_irq_mode() to restore boot irq mode before
> reboot or kexec/kdump jump.

Two things here.
a) This is missing a fixes tag and a CC stable.
b) What makes your change to the KEXEC_JUMP code path safe?
   Have the lapic and ioapic already been shut down?

The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
either need to be documented in the change long why they are safe
so that this change becomes obviously safe and correct.

Otherwise we risk and trivial and obvious looking change causing another
regression like changing the order of lapic_shutdown and disable_IOAPIC
did.

Eric


>
> Signed-off-by: Baoquan He 
> ---
>  arch/x86/include/asm/io_apic.h | 1 +
>  arch/x86/kernel/apic/io_apic.c | 2 +-
>  arch/x86/kernel/crash.c| 3 ++-
>  arch/x86/kernel/machine_kexec_32.c | 2 +-
>  arch/x86/kernel/machine_kexec_64.c | 2 +-
>  arch/x86/kernel/reboot.c   | 3 ++-
>  6 files changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> index 558d1a6a13ad..0fa95bfacb39 100644
> --- a/arch/x86/include/asm/io_apic.h
> +++ b/arch/x86/include/asm/io_apic.h
> @@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int 
> apic, unsigned int reg)
>  extern void setup_IO_APIC(void);
>  extern void enable_IO_APIC(void);
>  extern void disable_IO_APIC(void);
> +extern void clear_IO_APIC(void);
>  extern void restore_boot_irq_mode(void);
>  extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
>  extern void print_IO_APICs(void);
> diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> index 7b73b6b9b4b6..2d7cd2db77f5 100644
> --- a/arch/x86/kernel/apic/io_apic.c
> +++ b/arch/x86/kernel/apic/io_apic.c
> @@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, unsigned 
> int pin)
>  mpc_ioapic_id(apic), pin);
>  }
>  
> -static void clear_IO_APIC (void)
> +void clear_IO_APIC (void)
>  {
>   int apic, pin;
>  
> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> index 10e74d4778a1..1f6680427ff0 100644
> --- a/arch/x86/kernel/crash.c
> +++ b/arch/x86/kernel/crash.c
> @@ -199,9 

Re: [PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Eric W. Biederman]

Baoquan He  writes:

> This is a regression fix.
>
> Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
> I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
> calling after disable_IO_APIC(). This introdued a regression. The
> root cause is that disable_IO_APIC() not only clears IO_APIC, also
> restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
> after disable_IO_APIC() will disable LAPIC and ruin the possible
> virtual wire mode setting which the code has been trying to do all
> along.
>
> The consequence is, in KVM guest kernel always prints warning as below
> during kexec/kdump kernel boots up. That happened in setup_local_APIC()
> since 'do { xxx } while (queued && max_loops > 0)' loop does not function
> well any more if pending irq exists in APIC IRR after LAPIC is disabled.
> And people even saw casual kdump kernel hang once in ~30 attempts during
> stress testing of kdump on KVM machine.
>
> [0.001000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1467 
> setup_local_APIC+0x228/0x330
> [0.001000] Modules linked in:
> [0.001000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc5+ #3
> [0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
> 1.10.2-1.fc26 04/01/2014
> [0.001000] RIP: 0010:setup_local_APIC+0x228/0x330
> [0.001000] RSP: :b6e03eb8 EFLAGS: 00010286
> [0.001000] RAX: 009edb4c4d84 RBX:  RCX: 
> b099d800
> [0.001000] RDX: 009e RSI:  RDI: 
> 0810
> [0.001000] RBP:  R08:  R09: 
> 0001
> [0.001000] R10: 98ce6a801c00 R11: 0761076d072f0776 R12: 
> 0001
> [0.001000] R13: 00f0 R14: 4000 R15: 
> c6ff
> [0.001000] FS:  () GS:98ce6bc0() 
> knlGS:
> [0.001000] CS:  0010 DS:  ES:  CR0: 80050033
> [0.001000] CR2:  CR3: 22209000 CR4: 
> 000406b0
> [0.001000] Call Trace:
> [0.001000]  apic_bsp_setup+0x56/0x74
> [0.001000]  x86_late_time_init+0x11/0x16
> [0.001000]  start_kernel+0x3c9/0x486
> [0.001000]  secondary_startup_64+0xa5/0xb0
> [0.001000] Code: 00 85 c9 74 2d 0f 31 c1 e1 0a 48 c1 e2 20 41 89 cf 4c 03 
> 7c 24 08 48 09 d0 49 29 c7 4c 89 3c 24 48 83 3c 24 00 0f 8f 8f fe ff
> ff <0f> ff e9 10 ff ff ff 48 83 2c 24 01 eb e7 48 83 c4 18 5b 5d 41
> [0.001000] ---[ end trace b88e71b9a6ebebdd ]---
> [0.001000] masked ExtINT on CPU#0
>
> To fix this, just break down disable_IO_APIC(), then call
> clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
> and call restore_boot_irq_mode() to restore boot irq mode before
> reboot or kexec/kdump jump.

Two things here.
a) This is missing a fixes tag and a CC stable.
b) What makes your change to the KEXEC_JUMP code path safe?
   Have the lapic and ioapic already been shut down?

The KEXEC_JUMP changes to machine_kexec_32.c and machine_kexec_64.c
either need to be documented in the change long why they are safe
so that this change becomes obviously safe and correct.

Otherwise we risk and trivial and obvious looking change causing another
regression like changing the order of lapic_shutdown and disable_IOAPIC
did.

Eric


>
> Signed-off-by: Baoquan He 
> ---
>  arch/x86/include/asm/io_apic.h | 1 +
>  arch/x86/kernel/apic/io_apic.c | 2 +-
>  arch/x86/kernel/crash.c| 3 ++-
>  arch/x86/kernel/machine_kexec_32.c | 2 +-
>  arch/x86/kernel/machine_kexec_64.c | 2 +-
>  arch/x86/kernel/reboot.c   | 3 ++-
>  6 files changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
> index 558d1a6a13ad..0fa95bfacb39 100644
> --- a/arch/x86/include/asm/io_apic.h
> +++ b/arch/x86/include/asm/io_apic.h
> @@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int 
> apic, unsigned int reg)
>  extern void setup_IO_APIC(void);
>  extern void enable_IO_APIC(void);
>  extern void disable_IO_APIC(void);
> +extern void clear_IO_APIC(void);
>  extern void restore_boot_irq_mode(void);
>  extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
>  extern void print_IO_APICs(void);
> diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
> index 7b73b6b9b4b6..2d7cd2db77f5 100644
> --- a/arch/x86/kernel/apic/io_apic.c
> +++ b/arch/x86/kernel/apic/io_apic.c
> @@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, unsigned 
> int pin)
>  mpc_ioapic_id(apic), pin);
>  }
>  
> -static void clear_IO_APIC (void)
> +void clear_IO_APIC (void)
>  {
>   int apic, pin;
>  
> diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
> index 10e74d4778a1..1f6680427ff0 100644
> --- a/arch/x86/kernel/crash.c
> +++ b/arch/x86/kernel/crash.c
> @@ -199,9 +199,10 @@ void 

[PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

This is a regression fix.

Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
calling after disable_IO_APIC(). This introdued a regression. The
root cause is that disable_IO_APIC() not only clears IO_APIC, also
restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
after disable_IO_APIC() will disable LAPIC and ruin the possible
virtual wire mode setting which the code has been trying to do all
along.

The consequence is, in KVM guest kernel always prints warning as below
during kexec/kdump kernel boots up. That happened in setup_local_APIC()
since 'do { xxx } while (queued && max_loops > 0)' loop does not function
well any more if pending irq exists in APIC IRR after LAPIC is disabled.
And people even saw casual kdump kernel hang once in ~30 attempts during
stress testing of kdump on KVM machine.

[0.001000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1467 
setup_local_APIC+0x228/0x330
[0.001000] Modules linked in:
[0.001000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc5+ #3
[0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.10.2-1.fc26 04/01/2014
[0.001000] RIP: 0010:setup_local_APIC+0x228/0x330
[0.001000] RSP: :b6e03eb8 EFLAGS: 00010286
[0.001000] RAX: 009edb4c4d84 RBX:  RCX: b099d800
[0.001000] RDX: 009e RSI:  RDI: 0810
[0.001000] RBP:  R08:  R09: 0001
[0.001000] R10: 98ce6a801c00 R11: 0761076d072f0776 R12: 0001
[0.001000] R13: 00f0 R14: 4000 R15: c6ff
[0.001000] FS:  () GS:98ce6bc0() 
knlGS:
[0.001000] CS:  0010 DS:  ES:  CR0: 80050033
[0.001000] CR2:  CR3: 22209000 CR4: 000406b0
[0.001000] Call Trace:
[0.001000]  apic_bsp_setup+0x56/0x74
[0.001000]  x86_late_time_init+0x11/0x16
[0.001000]  start_kernel+0x3c9/0x486
[0.001000]  secondary_startup_64+0xa5/0xb0
[0.001000] Code: 00 85 c9 74 2d 0f 31 c1 e1 0a 48 c1 e2 20 41 89 cf 4c 03 
7c 24 08 48 09 d0 49 29 c7 4c 89 3c 24 48 83 3c 24 00 0f 8f 8f fe ff
ff <0f> ff e9 10 ff ff ff 48 83 2c 24 01 eb e7 48 83 c4 18 5b 5d 41
[0.001000] ---[ end trace b88e71b9a6ebebdd ]---
[0.001000] masked ExtINT on CPU#0

To fix this, just break down disable_IO_APIC(), then call
clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
and call restore_boot_irq_mode() to restore boot irq mode before
reboot or kexec/kdump jump.

Signed-off-by: Baoquan He 
---
 arch/x86/include/asm/io_apic.h | 1 +
 arch/x86/kernel/apic/io_apic.c | 2 +-
 arch/x86/kernel/crash.c| 3 ++-
 arch/x86/kernel/machine_kexec_32.c | 2 +-
 arch/x86/kernel/machine_kexec_64.c | 2 +-
 arch/x86/kernel/reboot.c   | 3 ++-
 6 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 558d1a6a13ad..0fa95bfacb39 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int apic, 
unsigned int reg)
 extern void setup_IO_APIC(void);
 extern void enable_IO_APIC(void);
 extern void disable_IO_APIC(void);
+extern void clear_IO_APIC(void);
 extern void restore_boot_irq_mode(void);
 extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
 extern void print_IO_APICs(void);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 7b73b6b9b4b6..2d7cd2db77f5 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, unsigned 
int pin)
   mpc_ioapic_id(apic), pin);
 }
 
-static void clear_IO_APIC (void)
+void clear_IO_APIC (void)
 {
int apic, pin;
 
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 10e74d4778a1..1f6680427ff0 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -199,9 +199,10 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 #ifdef CONFIG_X86_IO_APIC
/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
ioapic_zap_locks();
-   disable_IO_APIC();
+   clear_IO_APIC();
 #endif
lapic_shutdown();
+   restore_boot_irq_mode();
 #ifdef CONFIG_HPET_TIMER
hpet_disable();
 #endif
diff --git a/arch/x86/kernel/machine_kexec_32.c 
b/arch/x86/kernel/machine_kexec_32.c
index edfede768688..f78bb4432bfb 100644
--- a/arch/x86/kernel/machine_kexec_32.c
+++ b/arch/x86/kernel/machine_kexec_32.c
@@ -199,7 +199,7 @@ void machine_kexec(struct kimage *image)
 * one form or other. kexec jump path also need
 * one.
 */
-   

[PATCH v3 2/5] x86/apic: Fix restoring boot irq mode in reboot and kexec/kdump

[Baoquan He]

This is a regression fix.

Before, to fix erratum AVR31, commit 522e66464467 ("x86/apic: Disable
I/O APIC before shutdown of the local APIC") moved lapic_shutdown()
calling after disable_IO_APIC(). This introdued a regression. The
root cause is that disable_IO_APIC() not only clears IO_APIC, also
restore boot irq mode by setting LAPIC/APIC/IMCR, lapic_shutdown()
after disable_IO_APIC() will disable LAPIC and ruin the possible
virtual wire mode setting which the code has been trying to do all
along.

The consequence is, in KVM guest kernel always prints warning as below
during kexec/kdump kernel boots up. That happened in setup_local_APIC()
since 'do { xxx } while (queued && max_loops > 0)' loop does not function
well any more if pending irq exists in APIC IRR after LAPIC is disabled.
And people even saw casual kdump kernel hang once in ~30 attempts during
stress testing of kdump on KVM machine.

[0.001000] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1467 
setup_local_APIC+0x228/0x330
[0.001000] Modules linked in:
[0.001000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc5+ #3
[0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.10.2-1.fc26 04/01/2014
[0.001000] RIP: 0010:setup_local_APIC+0x228/0x330
[0.001000] RSP: :b6e03eb8 EFLAGS: 00010286
[0.001000] RAX: 009edb4c4d84 RBX:  RCX: b099d800
[0.001000] RDX: 009e RSI:  RDI: 0810
[0.001000] RBP:  R08:  R09: 0001
[0.001000] R10: 98ce6a801c00 R11: 0761076d072f0776 R12: 0001
[0.001000] R13: 00f0 R14: 4000 R15: c6ff
[0.001000] FS:  () GS:98ce6bc0() 
knlGS:
[0.001000] CS:  0010 DS:  ES:  CR0: 80050033
[0.001000] CR2:  CR3: 22209000 CR4: 000406b0
[0.001000] Call Trace:
[0.001000]  apic_bsp_setup+0x56/0x74
[0.001000]  x86_late_time_init+0x11/0x16
[0.001000]  start_kernel+0x3c9/0x486
[0.001000]  secondary_startup_64+0xa5/0xb0
[0.001000] Code: 00 85 c9 74 2d 0f 31 c1 e1 0a 48 c1 e2 20 41 89 cf 4c 03 
7c 24 08 48 09 d0 49 29 c7 4c 89 3c 24 48 83 3c 24 00 0f 8f 8f fe ff
ff <0f> ff e9 10 ff ff ff 48 83 2c 24 01 eb e7 48 83 c4 18 5b 5d 41
[0.001000] ---[ end trace b88e71b9a6ebebdd ]---
[0.001000] masked ExtINT on CPU#0

To fix this, just break down disable_IO_APIC(), then call
clear_IO_APIC() to stop IO_APIC where disable_IO_APIC() was called,
and call restore_boot_irq_mode() to restore boot irq mode before
reboot or kexec/kdump jump.

Signed-off-by: Baoquan He 
---
 arch/x86/include/asm/io_apic.h | 1 +
 arch/x86/kernel/apic/io_apic.c | 2 +-
 arch/x86/kernel/crash.c| 3 ++-
 arch/x86/kernel/machine_kexec_32.c | 2 +-
 arch/x86/kernel/machine_kexec_64.c | 2 +-
 arch/x86/kernel/reboot.c   | 3 ++-
 6 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/arch/x86/include/asm/io_apic.h b/arch/x86/include/asm/io_apic.h
index 558d1a6a13ad..0fa95bfacb39 100644
--- a/arch/x86/include/asm/io_apic.h
+++ b/arch/x86/include/asm/io_apic.h
@@ -193,6 +193,7 @@ static inline unsigned int io_apic_read(unsigned int apic, 
unsigned int reg)
 extern void setup_IO_APIC(void);
 extern void enable_IO_APIC(void);
 extern void disable_IO_APIC(void);
+extern void clear_IO_APIC(void);
 extern void restore_boot_irq_mode(void);
 extern int IO_APIC_get_PCI_irq_vector(int bus, int devfn, int pin);
 extern void print_IO_APICs(void);
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 7b73b6b9b4b6..2d7cd2db77f5 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -587,7 +587,7 @@ static void clear_IO_APIC_pin(unsigned int apic, unsigned 
int pin)
   mpc_ioapic_id(apic), pin);
 }
 
-static void clear_IO_APIC (void)
+void clear_IO_APIC (void)
 {
int apic, pin;
 
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 10e74d4778a1..1f6680427ff0 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -199,9 +199,10 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 #ifdef CONFIG_X86_IO_APIC
/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
ioapic_zap_locks();
-   disable_IO_APIC();
+   clear_IO_APIC();
 #endif
lapic_shutdown();
+   restore_boot_irq_mode();
 #ifdef CONFIG_HPET_TIMER
hpet_disable();
 #endif
diff --git a/arch/x86/kernel/machine_kexec_32.c 
b/arch/x86/kernel/machine_kexec_32.c
index edfede768688..f78bb4432bfb 100644
--- a/arch/x86/kernel/machine_kexec_32.c
+++ b/arch/x86/kernel/machine_kexec_32.c
@@ -199,7 +199,7 @@ void machine_kexec(struct kimage *image)
 * one form or other. kexec jump path also need
 * one.
 */
-   disable_IO_APIC();
+