Dear all,
Sorry for my poor explanations, I have tried to give as many details as
possible but reading my previous emails they are somehow confusing, and my
english skills are not yet very good.
I will try to expose the problem in a few lines with references to the
actual code. This might be better for the sake of understanding.
When an interruption arrives, The commit stage stops the fetch of new
instructions and waits till the rob is empty.
Once the rob is empty the commit stage can process the interruption
calling the interrupt->invoke method.
through the cpu->processInterrupts(cpu->getInterrupts()); call at
cpu/o3/commit_impl.hh
The invoke call will end up executing the code at
X86FaultBase::invoke(ThreadContext * tc, StaticInstPtr inst)
(arch/x86/faults.cc)
This call does 2 things, it stores the interrupt vector and the actual pc
in two registers:
tc->setIntReg(INTREG_MICRO(1), vector);
tc->setIntReg(INTREG_MICRO(7), pc);
And calls the code at arch/x86/isa/insts/romutil.py by setting the uPC:
entry = extern_label_longModeInterrupt;
...
pcState.upc(romMicroPC(entry));
pcState.nupc(romMicroPC(entry) + 1);
tc->pcState(pcState);
The code at that file just stores some of the processor status in the
stack and jumps to the OS interrupt handler using the interrupt vector
stored at INTREG_MICRO(1) register.
Now the simulator is done with the interrupt and it gets discarded. Being
its vector only accesible at INTREG_MICRO(1).
Now there is a very narrow time frame where we have set the register
there to hold the vector as a parameter to the longModeInterrupt routine,
but we hadn't complete a fetch & decode of the first instruction, so the
ROB remains empty.
If another interruption arrives at that moment, the commit stage will
detect an empty rob and will also do the invoke of the interrupt.
Overwriting that registers and losing the first interrupt vector forever.
Then the first interruption of that routine gets executed (Both interrupts
jump to the same code, so the uPC remains unchanged) and finds the vector
of the new interruption instead of the value stored when it was called.
What I did to solve the issue was to inhibit all the interruptions during
that time frame. It was the fastest thing that I could come up with ...
However, this is a extremely weird race that is extremely unlikely to
happen. I just had bad luck :).
Hope this info is useful and now is better explained than before.
Thank you all for your hard work on this wonderful tool!!
Best regards,
Emilio
________________________________________
De: gem5-dev [[email protected]] en nombre de Gabe Black via
gem5-dev [[email protected]]
Enviado: viernes, 28 de noviembre de 2014 21:33
Para: gem5 Developer List
Asunto: Re: [gem5-dev] Issue with O3 and interrupts
I feel like there should be a simple solution to this problem, but I
haven't had the time to really walk through your explanation and
understand
it yet.
Gabe
On Fri, Nov 28, 2014 at 9:30 AM, Castillo Villar, Emilio via gem5-dev <
[email protected]> wrote:
Hello,
The problem is that the first interrupt calls the invoke function of the
X86 Fault implementation arch/x86/fault.cc.
This function saves the PC and the interrupt vector in the micro arch
registers and calls the code at the Microcode rom (isa/insts/romutil.py)
"longModeInterrupt" by setting the uPC. This routine saves the pc in the
stack, and calculates the address to the OS interrupt service routine
using
the interrupt vector. Then the simulator will forget about this fault.
If we have a second interrupt when the first interrupt sets this
registers
but hadn't completed the fetch of the first instruction of the
longModeInterrupt routine, the O3 CPU will detect an empty rob and will
allow this interrupt to proceed. It will overwrite the registers holding
the pc and the interrupt vector, that had not the chance of being saved.
Therefore the first interruption data will be lost, and when the
longModeInterrupt code first instruction arrives, it sees the status
(vector) of the second interruption.
I did a hack to fix this situation where I completely disable the
interrupts during the time window between the set of this registers and
the
Microcode rom execution.
I added a new register that when set to 0x1, every single interruption
is
ignored at the x86/interrupt.cc checkInterrupts function. This had to be
done because setting the IF at the flags registers do not disable all
the
interruptions. Then I added a new microop at the x86 arch. that sets
this
register to 0. I modified the routine that does all the above to call
this
new instruction at the end. This way I made it work, its a bit hacky
solution so there might be some other elegant ways to solve this issue.
Hope this can be helpful.
Best regards.
________________________________________
De: gem5-dev [[email protected]] en nombre de Nilay Vaish via
gem5-dev [[email protected]]
Enviado: viernes, 28 de noviembre de 2014 16:03
Para: Castillo Villar, Emilio via gem5-dev
Asunto: Re: [gem5-dev] Issue with O3 and interrupts
Ok, I have not seen this problem, but I got the description below. So
what's your suggestion on fixing the problem? Should we add a stack of
pending interrupts instead of maintaining one single variable?
--
Nilay
On Wed, 26 Nov 2014, Castillo Villar, Emilio via gem5-dev wrote:
Good evening,
I am experiencing a weird issue with the O3CPU, X86 and the interrupt
handling. I am running in FS mode and one simulation just experienced
a
weird hang. The simulated machine is doing an spinlock over a value
that
an interrupt handler writes.
After some debug I found that when the APIC sends two interruptions to
the cpu in a very short time window, the first interruption is
completely ignored. It can not even complete a commit of the first
instruction in the service routine before all its values get replaced
by
the next interrupt. After this interrupt completes, the execution goes
back to the application code and do not execute the code for the first
interrupt.
The problem is that the Lapic has the vector of the first interruption
in the ISR register as it gets restored after the second interruption
completes. Therefore, it thinks that the cpu is currently processing
that
interruption, though the cpu went back to execute application code and
will
never clear this ISR register.
The Lapic uses this ISR value to filter incoming interruptions and in
several cases, it does not forward those to the cpu, leading to
unattended
interruptions and hangs.
I have seen this behavior in the kernels' native_flush_tlb_others
function when a page fault happens. The core in charge of executing it,
sends an interrupt to all the other cores in the system and it does a
loop
checking that every cores receive the interruption. When each core
receives
the interruption, they just execute the associated handler and perform
a
write to a variable, notifying the sender that the interruption was
processed and the tlb was flushed.
The problem is that one of the cores is ignoring this interruption,
which has a vector value of 0xf0. I found that this core lapic has a
value
of 0xf1 in the ISR, filtering every lower vector.
s
This 0xf1 vector value was set by an interruption that never got to
execute because of the problem explained before, hence the interruption
carrying the 0xf0 vector value will never be executed and the
native_flush_tlb_others function will not complete.
I just took a trace of the moment when the 0xf1 interruption gets
dropped. The flags used where Exec, Commit, Faults:
system.cpu00.interrupts: Interrupt 0xf1 sent to core.
7175754213000: External Interrupt: RIP 0xffffffff8027a0a0: vector
0xf1:
#INTR
7175754213000: system.cpu00.interrupts: NEW IRR 0 NEW ISR f1.
Now the interrupt 0xf3 gets to execute and drops all the first
interruption.
7175754217000: system.cpu00.interrupts: Got Trigger Interrupt message
with vector 0xf3.
7175754217000: system.cpu00.interrupts: Interrupt is an Fixed.
7175754220500: system.cpu00.commit: Interrupt detected.
7175754220500: system.cpu00.interrupts: Interrupt 0xf3 sent to core.
7175754220500: External Interrupt: RIP 0xffffffff8027a0a0: vector
0xf3:
#INTR
7175754220500: system.cpu00.interrupts: NEW IRR 0 NEW ISR f3.
It can be seen how for both interrupts the RIP is the same.
The first committed instruction after all this sequence of events is
7175754228500: system.cpu00 T0 : @handle_mm_fault+992.32768 :
Microcode_ROM : slli t4, t1, 0x4 : IntAlu : D=0x0000000000000f30
which indeed is from the 0xf3 interrupt.
The cpu executes all the handler and then writes to the APIC EOI
register
7175754418500: system.cpu00.interrupts: Writing Local APIC register 5
at
offset 0xb0 as 0.
7175754418500: system.cpu00.interrupts: WRITING TO EOI NEW ISRV IS
0xf1
Here the APIC believes it is servicing the 0xf1 interrupt. However the
cpu goes back to the code it was executing right before the 0xf1
interrupt,
and never services it.
I was wondering if someone has found this issue before.
Thanks a lot for your time.
Best regards,
---------------------------------------
Emilio Castillo
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
