Il giorno sabato 18 marzo 2017 09:32:19 UTC+1, Jan Kiszka ha scritto:
> On 2017-03-17 14:42, [email protected] wrote:
> > Il giorno venerdì 17 marzo 2017 13:43:32 UTC+1, J. Kiszka ha scritto:
> >> On 2017-03-17 13:06, Claudio Scordino wrote:
> >>> Dear all,
> >>>
> >>> we are facing an unexpected exception when running the apic timer to
> >>> drive a GPIO as a software PWM.
> >>>
> >>> The platform is x86. The software runs in a bare-metal cell. The PWM
> >>> frequency is 5 KHz.
> >>>
> >>> When the duty cycle is very high or very low (i.e., two subsequent
> >>> interrupts get closer) we face the following unexpected exception:
> >>>
> >>> FATAL: Unhandled VM-Exit, reason 2
> >>> qualification 0
> >>> vectoring info: 0 interrupt info: 0
> >>> RIP: 0x00000000000f15d6 RSP: 0x00000000000dff08 FLAGS: 10002
> >>
> >> "objdump -dS inmate-linked.o" can tell you which instruction at RIP
> >> causes this fault. It's a triple fault, likely started off by a general
> >> protection or page fault.
> >>
> >> Jan
> >
> > Hi I'm Errico, Claudio's coworker, and I'm actually playing with this issue.
> >
> > The fault happens when we re-arm the apic timer
> >
> > 00000000000f15b5 <apic_timer_set>:
> >
> > void apic_timer_set(unsigned long timeout_ns)
> > {
> > unsigned long long ticks =
> > (unsigned long long)timeout_ns * divided_apic_freq;
> > write_msr(X2APIC_TMICT, ticks / NS_PER_SEC);
> > f15b5: 48 89 f8 mov %rdi,%rax
> > f15b8: b9 00 ca 9a 3b mov $0x3b9aca00,%ecx
> > f15bd: 31 d2 xor %edx,%edx
> > f15bf: 48 0f af 05 f1 10 ff imul -0xef0f(%rip),%rax #
> > e26b8 <divided_apic_freq>
> > f15c6: ff
> > f15c7: 48 f7 f1 div %rcx
> > f15ca: b9 38 08 00 00 mov $0x838,%ecx
> > f15cf: 48 89 c2 mov %rax,%rdx
> > f15d2: 48 c1 ea 20 shr $0x20,%rdx
> > f15d6: 0f 30 wrmsr
> > f15d8: c3 retq
> >
> > It is the *wrmsr* inside the apic_timer_set to generate the fault.
> > Since I'm not expert of x86 (I'm more an embedded guy), I'm asking for tips
> > and ideas.
>
> Interesting. This writes to a 32-bit x2APIC register. The manual states:
> "The upper 32-bits of all x2APIC MSRs (except for the ICR) are
> reserved." But the timer value calculation let EDX (lower part of RDX)
> become non-zero.
>
> > RAX: 0x000000044b82f9d8 RBX: 0x00000000000f060f RCX: 0x0000000000000838
> > RDX: 0x0000000000000004 RSI: 0x0000000000000a36 RDI: 0xffffffffffffe134
>
> Never tested if hardware actually explodes over this, but it would have
> the right to do so. Simple check: confine ticks / NS_PER_SEC to 32 bits
> and see if that resolves the crash.
>
> But that may cause issues regarding the desired timeout. A careful
> analysis of what happens here /wrt timeout calculation will be needed.
> E.g. what is the timeout_ns value in those cases?
>
> As you copied from apic-demo and use the inmates library, those may
> share the issue.
>
> Jan
Thank You,
I was able to fix the previous issue.
Moreover I chaged the APIC Timer configuration, actually I'm using it as
TSC-Deadline, getting better frequency stability in PWM generation.
But adding features at my demo I discovered what I think be a race error that
could happen when there's a concurrence with a "Instruction Trap" (like the one
needed to handle In/Out instruction) and a local IRQ (the one generated by APIC
Timer).
Whe this scenario happens seams that the Context of main function is not
correctly restored (volatile registers are corrupted, in particular %EDX
register used as source register for In instruction is zeroed).
Exception Message:
FATAL: Invalid PIO read, port: 0 size: 1
RIP: 0x00000000000f0228 RSP: 0x00000000000dffd0 FLAGS: 246
RAX: 0x0000000000000000 RBX: 0x00000000000f05a8 RCX: 0x0000000000000000
RDX: 0x0000000000000000 RSI: 0x0000000000000a35 RDI: 0x0000000000000a36
CS: 10 BASE: 0x0000000000000000 AR-BYTES: a09b EFER.LMA 1
CR0: 0x0000000080010031 CR3: 0x00000000000f3000 CR4: 0x0000000000002020
EFER: 0x0000000000000500
Parking CPU 3 (Cell: "pwm-demo")
Closing cell "pwm-demo"
Page pool usage after cell destruction: mem 4316/16327, remap 16459/131072
CPU 3 received SIPI, vector 98
Faulty Code:
static inline u8 inb(u16 port)
{
f021a: 89 f8 mov %edi,%eax
f021c: 66 89 44 24 ec mov %ax,-0x14(%rsp)
u8 v;
asm volatile("inb %1,%0" : "=a" (v) : "dN" (port));
f0221: 0f b7 44 24 ec movzwl -0x14(%rsp),%eax
f0226: 89 c2 mov %eax,%edx
f0228: ec in (%dx),%al
f0229: 88 44 24 ff mov %al,-0x1(%rsp)
return v;
f022d: 0f b6 44 24 ff movzbl -0x1(%rsp),%eax
}
f0232: c3
Faulty Code Caller:
#include <inmate.h>
#define GPIO_PORT_COMMAND 0xA35U
#define GPIO_PORT_DATA 0xA36U
#define GPIO_COMMAND_READ_ALL 0xA2U
/** Return all I/O flags. */
static inline u8 gpio_get_pins(void) {
outb(GPIO_COMMAND_READ_ALL, GPIO_PORT_COMMAND);
return inb(GPIO_PORT_DATA);
}
I added an inmate code that re-generate the problem (I'm working on Industrial
PC with an I/O port, I'm trying to generate the same problem using a "standard
In port" for x86).
I added the following lines on apic-demo.c cell to make the code works
/* I need to access 0xa35 and 0xa36 only, but I have to open 0xa37 too,
don't know why... (mask should be ~0x30, not ~0x70).
Watchdog ports (0xa15, 0xa16, 0xa1a) are ok.
*/
.pio_bitmap = {
[ 0/8 ... 0x3f7/8] = -1,
[ 0x3f8/8 ... 0x3ff/8] = 0,
[ 0x400/8 ... 0xa0f/8] = -1,
[ 0xa10/8 ... 0xa17/8] = ~0x60,
[ 0xa18/8 ... 0xa1f/8] = ~0x4,
[ 0xa20/8 ... 0xa2f/8] = -1,
[ 0xa30/8 ... 0xa37/8] = ~0x70,
[ 0xa38/8 ... 0xe00f/8] = -1,
[0xe010/8 ... 0xe017/8] = 0,
[0xe018/8 ... 0xffff/8] = -1,
},
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
#include <inmate.h>
#define APIC_TIMER_VECTOR (32U)
#define KILO (1000U)
#define MEGA (1000000U)
#define TICKS_TO_NANO(TICKS, REF_FREQ_HZ) \
(((TICKS) / KILO)? \
((TICKS) * KILO) / ((REF_FREQ_HZ) / MEGA): \
((TICKS) * MEGA) / ((REF_FREQ_HZ) / KILO))
/*==============================================================================
GPIO Drv
=============================================================================*/
#define GPIO_PORT_COMMAND 0xA35U
#define GPIO_PORT_DATA 0xA36U
#define GPIO_COMMAND_READ_ALL 0xA2U
#define GPIO_COMMAND_READ_IGNITION 0xF2U
#define GPIO_COMMAND_WRITE_ALL 0xA1U
#define GPIO_PIN_0 0x0U
#define GPIO_PIN_1 0x1U
#define GPIO_PIN_2 0x2U
#define GPIO_PIN_3 0x3U
#define GPIO_PIN_4 0x4U
#define GPIO_PIN_5 0x5U
#define GPIO_PIN_6 0x6U
#define GPIO_PIN_7 0x7U
#define GPIO_PIN_0_MASK 0x1U
#define GPIO_PIN_1_MASK 0x2U
#define GPIO_PIN_2_MASK 0x4U
#define GPIO_PIN_3_MASK 0x8U
#define GPIO_PIN_4_MASK 0x10U
#define GPIO_PIN_5_MASK 0x20U
#define GPIO_PIN_6_MASK 0x40U
#define GPIO_PIN_7_MASK 0x80U
#define GPIO_PIN_INPUT_0 GPIO_PIN_0
#define GPIO_PIN_INPUT_1 GPIO_PIN_1
#define GPIO_PIN_INPUT_2 GPIO_PIN_2
#define GPIO_PIN_INPUT_3 GPIO_PIN_3
#define GPIO_PIN_OUTPUT_0 GPIO_PIN_4
#define GPIO_PIN_OUTPUT_1 GPIO_PIN_5
#define GPIO_PIN_OUTPUT_2 GPIO_PIN_6
#define GPIO_PIN_OUTPUT_3 GPIO_PIN_7
#define GPIO_PIN_IGNITION GPIO_PIN_INPUT_3
#define WATCHDOG_PORT_DATA 0xA16U
#define WATCHDOG_PORT_COMMAND 0xA15U
#define WATCHDOG_PORT_PME 0xA1AU
#define WATCHDOG_VALUE_ENABLE 0x20U
#define WATCHDOG_VALUE_PME_ENABLE 0x40U
#define WATCHDOG_VALUE_DISABLE 0x0U
#define WATCHDOG_TIME_SECONDS 0x0U
#define WATCHDOG_TIME_MINUTES 0x1U
/** Return all I/O flags. */
static inline u8 gpio_get_pins(void) {
outb(GPIO_COMMAND_READ_ALL, GPIO_PORT_COMMAND);
return inb(GPIO_PORT_DATA);
}
/** Write \b all I/O flags. */
static inline void gpio_set_pins(u8 value) {
outb(GPIO_COMMAND_WRITE_ALL, GPIO_PORT_COMMAND);
outb(value, GPIO_PORT_DATA);
}
/**
* \brief Read pin pin from flags.
* \param[in] pin The pin \b index, [0-7].
* \return The bit status, 0 or 1.
* \note It is recommended to use GPIO_PIN_X or GPIO_PIN_INPUT/OUTPUT_X macros.
*/
static inline u8 gpio_get_pin(u8 pin) {
return (gpio_get_pins() >> pin) & 1;
}
/** Return ignition status, 0 or 1. */
static inline u8 gpio_get_ignition(void) {
outb(GPIO_COMMAND_READ_IGNITION, GPIO_PORT_COMMAND);
return (gpio_get_pins() >> GPIO_PIN_IGNITION) & 1;
}
/**
* \brief Write the value value to pin pin.
* \param[in] pin The pin \b index, [0-7].
* \param[in] value The value of the pin to write, must be 0 or 1.
* \note It is recommended to use GPIO_PIN_X or GPIO_PIN_INPUT/OUTPUT_X macros.
*/
static inline void gpio_set_pin(u8 pin, u8 value) {
u8 pins = gpio_get_pins();
pins &= ~(1 << pin);
pins |= ((value & 1) << pin);
gpio_set_pins(pins);
}
/**
* \brief Toggle the pin bit.
* \param[in] pin The pin \b index, [0-7].
* \note It is recommended to use GPIO_PIN_X or GPIO_PIN_INPUT/OUTPUT_X macros.
*/
static inline void gpio_toggle_pin(u8 pin) {
gpio_set_pins(gpio_get_pins() ^ (1 << pin));
}
/**
* \brief Toggle flags with mask mask.
* \param[in] mask The mask to xor old flags value with.
*/
static inline void gpio_toggle_pins(u8 mask) {
gpio_set_pins(gpio_get_pins() ^ mask);
}
/*==============================================================================
Application Flag
=============================================================================*/
bool volatile app_terminate;
/*==============================================================================
BEGIN PWM Driver
=============================================================================*/
#define PIN_PWM GPIO_PIN_OUTPUT_1
#define PWM_MSK (1 << PIN_PWM)
static u64 tsc_freq_hz;
static volatile u64 jitter_cnt;
static volatile u64 jitter_max;
static volatile u64 jitter_avg;
static volatile u64 jitter_tot;
static volatile u64 jitter_min = ((u64)-1);
#define PWM_FREQUENCY_HZ (6000U)
#define DATA_XCHG_FREQ_HZ (20U)
#define DATA_XCHG_TICKS \
((PWM_FREQUENCY_HZ*2U)/DATA_XCHG_FREQ_HZ)
#define ENCODER_FREQ_HZ (PWM_FREQUENCY_HZ)
#define ENCODER_TICKS \
((PWM_FREQUENCY_HZ*2U)/ENCODER_FREQ_HZ)
/* 10s */
#define PWM_DUTY_SLICES (100U)
#define PWM_MIN_DUTY_SLICES (1U)
#define PWM_STARTING_DUTY (70U)
//#define PWM_STARTING_DUTY PWM_MIN_DUTY_SLICES
static volatile u16 pwm_duty;
static volatile u64 pwm_expected_ticks;
enum pwm_front {
PWM_ZERO_TO_ONE_FRONT = 0U,
PWM_ONE_TO_ZERO_FRONT
};
static u64 pwm_get_time_next_front(enum pwm_front front) {
u64 pwm_time_next_front;
u16 front_duty;
if (front == PWM_ONE_TO_ZERO_FRONT) {
front_duty = pwm_duty;
} else {
front_duty = (PWM_DUTY_SLICES - pwm_duty);
}
pwm_time_next_front = ((tsc_freq_hz * front_duty) +
((PWM_FREQUENCY_HZ * PWM_DUTY_SLICES)/2U)) /
(PWM_FREQUENCY_HZ * PWM_DUTY_SLICES);
return pwm_time_next_front;
}
static u64 rdtsc(void)
{
#ifdef __x86_64__
u32 lo, hi;
asm volatile("rdtsc" : "=a" (lo), "=d" (hi));
return (u64)lo | (((u64)hi) << 32);
#else
u64 v;
asm volatile("rdtsc" : "=A" (v));
return v;
#endif
}
#define X2APIC_LVTT (0x832U)
#define X2APIC_TSC_DL (0x6E0U)
static inline void apic_timer_tsc_dl_init(unsigned int vector) {
/* We set APIC Timer as TSC-Deadline (2U << 17U) + vector */
write_msr(X2APIC_LVTT, (2U << 17U) | vector);
}
static inline void apic_timer_tsc_dl(u64 dl)
{
write_msr(X2APIC_TSC_DL, dl);
}
static inline void pwm_setup_timer(u64 pwm_time_next_front) {
pwm_expected_ticks = rdtsc() + pwm_time_next_front;
apic_timer_tsc_dl(pwm_expected_ticks);
}
static void pwm_start ( u16 duty ) {
if ( duty > (PWM_DUTY_SLICES - PWM_MIN_DUTY_SLICES) ) {
duty = (PWM_DUTY_SLICES - PWM_MIN_DUTY_SLICES);
} if ( duty < PWM_MIN_DUTY_SLICES ) {
duty = PWM_MIN_DUTY_SLICES;
}
/* Set-up configuration */
pwm_duty = duty;
/* Assure that output is zero */
gpio_set_pin(PIN_PWM, 0);
/* Configure APIC Timer as TSC-Deadline */
apic_timer_tsc_dl_init(APIC_TIMER_VECTOR);
pwm_expected_ticks = rdtsc();
/* Start with a Zero-To-One Front */
pwm_setup_timer(pwm_get_time_next_front(PWM_ZERO_TO_ONE_FRONT));
}
static void pwm_front_irq( void )
{
u8 out;
u64 pwm_time_next_front;
/* Get Jitter measurement */
u64 const delta = rdtsc() - pwm_expected_ticks;
/* Handle Output */
/* Read OUT register */
out = gpio_get_pins();
if ( out & PWM_MSK ) {
out &= ~PWM_MSK;
pwm_time_next_front = pwm_get_time_next_front(PWM_ZERO_TO_ONE_FRONT);
} else {
out |= PWM_MSK;
pwm_time_next_front = pwm_get_time_next_front(PWM_ONE_TO_ZERO_FRONT);
}
/* Commit Changes */
gpio_set_pins(out);
/* Save min/MAX Jitter */
if (delta < jitter_min) {
jitter_min = delta;
}
if (delta > jitter_max) {
jitter_max = delta;
}
/* Increment the counter value (I DO NOT HANDLE WRAP AROUND,
I hope that in 8-10 hours do not wrap!!!) */
++jitter_cnt;
jitter_tot += delta;
jitter_avg = jitter_tot / jitter_cnt;
if (!app_terminate) {
pwm_setup_timer(pwm_time_next_front);
} else {
/* If the application is terminated: Assure that output is zero */
gpio_set_pin(PIN_PWM, 0);
}
}
/*==============================================================================
END PWM Driver
=============================================================================*/
void inmate_main(void)
{
comm_region->cell_state = JAILHOUSE_CELL_RUNNING_LOCKED;
tsc_freq_hz = tsc_init();
printk("Calibrated TSC frequency: %lu.%03u kHz\n", tsc_freq_hz / 1000,
tsc_freq_hz % 1000);
int_init();
int_set_handler(APIC_TIMER_VECTOR, pwm_front_irq);
pwm_start(PWM_STARTING_DUTY);
asm volatile("sti");
while (!app_terminate) {
u8 volatile gpio_in;
asm volatile("hlt");
/* Read Multiple Times GPIO => Issue Multiple TRAPS, to enanche the
chance of the RACE with local APIC Timer IRQ... */
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
gpio_in = gpio_get_pins();
((void)gpio_in);
{
static u32 decimator;
if (++decimator == DATA_XCHG_TICKS) {
printk(
"PWM parameters, duty:<%ld>\n"
"PWM timer jitter, avg:<%6ld> ns, min:<%6ld> ns, max:<%6ld> ns\n"
"PWM AVG on:<%d> Samples\n",
pwm_duty,
jitter_avg,
jitter_min,
jitter_max,
jitter_cnt
);
if ( ++pwm_duty >= PWM_DUTY_SLICES - PWM_MIN_DUTY_SLICES ) {
pwm_duty = PWM_MIN_DUTY_SLICES;
}
decimator = 0U;
}
}
switch (comm_region->msg_to_cell) {
case JAILHOUSE_MSG_SHUTDOWN_REQUEST:
gpio_set_pin(PIN_PWM, 0);
app_terminate = true;
break;
default:
jailhouse_send_reply_from_cell(comm_region, JAILHOUSE_MSG_UNKNOWN);
break;
}
}
printk("PWM demo Stopped\n");
comm_region->cell_state = JAILHOUSE_CELL_SHUT_DOWN;
}
