Re: X2APIC support

[Slawa Olhovchenkov]

On Wed, Sep 14, 2016 at 07:08:02PM +0300, Konstantin Belousov wrote:

> On Wed, Sep 14, 2016 at 05:02:21PM +0300, Andriy Gapon wrote:
> > On 14/09/2016 15:49, Slawa Olhovchenkov wrote:
> > > MSR_APICBASE = 0xfee00d00
> > > x2APIC is prohibited but turned on by BIOS
> > 
> > Kostik, ^
> 
> Well, the following might work, but I have no good idea what to do
> when BIOS does handoff with x2APIC enabled and directs us to not
> enable it.  Switching to xAPIC mode is not an option, I suspect.

CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
  
Features=0xbfebfbff
  
Features2=0x7ffefbff
  AMD Features=0x2c100800
  AMD Features2=0x121
  Structured Extended 
Features=0x21cbfbb
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
boot_cpu_id = 0
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 12 core(s)
random: unblocking device.
ioapic0  irqs 0-23 on motherboard
ioapic1  irqs 24-47 on motherboard
ioapic2  irqs 48-71 on motherboard
random: entropy device external interface
module_register_init: MOD_LOAD (vesa, 0x807bf2f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
netmap: loaded module
vtvga0:  on motherboard
cryptosoft0:  on motherboard
acpi0:  on motherboard
acpi0: Power Button (fixed)
[...]

boot OK, w/o any APIC-related messages

> diff --git a/sys/x86/acpica/madt.c b/sys/x86/acpica/madt.c
> index 241a769..3a93fd6 100644
> --- a/sys/x86/acpica/madt.c
> +++ b/sys/x86/acpica/madt.c
> @@ -138,7 +138,6 @@ madt_setup_local(void)
>  
>   madt = pmap_mapbios(madt_physaddr, madt_length);
>   if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
> - x2apic_mode = 1;
>   reason = NULL;
>  
>   /*
> @@ -150,21 +149,17 @@ madt_setup_local(void)
>   if (dmartbl_physaddr != 0) {
>   dmartbl = acpi_map_table(dmartbl_physaddr,
>   ACPI_SIG_DMAR);
> - if ((dmartbl->Flags & ACPI_DMAR_X2APIC_OPT_OUT) != 0) {
> - x2apic_mode = 0;
> + if ((dmartbl->Flags & ACPI_DMAR_X2APIC_OPT_OUT) != 0)
>   reason = "by DMAR table";
> - }
>   acpi_unmap_table(dmartbl);
>   }
>   if (vm_guest == VM_GUEST_VMWARE) {
>   vmware_hvcall(VMW_HVCMD_GETVCPU_INFO, p);
>   if ((p[0] & VMW_VCPUINFO_VCPU_RESERVED) != 0 ||
> - (p[0] & VMW_VCPUINFO_LEGACY_X2APIC) == 0) {
> - x2apic_mode = 0;
> - reason = "inside VMWare without intr redirection";
> - }
> + (p[0] & VMW_VCPUINFO_LEGACY_X2APIC) == 0)
> + reason =
> + "inside VMWare without intr redirection";
>   } else if (vm_guest == VM_GUEST_XEN) {
> - x2apic_mode = 0;
>   reason = "due to running under XEN";
>   } else if (vm_guest == VM_GUEST_NO &&
>   CPUID_TO_FAMILY(cpu_id) == 0x6 &&
> @@ -184,13 +179,21 @@ madt_setup_local(void)
>   if (!strcmp(hw_vendor, "LENOVO") ||
>   !strcmp(hw_vendor,
>   "ASUSTeK Computer Inc.")) {
> - x2apic_mode = 0;
>   reason =
>   "for a suspected SandyBridge BIOS bug";
>   }
>   freeenv(hw_vendor);
>   }
>   }
> + if (reason != NULL && lapic_is_x2apic()) {
> + if (bootverbose)
> + printf("x2APIC should be disabled %s but "
> + "already enabled by BIOS; enabling.\n",
> +  reason);
> + reason = NULL;
> + }
> + if (reason == NULL)
> + x2apic_mode = 1;
>   TUNABLE_INT_FETCH("hw.x2apic_enable", _mode);
>   if 

Re: X2APIC support

[Andriy Gapon]

On 14/09/2016 19:08, Konstantin Belousov wrote:
> Well, the following might work, but I have no good idea what to do
> when BIOS does handoff with x2APIC enabled and directs us to not
> enable it.  Switching to xAPIC mode is not an option, I suspect.

The specification describes a way to transition from x2APIC to xAPIC mode, but
it's not trivial, because it requires checking all CPU IDs and disabling CPUs
with too high ones.

Personally, I am not comfortable with your patch.  First, it seems that the
patch does not cover hw.x2apic_enable=0.  And I don't like that we leave x2APIC
mode enabled even when we otherwise would not enable it just because BIOS.

All in all, if we can't go x2APIC -> xAPIC, then I would prefer a meaningful
panic over a "compromise".

Just my opinion.  The topic probably warrants a wider discussion and a proper
review.

Slawa,
thank you for your persistence and testing.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Wed, Sep 14, 2016 at 05:02:21PM +0300, Andriy Gapon wrote:
> On 14/09/2016 15:49, Slawa Olhovchenkov wrote:
> > MSR_APICBASE = 0xfee00d00
> > x2APIC is prohibited but turned on by BIOS
> 
> Kostik, ^

Well, the following might work, but I have no good idea what to do
when BIOS does handoff with x2APIC enabled and directs us to not
enable it.  Switching to xAPIC mode is not an option, I suspect.

diff --git a/sys/x86/acpica/madt.c b/sys/x86/acpica/madt.c
index 241a769..3a93fd6 100644
--- a/sys/x86/acpica/madt.c
+++ b/sys/x86/acpica/madt.c
@@ -138,7 +138,6 @@ madt_setup_local(void)
 
madt = pmap_mapbios(madt_physaddr, madt_length);
if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
-   x2apic_mode = 1;
reason = NULL;
 
/*
@@ -150,21 +149,17 @@ madt_setup_local(void)
if (dmartbl_physaddr != 0) {
dmartbl = acpi_map_table(dmartbl_physaddr,
ACPI_SIG_DMAR);
-   if ((dmartbl->Flags & ACPI_DMAR_X2APIC_OPT_OUT) != 0) {
-   x2apic_mode = 0;
+   if ((dmartbl->Flags & ACPI_DMAR_X2APIC_OPT_OUT) != 0)
reason = "by DMAR table";
-   }
acpi_unmap_table(dmartbl);
}
if (vm_guest == VM_GUEST_VMWARE) {
vmware_hvcall(VMW_HVCMD_GETVCPU_INFO, p);
if ((p[0] & VMW_VCPUINFO_VCPU_RESERVED) != 0 ||
-   (p[0] & VMW_VCPUINFO_LEGACY_X2APIC) == 0) {
-   x2apic_mode = 0;
-   reason = "inside VMWare without intr redirection";
-   }
+   (p[0] & VMW_VCPUINFO_LEGACY_X2APIC) == 0)
+   reason =
+   "inside VMWare without intr redirection";
} else if (vm_guest == VM_GUEST_XEN) {
-   x2apic_mode = 0;
reason = "due to running under XEN";
} else if (vm_guest == VM_GUEST_NO &&
CPUID_TO_FAMILY(cpu_id) == 0x6 &&
@@ -184,13 +179,21 @@ madt_setup_local(void)
if (!strcmp(hw_vendor, "LENOVO") ||
!strcmp(hw_vendor,
"ASUSTeK Computer Inc.")) {
-   x2apic_mode = 0;
reason =
"for a suspected SandyBridge BIOS bug";
}
freeenv(hw_vendor);
}
}
+   if (reason != NULL && lapic_is_x2apic()) {
+   if (bootverbose)
+   printf("x2APIC should be disabled %s but "
+   "already enabled by BIOS; enabling.\n",
+reason);
+   reason = NULL;
+   }
+   if (reason == NULL)
+   x2apic_mode = 1;
TUNABLE_INT_FETCH("hw.x2apic_enable", _mode);
if (!x2apic_mode && reason != NULL && bootverbose)
printf("x2APIC available but disabled %s\n", reason);
diff --git a/sys/x86/include/apicvar.h b/sys/x86/include/apicvar.h
index 1ddb69e..09c3a63 100644
--- a/sys/x86/include/apicvar.h
+++ b/sys/x86/include/apicvar.h
@@ -206,6 +206,7 @@ struct apic_ops {
void(*create)(u_int, int);
void(*init)(vm_paddr_t);
void(*xapic_mode)(void);
+   bool(*is_x2apic)(void);
void(*setup)(int);
void(*dump)(const char *);
void(*disable)(void);
@@ -268,6 +269,13 @@ lapic_xapic_mode(void)
apic_ops.xapic_mode();
 }
 
+static inline bool
+lapic_is_x2apic(void)
+{
+
+   return (apic_ops.is_x2apic());
+}
+
 static inline void
 lapic_setup(int boot)
 {
diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
index cd774df..d9a3453 100644
--- a/sys/x86/x86/local_apic.c
+++ b/sys/x86/x86/local_apic.c
@@ -269,6 +269,16 @@ native_lapic_enable_x2apic(void)
wrmsr(MSR_APICBASE, apic_base);
 }
 
+static bool
+native_lapic_is_x2apic(void)
+{
+   uint64_t apic_base;
+
+   apic_base = rdmsr(MSR_APICBASE);
+   return ((apic_base & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
+   (APICBASE_X2APIC | APICBASE_ENABLED));
+}
+
 static voidlapic_enable(void);
 static voidlapic_resume(struct pic *pic, bool suspend_cancelled);
 static voidlapic_timer_oneshot(struct lapic *);
@@ -329,6 +339,7 @@ struct apic_ops apic_ops = {
.create = native_lapic_create,
.init   = native_lapic_init,
.xapic_mode = native_lapic_xapic_mode,
+   .is_x2apic  = 

Re: X2APIC support

[Andriy Gapon]

On 14/09/2016 15:49, Slawa Olhovchenkov wrote:
> MSR_APICBASE = 0xfee00d00
> x2APIC is prohibited but turned on by BIOS

Kostik, ^

P.S. the format string for the value should have been 0x%016jx.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Wed, Sep 14, 2016 at 03:35:39PM +0300, Andriy Gapon wrote:

> On 14/09/2016 15:33, Slawa Olhovchenkov wrote:
> > On Wed, Sep 14, 2016 at 03:22:17PM +0300, Andriy Gapon wrote:
> > 
> >> On 14/09/2016 14:36, Konstantin Belousov wrote:
> >>> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
>  On 13/09/2016 18:22, Konstantin Belousov wrote:
> > Any access
> > to the LAPIC registers page in x2APIC mode faults.
> 
>  Is this a fact?
>  I read the following in the specification:
> 
>    In x2APIC mode, the memory mapped interface is not available and any
>    access to the MMIO interface will behave similar to that of a legacy 
>  xAPIC
>    in globally disabled state.
> 
>  But I couldn't find what actually happens for the legacy xAPIC in 
>  globally
>  disabled state.  For AMD processors it is documented that if xAPIC is 
>  disabled
>  then accessing the APIC memory range works the same as accessing the 
>  regular
>  memory.  That can be different for Intel processors, of course.
> >>>
> >>> Look at the native_lapic_init(), very beginning of the function.  If
> >>> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
> >>> attributes are changed.
> >>
> >> Right, but we are talking about the case where x2apic_mode *is zero* while 
> >> the
> >> hardware in x2APIC mode.
> >>
> >>> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
> >>> because no LAPIC accesses are done until native_lapic_enable_x2apic() is
> >>> called. This just happens, I thought about more organized receipt of the
> >>> current LAPIC mode. Issue is that decision for LAPIC mode is performed
> >>> by madt enumerator which pref. should not read LAPIC config (too early).
> >>> And it is not clear at all, what to do if there is reason to use xAPIC,
> >>> as checked by madt_setup_local(), but we are in x2APIC mode already.
> >>
> >> Yes.  As I mentioned earlier we should at least panic with an informative
> >> message.  Maybe we could add some code to so something smarter later.
> >>
> >>> Anyway, returning to the original issue, I do not believe that the
> >>> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
> >>> is specified. Kernel would fault in that case.
> >>
> >> I think that it should be easy to check that if Slawa is still willing to 
> >> try
> >> another diagnostic patch.
> > 
> > What combination of BIOS setting need?
> 
> The one that causes the crash.

VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
  
Features=0xbfebfbff
  
Features2=0x7ffefbff
  AMD Features=0x2c100800
  AMD Features2=0x121
  Structured Extended 
Features=0x21cbfbb
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)
MSR_APICBASE = 0xfee00d00
x2APIC is prohibited but turned on by BIOS
Event timer "LAPIC" quality 600
ACPI APIC Table: 
boot_cpu_id = 255
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = ff
fault virtual address   = 0x0
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x80537e74
stack pointer   = 0x28:0x814b3a60
frame pointer   = 0x28:0x814b3a70
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0x805272e7 at kdb_backtrace+0x67
#1 0x804dd662 at vpanic+0x182
#2 0x804dd4d3 at panic+0x43
#3 0x807a37a1 at trap_fatal+0x351
#4 0x807a3993 at trap_pfault+0x1e3
#5 0x807a2f1c at trap+0x26c
#6 0x80787ca1 at calltrap+0x8
#7 0x8083b57a at topo_probe+0x61a
#8 0x8078fe93 at cpu_mp_start+0x1c3
#9 0x805382ca at mp_start+0x3a
#10 0x80465cd8 at mi_startup+0x118
#11 0x8028dfac at btext+0x2c
Uptime: 1s

> >> diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
> >> index 203e9d00e8acc..37ac03fb9d811 100644
> >> --- a/sys/x86/x86/local_apic.c
> >> +++ b/sys/x86/x86/local_apic.c
> >> @@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr)
> >>if 

Re: X2APIC support

[Andriy Gapon]

On 14/09/2016 15:33, Slawa Olhovchenkov wrote:
> On Wed, Sep 14, 2016 at 03:22:17PM +0300, Andriy Gapon wrote:
> 
>> On 14/09/2016 14:36, Konstantin Belousov wrote:
>>> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
 On 13/09/2016 18:22, Konstantin Belousov wrote:
> Any access
> to the LAPIC registers page in x2APIC mode faults.

 Is this a fact?
 I read the following in the specification:

   In x2APIC mode, the memory mapped interface is not available and any
   access to the MMIO interface will behave similar to that of a legacy 
 xAPIC
   in globally disabled state.

 But I couldn't find what actually happens for the legacy xAPIC in globally
 disabled state.  For AMD processors it is documented that if xAPIC is 
 disabled
 then accessing the APIC memory range works the same as accessing the 
 regular
 memory.  That can be different for Intel processors, of course.
>>>
>>> Look at the native_lapic_init(), very beginning of the function.  If
>>> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
>>> attributes are changed.
>>
>> Right, but we are talking about the case where x2apic_mode *is zero* while 
>> the
>> hardware in x2APIC mode.
>>
>>> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
>>> because no LAPIC accesses are done until native_lapic_enable_x2apic() is
>>> called. This just happens, I thought about more organized receipt of the
>>> current LAPIC mode. Issue is that decision for LAPIC mode is performed
>>> by madt enumerator which pref. should not read LAPIC config (too early).
>>> And it is not clear at all, what to do if there is reason to use xAPIC,
>>> as checked by madt_setup_local(), but we are in x2APIC mode already.
>>
>> Yes.  As I mentioned earlier we should at least panic with an informative
>> message.  Maybe we could add some code to so something smarter later.
>>
>>> Anyway, returning to the original issue, I do not believe that the
>>> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
>>> is specified. Kernel would fault in that case.
>>
>> I think that it should be easy to check that if Slawa is still willing to try
>> another diagnostic patch.
> 
> What combination of BIOS setting need?

The one that causes the crash.

>> diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
>> index 203e9d00e8acc..37ac03fb9d811 100644
>> --- a/sys/x86/x86/local_apic.c
>> +++ b/sys/x86/x86/local_apic.c
>> @@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr)
>>  if (x2apic_mode) {
>>  native_lapic_enable_x2apic();
>>  lapic_map = NULL;
>> +} else if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
>> +r = rdmsr(MSR_APICBASE);
>> +printf("MSR_APICBASE = 0x%16jx\n", (uintmax_t)r);
>> +if ((r & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
>> +(APICBASE_X2APIC | APICBASE_ENABLED))
>> +printf("x2APIC is prohibited but turned on by BIOS\n");
>>  }
>>
>>  /* Setup the spurious interrupt handler. */
>>
>>
>>
>> -- 
>> Andriy Gapon


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Wed, Sep 14, 2016 at 03:22:17PM +0300, Andriy Gapon wrote:

> On 14/09/2016 14:36, Konstantin Belousov wrote:
> > On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
> >> On 13/09/2016 18:22, Konstantin Belousov wrote:
> >>> Any access
> >>> to the LAPIC registers page in x2APIC mode faults.
> >>
> >> Is this a fact?
> >> I read the following in the specification:
> >>
> >>   In x2APIC mode, the memory mapped interface is not available and any
> >>   access to the MMIO interface will behave similar to that of a legacy 
> >> xAPIC
> >>   in globally disabled state.
> >>
> >> But I couldn't find what actually happens for the legacy xAPIC in globally
> >> disabled state.  For AMD processors it is documented that if xAPIC is 
> >> disabled
> >> then accessing the APIC memory range works the same as accessing the 
> >> regular
> >> memory.  That can be different for Intel processors, of course.
> > 
> > Look at the native_lapic_init(), very beginning of the function.  If
> > x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
> > attributes are changed.
> 
> Right, but we are talking about the case where x2apic_mode *is zero* while the
> hardware in x2APIC mode.
> 
> > Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
> > because no LAPIC accesses are done until native_lapic_enable_x2apic() is
> > called. This just happens, I thought about more organized receipt of the
> > current LAPIC mode. Issue is that decision for LAPIC mode is performed
> > by madt enumerator which pref. should not read LAPIC config (too early).
> > And it is not clear at all, what to do if there is reason to use xAPIC,
> > as checked by madt_setup_local(), but we are in x2APIC mode already.
> 
> Yes.  As I mentioned earlier we should at least panic with an informative
> message.  Maybe we could add some code to so something smarter later.
> 
> > Anyway, returning to the original issue, I do not believe that the
> > hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
> > is specified. Kernel would fault in that case.
> 
> I think that it should be easy to check that if Slawa is still willing to try
> another diagnostic patch.

What combination of BIOS setting need?

> diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
> index 203e9d00e8acc..37ac03fb9d811 100644
> --- a/sys/x86/x86/local_apic.c
> +++ b/sys/x86/x86/local_apic.c
> @@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr)
>   if (x2apic_mode) {
>   native_lapic_enable_x2apic();
>   lapic_map = NULL;
> + } else if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
> + r = rdmsr(MSR_APICBASE);
> + printf("MSR_APICBASE = 0x%16jx\n", (uintmax_t)r);
> + if ((r & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
> + (APICBASE_X2APIC | APICBASE_ENABLED))
> + printf("x2APIC is prohibited but turned on by BIOS\n");
>   }
> 
>   /* Setup the spurious interrupt handler. */
> 
> 
> 
> -- 
> Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 13/09/2016 17:59, Slawa Olhovchenkov wrote:
> Goggling on X2APIC_OPT_OUT take same result: other OS in this case
> downgrade to xAPIC mode.

It still does not make any sense for BIOS to turn on x2APIC and then instruct
the OS that x2APIC must not be used.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 14/09/2016 14:36, Konstantin Belousov wrote:
> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
>> On 13/09/2016 18:22, Konstantin Belousov wrote:
>>> Any access
>>> to the LAPIC registers page in x2APIC mode faults.
>>
>> Is this a fact?
>> I read the following in the specification:
>>
>>   In x2APIC mode, the memory mapped interface is not available and any
>>   access to the MMIO interface will behave similar to that of a legacy xAPIC
>>   in globally disabled state.
>>
>> But I couldn't find what actually happens for the legacy xAPIC in globally
>> disabled state.  For AMD processors it is documented that if xAPIC is 
>> disabled
>> then accessing the APIC memory range works the same as accessing the regular
>> memory.  That can be different for Intel processors, of course.
> 
> Look at the native_lapic_init(), very beginning of the function.  If
> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
> attributes are changed.

Right, but we are talking about the case where x2apic_mode *is zero* while the
hardware in x2APIC mode.

> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
> because no LAPIC accesses are done until native_lapic_enable_x2apic() is
> called. This just happens, I thought about more organized receipt of the
> current LAPIC mode. Issue is that decision for LAPIC mode is performed
> by madt enumerator which pref. should not read LAPIC config (too early).
> And it is not clear at all, what to do if there is reason to use xAPIC,
> as checked by madt_setup_local(), but we are in x2APIC mode already.

Yes.  As I mentioned earlier we should at least panic with an informative
message.  Maybe we could add some code to so something smarter later.

> Anyway, returning to the original issue, I do not believe that the
> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
> is specified. Kernel would fault in that case.

I think that it should be easy to check that if Slawa is still willing to try
another diagnostic patch.

diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
index 203e9d00e8acc..37ac03fb9d811 100644
--- a/sys/x86/x86/local_apic.c
+++ b/sys/x86/x86/local_apic.c
@@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr)
if (x2apic_mode) {
native_lapic_enable_x2apic();
lapic_map = NULL;
+   } else if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
+   r = rdmsr(MSR_APICBASE);
+   printf("MSR_APICBASE = 0x%16jx\n", (uintmax_t)r);
+   if ((r & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
+   (APICBASE_X2APIC | APICBASE_ENABLED))
+   printf("x2APIC is prohibited but turned on by BIOS\n");
}

/* Setup the spurious interrupt handler. */



-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Wed, Sep 14, 2016 at 02:36:34PM +0300, Konstantin Belousov wrote:

> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
> > On 13/09/2016 18:22, Konstantin Belousov wrote:
> > > Any access
> > > to the LAPIC registers page in x2APIC mode faults.
> > 
> > Is this a fact?
> > I read the following in the specification:
> > 
> >   In x2APIC mode, the memory mapped interface is not available and any
> >   access to the MMIO interface will behave similar to that of a legacy xAPIC
> >   in globally disabled state.
> > 
> > But I couldn't find what actually happens for the legacy xAPIC in globally
> > disabled state.  For AMD processors it is documented that if xAPIC is 
> > disabled
> > then accessing the APIC memory range works the same as accessing the regular
> > memory.  That can be different for Intel processors, of course.
> 
> Look at the native_lapic_init(), very beginning of the function.  If
> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
> attributes are changed.
> 
> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
> because no LAPIC accesses are done until native_lapic_enable_x2apic() is
> called. This just happens, I thought about more organized receipt of the
> current LAPIC mode. Issue is that decision for LAPIC mode is performed
> by madt enumerator which pref. should not read LAPIC config (too early).
> And it is not clear at all, what to do if there is reason to use xAPIC,
> as checked by madt_setup_local(), but we are in x2APIC mode already.
> 
> Anyway, returning to the original issue, I do not believe that the
> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
> is specified. Kernel would fault in that case.

As I assume, other OS don't fault in this case.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
> On 13/09/2016 18:22, Konstantin Belousov wrote:
> > Any access
> > to the LAPIC registers page in x2APIC mode faults.
> 
> Is this a fact?
> I read the following in the specification:
> 
>   In x2APIC mode, the memory mapped interface is not available and any
>   access to the MMIO interface will behave similar to that of a legacy xAPIC
>   in globally disabled state.
> 
> But I couldn't find what actually happens for the legacy xAPIC in globally
> disabled state.  For AMD processors it is documented that if xAPIC is disabled
> then accessing the APIC memory range works the same as accessing the regular
> memory.  That can be different for Intel processors, of course.

Look at the native_lapic_init(), very beginning of the function.  If
x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
attributes are changed.

Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
because no LAPIC accesses are done until native_lapic_enable_x2apic() is
called. This just happens, I thought about more organized receipt of the
current LAPIC mode. Issue is that decision for LAPIC mode is performed
by madt enumerator which pref. should not read LAPIC config (too early).
And it is not clear at all, what to do if there is reason to use xAPIC,
as checked by madt_setup_local(), but we are in x2APIC mode already.

Anyway, returning to the original issue, I do not believe that the
hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
is specified. Kernel would fault in that case.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 13/09/2016 18:22, Konstantin Belousov wrote:
> Any access
> to the LAPIC registers page in x2APIC mode faults.

Is this a fact?
I read the following in the specification:

  In x2APIC mode, the memory mapped interface is not available and any
  access to the MMIO interface will behave similar to that of a legacy xAPIC
  in globally disabled state.

But I couldn't find what actually happens for the legacy xAPIC in globally
disabled state.  For AMD processors it is documented that if xAPIC is disabled
then accessing the APIC memory range works the same as accessing the regular
memory.  That can be different for Intel processors, of course.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Tue, Sep 13, 2016 at 05:54:26PM +0300, Andriy Gapon wrote:
> On 13/09/2016 17:21, Slawa Olhovchenkov wrote:
> > boot failed:
> > 
> > set hw.x2apic_enable=0
> > loading required module 'krpc'
> > /boot/kernel.VSTREAM/krpc.ko size 0x2a210 at 0x134e000
> > loading required module 'opensolaris'
> >  ^@/boot/kernel.VSTREAM/opensolaris.ko size 0xadb8 at 0x1379000
> > /boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x1384000
> > can't find 'if_ixgbe'
> > /boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x13ee000^M ^@
> > +/boot/kernel.VSTREAM/ukbd.ko size 0xe128 at 0x1404000
> > loading required module 'usb'
> > /boot/kernel.VSTREAM/usb.ko size 0x458b0 at 0x1413000^M|
> >  
> > /boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x1459000
> > /boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x1464000
> > /boot/kernel.VSTREAM/uhci.ko size 0xd508 at 0x1467000
> > /boot/kernel.VSTREAM/ohci.ko size 0xc9d0 at 0x1475000^M
> > /boot/kernel.VSTREAM/ehci.ko size 0xfc40 at 0x1482000
> > /boot/kernel.VSTREAM/xhci.ko size 0x11068 at 0x1492000
> > /boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x14a4000
> > Booting...
> > Copyright (c) 1992-2016 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> > The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 11.0-RELEASE-p305117 #0: Mon Sep 12 20:38:53 MSK 2016
> > s...@edge21.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
> > FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
> > 3.8.0)
> > VT(vga): text 80x25
> > CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
> >   Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
> >   
> > Features=0xbfebfbff
> >   
> > Features2=0x7ffefbff
> >   AMD Features=0x2c100800
> >   AMD Features2=0x121
> >   Structured Extended 
> > Features=0x21cbfbb
> >   XSAVE Features=0x1
> >   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> >   TSC: P-state invariant, performance statistics
> > real memory  = 137438953472 (131072 MB)
> > avail memory = 133407973376 (127227 MB)
> > Event timer "LAPIC" quality 600
> > ACPI APIC Table: 
> > boot_cpu_id = 255
> > kernel trap 12 with interrupts disabled
> > 
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = ff
> > fault virtual address   = 0x0
> > fault code  = supervisor read data, page not present
> > instruction pointer = 0x20:0x80537e74
> > stack pointer   = 0x28:0x814b3a60
> > frame pointer   = 0x28:0x814b3a70
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags= resume, IOPL = 0
> > current process = 0 ()
> > trap number = 12
> > panic: page fault
> > cpuid = 0
> > KDB: stack backtrace:
> > #0 0x805272e7 at kdb_backtrace+0x67
> > #1 0x804dd662 at vpanic+0x182
> > #2 0x804dd4d3 at panic+0x43
> > #3 0x807a37a1 at trap_fatal+0x351
> > #4 0x807a3993 at trap_pfault+0x1e3
> > #5 0x807a2f1c at trap+0x26c
> > #6 0x80787ca1 at calltrap+0x8
> > #7 0x8083b53a at topo_probe+0x61a
> > #8 0x8078fe93 at cpu_mp_start+0x1c3
> > #9 0x805382ca at mp_start+0x3a
> > #10 0x80465cd8 at mi_startup+0x118
> > #11 0x8028dfac at btext+0x2c
> > Uptime: 1s
> 
> Thank you!
> It seems like exactly the same behavior that happens when you toggle that BIOS
> option.
> 
> My theory is that in both cases, hw.x2apic_enable=0 and X2APIC_OPT_OUT is on,
> the BIOS turns on x2APIC mode and transitions to OS in that mode.
> In the case when X2APIC_OPT_OUT is on it's clearly a BIOS bug.
> But maybe we could do a little bit better in both cases.  At the very least we
> could detect the situation and panic with a helpful message (e.g. "x2APIC mode
> is disabled but turn on by BIOS").  Perhaps we could even try to downgrade to
> xAPIC mode.

If hw.x2apic_enable=0 and machine booted to the stage of topo probe,
BIOS definitely did not made hand-off with x2APIC enabled.  Any access
to the LAPIC registers page in x2APIC mode faults.

And X2APIC_OPT_OUT also does not result in x2APIC mode hand-off, for the
same reason.  System would panic much earlier, while dmesg indicates that
LAPIC ICR calibration was succesful.

It is invalid 

Re: X2APIC support

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 05:54:26PM +0300, Andriy Gapon wrote:

> On 13/09/2016 17:21, Slawa Olhovchenkov wrote:
> > boot failed:
> > 
> > set hw.x2apic_enable=0
> > loading required module 'krpc'
> > /boot/kernel.VSTREAM/krpc.ko size 0x2a210 at 0x134e000
> > loading required module 'opensolaris'
> >  ^@/boot/kernel.VSTREAM/opensolaris.ko size 0xadb8 at 0x1379000
> > /boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x1384000
> > can't find 'if_ixgbe'
> > /boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x13ee000^M ^@
> > +/boot/kernel.VSTREAM/ukbd.ko size 0xe128 at 0x1404000
> > loading required module 'usb'
> > /boot/kernel.VSTREAM/usb.ko size 0x458b0 at 0x1413000^M|
> >  
> > /boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x1459000
> > /boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x1464000
> > /boot/kernel.VSTREAM/uhci.ko size 0xd508 at 0x1467000
> > /boot/kernel.VSTREAM/ohci.ko size 0xc9d0 at 0x1475000^M
> > /boot/kernel.VSTREAM/ehci.ko size 0xfc40 at 0x1482000
> > /boot/kernel.VSTREAM/xhci.ko size 0x11068 at 0x1492000
> > /boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x14a4000
> > Booting...
> > Copyright (c) 1992-2016 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> > The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 11.0-RELEASE-p305117 #0: Mon Sep 12 20:38:53 MSK 2016
> > s...@edge21.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
> > FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
> > 3.8.0)
> > VT(vga): text 80x25
> > CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
> >   Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
> >   
> > Features=0xbfebfbff
> >   
> > Features2=0x7ffefbff
> >   AMD Features=0x2c100800
> >   AMD Features2=0x121
> >   Structured Extended 
> > Features=0x21cbfbb
> >   XSAVE Features=0x1
> >   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
> >   TSC: P-state invariant, performance statistics
> > real memory  = 137438953472 (131072 MB)
> > avail memory = 133407973376 (127227 MB)
> > Event timer "LAPIC" quality 600
> > ACPI APIC Table: 
> > boot_cpu_id = 255
> > kernel trap 12 with interrupts disabled
> > 
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = ff
> > fault virtual address   = 0x0
> > fault code  = supervisor read data, page not present
> > instruction pointer = 0x20:0x80537e74
> > stack pointer   = 0x28:0x814b3a60
> > frame pointer   = 0x28:0x814b3a70
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags= resume, IOPL = 0
> > current process = 0 ()
> > trap number = 12
> > panic: page fault
> > cpuid = 0
> > KDB: stack backtrace:
> > #0 0x805272e7 at kdb_backtrace+0x67
> > #1 0x804dd662 at vpanic+0x182
> > #2 0x804dd4d3 at panic+0x43
> > #3 0x807a37a1 at trap_fatal+0x351
> > #4 0x807a3993 at trap_pfault+0x1e3
> > #5 0x807a2f1c at trap+0x26c
> > #6 0x80787ca1 at calltrap+0x8
> > #7 0x8083b53a at topo_probe+0x61a
> > #8 0x8078fe93 at cpu_mp_start+0x1c3
> > #9 0x805382ca at mp_start+0x3a
> > #10 0x80465cd8 at mi_startup+0x118
> > #11 0x8028dfac at btext+0x2c
> > Uptime: 1s
> 
> Thank you!
> It seems like exactly the same behavior that happens when you toggle that BIOS
> option.
> 
> My theory is that in both cases, hw.x2apic_enable=0 and X2APIC_OPT_OUT is on,
> the BIOS turns on x2APIC mode and transitions to OS in that mode.
> In the case when X2APIC_OPT_OUT is on it's clearly a BIOS bug.
> But maybe we could do a little bit better in both cases.  At the very least we
> could detect the situation and panic with a helpful message (e.g. "x2APIC mode
> is disabled but turn on by BIOS").  Perhaps we could even try to downgrade to
> xAPIC mode.

Goggling on X2APIC_OPT_OUT take same result: other OS in this case
downgrade to xAPIC mode.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 13/09/2016 17:21, Slawa Olhovchenkov wrote:
> boot failed:
> 
> set hw.x2apic_enable=0
> loading required module 'krpc'
> /boot/kernel.VSTREAM/krpc.ko size 0x2a210 at 0x134e000
> loading required module 'opensolaris'
>  ^@/boot/kernel.VSTREAM/opensolaris.ko size 0xadb8 at 0x1379000
> /boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x1384000
> can't find 'if_ixgbe'
> /boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x13ee000^M ^@
> +/boot/kernel.VSTREAM/ukbd.ko size 0xe128 at 0x1404000
> loading required module 'usb'
> /boot/kernel.VSTREAM/usb.ko size 0x458b0 at 0x1413000^M|
>  
> /boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x1459000
> /boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x1464000
> /boot/kernel.VSTREAM/uhci.ko size 0xd508 at 0x1467000
> /boot/kernel.VSTREAM/ohci.ko size 0xc9d0 at 0x1475000^M
> /boot/kernel.VSTREAM/ehci.ko size 0xfc40 at 0x1482000
> /boot/kernel.VSTREAM/xhci.ko size 0x11068 at 0x1492000
> /boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x14a4000
> Booting...
> Copyright (c) 1992-2016 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.0-RELEASE-p305117 #0: Mon Sep 12 20:38:53 MSK 2016
> s...@edge21.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
> FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
> 3.8.0)
> VT(vga): text 80x25
> CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
>   
> Features=0xbfebfbff
>   
> Features2=0x7ffefbff
>   AMD Features=0x2c100800
>   AMD Features2=0x121
>   Structured Extended 
> Features=0x21cbfbb
>   XSAVE Features=0x1
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
>   TSC: P-state invariant, performance statistics
> real memory  = 137438953472 (131072 MB)
> avail memory = 133407973376 (127227 MB)
> Event timer "LAPIC" quality 600
> ACPI APIC Table: 
> boot_cpu_id = 255
> kernel trap 12 with interrupts disabled
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = ff
> fault virtual address   = 0x0
> fault code  = supervisor read data, page not present
> instruction pointer = 0x20:0x80537e74
> stack pointer   = 0x28:0x814b3a60
> frame pointer   = 0x28:0x814b3a70
> code segment= base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags= resume, IOPL = 0
> current process = 0 ()
> trap number = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> #0 0x805272e7 at kdb_backtrace+0x67
> #1 0x804dd662 at vpanic+0x182
> #2 0x804dd4d3 at panic+0x43
> #3 0x807a37a1 at trap_fatal+0x351
> #4 0x807a3993 at trap_pfault+0x1e3
> #5 0x807a2f1c at trap+0x26c
> #6 0x80787ca1 at calltrap+0x8
> #7 0x8083b53a at topo_probe+0x61a
> #8 0x8078fe93 at cpu_mp_start+0x1c3
> #9 0x805382ca at mp_start+0x3a
> #10 0x80465cd8 at mi_startup+0x118
> #11 0x8028dfac at btext+0x2c
> Uptime: 1s

Thank you!
It seems like exactly the same behavior that happens when you toggle that BIOS
option.

My theory is that in both cases, hw.x2apic_enable=0 and X2APIC_OPT_OUT is on,
the BIOS turns on x2APIC mode and transitions to OS in that mode.
In the case when X2APIC_OPT_OUT is on it's clearly a BIOS bug.
But maybe we could do a little bit better in both cases.  At the very least we
could detect the situation and panic with a helpful message (e.g. "x2APIC mode
is disabled but turn on by BIOS").  Perhaps we could even try to downgrade to
xAPIC mode.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 03:57:39PM +0300, Andriy Gapon wrote:

> On 13/09/2016 15:42, Slawa Olhovchenkov wrote:
> > On Tue, Sep 13, 2016 at 03:38:17PM +0300, Andriy Gapon wrote:
> > 
> >> On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
> >>> On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
> >>>
>  On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> > boot_cpu_id = 255
> 
>  I think that this points towards the BIOS not configuring the BSP LAPIC
>  correctly when you select that combination of BIOS options.
>  That's weird, but I fail to find any other explanation.
> 
>  If wonder what would happen if you disable X2APIC_OPT_OUT and set
>  hw.x2apic_enable=0 in loader.conf or at the loader prompt.
> >>>
> >>> Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
> >>>
> >>
> >> This doesn't answer my question and doesn't add any new information, 
> >> correct?
> > 
> > I am don't see any question. I am see only suggestion of workaround.
> > Sorry if missunderstund.
> > You need some additional debug print?
> > 
> 
> I already knew from you that not enabling X2APIC_OPT_OUT fixed the problem.
> "If wonder what would happen if you disable X2APIC_OPT_OUT and set
> hw.x2apic_enable=0 in loader.conf or at the loader prompt."
> This was a question even though there is no question mark.

boot failed:

set hw.x2apic_enable=0
loading required module 'krpc'
/boot/kernel.VSTREAM/krpc.ko size 0x2a210 at 0x134e000
loading required module 'opensolaris'
 ^@/boot/kernel.VSTREAM/opensolaris.ko size 0xadb8 at 0x1379000
/boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x1384000
can't find 'if_ixgbe'
/boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x13ee000^M ^@
+/boot/kernel.VSTREAM/ukbd.ko size 0xe128 at 0x1404000
loading required module 'usb'
/boot/kernel.VSTREAM/usb.ko size 0x458b0 at 0x1413000^M|
 
/boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x1459000
/boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x1464000
/boot/kernel.VSTREAM/uhci.ko size 0xd508 at 0x1467000
/boot/kernel.VSTREAM/ohci.ko size 0xc9d0 at 0x1475000^M
/boot/kernel.VSTREAM/ehci.ko size 0xfc40 at 0x1482000
/boot/kernel.VSTREAM/xhci.ko size 0x11068 at 0x1492000
/boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x14a4000
Booting...
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p305117 #0: Mon Sep 12 20:38:53 MSK 2016
s...@edge21.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
3.8.0)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
  
Features=0xbfebfbff
  
Features2=0x7ffefbff
  AMD Features=0x2c100800
  AMD Features2=0x121
  Structured Extended 
Features=0x21cbfbb
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
boot_cpu_id = 255
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = ff
fault virtual address   = 0x0
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x80537e74
stack pointer   = 0x28:0x814b3a60
frame pointer   = 0x28:0x814b3a70
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0x805272e7 at kdb_backtrace+0x67
#1 0x804dd662 at vpanic+0x182
#2 0x804dd4d3 at panic+0x43
#3 0x807a37a1 at trap_fatal+0x351
#4 0x807a3993 at trap_pfault+0x1e3
#5 0x807a2f1c at trap+0x26c
#6 0x80787ca1 at calltrap+0x8
#7 0x8083b53a at topo_probe+0x61a
#8 0x8078fe93 at cpu_mp_start+0x1c3
#9 0x805382ca at mp_start+0x3a
#10 0x80465cd8 at mi_startup+0x118
#11 0x8028dfac at btext+0x2c
Uptime: 1s
___

Re: X2APIC support

[Andriy Gapon]

On 13/09/2016 15:42, Slawa Olhovchenkov wrote:
> On Tue, Sep 13, 2016 at 03:38:17PM +0300, Andriy Gapon wrote:
> 
>> On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
>>> On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
>>>
 On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> boot_cpu_id = 255

 I think that this points towards the BIOS not configuring the BSP LAPIC
 correctly when you select that combination of BIOS options.
 That's weird, but I fail to find any other explanation.

 If wonder what would happen if you disable X2APIC_OPT_OUT and set
 hw.x2apic_enable=0 in loader.conf or at the loader prompt.
>>>
>>> Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
>>>
>>
>> This doesn't answer my question and doesn't add any new information, correct?
> 
> I am don't see any question. I am see only suggestion of workaround.
> Sorry if missunderstund.
> You need some additional debug print?
> 

I already knew from you that not enabling X2APIC_OPT_OUT fixed the problem.
"If wonder what would happen if you disable X2APIC_OPT_OUT and set
hw.x2apic_enable=0 in loader.conf or at the loader prompt."
This was a question even though there is no question mark.


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 03:38:17PM +0300, Andriy Gapon wrote:

> On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
> > On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
> > 
> >> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> >>> boot_cpu_id = 255
> >>
> >> I think that this points towards the BIOS not configuring the BSP LAPIC
> >> correctly when you select that combination of BIOS options.
> >> That's weird, but I fail to find any other explanation.
> >>
> >> If wonder what would happen if you disable X2APIC_OPT_OUT and set
> >> hw.x2apic_enable=0 in loader.conf or at the loader prompt.
> > 
> > Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
> > 
> 
> This doesn't answer my question and doesn't add any new information, correct?

I am don't see any question. I am see only suggestion of workaround.
Sorry if missunderstund.
You need some additional debug print?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
> On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
> 
>> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
>>> boot_cpu_id = 255
>>
>> I think that this points towards the BIOS not configuring the BSP LAPIC
>> correctly when you select that combination of BIOS options.
>> That's weird, but I fail to find any other explanation.
>>
>> If wonder what would happen if you disable X2APIC_OPT_OUT and set
>> hw.x2apic_enable=0 in loader.conf or at the loader prompt.
> 
> Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
> 

This doesn't answer my question and doesn't add any new information, correct?

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:

> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> > boot_cpu_id = 255
> 
> I think that this points towards the BIOS not configuring the BSP LAPIC
> correctly when you select that combination of BIOS options.
> That's weird, but I fail to find any other explanation.
> 
> If wonder what would happen if you disable X2APIC_OPT_OUT and set
> hw.x2apic_enable=0 in loader.conf or at the loader prompt.

Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> boot_cpu_id = 255

I think that this points towards the BIOS not configuring the BSP LAPIC
correctly when you select that combination of BIOS options.
That's weird, but I fail to find any other explanation.

If wonder what would happen if you disable X2APIC_OPT_OUT and set
hw.x2apic_enable=0 in loader.conf or at the loader prompt.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Mon, Sep 12, 2016 at 08:05:33PM +0300, Andriy Gapon wrote:

> On 12/09/2016 19:44, Slawa Olhovchenkov wrote:
> > I am not kernel developer: please point what I am need insert and file
> > for edit.
> 
> In sys/amd64/amd64/mp_machdep.c (assuming you use amd64), in function
> cpu_mp_start, in this place
>   /* Set boot_cpu_id if needed. */
>   if (boot_cpu_id == -1) {
>   boot_cpu_id = PCPU_GET(apic_id);
>   cpu_info[boot_cpu_id].cpu_bsp = 1;
>   } else
> 
> right after boot_cpu_id = PCPU_GET(apic_id) line please insert
>   printf("boot_cpu_id = %d\n", boot_cpu_id);

VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
  
Features=0xbfebfbff
  
Features2=0x7ffefbff
  AMD Features=0x2c100800
  AMD Features2=0x121
  Structured Extended 
Features=0x21cbfbb
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
boot_cpu_id = 255
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = ff
fault virtual address   = 0x0
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x80537e74
stack pointer   = 0x28:0x814b3a60
frame pointer   = 0x28:0x814b3a70
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0x805272e7 at kdb_backtrace+0x67
#1 0x804dd662 at vpanic+0x182
#2 0x804dd4d3 at panic+0x43
#3 0x807a37a1 at trap_fatal+0x351
#4 0x807a3993 at trap_pfault+0x1e3
#5 0x807a2f1c at trap+0x26c
#6 0x80787ca1 at calltrap+0x8
#7 0x8083b53a at topo_probe+0x61a
#8 0x8078fe93 at cpu_mp_start+0x1c3
#9 0x805382ca at mp_start+0x3a
#10 0x80465cd8 at mi_startup+0x118
#11 0x8028dfac at btext+0x2c
Uptime: 1s
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 12/09/2016 19:44, Slawa Olhovchenkov wrote:
> I am not kernel developer: please point what I am need insert and file
> for edit.

In sys/amd64/amd64/mp_machdep.c (assuming you use amd64), in function
cpu_mp_start, in this place
/* Set boot_cpu_id if needed. */
if (boot_cpu_id == -1) {
boot_cpu_id = PCPU_GET(apic_id);
cpu_info[boot_cpu_id].cpu_bsp = 1;
} else

right after boot_cpu_id = PCPU_GET(apic_id) line please insert
printf("boot_cpu_id = %d\n", boot_cpu_id);

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Mon, Sep 12, 2016 at 04:44:09PM +0300, Andriy Gapon wrote:

> On 12/09/2016 12:39, Slawa Olhovchenkov wrote:
> > On Sun, Sep 04, 2016 at 08:14:07PM +0300, Andriy Gapon wrote:
> > 
> >> On 04/09/2016 19:29, Konstantin Belousov wrote:
> >>> This is possible, of course.  But it would not affect "SMP: Added CPU ..."
> >>> lines.
> >>
> >> Well, looking at the code it seems that only if mptable is used, then those
> >> lines are expected to correctly identify a BSP.  With MADT there is no
> >> information to identify the BSP and that is supposed to happen in 
> >> cpu_mp_start().
> >>
> >>
> >> static void
> >> madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
> >> {
> >> struct lapic_info *la;
> >>
> >> /*
> >>  * The MADT does not include a BSP flag, so we have to let the
> >>  * MP code figure out which CPU is the BSP on its own.
> >>  */
> >> ...
> >>
> >> In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
> >> And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
> >> problematic configuration.
> > 
> > For next day or two I am have new server with same hardware before put
> > in prodution.
> > Can I do some test for you?
> > 
> 
> >From my earlier email:
> "my guess can be checked by adding a printf to cpu_mp_start() right after
> boot_cpu_id assignment".

I am not kernel developer: please point what I am need insert and file
for edit.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 12/09/2016 12:39, Slawa Olhovchenkov wrote:
> On Sun, Sep 04, 2016 at 08:14:07PM +0300, Andriy Gapon wrote:
> 
>> On 04/09/2016 19:29, Konstantin Belousov wrote:
>>> This is possible, of course.  But it would not affect "SMP: Added CPU ..."
>>> lines.
>>
>> Well, looking at the code it seems that only if mptable is used, then those
>> lines are expected to correctly identify a BSP.  With MADT there is no
>> information to identify the BSP and that is supposed to happen in 
>> cpu_mp_start().
>>
>>
>> static void
>> madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
>> {
>> struct lapic_info *la;
>>
>> /*
>>  * The MADT does not include a BSP flag, so we have to let the
>>  * MP code figure out which CPU is the BSP on its own.
>>  */
>> ...
>>
>> In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
>> And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
>> problematic configuration.
> 
> For next day or two I am have new server with same hardware before put
> in prodution.
> Can I do some test for you?
> 

>From my earlier email:
"my guess can be checked by adding a printf to cpu_mp_start() right after
boot_cpu_id assignment".

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Sun, Sep 04, 2016 at 08:14:07PM +0300, Andriy Gapon wrote:

> On 04/09/2016 19:29, Konstantin Belousov wrote:
> > This is possible, of course.  But it would not affect "SMP: Added CPU ..."
> > lines.
> 
> Well, looking at the code it seems that only if mptable is used, then those
> lines are expected to correctly identify a BSP.  With MADT there is no
> information to identify the BSP and that is supposed to happen in 
> cpu_mp_start().
> 
> 
> static void
> madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
> {
> struct lapic_info *la;
> 
> /*
>  * The MADT does not include a BSP flag, so we have to let the
>  * MP code figure out which CPU is the BSP on its own.
>  */
> ...
> 
> In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
> And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
> problematic configuration.

For next day or two I am have new server with same hardware before put
in prodution.
Can I do some test for you?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Tue, Sep 06, 2016 at 05:08:28PM +0300, Konstantin Belousov wrote:

> On Tue, Sep 06, 2016 at 04:13:05PM +0300, Slawa Olhovchenkov wrote:
> > On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> > 
> > > On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > > > DMAR: Found table at 0x79b32798
> > > > x2APIC available but disabled by DMAR table
> > > 
> > > > Event timer "LAPIC" quality 600
> > > > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > > > ACPI APIC Table: 
> > 
> > Is this significant? On carsh case present 'x2APIC available but
> > disabled by DMAR table' and 'LAPIC: ipi_wait() us multiplier 1 (r
> > 116268019 tsc 2200043851)'
> Significant for what ?  First message mostly repeat the setting in the

For cause of crash.
May be not all of code correct understund this combinations of setting?

> BIOS which you frobbed.  The second message indicates that xAPIC
> timings for ICR reads were calibrated, i.e. x2APIC mode was indeed
> disabled.

OK

> > 
> > On successful boot both messages absent:
> > 
> > Table 'DMAR' at 0x79b32798
> > DMAR: Found table at 0x79b32798
> > Event timer "LAPIC" quality 600
> > ACPI APIC Table: 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Tue, Sep 06, 2016 at 04:13:05PM +0300, Slawa Olhovchenkov wrote:
> On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> 
> > On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > > DMAR: Found table at 0x79b32798
> > > x2APIC available but disabled by DMAR table
> > 
> > > Event timer "LAPIC" quality 600
> > > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > > ACPI APIC Table: 
> 
> Is this significant? On carsh case present 'x2APIC available but
> disabled by DMAR table' and 'LAPIC: ipi_wait() us multiplier 1 (r
> 116268019 tsc 2200043851)'
Significant for what ?  First message mostly repeat the setting in the
BIOS which you frobbed.  The second message indicates that xAPIC
timings for ICR reads were calibrated, i.e. x2APIC mode was indeed
disabled.

> 
> On successful boot both messages absent:
> 
> Table 'DMAR' at 0x79b32798
> DMAR: Found table at 0x79b32798
> Event timer "LAPIC" quality 600
> ACPI APIC Table: 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:

> On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > DMAR: Found table at 0x79b32798
> > x2APIC available but disabled by DMAR table
> 
> > Event timer "LAPIC" quality 600
> > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > ACPI APIC Table: 

Is this significant? On carsh case present 'x2APIC available but
disabled by DMAR table' and 'LAPIC: ipi_wait() us multiplier 1 (r
116268019 tsc 2200043851)'

On successful boot both messages absent:

Table 'DMAR' at 0x79b32798
DMAR: Found table at 0x79b32798
Event timer "LAPIC" quality 600
ACPI APIC Table: 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 04/09/2016 19:29, Konstantin Belousov wrote:
> This is possible, of course.  But it would not affect "SMP: Added CPU ..."
> lines.

Well, looking at the code it seems that only if mptable is used, then those
lines are expected to correctly identify a BSP.  With MADT there is no
information to identify the BSP and that is supposed to happen in 
cpu_mp_start().


static void
madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
{
struct lapic_info *la;

/*
 * The MADT does not include a BSP flag, so we have to let the
 * MP code figure out which CPU is the BSP on its own.
 */
...

In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
problematic configuration.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Sun, Sep 04, 2016 at 06:49:43PM +0300, Andriy Gapon wrote:
> On 04/09/2016 18:14, Konstantin Belousov wrote:
> > On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> >> Kostik, I see one strange thing which is common to both successful and
> >> unsuccessful configurations.  All "SMP: Added CPU..." lines have "AP" in 
> >> them.
> >> It seems like the platform does not tell explicitly tell which CPU is the 
> >> BSP,
> >> see cpu_add() function.  This can break quite a few assumption.  And I am 
> >> not
> >> even sure how the successful scenario works.
> >> Ah... I see that there is a backup code in cpu_mp_start() where 
> >> boot_cpu_id is
> >> set based on the current CPU's Local APIC ID.  I suspect then that this
> >> information is incorrect in the failing case.
> >>
> > Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
> > are initialized. BIOS might reprogram LAPIC Ids, so reading from
> > CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
> > it might be in the x2APIC state, since 8 bits are not enough for 32bit
> > x2APIC Id.
> 
> Hmm, I am not sure how what you are saying here is relevant to the problem.
> I believe that cpu_mp_start() is executed (on the BSP) after the BSP's LAPIC 
> is
> initialized.  So, the code should just work.
The order is madt_probe()->madt_probe_cpus()->madt_setup_local().

madt_probe() and madt_probe_cpus() are called from apic_init9) at
SI_SUB_TUNABLEs, and madt_probe_cpus()->madt_add_cpu()->lapic_create()->
cpu_add() is how the SMP: ... lines are printed.

The madt_setup_local() code is called from apic_setup_local() at SI_SUB_CPU,
this is where APIC window is mapped or x2APIC mode is enabled by the call
to native_lapic_init().

You cannot get BSP LAPIC Id earlier than native_lapic_init() was executed.

mp_start()->cpu_mp_start() is called right after the madt_setup_local().

> 
> My theory was that the BSP's LAPIC ID was incorrectly programmed by BIOS 
> (firmware).

This is possible, of course.  But it would not affect "SMP: Added CPU ..."
lines.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Andriy Gapon]

On 04/09/2016 18:14, Konstantin Belousov wrote:
> On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
>> Kostik, I see one strange thing which is common to both successful and
>> unsuccessful configurations.  All "SMP: Added CPU..." lines have "AP" in 
>> them.
>> It seems like the platform does not tell explicitly tell which CPU is the 
>> BSP,
>> see cpu_add() function.  This can break quite a few assumption.  And I am not
>> even sure how the successful scenario works.
>> Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id 
>> is
>> set based on the current CPU's Local APIC ID.  I suspect then that this
>> information is incorrect in the failing case.
>>
> Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
> are initialized. BIOS might reprogram LAPIC Ids, so reading from
> CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
> it might be in the x2APIC state, since 8 bits are not enough for 32bit
> x2APIC Id.

Hmm, I am not sure how what you are saying here is relevant to the problem.
I believe that cpu_mp_start() is executed (on the BSP) after the BSP's LAPIC is
initialized.  So, the code should just work.

My theory was that the BSP's LAPIC ID was incorrectly programmed by BIOS 
(firmware).

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > DMAR: Found table at 0x79b32798
> > x2APIC available but disabled by DMAR table
> 
> > Event timer "LAPIC" quality 600
> > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > ACPI APIC Table: 
> > Package ID shift: 5
> > L3 cache ID shift: 5
> > L2 cache ID shift: 1
> > L1 cache ID shift: 1
> > Core ID shift: 1
> > kernel trap 12 with interrupts disabled
> > 
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = ff
> 
> > fault virtual address   = 0x0
> > fault code  = supervisor read data, page not present
> > instruction pointer = 0x20:0x80537e74
> > stack pointer   = 0x28:0x814b4a60
> > frame pointer   = 0x28:0x814b4a70
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags= resume, IOPL = 0
> > current process = 0 ()
> > trap number = 12
> > panic: page fault
> > cpuid = 0
> > KDB: stack backtrace:
> > #0 0x805272e7 at kdb_backtrace+0x67
> > #1 0x804dd662 at vpanic+0x182
> > #2 0x804dd4d3 at panic+0x43
> > #3 0x807a3791 at trap_fatal+0x351
> > #4 0x807a3983 at trap_pfault+0x1e3
> > #5 0x807a2f0c at trap+0x26c
> > #6 0x80787ca1 at calltrap+0x8
> > #7 0x8083b52a at topo_probe+0x61a
> 
> Interesting.  Could you please do 'list *topo_probe+0x61a' in kgdb, so that I
> can see what code is being executed when the trap happens?  Also, disassembly 
> of
> the function could be useful as well.
> 
> Wait...
> Kostik, I see one strange thing which is common to both successful and
> unsuccessful configurations.  All "SMP: Added CPU..." lines have "AP" in them.
> It seems like the platform does not tell explicitly tell which CPU is the BSP,
> see cpu_add() function.  This can break quite a few assumption.  And I am not
> even sure how the successful scenario works.
> Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is
> set based on the current CPU's Local APIC ID.  I suspect then that this
> information is incorrect in the failing case.
> 
Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
are initialized. BIOS might reprogram LAPIC Ids, so reading from
CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
it might be in the x2APIC state, since 8 bits are not enough for 32bit
x2APIC Id.

> Slawa,
> my guess can be checked by adding a printf to cpu_mp_start() right after
> boot_cpu_id assignment.
> 
> > #8 0x8078fe81 at cpu_mp_start+0x1b1
> > #9 0x805382ca at mp_start+0x3a
> > #10 0x80465cd8 at mi_startup+0x118
> > #11 0x8028dfac at btext+0x2c
> > Uptime: 1s
> 
> 
> -- 
> Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:

> On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > DMAR: Found table at 0x79b32798
> > x2APIC available but disabled by DMAR table
> 
> > Event timer "LAPIC" quality 600
> > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > ACPI APIC Table: 
> > Package ID shift: 5
> > L3 cache ID shift: 5
> > L2 cache ID shift: 1
> > L1 cache ID shift: 1
> > Core ID shift: 1
> > kernel trap 12 with interrupts disabled
> > 
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = ff
> 
> > fault virtual address   = 0x0
> > fault code  = supervisor read data, page not present
> > instruction pointer = 0x20:0x80537e74
> > stack pointer   = 0x28:0x814b4a60
> > frame pointer   = 0x28:0x814b4a70
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags= resume, IOPL = 0
> > current process = 0 ()
> > trap number = 12
> > panic: page fault
> > cpuid = 0
> > KDB: stack backtrace:
> > #0 0x805272e7 at kdb_backtrace+0x67
> > #1 0x804dd662 at vpanic+0x182
> > #2 0x804dd4d3 at panic+0x43
> > #3 0x807a3791 at trap_fatal+0x351
> > #4 0x807a3983 at trap_pfault+0x1e3
> > #5 0x807a2f0c at trap+0x26c
> > #6 0x80787ca1 at calltrap+0x8
> > #7 0x8083b52a at topo_probe+0x61a
> 
> Interesting.  Could you please do 'list *topo_probe+0x61a' in kgdb, so that I

(kgdb) list *topo_probe+0x61a
0x8083b52a is in topo_probe (/usr/src/sys/x86/x86/mp_x86.c:540).
535 topo_layers[layer].subtype);
536 }
537 }
538
539 parent = _root;
540 for (layer = 0; layer < nlayers; ++layer) {
541 node_id = boot_cpu_id >> topo_layers[layer].id_shift;
542 node = topo_find_node_by_hwid(parent, node_id,
543 topo_layers[layer].type,
544 topo_layers[layer].subtype);
Current language:  auto; currently minimal

> can see what code is being executed when the trap happens?  Also, disassembly 
> of
> the function could be useful as well.

(kgdb) x/40i *topo_probe+0x600
0x8083b510 :   and$0xf8,%al
0x8083b512 :   movslq -0x4(%r12),%rcx
0x8083b517 :   mov%rbx,%rdi
0x8083b51a :   callq  0x80537e30 

0x8083b51f :   mov%rax,%rbx
0x8083b522 :   mov%rbx,%rdi
0x8083b525 :   callq  0x80537e70 

0x8083b52a :   add$0xc,%r12
0x8083b52e :   dec%r14d
0x8083b531 :   jne0x8083b500 

0x8083b533 :   movb   $0x1,0x80dfa664
0x8083b53b :   add$0x68,%rsp
0x8083b53f :   pop%rbx
0x8083b540 :   pop%r12
0x8083b542 :   pop%r13
0x8083b544 :   pop%r14
0x8083b546 :   pop%r15
0x8083b548 :   pop%rbp
0x8083b549 :   retq
0x8083b54a :   nopw   0x0(%rax,%rax,1)


> Wait...
> Kostik, I see one strange thing which is common to both successful and
> unsuccessful configurations.  All "SMP: Added CPU..." lines have "AP" in them.

for #1..#23
no line 'SMP: AP CPU #0 Launched!'

> It seems like the platform does not tell explicitly tell which CPU is the BSP,
> see cpu_add() function.  This can break quite a few assumption.  And I am not
> even sure how the successful scenario works.

# mptable 

===

MPTable

---

MP Floating Pointer Structure:

  location: BIOS
  physical address: 0x000fd050
  signature:'_MP_'
  length:   16 bytes
  version:  1.4
  checksum: 0x27
  mode: Virtual Wire

---

MP Config Table Header:

  physical address: 0x000fcaa0
  signature:'PCMP'
  base table length:1228
  version:  1.4
  checksum: 0x95
  OEM ID:   'A M I'
  Product ID:   'ALASKA'
  OEM table pointer:0x
  OEM table size:   0
  entry count:  112
  local APIC address:   0xfee0
  extended 

Re: X2APIC support

[Andriy Gapon]

On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> DMAR: Found table at 0x79b32798
> x2APIC available but disabled by DMAR table

> Event timer "LAPIC" quality 600
> LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> ACPI APIC Table: 
> Package ID shift: 5
> L3 cache ID shift: 5
> L2 cache ID shift: 1
> L1 cache ID shift: 1
> Core ID shift: 1
> kernel trap 12 with interrupts disabled
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = ff

> fault virtual address   = 0x0
> fault code  = supervisor read data, page not present
> instruction pointer = 0x20:0x80537e74
> stack pointer   = 0x28:0x814b4a60
> frame pointer   = 0x28:0x814b4a70
> code segment= base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags= resume, IOPL = 0
> current process = 0 ()
> trap number = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> #0 0x805272e7 at kdb_backtrace+0x67
> #1 0x804dd662 at vpanic+0x182
> #2 0x804dd4d3 at panic+0x43
> #3 0x807a3791 at trap_fatal+0x351
> #4 0x807a3983 at trap_pfault+0x1e3
> #5 0x807a2f0c at trap+0x26c
> #6 0x80787ca1 at calltrap+0x8
> #7 0x8083b52a at topo_probe+0x61a

Interesting.  Could you please do 'list *topo_probe+0x61a' in kgdb, so that I
can see what code is being executed when the trap happens?  Also, disassembly of
the function could be useful as well.

Wait...
Kostik, I see one strange thing which is common to both successful and
unsuccessful configurations.  All "SMP: Added CPU..." lines have "AP" in them.
It seems like the platform does not tell explicitly tell which CPU is the BSP,
see cpu_add() function.  This can break quite a few assumption.  And I am not
even sure how the successful scenario works.
Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is
set based on the current CPU's Local APIC ID.  I suspect then that this
information is incorrect in the failing case.

Slawa,
my guess can be checked by adding a printf to cpu_mp_start() right after
boot_cpu_id assignment.

> #8 0x8078fe81 at cpu_mp_start+0x1b1
> #9 0x805382ca at mp_start+0x3a
> #10 0x80465cd8 at mi_startup+0x118
> #11 0x8028dfac at btext+0x2c
> Uptime: 1s


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[sthaug]

> > > Sorry, don't cleanly understund, what combination of BIOS setting I am 
> > > need to probe?
> > > And what I am need to check?
> > 
> > Set 'Hyper-Threading' to Enabled.
> > Set 'X2APIC_OPT_OUT' to Enabled.
> > Try to boot.
> 
> Crashed at same point.

A comment about X2APIC, on a different type of system: I had to disable
X2APIC on HP ProLiant DL360 Gen9 servers in order to avoid 10.3-STABLE
crashing during boot.

Steinar Haug, Nethelp consulting, sth...@nethelp.no
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 09:37:29PM +0200, sth...@nethelp.no wrote:

> > > > Sorry, don't cleanly understund, what combination of BIOS setting I am 
> > > > need to probe?
> > > > And what I am need to check?
> > > 
> > > Set 'Hyper-Threading' to Enabled.
> > > Set 'X2APIC_OPT_OUT' to Enabled.
> > > Try to boot.
> > 
> > Crashed at same point.
> 
> A comment about X2APIC, on a different type of system: I had to disable
> X2APIC on HP ProLiant DL360 Gen9 servers in order to avoid 10.3-STABLE
> crashing during boot.
> 
> Steinar Haug, Nethelp consulting, sth...@nethelp.no

In this test I am use 11.0
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 09:19:15PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 09:00:14PM +0300, Slawa Olhovchenkov wrote:
> > Sorry, don't cleanly understund, what combination of BIOS setting I am need 
> > to probe?
> > And what I am need to check?
> 
> Set 'Hyper-Threading' to Enabled.
> Set 'X2APIC_OPT_OUT' to Enabled.
> Try to boot.

Crashed at same point.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 09:00:14PM +0300, Slawa Olhovchenkov wrote:
> Sorry, don't cleanly understund, what combination of BIOS setting I am need 
> to probe?
> And what I am need to check?

Set 'Hyper-Threading' to Enabled.
Set 'X2APIC_OPT_OUT' to Enabled.
Try to boot.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 08:50:49PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 08:31:49PM +0300, Slawa Olhovchenkov wrote:
> > On Thu, Sep 01, 2016 at 08:26:32PM +0300, Konstantin Belousov wrote:
> > 
> > > On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> > > > On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> > > > As I point before: 'X2APIC_OPT_OUT Flag'.
> > > > Hidden before 'X2APIC' set to '[Enable]'
> > > > 
> > > > https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> > > > 
> > > > > Show complete verbose dmesg from the first line to backtrace.
> > > 
> > > And post the verbose dmesg for successfull boot as well, please.
> > 
> > http://m.uploadedit.com/ba3s/147275109329.txt
> 
> >From pure curiousity, if you enable hyperthreading, does system survives
> frobbing of X2APIC_OPT_OUT ?

Sorry, don't cleanly understund, what combination of BIOS setting I am need to 
probe?
And what I am need to check?

> Also, please show the output of acpidump -t.

/*
  RSD PTR: OEM=ALASKA, ACPI_Rev=2.0x (2)
XSDT=0x79ae50a0, length=36, cksum=197
 */
/*
  XSDT: Length=188, Revision=1, Checksum=37,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
Entries={ 0x79b189a8, 0x79b18ab8, 0x79b18c70, 
0x79b18cb8, 0x79b18d58, 0x79b18da0, 0x79b18de0, 
0x79b18e28, 0x79b18f58, 0x79b18f90, 0x79b18fd0, 
0x79b30020, 0x79b32670, 0x79b326d8,
  0x79b32798, 0x79b328e8, 0x79b32990, 
0x79b329c0, 0x79b32bf0 }
 */
/*
  FACP: Length=268, Revision=5, Checksum=66,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
FACS=0x79fa8f80, DSDT=0x79ae51f0
INT_MODEL=APIC
Preferred_PM_Profile=Enterprise Server (4)
SCI_INT=9
SMI_CMD=0xb2, ACPI_ENABLE=0xa0, ACPI_DISABLE=0xa1, S4BIOS_REQ=0x0
PSTATE_CNT=0x0
PM1a_EVT_BLK=0x400-0x403
PM1a_CNT_BLK=0x404-0x405
PM2_CNT_BLK=0x450-0x450
PM_TMR_BLK=0x408-0x40b
GPE0_BLK=0x420-0x42f
P_LVL2_LAT=101 us, P_LVL3_LAT=1001 us
FLUSH_SIZE=1024, FLUSH_STRIDE=16
DUTY_OFFSET=1, DUTY_WIDTH=3
DAY_ALRM=13, MON_ALRM=0, CENTURY=50
IAPC_BOOT_ARCH={LEGACY_DEVICES,NO_ASPM}

Flags={WBINVD,C1_SUPPORTED,SLEEP_BUTTON,S4_RTC_WAKE,RESET_REGISTER,PLATFORM_CLOCK}
RESET_REG=0xcf9:0[8] (IO), RESET_VALUE=0x6
 */
/*
  FACS: Length=64, HwSig=0x0ee12c64, Firm_Wake_Vec=0x
Global_Lock=
Flags=
Version=2
 */
/*
  DSDT: Length=210868, Revision=2, Checksum=0,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=INTL, Creator Revision=0x20091013
 */
/*
  APIC: Length=436, Revision=3, Checksum=176,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
Local APIC ADDR=0xfee0
Flags={PC-AT}

Type=Local APIC
ACPI CPU=0
Flags={ENABLED}
APIC ID=0

Type=Local APIC
ACPI CPU=2
Flags={ENABLED}
APIC ID=2

Type=Local APIC
ACPI CPU=4
Flags={ENABLED}
APIC ID=4

Type=Local APIC
ACPI CPU=6
Flags={ENABLED}
APIC ID=6

Type=Local APIC
ACPI CPU=8
Flags={ENABLED}
APIC ID=8

Type=Local APIC
ACPI CPU=10
Flags={ENABLED}
APIC ID=10

Type=Local APIC
ACPI CPU=16
Flags={ENABLED}
APIC ID=16

Type=Local APIC
ACPI CPU=18
Flags={ENABLED}
APIC ID=18

Type=Local APIC
ACPI CPU=20
Flags={ENABLED}
APIC ID=20

Type=Local APIC
ACPI CPU=22
Flags={ENABLED}
APIC ID=22

Type=Local APIC
ACPI CPU=24
Flags={ENABLED}
APIC ID=24

Type=Local APIC
ACPI CPU=26
Flags={ENABLED}
APIC ID=26

Type=Local APIC
ACPI CPU=32
Flags={ENABLED}
APIC ID=32

Type=Local APIC
ACPI CPU=34
Flags={ENABLED}
APIC ID=34

Type=Local APIC
ACPI CPU=36
Flags={ENABLED}
APIC ID=36

Type=Local APIC
ACPI CPU=38
Flags={ENABLED}
APIC ID=38

Type=Local APIC
ACPI CPU=40
Flags={ENABLED}
APIC ID=40

Type=Local APIC
ACPI CPU=42
Flags={ENABLED}
APIC ID=42

Type=Local APIC
ACPI CPU=48
Flags={ENABLED}
APIC ID=48

Type=Local APIC
ACPI CPU=50
Flags={ENABLED}
APIC ID=50

Type=Local APIC
ACPI CPU=52
Flags={ENABLED}
APIC ID=52

Type=Local APIC

Re: X2APIC support

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 08:31:49PM +0300, Slawa Olhovchenkov wrote:
> On Thu, Sep 01, 2016 at 08:26:32PM +0300, Konstantin Belousov wrote:
> 
> > On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> > > On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> > > As I point before: 'X2APIC_OPT_OUT Flag'.
> > > Hidden before 'X2APIC' set to '[Enable]'
> > > 
> > > https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> > > 
> > > > Show complete verbose dmesg from the first line to backtrace.
> > 
> > And post the verbose dmesg for successfull boot as well, please.
> 
> http://m.uploadedit.com/ba3s/147275109329.txt

>From pure curiousity, if you enable hyperthreading, does system survives
frobbing of X2APIC_OPT_OUT ?

Also, please show the output of acpidump -t.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 08:26:32PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> > On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> > As I point before: 'X2APIC_OPT_OUT Flag'.
> > Hidden before 'X2APIC' set to '[Enable]'
> > 
> > https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> > 
> > > Show complete verbose dmesg from the first line to backtrace.
> 
> And post the verbose dmesg for successfull boot as well, please.

http://m.uploadedit.com/ba3s/147275109329.txt
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> As I point before: 'X2APIC_OPT_OUT Flag'.
> Hidden before 'X2APIC' set to '[Enable]'
> 
> https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> 
> > Show complete verbose dmesg from the first line to backtrace.

And post the verbose dmesg for successfull boot as well, please.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 02:27:24PM +0300, Slawa Olhovchenkov wrote:
> > On Sat, Dec 12, 2015 at 03:35:13PM +0200, Konstantin Belousov wrote:
> > 
> > > On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> > > > Does STABLE support X2APIC?
> > > > I see X2APIC related commits in CURRENT, what is status for STABLE?
> > > > I am try to enable X2APIC support on X10DRi and see kernel trap on
> > > > boot.
> > > 
> > > x2APIC is only supported in HEAD.  The code to parse processor x2APIC
> > > structures in MADT was committed, but AFAIK never tested on real hardware.
> > > 
> > > There are no plans to merge this to stable/10.
> > 
> > I am try now on real hardware and got kernel fault at boot when set in
> > BIOS X2APIC_OPT_OUT Flag to [Enable].
> > https://s9.postimg.org/6mt8bu21b/fault2.jpg
> How do you know that the issue is due to that flag ?

Just by turn on and off this flag and try boot at both cases.

> I doubt that BIOS would provide direct knob to manage that flag, since 
> requesting that
> out-out only makes sense if BIOS is unable to handle x2APIC mode, e.g.
> in SMI handlers.  How precisely the BIOS setting called which you frobbed ?

As I point before: 'X2APIC_OPT_OUT Flag'.
Hidden before 'X2APIC' set to '[Enable]'

https://s15.postimg.org/3k5l7z397/bios_x2.jpg

> Show complete verbose dmesg from the first line to backtrace.

Booting...
\01;00Table 'FACP' at 0x79b189a8
Table 'APIC' at 0x79b18ab8
APIC: Found table at 0x79b18ab8
APIC: Using the MADT enumerator.
MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 2 ACPI ID 2: enabled
SMP: Added CPU 2 (AP)
MADT: Found CPU APIC ID 4 ACPI ID 4: enabled
SMP: Added CPU 4 (AP)
MADT: Found CPU APIC ID 6 ACPI ID 6: enabled
SMP: Added CPU 6 (AP)
MADT: Found CPU APIC ID 8 ACPI ID 8: enabled
SMP: Added CPU 8 (AP)
MADT: Found CPU APIC ID 10 ACPI ID 10: enabled
SMP: Added CPU 10 (AP)
MADT: Found CPU APIC ID 16 ACPI ID 16: enabled
SMP: Added CPU 16 (AP)
MADT: Found CPU APIC ID 18 ACPI ID 18: enabled
SMP: Added CPU 18 (AP)
MADT: Found CPU APIC ID 20 ACPI ID 20: enabled
SMP: Added CPU 20 (AP)
MADT: Found CPU APIC ID 22 ACPI ID 22: enabled
SMP: Added CPU 22 (AP)
MADT: Found CPU APIC ID 24 ACPI ID 24: enabled
SMP: Added CPU 24 (AP)
MADT: Found CPU APIC ID 26 ACPI ID 26: enabled
SMP: Added CPU 26 (AP)
MADT: Found CPU APIC ID 32 ACPI ID 32: enabled
SMP: Added CPU 32 (AP)
MADT: Found CPU APIC ID 34 ACPI ID 34: enabled
SMP: Added CPU 34 (AP)
MADT: Found CPU APIC ID 36 ACPI ID 36: enabled
SMP: Added CPU 36 (AP)
MADT: Found CPU APIC ID 38 ACPI ID 38: enabled
SMP: Added CPU 38 (AP)
MADT: Found CPU APIC ID 40 ACPI ID 40: enabled
SMP: Added CPU 40 (AP)
MADT: Found CPU APIC ID 42 ACPI ID 42: enabled
SMP: Added CPU 42 (AP)
MADT: Found CPU APIC ID 48 ACPI ID 48: enabled
SMP: Added CPU 48 (AP)
MADT: Found CPU APIC ID 50 ACPI ID 50: enabled
SMP: Added CPU 50 (AP)
MADT: Found CPU APIC ID 52 ACPI ID 52: enabled
SMP: Added CPU 52 (AP)
MADT: Found CPU APIC ID 54 ACPI ID 54: enabled
SMP: Added CPU 54 (AP)
MADT: Found CPU APIC ID 56 ACPI ID 56: enabled
SMP: Added CPU 56 (AP)
MADT: Found CPU APIC ID 58 ACPI ID 58: enabled
SMP: Added CPU 58 (AP)
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p305117 #0: Wed Aug 31 17:58:35 UTC 2016
s...@storage01.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 
3.8.0)
Table 'FACP' at 0x79b189a8
Table 'APIC' at 0x79b18ab8
Table 'FPDT' at 0x79b18c70
Table 'FIDT' at 0x79b18cb8
Table 'SPMI' at 0x79b18d58
Table 'MCFG' at 0x79b18da0
Table 'UEFI' at 0x79b18de0
Table 'MCEJ' at 0x79b18e28
Table 'HPET' at 0x79b18f58
Table 'WDDT' at 0x79b18f90
Table 'SSDT' at 0x79b18fd0
Table 'SSDT' at 0x79b30020
Table 'SSDT' at 0x79b32670
Table 'PRAD' at 0x79b326d8
Table 'DMAR' at 0x79b32798
Table 'HEST' at 0x79b328e8
Table 'BERT' at 0x79b32990
Table 'ERST' at 0x79b329c0
Table 'EINJ' at 0x79b32bf0
ACPI: No SRAT table found
PPIM 0: PA=0xb8000, VA=0x8161, size=0x8000, mode=0
VT(vga): text 80x25
Preloaded elf kernel "/boot/kernel.VSTREAM/kernel" at 0x814a8000.
Preloaded /boot/entropy "/boot/entropy" at 0x814a8ac0.
Preloaded elf obj module "/boot/kernel.VSTREAM/zfs.ko" at 0x814a8b10.
Preloaded elf obj module "/boot/kernel.VSTREAM/krpc.ko" at 0x814a9300.
Preloaded elf obj module "/boot/kernel.VSTREAM/opensolaris.ko" at 
0x814a9970.
Preloaded elf obj module "/boot/kernel.VSTREAM/if_igb.ko&quo

Re: X2APIC support

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 02:27:24PM +0300, Slawa Olhovchenkov wrote:
> On Sat, Dec 12, 2015 at 03:35:13PM +0200, Konstantin Belousov wrote:
> 
> > On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> > > Does STABLE support X2APIC?
> > > I see X2APIC related commits in CURRENT, what is status for STABLE?
> > > I am try to enable X2APIC support on X10DRi and see kernel trap on
> > > boot.
> > 
> > x2APIC is only supported in HEAD.  The code to parse processor x2APIC
> > structures in MADT was committed, but AFAIK never tested on real hardware.
> > 
> > There are no plans to merge this to stable/10.
> 
> I am try now on real hardware and got kernel fault at boot when set in
> BIOS X2APIC_OPT_OUT Flag to [Enable].
> https://s9.postimg.org/6mt8bu21b/fault2.jpg
How do you know that the issue is due to that flag ? I doubt that BIOS
would provide direct knob to manage that flag, since requesting that
out-out only makes sense if BIOS is unable to handle x2APIC mode, e.g.
in SMI handlers.  How precisely the BIOS setting called which you frobbed ?

Show complete verbose dmesg from the first line to backtrace.

> 
> 0x80537e74 is /usr/src/sys/kern/subr_smp.c:959
And this is a topology probe code, which should not be affected by the
x2APIC mode at all, except possibly due to longer format of the APIC IDs.

I added Andrey to Cc:, he might give some more guidance.
> 
> Boot success if X2APIC_OPT_OUT Flag set to [Disable]
> Supermicro X10DRi, dual E5-2650v4
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Slawa Olhovchenkov]

On Sat, Dec 12, 2015 at 03:35:13PM +0200, Konstantin Belousov wrote:

> On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> > Does STABLE support X2APIC?
> > I see X2APIC related commits in CURRENT, what is status for STABLE?
> > I am try to enable X2APIC support on X10DRi and see kernel trap on
> > boot.
> 
> x2APIC is only supported in HEAD.  The code to parse processor x2APIC
> structures in MADT was committed, but AFAIK never tested on real hardware.
> 
> There are no plans to merge this to stable/10.

I am try now on real hardware and got kernel fault at boot when set in
BIOS X2APIC_OPT_OUT Flag to [Enable].
https://s9.postimg.org/6mt8bu21b/fault2.jpg

0x80537e74 is /usr/src/sys/kern/subr_smp.c:959

Boot success if X2APIC_OPT_OUT Flag set to [Disable]
Supermicro X10DRi, dual E5-2650v4

PS: addr2line don't support debug symbol location at /usr/lib/debug
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

X2APIC support

[Slawa Olhovchenkov]

Does STABLE support X2APIC?
I see X2APIC related commits in CURRENT, what is status for STABLE?
I am try to enable X2APIC support on X10DRi and see kernel trap on
boot.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: X2APIC support

[Konstantin Belousov]

On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> Does STABLE support X2APIC?
> I see X2APIC related commits in CURRENT, what is status for STABLE?
> I am try to enable X2APIC support on X10DRi and see kernel trap on
> boot.

x2APIC is only supported in HEAD.  The code to parse processor x2APIC
structures in MADT was committed, but AFAIK never tested on real hardware.

There are no plans to merge this to stable/10.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"