Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-28 Thread John Baldwin 
On Tuesday, March 27, 2018 11:07:22 AM Jeff Roberson wrote:
> On Tue, 27 Mar 2018, Li-Wen Hsu wrote:
> 
> > On Mon, Mar 26, 2018 at 20:35:12 -1000, Jeff Roberson wrote:
> >> The patch has been on my branch for weeks and has been tested by a half
> >> dozen people.  I'm sorry it does not work for you.  If you reverted 331605
> >> the change that followed should not have built properly.  Did you build
> >> cleanly?  Can you share your kernel config?
> >>
> >> I tried with and without EARLY_AP_STARTUP and with and without NUMA.  I'm
> >> not having any trouble booting.  Did you make cleandepend && make depend?
> >
> > It also hangs in our testing system:
> >
> > https://ci.freebsd.org/job/FreeBSD-head-amd64-test/6817/console
> > https://ci.freebsd.org/job/FreeBSD-head-i386-test/1000/console
> >
> > It is built cleanly and uses unmodified GENERIC config.
> >
> > The artifacts used are available here:
> >
> > https://artifact.ci.freebsd.org/snapshot/head/r331606/amd64/amd64/
> > https://artifact.ci.freebsd.org/snapshot/head/r331606/i386/i386/
> >
> > Hope these information help.
> 
> Could someone who was experiencing the hang try the enclosed patch?  I can 
> only verify that it continues to boot for me but I believe this fixes the 
> bug on other systems.
> 
> I believe the issue was that cpuset_domain[0] was initialized too late on 
> some systems.  It depended on what kind of hardware was present and which 
> sysinit with SI_ORDER_ANY ran first.

Hmm, does this work if you don't use EARLY_AP_STARTUP?  I think all_cpus
isn't set yet (or only contains the BSP) when cpuset_thread0 is run if you
don't have EARLY_AP_STARTUP (and EARLY_AP_STARTUP is only supported on
x86 currently)

-- 
John Baldwin
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread Eitan Adler 
On 27 March 2018 at 14:07, Jeff Roberson  wrote:
> On Tue, 27 Mar 2018, Li-Wen Hsu wrote:
>
>> On Mon, Mar 26, 2018 at 20:35:12 -1000, Jeff Roberson wrote:
>>>
>>> The patch has been on my branch for weeks and has been tested by a half
>>> dozen people.  I'm sorry it does not work for you.  If you reverted
>>> 331605
>>> the change that followed should not have built properly.  Did you build
>>> cleanly?  Can you share your kernel config?
>>>
>>> I tried with and without EARLY_AP_STARTUP and with and without NUMA.  I'm
>>> not having any trouble booting.  Did you make cleandepend && make depend?
>>
>>
>> It also hangs in our testing system:
>>
>> https://ci.freebsd.org/job/FreeBSD-head-amd64-test/6817/console
>> https://ci.freebsd.org/job/FreeBSD-head-i386-test/1000/console
>>
>> It is built cleanly and uses unmodified GENERIC config.
>>
>> The artifacts used are available here:
>>
>> https://artifact.ci.freebsd.org/snapshot/head/r331606/amd64/amd64/
>> https://artifact.ci.freebsd.org/snapshot/head/r331606/i386/i386/
>>
>> Hope these information help.
>
>
> Could someone who was experiencing the hang try the enclosed patch?  I can
> only verify that it continues to boot for me but I believe this fixes the
> bug on other systems.

This patch works for me on r331656M.




-- 
Eitan Adler
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread Jeff Roberson 

On Tue, 27 Mar 2018, Li-Wen Hsu wrote:


On Mon, Mar 26, 2018 at 20:35:12 -1000, Jeff Roberson wrote:

The patch has been on my branch for weeks and has been tested by a half
dozen people.  I'm sorry it does not work for you.  If you reverted 331605
the change that followed should not have built properly.  Did you build
cleanly?  Can you share your kernel config?

I tried with and without EARLY_AP_STARTUP and with and without NUMA.  I'm
not having any trouble booting.  Did you make cleandepend && make depend?


It also hangs in our testing system:

https://ci.freebsd.org/job/FreeBSD-head-amd64-test/6817/console
https://ci.freebsd.org/job/FreeBSD-head-i386-test/1000/console

It is built cleanly and uses unmodified GENERIC config.

The artifacts used are available here:

https://artifact.ci.freebsd.org/snapshot/head/r331606/amd64/amd64/
https://artifact.ci.freebsd.org/snapshot/head/r331606/i386/i386/

Hope these information help.


Could someone who was experiencing the hang try the enclosed patch?  I can 
only verify that it continues to boot for me but I believe this fixes the 
bug on other systems.


I believe the issue was that cpuset_domain[0] was initialized too late on 
some systems.  It depended on what kind of hardware was present and which 
sysinit with SI_ORDER_ANY ran first.


Thanks,
Jeff



Li-Wen

--
Li-Wen Hsu 
https://lwhsu.org
Index: amd64/include/intr_machdep.h
===
--- amd64/include/intr_machdep.h(revision 331610)
+++ amd64/include/intr_machdep.h(working copy)
@@ -132,6 +132,7 @@ struct intsrc {
u_long *is_straycount;
u_int is_index;
u_int is_handlers;
+   u_int is_domain;
u_int is_cpu;
 };
 
@@ -168,7 +169,7 @@ voidintr_add_cpu(u_int cpu);
 #endif
 intintr_add_handler(const char *name, int vector, driver_filter_t filter, 
 driver_intr_t handler, void *arg, enum intr_type 
flags, 
-void **cookiep);
+void **cookiep, int domain);
 #ifdef SMP
 intintr_bind(u_int vector, u_char cpu);
 #endif
@@ -176,7 +177,7 @@ int intr_config_intr(int vector, enum intr_trigger
 enum intr_polarity pol);
 intintr_describe(u_int vector, void *ih, const char *descr);
 void   intr_execute_handlers(struct intsrc *isrc, struct trapframe *frame);
-u_int  intr_next_cpu(void);
+u_int  intr_next_cpu(int domain);
 struct intsrc *intr_lookup_source(int vector);
 intintr_register_pic(struct pic *pic);
 intintr_register_source(struct intsrc *isrc);
Index: i386/include/intr_machdep.h
===
--- i386/include/intr_machdep.h (revision 331610)
+++ i386/include/intr_machdep.h (working copy)
@@ -132,6 +132,7 @@ struct intsrc {
u_long *is_straycount;
u_int is_index;
u_int is_handlers;
+   u_int is_domain;
u_int is_cpu;
 };
 
@@ -158,7 +159,8 @@ voidelcr_write_trigger(u_int irq, enum intr_trigg
 void   intr_add_cpu(u_int cpu);
 #endif
 intintr_add_handler(const char *name, int vector, driver_filter_t filter,
-driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep);
+driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
+int domain);
 #ifdef SMP
 intintr_bind(u_int vector, u_char cpu);
 #endif
@@ -166,7 +168,7 @@ int intr_config_intr(int vector, enum intr_trigger
 enum intr_polarity pol);
 intintr_describe(u_int vector, void *ih, const char *descr);
 void   intr_execute_handlers(struct intsrc *isrc, struct trapframe *frame);
-u_int  intr_next_cpu(void);
+u_int  intr_next_cpu(int domain);
 struct intsrc *intr_lookup_source(int vector);
 intintr_register_pic(struct pic *pic);
 intintr_register_source(struct intsrc *isrc);
Index: kern/kern_cpuset.c
===
--- kern/kern_cpuset.c  (revision 331604)
+++ kern/kern_cpuset.c  (working copy)
@@ -1363,6 +1363,7 @@ cpuset_thread0(void)
 {
struct cpuset *set;
int error;
+   int i;
 
cpuset_zone = uma_zcreate("cpuset", sizeof(struct cpuset), NULL, NULL,
NULL, NULL, UMA_ALIGN_PTR, 0);
@@ -1374,11 +1375,11 @@ cpuset_thread0(void)
 * cpuset_create() due to NULL parent.
 */
set = uma_zalloc(cpuset_zone, M_WAITOK | M_ZERO);
-   CPU_FILL(>cs_mask);
+   CPU_COPY(_cpus, >cs_mask);
LIST_INIT(>cs_children);
LIST_INSERT_HEAD(_ids, set, cs_link);
set->cs_ref = 1;
-   set->cs_flags = CPU_SET_ROOT;
+   set->cs_flags = CPU_SET_ROOT | CPU_SET_RDONLY;
set->cs_domain = 
cpuset_zero = set;
cpuset_root = >cs_mask;
@@ -1396,6 +1397,16 @@ cpuset_thread0(void)
 */
cpuset_unr = new_unrhdr(2, INT_MAX, NULL);
 
+   /*
+* If MD code has not initialized per-domain cpusets, place all
+*

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread Li-Wen Hsu 
On Mon, Mar 26, 2018 at 20:35:12 -1000, Jeff Roberson wrote:
> The patch has been on my branch for weeks and has been tested by a half 
> dozen people.  I'm sorry it does not work for you.  If you reverted 331605 
> the change that followed should not have built properly.  Did you build 
> cleanly?  Can you share your kernel config?
> 
> I tried with and without EARLY_AP_STARTUP and with and without NUMA.  I'm 
> not having any trouble booting.  Did you make cleandepend && make depend?

It also hangs in our testing system:

https://ci.freebsd.org/job/FreeBSD-head-amd64-test/6817/console
https://ci.freebsd.org/job/FreeBSD-head-i386-test/1000/console

It is built cleanly and uses unmodified GENERIC config.

The artifacts used are available here:

https://artifact.ci.freebsd.org/snapshot/head/r331606/amd64/amd64/
https://artifact.ci.freebsd.org/snapshot/head/r331606/i386/i386/

Hope these information help.

Li-Wen

-- 
Li-Wen Hsu 
https://lwhsu.org
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread Eitan Adler 
On 27 March 2018 at 01:45, Hans Petter Selasky  wrote:
> On 03/27/18 05:37, Jeff Roberson wrote:
>>
>> do {
>> -   current_cpu++;
>> -   if (current_cpu > mp_maxid)
>> -   current_cpu = 0;
>> -   } while (!CPU_ISSET(current_cpu, _cpus));
>> +   current_cpu[domain]++;
>> +   if (current_cpu[domain] > mp_maxid)
>> +   current_cpu[domain] = 0;
>> +   } while (!CPU_ISSET(current_cpu[domain], _cpus) ||
>> +   !CPU_ISSET(current_cpu[domain], _domain[domain]));
>
>
> Can it happen that cpuset_domain[domain] is all zero? Won't this loop hang
> then?

I can replicate the hang. Let me know what other debugging information
you might need.



-- 
Eitan Adler
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread Hans Petter Selasky 

On 03/27/18 05:37, Jeff Roberson wrote:

do {
-   current_cpu++;
-   if (current_cpu > mp_maxid)
-   current_cpu = 0;
-   } while (!CPU_ISSET(current_cpu, _cpus));
+   current_cpu[domain]++;
+   if (current_cpu[domain] > mp_maxid)
+   current_cpu[domain] = 0;
+   } while (!CPU_ISSET(current_cpu[domain], _cpus) ||
+   !CPU_ISSET(current_cpu[domain], _domain[domain]));


Can it happen that cpuset_domain[domain] is all zero? Won't this loop 
hang then?


--HPS
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread Jeff Roberson 

On Tue, 27 Mar 2018, O. Hartmann wrote:


On Tue, 27 Mar 2018 08:15:35 +0200
"O. Hartmann"  wrote:


On Tue, 27 Mar 2018 03:37:04 + (UTC)
Jeff Roberson  wrote:


Author: jeff
Date: Tue Mar 27 03:37:04 2018
New Revision: 331606
URL: https://svnweb.freebsd.org/changeset/base/331606

Log:
  Only use CPUs in the domain the device is attached to for default
  assignment.  Device drivers are able to override the default assignment
  if they bind directly.  There are severe performance penalties for
  handling interrupts on remote CPUs and this should only be done in
  very controlled circumstances.

  Reviewed by:  jhb, kib
  Tested by:pho (earlier version)
  Sponsored by: Netflix, Dell/EMC Isilon
  Differential Revision:https://reviews.freebsd.org/D14838

Modified:
  head/sys/amd64/include/intr_machdep.h
  head/sys/i386/include/intr_machdep.h
  head/sys/x86/x86/intr_machdep.c
  head/sys/x86/x86/io_apic.c
  head/sys/x86/x86/msi.c
  head/sys/x86/x86/nexus.c
  head/sys/x86/xen/xen_intr.c

Modified: head/sys/amd64/include/intr_machdep.h
==
--- head/sys/amd64/include/intr_machdep.h   Tue Mar 27 03:27:02
2018(r331605) +++ head/sys/amd64/include/intr_machdep.h Tue
Mar 27 03:37:04 2018(r331606) @@ -132,6 +132,7 @@ struct intsrc {
u_long *is_straycount;
u_int is_index;
u_int is_handlers;
+   u_int is_domain;
u_int is_cpu;
 };

@@ -168,7 +169,7 @@ voidintr_add_cpu(u_int cpu);
 #endif
 intintr_add_handler(const char *name, int vector, driver_filter_t
filter, driver_intr_t handler, void *arg, enum intr_type flags,
-void **cookiep);
+void **cookiep, int domain);
 #ifdef SMP
 intintr_bind(u_int vector, u_char cpu);
 #endif
@@ -176,7 +177,7 @@ int intr_config_intr(int vector, enum
intr_trigger tri enum intr_polarity pol);
 intintr_describe(u_int vector, void *ih, const char *descr);
 void   intr_execute_handlers(struct intsrc *isrc, struct trapframe
*frame); -u_int intr_next_cpu(void);
+u_int  intr_next_cpu(int domain);
 struct intsrc *intr_lookup_source(int vector);
 intintr_register_pic(struct pic *pic);
 intintr_register_source(struct intsrc *isrc);

Modified: head/sys/i386/include/intr_machdep.h
==
--- head/sys/i386/include/intr_machdep.hTue Mar 27 03:27:02
2018(r331605) +++ head/sys/i386/include/intr_machdep.h  Tue
Mar 27 03:37:04 2018(r331606) @@ -132,6 +132,7 @@ struct intsrc {
u_long *is_straycount;
u_int is_index;
u_int is_handlers;
+   u_int is_domain;
u_int is_cpu;
 };

@@ -158,7 +159,8 @@ voidelcr_write_trigger(u_int irq, enum
intr_trigger t void intr_add_cpu(u_int cpu);
 #endif
 intintr_add_handler(const char *name, int vector, driver_filter_t
filter,
-driver_intr_t handler, void *arg, enum intr_type flags, void
**cookiep);
+driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
+int domain);
 #ifdef SMP
 intintr_bind(u_int vector, u_char cpu);
 #endif
@@ -166,7 +168,7 @@ int intr_config_intr(int vector, enum
intr_trigger tri enum intr_polarity pol);
 intintr_describe(u_int vector, void *ih, const char *descr);
 void   intr_execute_handlers(struct intsrc *isrc, struct trapframe
*frame); -u_int intr_next_cpu(void);
+u_int  intr_next_cpu(int domain);
 struct intsrc *intr_lookup_source(int vector);
 intintr_register_pic(struct pic *pic);
 intintr_register_source(struct intsrc *isrc);

Modified: head/sys/x86/x86/intr_machdep.c
==
--- head/sys/x86/x86/intr_machdep.c Tue Mar 27 03:27:02 2018
(r331605) +++ head/sys/x86/x86/intr_machdep.c   Tue Mar 27 03:37:04
2018(r331606) @@ -71,6 +71,8 @@
 #include 
 #endif

+#include 
+
 #defineMAX_STRAY_LOG   5

 typedef void (*mask_fn)(void *);
@@ -185,7 +187,8 @@ intr_lookup_source(int vector)

 int
 intr_add_handler(const char *name, int vector, driver_filter_t filter,
-driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep)
+driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
+int domain)
 {
struct intsrc *isrc;
int error;
@@ -200,6 +203,7 @@ intr_add_handler(const char *name, int vector, driver_
intrcnt_updatename(isrc);
isrc->is_handlers++;
if (isrc->is_handlers == 1) {
+   isrc->is_domain = domain;
isrc->is_pic->pic_enable_intr(isrc);
isrc->is_pic->pic_enable_source(isrc);
}
@@ -507,14 +511,27 @@ DB_SHOW_COMMAND(irqs, db_show_irqs)
  */

 cpuset_t intr_cpus = CPUSET_T_INITIALIZER(0x1);
-static int current_cpu;
+static int current_cpu[MAXMEMDOM];

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread O. Hartmann 
On Tue, 27 Mar 2018 08:15:35 +0200
"O. Hartmann"  wrote:

> On Tue, 27 Mar 2018 03:37:04 + (UTC)
> Jeff Roberson  wrote:
> 
> > Author: jeff
> > Date: Tue Mar 27 03:37:04 2018
> > New Revision: 331606
> > URL: https://svnweb.freebsd.org/changeset/base/331606
> > 
> > Log:
> >   Only use CPUs in the domain the device is attached to for default
> >   assignment.  Device drivers are able to override the default assignment
> >   if they bind directly.  There are severe performance penalties for
> >   handling interrupts on remote CPUs and this should only be done in
> >   very controlled circumstances.
> >   
> >   Reviewed by:  jhb, kib
> >   Tested by:pho (earlier version)
> >   Sponsored by: Netflix, Dell/EMC Isilon
> >   Differential Revision:https://reviews.freebsd.org/D14838
> > 
> > Modified:
> >   head/sys/amd64/include/intr_machdep.h
> >   head/sys/i386/include/intr_machdep.h
> >   head/sys/x86/x86/intr_machdep.c
> >   head/sys/x86/x86/io_apic.c
> >   head/sys/x86/x86/msi.c
> >   head/sys/x86/x86/nexus.c
> >   head/sys/x86/xen/xen_intr.c
> > 
> > Modified: head/sys/amd64/include/intr_machdep.h
> > ==
> > --- head/sys/amd64/include/intr_machdep.h   Tue Mar 27 03:27:02
> > 2018(r331605) +++ head/sys/amd64/include/intr_machdep.h Tue
> > Mar 27 03:37:04 2018(r331606) @@ -132,6 +132,7 @@ struct intsrc {
> > u_long *is_straycount;
> > u_int is_index;
> > u_int is_handlers;
> > +   u_int is_domain;
> > u_int is_cpu;
> >  };
> >  
> > @@ -168,7 +169,7 @@ voidintr_add_cpu(u_int cpu);
> >  #endif
> >  intintr_add_handler(const char *name, int vector, driver_filter_t
> > filter, driver_intr_t handler, void *arg, enum intr_type flags, 
> > -void **cookiep);
> > +void **cookiep, int domain);
> >  #ifdef SMP
> >  intintr_bind(u_int vector, u_char cpu);
> >  #endif
> > @@ -176,7 +177,7 @@ int intr_config_intr(int vector, enum
> > intr_trigger tri enum intr_polarity pol);
> >  intintr_describe(u_int vector, void *ih, const char *descr);
> >  void   intr_execute_handlers(struct intsrc *isrc, struct trapframe
> > *frame); -u_int intr_next_cpu(void);
> > +u_int  intr_next_cpu(int domain);
> >  struct intsrc *intr_lookup_source(int vector);
> >  intintr_register_pic(struct pic *pic);
> >  intintr_register_source(struct intsrc *isrc);
> > 
> > Modified: head/sys/i386/include/intr_machdep.h
> > ==
> > --- head/sys/i386/include/intr_machdep.hTue Mar 27 03:27:02
> > 2018(r331605) +++ head/sys/i386/include/intr_machdep.h  Tue
> > Mar 27 03:37:04 2018(r331606) @@ -132,6 +132,7 @@ struct intsrc {
> > u_long *is_straycount;
> > u_int is_index;
> > u_int is_handlers;
> > +   u_int is_domain;
> > u_int is_cpu;
> >  };
> >  
> > @@ -158,7 +159,8 @@ voidelcr_write_trigger(u_int irq, enum
> > intr_trigger t void intr_add_cpu(u_int cpu);
> >  #endif
> >  intintr_add_handler(const char *name, int vector, driver_filter_t
> > filter,
> > -driver_intr_t handler, void *arg, enum intr_type flags, void
> > **cookiep);
> > +driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
> > +int domain);
> >  #ifdef SMP
> >  intintr_bind(u_int vector, u_char cpu);
> >  #endif
> > @@ -166,7 +168,7 @@ int intr_config_intr(int vector, enum
> > intr_trigger tri enum intr_polarity pol);
> >  intintr_describe(u_int vector, void *ih, const char *descr);
> >  void   intr_execute_handlers(struct intsrc *isrc, struct trapframe
> > *frame); -u_int intr_next_cpu(void);
> > +u_int  intr_next_cpu(int domain);
> >  struct intsrc *intr_lookup_source(int vector);
> >  intintr_register_pic(struct pic *pic);
> >  intintr_register_source(struct intsrc *isrc);
> > 
> > Modified: head/sys/x86/x86/intr_machdep.c
> > ==
> > --- head/sys/x86/x86/intr_machdep.c Tue Mar 27 03:27:02 2018
> > (r331605) +++ head/sys/x86/x86/intr_machdep.c   Tue Mar 27 03:37:04
> > 2018(r331606) @@ -71,6 +71,8 @@
> >  #include 
> >  #endif
> >  
> > +#include 
> > +
> >  #defineMAX_STRAY_LOG   5
> >  
> >  typedef void (*mask_fn)(void *);
> > @@ -185,7 +187,8 @@ intr_lookup_source(int vector)
> >  
> >  int
> >  intr_add_handler(const char *name, int vector, driver_filter_t filter,
> > -driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep)
> > +driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
> > +int domain)
> >  {
> > struct intsrc *isrc;
> > int error;
> > @@ -200,6 +203,7 @@ intr_add_handler(const char *name, int vector, driver_
> >

Re: svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-27 Thread O. Hartmann 
On Tue, 27 Mar 2018 03:37:04 + (UTC)
Jeff Roberson  wrote:

> Author: jeff
> Date: Tue Mar 27 03:37:04 2018
> New Revision: 331606
> URL: https://svnweb.freebsd.org/changeset/base/331606
> 
> Log:
>   Only use CPUs in the domain the device is attached to for default
>   assignment.  Device drivers are able to override the default assignment
>   if they bind directly.  There are severe performance penalties for
>   handling interrupts on remote CPUs and this should only be done in
>   very controlled circumstances.
>   
>   Reviewed by:jhb, kib
>   Tested by:  pho (earlier version)
>   Sponsored by:   Netflix, Dell/EMC Isilon
>   Differential Revision:  https://reviews.freebsd.org/D14838
> 
> Modified:
>   head/sys/amd64/include/intr_machdep.h
>   head/sys/i386/include/intr_machdep.h
>   head/sys/x86/x86/intr_machdep.c
>   head/sys/x86/x86/io_apic.c
>   head/sys/x86/x86/msi.c
>   head/sys/x86/x86/nexus.c
>   head/sys/x86/xen/xen_intr.c
> 
> Modified: head/sys/amd64/include/intr_machdep.h
> ==
> --- head/sys/amd64/include/intr_machdep.h Tue Mar 27 03:27:02
> 2018  (r331605) +++ head/sys/amd64/include/intr_machdep.h Tue
> Mar 27 03:37:04 2018  (r331606) @@ -132,6 +132,7 @@ struct intsrc {
>   u_long *is_straycount;
>   u_int is_index;
>   u_int is_handlers;
> + u_int is_domain;
>   u_int is_cpu;
>  };
>  
> @@ -168,7 +169,7 @@ void  intr_add_cpu(u_int cpu);
>  #endif
>  int  intr_add_handler(const char *name, int vector, driver_filter_t
> filter, driver_intr_t handler, void *arg, enum intr_type flags, 
> -  void **cookiep);
> +  void **cookiep, int domain);
>  #ifdef SMP
>  int  intr_bind(u_int vector, u_char cpu);
>  #endif
> @@ -176,7 +177,7 @@ int   intr_config_intr(int vector, enum intr_trigger
> tri enum intr_polarity pol);
>  int  intr_describe(u_int vector, void *ih, const char *descr);
>  void intr_execute_handlers(struct intsrc *isrc, struct trapframe
> *frame); -u_int   intr_next_cpu(void);
> +u_intintr_next_cpu(int domain);
>  struct intsrc *intr_lookup_source(int vector);
>  int  intr_register_pic(struct pic *pic);
>  int  intr_register_source(struct intsrc *isrc);
> 
> Modified: head/sys/i386/include/intr_machdep.h
> ==
> --- head/sys/i386/include/intr_machdep.h  Tue Mar 27 03:27:02
> 2018  (r331605) +++ head/sys/i386/include/intr_machdep.h  Tue Mar
> 27 03:37:04 2018  (r331606) @@ -132,6 +132,7 @@ struct intsrc {
>   u_long *is_straycount;
>   u_int is_index;
>   u_int is_handlers;
> + u_int is_domain;
>   u_int is_cpu;
>  };
>  
> @@ -158,7 +159,8 @@ void  elcr_write_trigger(u_int irq, enum
> intr_trigger t void   intr_add_cpu(u_int cpu);
>  #endif
>  int  intr_add_handler(const char *name, int vector, driver_filter_t
> filter,
> -driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep);
> +driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
> +int domain);
>  #ifdef SMP
>  int  intr_bind(u_int vector, u_char cpu);
>  #endif
> @@ -166,7 +168,7 @@ int   intr_config_intr(int vector, enum intr_trigger
> tri enum intr_polarity pol);
>  int  intr_describe(u_int vector, void *ih, const char *descr);
>  void intr_execute_handlers(struct intsrc *isrc, struct trapframe
> *frame); -u_int   intr_next_cpu(void);
> +u_intintr_next_cpu(int domain);
>  struct intsrc *intr_lookup_source(int vector);
>  int  intr_register_pic(struct pic *pic);
>  int  intr_register_source(struct intsrc *isrc);
> 
> Modified: head/sys/x86/x86/intr_machdep.c
> ==
> --- head/sys/x86/x86/intr_machdep.c   Tue Mar 27 03:27:02 2018
> (r331605) +++ head/sys/x86/x86/intr_machdep.c Tue Mar 27 03:37:04
> 2018  (r331606) @@ -71,6 +71,8 @@
>  #include 
>  #endif
>  
> +#include 
> +
>  #define  MAX_STRAY_LOG   5
>  
>  typedef void (*mask_fn)(void *);
> @@ -185,7 +187,8 @@ intr_lookup_source(int vector)
>  
>  int
>  intr_add_handler(const char *name, int vector, driver_filter_t filter,
> -driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep)
> +driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
> +int domain)
>  {
>   struct intsrc *isrc;
>   int error;
> @@ -200,6 +203,7 @@ intr_add_handler(const char *name, int vector, driver_
>   intrcnt_updatename(isrc);
>   isrc->is_handlers++;
>   if (isrc->is_handlers == 1) {
> + isrc->is_domain = domain;
>   isrc->is_pic->pic_enable_intr(isrc);
>   isrc->is_pic->pic_enable_source(isrc);
>   }
> @@ -507,14 +511,27 @@ DB_SHOW_COMMAND(irqs, db_show_irqs)
>   */
>  
>  cpuset_t

svn commit: r331606 - in head/sys: amd64/include i386/include x86/x86 x86/xen

2018-03-26 Thread Jeff Roberson 
Author: jeff
Date: Tue Mar 27 03:37:04 2018
New Revision: 331606
URL: https://svnweb.freebsd.org/changeset/base/331606

Log:
  Only use CPUs in the domain the device is attached to for default
  assignment.  Device drivers are able to override the default assignment
  if they bind directly.  There are severe performance penalties for
  handling interrupts on remote CPUs and this should only be done in
  very controlled circumstances.
  
  Reviewed by:  jhb, kib
  Tested by:pho (earlier version)
  Sponsored by: Netflix, Dell/EMC Isilon
  Differential Revision:https://reviews.freebsd.org/D14838

Modified:
  head/sys/amd64/include/intr_machdep.h
  head/sys/i386/include/intr_machdep.h
  head/sys/x86/x86/intr_machdep.c
  head/sys/x86/x86/io_apic.c
  head/sys/x86/x86/msi.c
  head/sys/x86/x86/nexus.c
  head/sys/x86/xen/xen_intr.c

Modified: head/sys/amd64/include/intr_machdep.h
==
--- head/sys/amd64/include/intr_machdep.h   Tue Mar 27 03:27:02 2018
(r331605)
+++ head/sys/amd64/include/intr_machdep.h   Tue Mar 27 03:37:04 2018
(r331606)
@@ -132,6 +132,7 @@ struct intsrc {
u_long *is_straycount;
u_int is_index;
u_int is_handlers;
+   u_int is_domain;
u_int is_cpu;
 };
 
@@ -168,7 +169,7 @@ voidintr_add_cpu(u_int cpu);
 #endif
 intintr_add_handler(const char *name, int vector, driver_filter_t filter, 
 driver_intr_t handler, void *arg, enum intr_type 
flags, 
-void **cookiep);
+void **cookiep, int domain);
 #ifdef SMP
 intintr_bind(u_int vector, u_char cpu);
 #endif
@@ -176,7 +177,7 @@ int intr_config_intr(int vector, enum intr_trigger tri
 enum intr_polarity pol);
 intintr_describe(u_int vector, void *ih, const char *descr);
 void   intr_execute_handlers(struct intsrc *isrc, struct trapframe *frame);
-u_int  intr_next_cpu(void);
+u_int  intr_next_cpu(int domain);
 struct intsrc *intr_lookup_source(int vector);
 intintr_register_pic(struct pic *pic);
 intintr_register_source(struct intsrc *isrc);

Modified: head/sys/i386/include/intr_machdep.h
==
--- head/sys/i386/include/intr_machdep.hTue Mar 27 03:27:02 2018
(r331605)
+++ head/sys/i386/include/intr_machdep.hTue Mar 27 03:37:04 2018
(r331606)
@@ -132,6 +132,7 @@ struct intsrc {
u_long *is_straycount;
u_int is_index;
u_int is_handlers;
+   u_int is_domain;
u_int is_cpu;
 };
 
@@ -158,7 +159,8 @@ voidelcr_write_trigger(u_int irq, enum intr_trigger 
t
 void   intr_add_cpu(u_int cpu);
 #endif
 intintr_add_handler(const char *name, int vector, driver_filter_t filter,
-driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep);
+driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
+int domain);
 #ifdef SMP
 intintr_bind(u_int vector, u_char cpu);
 #endif
@@ -166,7 +168,7 @@ int intr_config_intr(int vector, enum intr_trigger tri
 enum intr_polarity pol);
 intintr_describe(u_int vector, void *ih, const char *descr);
 void   intr_execute_handlers(struct intsrc *isrc, struct trapframe *frame);
-u_int  intr_next_cpu(void);
+u_int  intr_next_cpu(int domain);
 struct intsrc *intr_lookup_source(int vector);
 intintr_register_pic(struct pic *pic);
 intintr_register_source(struct intsrc *isrc);

Modified: head/sys/x86/x86/intr_machdep.c
==
--- head/sys/x86/x86/intr_machdep.c Tue Mar 27 03:27:02 2018
(r331605)
+++ head/sys/x86/x86/intr_machdep.c Tue Mar 27 03:37:04 2018
(r331606)
@@ -71,6 +71,8 @@
 #include 
 #endif
 
+#include 
+
 #defineMAX_STRAY_LOG   5
 
 typedef void (*mask_fn)(void *);
@@ -185,7 +187,8 @@ intr_lookup_source(int vector)
 
 int
 intr_add_handler(const char *name, int vector, driver_filter_t filter,
-driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep)
+driver_intr_t handler, void *arg, enum intr_type flags, void **cookiep,
+int domain)
 {
struct intsrc *isrc;
int error;
@@ -200,6 +203,7 @@ intr_add_handler(const char *name, int vector, driver_
intrcnt_updatename(isrc);
isrc->is_handlers++;
if (isrc->is_handlers == 1) {
+   isrc->is_domain = domain;
isrc->is_pic->pic_enable_intr(isrc);
isrc->is_pic->pic_enable_source(isrc);
}
@@ -507,14 +511,27 @@ DB_SHOW_COMMAND(irqs, db_show_irqs)
  */
 
 cpuset_t intr_cpus = CPUSET_T_INITIALIZER(0x1);
-static int current_cpu;
+static int current_cpu[MAXMEMDOM];
 
+static void
+intr_init_cpus(void)
+{
+   int i;
+
+   for (i = 0; i < vm_ndomains; i++) {
+