Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
On 24/04/18 08:10, Dongli Zhang wrote: > > > On 04/24/2018 02:03 PM, Juergen Gross wrote: >> On 24/04/18 07:52, Dongli Zhang wrote: >>> Hi Juergen, >>> >>> On 04/24/2018 01:22 PM, Juergen Gross wrote: On 24/04/18 01:55, Dongli Zhang wrote: > Hi Wei, > > On 04/23/2018 10:09 PM, Wei Liu wrote: >> On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: >>> About per-domU xenwatch thread create/destroy, a new type of xenstore >>> node is >>> introduced: '/local/domain/0/mtwatch/'. >>> >>> Suppose the new domid id 7. During the domU (domid=7) creation, the xen >>> toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the >>> insertion >>> of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore >>> operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. >>> >>> The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. >>> Kernel >>> thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is >>> inserted, >>> while this kernel thread is destroyed when the corresponding xenstore >>> node is >>> removed. >> >> Instead of inventing yet another node, can you not watch /local/domain >> directly? > > Would you like to watch at /local/domain directly? Or is your question > "is there > any other way to not watch at /local/domain, while no extra xenstore node > will > be introduced"? > > Actually, the first prototype of this idea was to watch at /local/domain > directly to get aware of the domU create/destroy, so that xen toolstack > will not > get involved. Joao Martins (CCed) had a concern on the performance as > watching > at /local/domain would lead to large amount of xenwatch events. That's what the special watches "@introduceDomain" and "@releaseDomain" are meant for. >>> >>> I used to consider to watch at "@introduceDomain". However, there is no >>> domain >>> information appended with "@introduceDomain" and it is still required for >>> dom0 >>> kernel to proactively confirm who is created. >> >> That isn't too hard, right? You just need to read /local/domain to get >> the list of its children and look for new domains there. > > You are right. I will try to limit the modification within linux kernel, and > try > to not dirty xen toolstack. > > Thank you very much for the suggestion. When going that route you should extend xenbus_directory() in the kernel to use XS_DIRECTORY_PART in case XS_DIRECTORY returns E2BIG (see xs_directory() in Xen's tools/xenstore/xs.c ). This will enable the kernel to support more than about 1000 domains. Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
On 04/24/2018 02:03 PM, Juergen Gross wrote: > On 24/04/18 07:52, Dongli Zhang wrote: >> Hi Juergen, >> >> On 04/24/2018 01:22 PM, Juergen Gross wrote: >>> On 24/04/18 01:55, Dongli Zhang wrote: Hi Wei, On 04/23/2018 10:09 PM, Wei Liu wrote: > On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: >> About per-domU xenwatch thread create/destroy, a new type of xenstore >> node is >> introduced: '/local/domain/0/mtwatch/'. >> >> Suppose the new domid id 7. During the domU (domid=7) creation, the xen >> toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the >> insertion >> of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore >> operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. >> >> The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. >> Kernel >> thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is >> inserted, >> while this kernel thread is destroyed when the corresponding xenstore >> node is >> removed. > > Instead of inventing yet another node, can you not watch /local/domain > directly? Would you like to watch at /local/domain directly? Or is your question "is there any other way to not watch at /local/domain, while no extra xenstore node will be introduced"? Actually, the first prototype of this idea was to watch at /local/domain directly to get aware of the domU create/destroy, so that xen toolstack will not get involved. Joao Martins (CCed) had a concern on the performance as watching at /local/domain would lead to large amount of xenwatch events. >>> >>> That's what the special watches "@introduceDomain" and "@releaseDomain" >>> are meant for. >> >> I used to consider to watch at "@introduceDomain". However, there is no >> domain >> information appended with "@introduceDomain" and it is still required for >> dom0 >> kernel to proactively confirm who is created. > > That isn't too hard, right? You just need to read /local/domain to get > the list of its children and look for new domains there. You are right. I will try to limit the modification within linux kernel, and try to not dirty xen toolstack. Thank you very much for the suggestion. Dongli Zhang ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
On 24/04/18 07:52, Dongli Zhang wrote: > Hi Juergen, > > On 04/24/2018 01:22 PM, Juergen Gross wrote: >> On 24/04/18 01:55, Dongli Zhang wrote: >>> Hi Wei, >>> >>> On 04/23/2018 10:09 PM, Wei Liu wrote: On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: > About per-domU xenwatch thread create/destroy, a new type of xenstore > node is > introduced: '/local/domain/0/mtwatch/'. > > Suppose the new domid id 7. During the domU (domid=7) creation, the xen > toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the > insertion > of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore > operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. > > The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. > Kernel > thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is > inserted, > while this kernel thread is destroyed when the corresponding xenstore > node is > removed. Instead of inventing yet another node, can you not watch /local/domain directly? >>> >>> Would you like to watch at /local/domain directly? Or is your question "is >>> there >>> any other way to not watch at /local/domain, while no extra xenstore node >>> will >>> be introduced"? >>> >>> Actually, the first prototype of this idea was to watch at /local/domain >>> directly to get aware of the domU create/destroy, so that xen toolstack >>> will not >>> get involved. Joao Martins (CCed) had a concern on the performance as >>> watching >>> at /local/domain would lead to large amount of xenwatch events. >> >> That's what the special watches "@introduceDomain" and "@releaseDomain" >> are meant for. > > I used to consider to watch at "@introduceDomain". However, there is no domain > information appended with "@introduceDomain" and it is still required for dom0 > kernel to proactively confirm who is created. That isn't too hard, right? You just need to read /local/domain to get the list of its children and look for new domains there. Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
Hi Juergen, On 04/24/2018 01:22 PM, Juergen Gross wrote: > On 24/04/18 01:55, Dongli Zhang wrote: >> Hi Wei, >> >> On 04/23/2018 10:09 PM, Wei Liu wrote: >>> On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: About per-domU xenwatch thread create/destroy, a new type of xenstore node is introduced: '/local/domain/0/mtwatch/'. Suppose the new domid id 7. During the domU (domid=7) creation, the xen toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the insertion of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. Kernel thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is inserted, while this kernel thread is destroyed when the corresponding xenstore node is removed. >>> >>> Instead of inventing yet another node, can you not watch /local/domain >>> directly? >> >> Would you like to watch at /local/domain directly? Or is your question "is >> there >> any other way to not watch at /local/domain, while no extra xenstore node >> will >> be introduced"? >> >> Actually, the first prototype of this idea was to watch at /local/domain >> directly to get aware of the domU create/destroy, so that xen toolstack will >> not >> get involved. Joao Martins (CCed) had a concern on the performance as >> watching >> at /local/domain would lead to large amount of xenwatch events. > > That's what the special watches "@introduceDomain" and "@releaseDomain" > are meant for. I used to consider to watch at "@introduceDomain". However, there is no domain information appended with "@introduceDomain" and it is still required for dom0 kernel to proactively confirm who is created. Dongli Zhang ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
On 24/04/18 01:55, Dongli Zhang wrote: > Hi Wei, > > On 04/23/2018 10:09 PM, Wei Liu wrote: >> On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: >>> About per-domU xenwatch thread create/destroy, a new type of xenstore node >>> is >>> introduced: '/local/domain/0/mtwatch/'. >>> >>> Suppose the new domid id 7. During the domU (domid=7) creation, the xen >>> toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the >>> insertion >>> of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore >>> operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. >>> >>> The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. >>> Kernel >>> thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is >>> inserted, >>> while this kernel thread is destroyed when the corresponding xenstore node >>> is >>> removed. >> >> Instead of inventing yet another node, can you not watch /local/domain >> directly? > > Would you like to watch at /local/domain directly? Or is your question "is > there > any other way to not watch at /local/domain, while no extra xenstore node will > be introduced"? > > Actually, the first prototype of this idea was to watch at /local/domain > directly to get aware of the domU create/destroy, so that xen toolstack will > not > get involved. Joao Martins (CCed) had a concern on the performance as watching > at /local/domain would lead to large amount of xenwatch events. That's what the special watches "@introduceDomain" and "@releaseDomain" are meant for. Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
Hi Wei, On 04/23/2018 10:09 PM, Wei Liu wrote: > On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: >> About per-domU xenwatch thread create/destroy, a new type of xenstore node is >> introduced: '/local/domain/0/mtwatch/'. >> >> Suppose the new domid id 7. During the domU (domid=7) creation, the xen >> toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the insertion >> of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore >> operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. >> >> The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. Kernel >> thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is >> inserted, >> while this kernel thread is destroyed when the corresponding xenstore node is >> removed. > > Instead of inventing yet another node, can you not watch /local/domain > directly? Would you like to watch at /local/domain directly? Or is your question "is there any other way to not watch at /local/domain, while no extra xenstore node will be introduced"? Actually, the first prototype of this idea was to watch at /local/domain directly to get aware of the domU create/destroy, so that xen toolstack will not get involved. Joao Martins (CCed) had a concern on the performance as watching at /local/domain would lead to large amount of xenwatch events. Dongli Zhang ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
On Sat, Apr 07, 2018 at 07:25:53PM +0800, Dongli Zhang wrote: > About per-domU xenwatch thread create/destroy, a new type of xenstore node is > introduced: '/local/domain/0/mtwatch/'. > > Suppose the new domid id 7. During the domU (domid=7) creation, the xen > toolstack writes '/local/domain/0/mtwatch/7' to xenstore before the insertion > of '/local/domain/7'. When the domid=7 is destroyed, the last xenstore > operation by xen toolstack is to remove '/local/domain/0/mtwatch/7'. > > The dom0 kernel subscribes a watch at node '/local/domain/0/mtwatch'. Kernel > thread [xen-mtwatch-7] is created when '/local/domain/0/mtwatch/7' is > inserted, > while this kernel thread is destroyed when the corresponding xenstore node is > removed. Instead of inventing yet another node, can you not watch /local/domain directly? Wei. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
Hi Juergen, On 04/07/2018 07:59 PM, Juergen Gross wrote: > On 07/04/18 13:25, Dongli Zhang wrote: >> This is to introduce "xenwatch multithreading" (or "multithreaded xenwatch", >> abbreviated as 'mtwatch'). The implementation of xen mtwatch involves below >> components: >> >> * dom0 linux kernel >> * xen toolstack >> >> Here are what the RFC is going to discuss: >> >> - what is the problem >> - what is the objective >> - what is the solution > > Instead of creating one thread per domU, wouldn't it make much more > sense to use another mechanism, e.g. a workqueue, to deliver the > watch events? This would be one central place, wouldn't need any > changes in Xen tools or complex mechanisms to select the correct > thread, save resources, and domUs would benefit, too. I think the xenwatch events of the same domU are expected to be processed in order. I do not think we are able to guarantee the events of same domU are processed in order with a central workqueue (unless it is an ordered workqueue). Suppose an event callback function is stalled, we expected all following events belong to the same domU are delayed until there is workaround/solution to unblock the stalled function. For instance, as mentioned in below mailing list archive, once the NIC is reset, all in-flight packets are flushed and the stalled callback function moves forward and complete. As all events (especially the events belong to the same domU) are still queued, the pv device backend can disconnect correctly and successfully. With a central workqueue, the following events belong to the same domU are already processed so that the pv device backend may not disconnect successfully. https://lists.xenproject.org/archives/html/xen-devel/2016-06/msg00195.html Dongli Zhang ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [RFC 0/2] To introduce xenwatch multithreading (xen mtwatch)
On 07/04/18 13:25, Dongli Zhang wrote: > This is to introduce "xenwatch multithreading" (or "multithreaded xenwatch", > abbreviated as 'mtwatch'). The implementation of xen mtwatch involves below > components: > > * dom0 linux kernel > * xen toolstack > > Here are what the RFC is going to discuss: > > - what is the problem > - what is the objective > - what is the solution Instead of creating one thread per domU, wouldn't it make much more sense to use another mechanism, e.g. a workqueue, to deliver the watch events? This would be one central place, wouldn't need any changes in Xen tools or complex mechanisms to select the correct thread, save resources, and domUs would benefit, too. Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
