Re: [Xen-devel] Outreachy bite-sized tasks
On Wed, Apr 20, 2016 at 12:06:48PM +0200, Paulina Szubarczyk wrote: > On Fri, 2016-04-01 at 15:35 +0200, Roger Pau Monné wrote: > > Please don't top post, it breaks the flow of the conversation. > > > > I'm also adding Anthony (one of the QEMU/Xen maintainers). > > On Fri, 1 Apr 2016, Paulina Szubarczyk wrote: > > > > > Hi Roger, > > > > > > An another point that needs to be filled up in the application is > > > timeline. > > > I made the proposition of it, I am not sure if it should be more complex. > > > Could you look at it in your free time? > > > > > > Timeline: > > > > > > 22 April - 22 May [Before the intership] > > > * Elaborate performance measurement methodology. That may include > > > putting trace > > > points using TRDCS and performing read operation with varying block > > > sizes on > > > different storage devices as in [3]. > > > > Could you elaborate on TRDCS? I've tried googling for it, but nothing came > > up. > I mistyped TRDCS, I meant putting trace points in the application to > measure the time spent in each place as shown in the talk[3]. But is I > did more research concerning the measurement I think that the good idea > would be using fio and make similar test as Adriana did during her > OWP[4] when she worked on implementing multi-queue support for > xen_backend in Linux. > > > Also, I think it would be good to write down which tool are you going to > > use in order to perform those measurements, and how/which block sizes are > > you going to test. Also keep into account that apart from block sizes you > > also have to take into account the number of parallel workers and the > > queue depth of each one. > > > > Regarding the different storage devices, we usually did benchmarks using a > > ramdisk, but Linux intorduced a null-block device [0] not long ago that I > > think could be used to model different storage devices without actually > > having to buy them. > I make myself familiar with the protocol and I debug the qdisk during > attaching one of ram devices to follow how it works. I also wondered if > maybe I could make a change to 'xl create' such as when it is run in the > qemu debug mode doesn't terminate when Device Mode isn't ready yet, > because sometimes is hard to catch the moment when the connection to > gdbserver is possible. > > Whereas I saw the implementation of indirect descriptors and the > multi-queue support by creating multiple rings, I couldn't find the > information of grant copy. Do you mean by that hypervisor-driven copy of > grant pages between domains instead of mapping them? Yup. The hypervisor does the memcpy. The grant mapping system has an impact that is felt when you are done with the grant - you need to unmap it. And when you unmap it - you need to flush out the virtual address out of the guests vCPUS (if the guests are runninig) - otherwise the CPU could use an stale virtual address (they are cached) and reference an memory that now points to somewhere else. Hence the unmap means also doing an TLB shootdown which is expensive as it flushes out the CPU cache. There have been improvements in this - David Vrabel had posted some patches for this I think.. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Outreachy bite-sized tasks
On Fri, 2016-04-01 at 15:35 +0200, Roger Pau Monné wrote: > Please don't top post, it breaks the flow of the conversation. > > I'm also adding Anthony (one of the QEMU/Xen maintainers). > On Fri, 1 Apr 2016, Paulina Szubarczyk wrote: > > > Hi Roger, > > > > An another point that needs to be filled up in the application is timeline. > > I made the proposition of it, I am not sure if it should be more complex. > > Could you look at it in your free time? > > > > Timeline: > > > > 22 April - 22 May [Before the intership] > > * Elaborate performance measurement methodology. That may include putting > > trace > > points using TRDCS and performing read operation with varying block > > sizes on > > different storage devices as in [3]. > > Could you elaborate on TRDCS? I've tried googling for it, but nothing came > up. I mistyped TRDCS, I meant putting trace points in the application to measure the time spent in each place as shown in the talk[3]. But is I did more research concerning the measurement I think that the good idea would be using fio and make similar test as Adriana did during her OWP[4] when she worked on implementing multi-queue support for xen_backend in Linux. > Also, I think it would be good to write down which tool are you going to > use in order to perform those measurements, and how/which block sizes are > you going to test. Also keep into account that apart from block sizes you > also have to take into account the number of parallel workers and the > queue depth of each one. > > Regarding the different storage devices, we usually did benchmarks using a > ramdisk, but Linux intorduced a null-block device [0] not long ago that I > think could be used to model different storage devices without actually > having to buy them. I make myself familiar with the protocol and I debug the qdisk during attaching one of ram devices to follow how it works. I also wondered if maybe I could make a change to 'xl create' such as when it is run in the qemu debug mode doesn't terminate when Device Mode isn't ready yet, because sometimes is hard to catch the moment when the connection to gdbserver is possible. Whereas I saw the implementation of indirect descriptors and the multi-queue support by creating multiple rings, I couldn't find the information of grant copy. Do you mean by that hypervisor-driven copy of grant pages between domains instead of mapping them? > > I don't expect that you do all this now, but it is important to think > about the different scenarios/benchmarks and plan ahead, so that you > don't end up running benchmarks that are not useful. > > > * Gain a profound knowledge about implementation of xen-blkback in Qemu > > and Linux kernel. > > > > 23 May - 6 June [Begin of the intership - up to two weeks] > > * Prepare and improve performance measurement test. > > * Gather and document the results. > > > > 6 June - 23 August > > * Base on the analysis and the task description implement performance > > improvement future > >inside of QEMU's xen-blkback. > > * Prepare and run regression tests and continue performance measurement > > for > >each implemented feature. > > * Document the results. > > > > 15 August - 23 August [Close to the end] > > * Sum up the work: > >- indicate the obstacles > >- form the conclusion and > >- indicate future work. > > > > References: > > [1] http://lxr.free-electrons.com/source/drivers/block/xen-blkback/blkback.c > > [2] > > http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/blkif.h > > [3] > > https://events.linuxfoundation.org/sites/events/files/slides/20131025%20-%20Storage%20Performance%20PDF.pdf > > [0] https://www.kernel.org/doc/Documentation/block/null_blk.txt [4] https://github.com/ariava/xen-blk-mq-benchmark Paulina ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Outreachy bite-sized tasks
Please don't top post, it breaks the flow of the conversation. I'm also adding Anthony (one of the QEMU/Xen maintainers). On Fri, 1 Apr 2016, Paulina Szubarczyk wrote: > Hi Roger, > > An another point that needs to be filled up in the application is timeline. > I made the proposition of it, I am not sure if it should be more complex. > Could you look at it in your free time? > > Timeline: > > 22 April - 22 May [Before the intership] > * Elaborate performance measurement methodology. That may include putting > trace > points using TRDCS and performing read operation with varying block sizes > on > different storage devices as in [3]. Could you elaborate on TRDCS? I've tried googling for it, but nothing came up. Also, I think it would be good to write down which tool are you going to use in order to perform those measurements, and how/which block sizes are you going to test. Also keep into account that apart from block sizes you also have to take into account the number of parallel workers and the queue depth of each one. Regarding the different storage devices, we usually did benchmarks using a ramdisk, but Linux intorduced a null-block device [0] not long ago that I think could be used to model different storage devices without actually having to buy them. I don't expect that you do all this now, but it is important to think about the different scenarios/benchmarks and plan ahead, so that you don't end up running benchmarks that are not useful. > * Gain a profound knowledge about implementation of xen-blkback in Qemu and > Linux kernel. > > 23 May - 6 June [Begin of the intership - up to two weeks] > * Prepare and improve performance measurement test. > * Gather and document the results. > > 6 June - 23 August > * Base on the analysis and the task description implement performance > improvement future > inside of QEMU's xen-blkback. > * Prepare and run regression tests and continue performance measurement for > each implemented feature. > * Document the results. > > 15 August - 23 August [Close to the end] > * Sum up the work: > - indicate the obstacles > - form the conclusion and > - indicate future work. > > References: > [1] http://lxr.free-electrons.com/source/drivers/block/xen-blkback/blkback.c > [2] > http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/blkif.h > [3] > https://events.linuxfoundation.org/sites/events/files/slides/20131025%20-%20Storage%20Performance%20PDF.pdf [0] https://www.kernel.org/doc/Documentation/block/null_blk.txt___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Outreachy bite-sized tasks
Hi Roger, An another point that needs to be filled up in the application is timeline. I made the proposition of it, I am not sure if it should be more complex. Could you look at it in your free time? Timeline: 22 April - 22 May [Before the intership] * Elaborate performance measurement methodology. That may include putting trace points using TRDCS and performing read operation with varying block sizes on different storage devices as in [3]. * Gain a profound knowledge about implementation of xen-blkback in Qemu and Linux kernel. 23 May - 6 June [Begin of the intership - up to two weeks] * Prepare and improve performance measurement test. * Gather and document the results. 6 June - 23 August * Base on the analysis and the task description implement performance improvement future inside of QEMU's xen-blkback. * Prepare and run regression tests and continue performance measurement for each implemented feature. * Document the results. 15 August - 23 August [Close to the end] * Sum up the work: - indicate the obstacles - form the conclusion and - indicate future work. References: [1] http://lxr.free-electrons.com/source/drivers/block/xen-blkback/blkback.c [2] http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/io/blkif.h [3] https://events.linuxfoundation.org/sites/events/files/slides/20131025%20-%20Storage%20Performance%20PDF.pdf On 23 March 2016 at 19:21, Dario Faggioliwrote: > Hey, > > please, do not use HTML for emails to this list. > > On Wed, 2016-03-23 at 17:38 +0100, Paulina Szubarczyk wrote: >> Hi, >> >> Thank you for the proposed tasks. I would like to work on the second >> one, >> fixing the return codes in xl. >> > I just wanted to say that, since I've done (and mentored) some similar > activity before, so, if you go for this, feel free to ask and/or to Cc > me to the patches as well. :-) > > Regards, > Dario > -- > <> (Raistlin Majere) > - > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK) > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Outreachy bite-sized tasks
Hey, please, do not use HTML for emails to this list. On Wed, 2016-03-23 at 17:38 +0100, Paulina Szubarczyk wrote: > Hi, > > Thank you for the proposed tasks. I would like to work on the second > one, > fixing the return codes in xl. > I just wanted to say that, since I've done (and mentored) some similar activity before, so, if you go for this, feel free to ask and/or to Cc me to the patches as well. :-) Regards, Dario -- <> (Raistlin Majere) - Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R Ltd., Cambridge (UK) signature.asc Description: This is a digitally signed message part ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Outreachy bite-sized tasks
Hi, Thank you for the proposed tasks. I would like to work on the second one, fixing the return codes in xl. Regards, Paulina Szubarczyk On 23 March 2016 at 16:32, Roger Pau Monnéwrote: > Hello, > > First of all, thanks for your interest in the Xen Project, and for wanting > to participate in Outreachy. > > Both of you have expressed interest in the "QEMU xen-blkback performance > analysis and improvements" Outreachy project, and AFAIK both of you still > need to perform your initial contribution. > > I've found a couple of small tasks that you can perform and should allow > you to complete your initial contribution to the project: > > - The first one is related to xenalyze, and it consist in creating a >header file that can be shared by both the Xen kernel and xenalyze. >You will need to move the TRC_ defines found in sched_credit.c and >sched_credit2.c to a header that's shared with xenalyze and then >replace the usage of TRC_SCHED_CLASS_EVT with the defines in the header >file [0]. > > - The second one consist in fixing the return codes of certain xl >commands. There are commands in xl that will return 0 (SUCCESS) even >when failing, which makes it very hard to write scripts that make use >of xl. A list of those commands can be found in [1], together with some >preliminary patches. Please note that those patches have comments that >you will need to address, and that you should also need to preserve the >original authorship of the patches plus yours. > > I encourage you to read the wiki page about sending patches to xen-devel > [2], it should guide you through your first steps on using git and > creating suitable patches. > > Also, please note that this is an open source project, so you will need to > coordinate in order to figure out which task are you going to take, in > order to avoid clashes or duplication of efforts. > > If you have further questions, either reply to this thread (keeping > the xen-devel mailing list on the Cc), or feel free to start another one > if you think it's more suited. > > Roger. > > [0] > http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg02624.html > [1] > http://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02246.html > [2] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches > ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
[Xen-devel] Outreachy bite-sized tasks
Hello, First of all, thanks for your interest in the Xen Project, and for wanting to participate in Outreachy. Both of you have expressed interest in the "QEMU xen-blkback performance analysis and improvements" Outreachy project, and AFAIK both of you still need to perform your initial contribution. I've found a couple of small tasks that you can perform and should allow you to complete your initial contribution to the project: - The first one is related to xenalyze, and it consist in creating a header file that can be shared by both the Xen kernel and xenalyze. You will need to move the TRC_ defines found in sched_credit.c and sched_credit2.c to a header that's shared with xenalyze and then replace the usage of TRC_SCHED_CLASS_EVT with the defines in the header file [0]. - The second one consist in fixing the return codes of certain xl commands. There are commands in xl that will return 0 (SUCCESS) even when failing, which makes it very hard to write scripts that make use of xl. A list of those commands can be found in [1], together with some preliminary patches. Please note that those patches have comments that you will need to address, and that you should also need to preserve the original authorship of the patches plus yours. I encourage you to read the wiki page about sending patches to xen-devel [2], it should guide you through your first steps on using git and creating suitable patches. Also, please note that this is an open source project, so you will need to coordinate in order to figure out which task are you going to take, in order to avoid clashes or duplication of efforts. If you have further questions, either reply to this thread (keeping the xen-devel mailing list on the Cc), or feel free to start another one if you think it's more suited. Roger. [0] http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg02624.html [1] http://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02246.html [2] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel