Re: [Xen-ia64-devel] One unstablity in fast syscall path
Le Lundi 19 Juin 2006 15:35, Magenheimer, Dan (HP Labs Fort Collins) a écrit : On this point VTi may have a real advantage over paravirtualization. Could you explain further? Yes. The virtualization handler has the opcode in GR25. I don't know the amount of PAL code involved, but the hardware might provides the opcode directly from the pipeline. The VM doesn't have to do the fecth, which is time and cache consuming. No the hardware does not provide the opcode directly, though perhaps it will in future (post-Montecito) implementations. I believe that PAL is doing the same thing that Xen is doing to fetch the opcode, except since it is doing it sooner, there is a higher probability that it will be successful. And I suspect that it may be doing the slow path every time as it is probably not capable of handling (and recovering from) TLB misses. I'm not sure I would call this an advantage. I suspected this behavior too. So, this is only a *potential* advantage ;-) Perhaps Intel can comment more here... I'm not sure what info is public. Yes, I suppose this info is not public... Tristan. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] One unstablity in fast syscall path
Le Mardi 13 Juin 2006 22:11, Magenheimer, Dan (HP Labs Fort Collins) a écrit : After reading some Xen/ia64 source, I think we'd better not to use vpsr.ic for fast hypercall: it has some interractions with vpsr.ic flag! I'd vote for creating a new paravirtualized register: xen_break (or xen_hyperprivop): when this new register is set, breaks are hyperprivop, when not set breaks are reflected. The advantage of using vpsr.ic is that the vast majority of hyperprivops (dynamically) are in the ivt where vpsr.ic is already off. Using a different virtual register requires the new virtual register to be set and cleared around the use of each hyperprivop (or each group of hyperprivops). While I was updating Linux entry.S and ivt.S, I didn't have this *feeling*. However you may be right and in any case this is a real point. [...] B. introduce new flag for hyperprivop instread VPSR.ic as Tristan proposed. See above. English expression: Don't throw out the baby with the bath water. [We have the same expression in French :-) Meaning, just because hyperprivops can't be used in every situation doesn't mean they should be rewritten so that they can. I'll bet this suggestion will have a measurable negative impact on performance; changing vDSO to use a hypercall rather than a hyperprivop may not have any negative impact. Just a suggestion: you may like to add a new flag. breaks will be hyperprivop if either VPSR.ic is clear or this flag is set. However I think this solution must not be the first one. [...] E. use hyperprivops for vDSO but via a function call to code in hypercall.S (which *is* pinned). Clean an elegant. My first choice! Tristan. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] One unstablity in fast syscall path
Le Mercredi 14 Juin 2006 05:02, Isaku Yamahata a écrit : On Tue, Jun 13, 2006 at 01:11:04PM -0700, Magenheimer, Dan (HP Labs Fort Collins) wrote: There are two purposes of paravirtualization: one is correctness and the other is performance. If fully virtualized vDSO works properly and there's no impact on performance, it shouldn't be paravirtualized. If there is a measurable impact, it should be paravirtualized. I fully agree with you that paravirtualization for performance must be backed by a sort of measurement. I have a question on priv_handle_op(). I changed the function so that xen/ia64 reflects itlb miss to a domain when xen/ia64 fails to read a bundle. Xen/ia64 reflected dtlb miss before my change. Is it correct to reflect dtlb miss? I guess you already encountered the same problem and gave the consideration on it. Just a small comment here: before your patch itlb was useless: it was never read. This works because Linux assume I space == D space. But this may be wrong for other OS. It was of course wrong to read the bundle directly in the D space, but for sure it was faster than doing manually the translation. On this point VTi may have a real advantage over paravirtualization. Tristan. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-ia64-devel] One unstablity in fast syscall path
I have a question on priv_handle_op(). I changed the function so that xen/ia64 reflects itlb miss to a domain when xen/ia64 fails to read a bundle. Xen/ia64 reflected dtlb miss before my change. Is it correct to reflect dtlb miss? No this was just a hack that worked. It does need to be fixed. This was the intent of the one-entry itlb but it never got fully implemented. I guess you already encountered the same problem and gave the consideration on it. Just a small comment here: before your patch itlb was useless: it was never read. This works because Linux assume I space == D space. But this may be wrong for other OS. Very true. It was of course wrong to read the bundle directly in the D space, but for sure it was faster than doing manually the translation. Is it possible to do the translation in Xen without either requesting information from the guest (deliver a TLB miss) or hope that the information is cached? On this point VTi may have a real advantage over paravirtualization. Could you explain further? Dan ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] One unstablity in fast syscall path
Le Mardi 13 Juin 2006 02:36, Tian, Kevin a écrit : Recently we kept seeing intermittent domU creation failure after creating VTI domain. After some debug, we found it caused by following changeset: changeset: 10238:b27139d8c1e1 user:[EMAIL PROTECTED] date:Sat Jun 03 11:16:47 2006 -0600 summary: [IA64] paravirtualize vdso There're several privileged accesses to vPSR in gate page, and previously it's done directly by triggering GP fault without xenlinux changes. Rev 10238 tries to para-virtualize those instructions by introducing two new hyperprivops. Now a simple assumption is forced to distinguish break immediate by checking vPSR.ic. People thought there's no break instruction with psr.ic off in xenlinux. So any break with vpsr.ic off is considered as a hyperprivop between domain and hypervisor. Based on this assumption, explicit vpsr.ic clearance is required before hypercall, and then recover required after. OK, that works, for most time because normally those hypercalls are issued within kernel text segment which is mapped by TR. So after hypervisor finished service for fast hypercall, xen can always RFI to exact resume point in xenlinux even when ITLB miss happens since that resume point can be found by vTR. However, gate page is not mapped by TR, and thus populating PSR.ic at non-TR-mapped segment is even mad in native environment. In our observation, when backing to gate page after servicing fast hypercall, ITLB miss happens but there's no guest entry in vTLB and VHPT. Thus xen injects a nested dtlb fault to xenlinux (vPSR.ic is still off) which is undesired to crash domU. That's the real problem, though we're not sure why this phenomenon is easier to be reproduced after creating VTI domain. Quick/easy solution can be to roll back above changeset to ensure tree stability first, and then community needs to think more robust solution later. Good work ! After reading some Xen/ia64 source, I think we'd better not to use vpsr.ic for fast hypercall: it has some interractions with vpsr.ic flag! I'd vote for creating a new paravirtualized register: xen_break (or xen_hyperprivop): when this new register is set, breaks are hyperprivop, when not set breaks are reflected. Of course, changing the logic requires some work and careful checks. Tristan. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] One unstablity in fast syscall path
On Mon, Jun 12, 2006 at 09:49:17PM -0600, Alex Williamson wrote: On Tue, 2006-06-13 at 08:36 +0800, Tian, Kevin wrote: That's the real problem, though we're not sure why this phenomenon is easier to be reproduced after creating VTI domain. Quick/easy solution can be to roll back above changeset to ensure tree stability first, and then community needs to think more robust solution later. Hi Kevin, Thanks for tracking this down, I've seen some instability lately too. Isaku, should we back this one out until it can be stabilized or do you have a more direct solution? Thanks, Sorry for the regression. Please back it out. The correctness has higher priority than performance, the patch should be backed out since it focuses on only performance. There are the following choices for right fix, I think. A. use hypercall instead of hyperprivop. B. introduce new flag for hyperprivop instread VPSR.ic as Tristan proposed. C. use one of itr to map vDSO. D. abondan vDSO paravirtualization. Thanks. -- yamahata ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
RE: [Xen-ia64-devel] One unstablity in fast syscall path
From: Isaku Yamahata [mailto:[EMAIL PROTECTED] Sent: 2006年6月13日 19:13 There are the following choices for right fix, I think. A. use hypercall instead of hyperprivop. B. introduce new flag for hyperprivop instread VPSR.ic as Tristan proposed. C. use one of itr to map vDSO. D. abondan vDSO paravirtualization. Option B seems reasonable since original vPSR.ic is also borrowed to serve as an indicator. Thanks, Kevin ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] One unstablity in fast syscall path
On Tue, Jun 13, 2006 at 01:11:04PM -0700, Magenheimer, Dan (HP Labs Fort Collins) wrote: There are two purposes of paravirtualization: one is correctness and the other is performance. If fully virtualized vDSO works properly and there's no impact on performance, it shouldn't be paravirtualized. If there is a measurable impact, it should be paravirtualized. I fully agree with you that paravirtualization for performance must be backed by a sort of measurement. I have a question on priv_handle_op(). I changed the function so that xen/ia64 reflects itlb miss to a domain when xen/ia64 fails to read a bundle. Xen/ia64 reflected dtlb miss before my change. Is it correct to reflect dtlb miss? I guess you already encountered the same problem and gave the consideration on it. With my change, the following sequence happens on every system call entry. It would causes performance loss. So I thought it was worthwhile to paravirtualize vDSO area. 1. a domain enters to vDSO 2. execute privilieged op. (rsm or reading psr) 3. traps to Xen/IA64 4. Xen/IA64 tries to read the operation, but fails because the vDSO page is only mapped by itlb cache. (This depends on processor tlb cache implementation) So it reflects itlb. 5. a domain issued itlb insert and xen/ia64 cached vcpu-arch.itlb 6. a domain re-execute the privileged op. 7. traps to Xen/IA64 8. Xen/IA64 successes to read the op with vcpu-arch.itlb and emulates the op. If I understand the problem (not sure I do fully), there is: E. use hyperprivops for vDSO but via a function call to code in hypercall.S (which *is* pinned). This options sounds good. Thanks. -- yamahata ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
Re: [Xen-ia64-devel] One unstablity in fast syscall path
On Tue, 2006-06-13 at 08:36 +0800, Tian, Kevin wrote: That's the real problem, though we're not sure why this phenomenon is easier to be reproduced after creating VTI domain. Quick/easy solution can be to roll back above changeset to ensure tree stability first, and then community needs to think more robust solution later. Hi Kevin, Thanks for tracking this down, I've seen some instability lately too. Isaku, should we back this one out until it can be stabilized or do you have a more direct solution? Thanks, Alex -- Alex Williamson HP Open Source Linux Org. ___ Xen-ia64-devel mailing list Xen-ia64-devel@lists.xensource.com http://lists.xensource.com/xen-ia64-devel
