RE: [XenPPC] systemsim-gpul problems
Not sure this sheds any light on the timing problem but it looks like Xen is in it's idle_loop() during the time things are running unexpectedly slow. Stuart > -Original Message- > From: Hollis Blanchard [mailto:[EMAIL PROTECTED] > Sent: Friday, January 05, 2007 1:06 PM > To: Yoder Stuart-B08248 > Cc: Jimi Xenidis; xen-ppc-devel@lists.xensource.com > Subject: RE: [XenPPC] systemsim-gpul problems > > On Fri, 2007-01-05 at 09:54 -0700, Yoder Stuart-B08248 wrote: > > > > 1449668513: (1449502549): IPv4 over IPv4 tunneling driver > > 1450457186: (1450290832): TCP bic registered > > 1450527304: (1450360932): NET: Registered protocol family 1 > > 1450557364: (1450390979): NET: Registered protocol family 17 > > 146000: [0]: (PC:0x0042FA80) : 7997.4 Kilo-In... > > 148000: [0]: (PC:0x0042FA80) : .3 Kilo-In... > > 15: [0]: (PC:0x004224B0) : .3 Kilo-In... > > This is interesting because it seems to indicate that we're spending > lots of time in Xen (that's a Xen address). nm xen-syms | sort should > help you figure out what function that is. > > -- > Hollis Blanchard > IBM Linux Technology Center > > ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
> -Original Message- > From: Jimi Xenidis [mailto:[EMAIL PROTECTED] > > This one does not make sense, is there garage that matches this in > your available property? > > boot_of_alloc_init: marking 0x0 - 0x400^M > This particular print is actually gone in top of tree Xen. I had been playing with different changesets and the trace I put in the email was from an old version that had four regions marked in boot_of_alloc_init. Stuart ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
On Fri, 2007-01-05 at 09:54 -0700, Yoder Stuart-B08248 wrote: > > 1449668513: (1449502549): IPv4 over IPv4 tunneling driver > 1450457186: (1450290832): TCP bic registered > 1450527304: (1450360932): NET: Registered protocol family 1 > 1450557364: (1450390979): NET: Registered protocol family 17 > 146000: [0]: (PC:0x0042FA80) : 7997.4 Kilo-In... > 148000: [0]: (PC:0x0042FA80) : .3 Kilo-In... > 15: [0]: (PC:0x004224B0) : .3 Kilo-In... This is interesting because it seems to indicate that we're spending lots of time in Xen (that's a Xen address). nm xen-syms | sort should help you figure out what function that is. -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
> -Original Message- > From: Jimi Xenidis [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 04, 2007 7:54 PM > To: Hollis Blanchard > Cc: Yoder Stuart-B08248; xen-ppc-devel@lists.xensource.com > Subject: Re: [XenPPC] systemsim-gpul problems > > > On Jan 4, 2007, at 7:19 PM, Hollis Blanchard wrote: > > > On Thu, 2007-01-04 at 16:48 -0700, Yoder Stuart-B08248 wrote: > >> Whad'ya know...if you wait long enough it works! > > > > So simulator performance is acceptable until mid-way through dom0 > > boot? > > It would be good to figure out the source of the slowdown. > > the last time we had this problem it was because Linux takes to long > figuring out instructions-per-jiffy and then the result is pain. > > can try the workaround described in: >http://lists.xensource.com/archives/html/xen-ppc-devel/2006-03/ > msg00119.html > > We may have a systemsym regression here, because this was no longer > necessary. > -JX I tried the lpj= option and it didn't seem to affect the wall clock time the simluation takes. Here is an observation-- On the changset from 11/21 that performed more quickly here is an excerpt of the log: [snip] 1449091137: (1249159347): IPv4 over IPv4 tunneling driver 1449879806: (1249947628): TCP bic registered 1449948355: (1250016163): NET: Registered protocol family 1 1449978413: (1250046208): NET: Registered protocol family 17 2310238380: (1251708119): Bridge firewalling registered 2310251180: (1251720893): Ebtables v2.0 registered 2670267306: (1252438049): ebt_ulog: not logging via ulog... 2670949205: (1253119878): 802.1Q VLAN Support v1.8 Ben G... 2670966817: (1253137486): All bugs added by David S. Mil... [snip] After the systemsim-gpul fix went in on 12/11 here is the same excerpt of the log [snip 1449668513: (1449502549): IPv4 over IPv4 tunneling driver 1450457186: (1450290832): TCP bic registered 1450527304: (1450360932): NET: Registered protocol family 1 1450557364: (1450390979): NET: Registered protocol family 17 146000: [0]: (PC:0x0042FA80) : 7997.4 Kilo-In... 148000: [0]: (PC:0x0042FA80) : .3 Kilo-In... 15: [0]: (PC:0x004224B0) : .3 Kilo-In... 152000: [0]: (PC:0x004224AC) : .3 Kilo-In... 154000: [0]: (PC:0x0042FA84) : .2 Kilo-In... 156000: [0]: (PC:0x004224B8) : .3 Kilo-In... 158000: [0]: (PC:0x0042FA74) : .3 Kilo-In... 16: [0]: (PC:0x0042FA80) : 2500.1 Kilo-In... 162000: [0]: (PC:0x004224B0) : .3 Kilo-In... 164000: [0]: (PC:0x0042FA80) : .3 Kilo-In... 166000: [0]: (PC:0x004224A8) : .3 Kilo-In... 168000: [0]: (PC:0x0042FA84) : .3 Kilo-In... 17: [0]: (PC:0x0042FA78) : .3 Kilo-In... 172000: [0]: (PC:0x0042FA78) : .3 Kilo-In... 174000: [0]: (PC:0x004224B8) : .3 Kilo-In... 176000: [0]: (PC:0x004224B0) : 2500.1 Kilo-In... 178000: [0]: (PC:0x0042FA80) : .3 Kilo-In... 18: [0]: (PC:0x004224A8) : .3 Kilo-In... 182000: [0]: (PC:0x004224B0) : .3 Kilo-In... 184000: [0]: (PC:0x0042FA74) : .3 Kilo-In... 186000: [0]: (PC:0x0042FA74) : .3 Kilo-In... 188000: [0]: (PC:0x0042FA74) : .3 Kilo-In... 19: [0]: (PC:0x004224A8) : .3 Kilo-In... 192000: [0]: (PC:0x004224AC) : 2500.1 Kilo-In... 194000: [0]: (PC:0x0042FA78) : .3 Kilo-In... 196000: [0]: (PC:0x0042FA80) : .3 Kilo-In... 198000: [0]: (PC:0x0042FA74) : .3 Kilo-In... 20: [0]: (PC:0x0042FA84) : .3 Kilo-In... 202000: [0]: (PC:0x004224B0) : .3 Kilo-In... 204000: [0]: (PC:0x0042FA78) : .3 Kilo-In... 206000: [0]: (PC:0x004224B8) : .3 Kilo-In... 208000: [0]: (PC:0x0042FA78) : .3 Kilo-In... 21: [0]: (PC:0x004224B0) : .3 Kilo-In... 212000: [0]: (PC:0x004224B0) : 2500.1 Kilo-In... 214000: [0]: (PC:0x0042FA80) : .3 Kilo-In... 216000: [0]: (PC:0x004224AC) : .3 Kilo-In... 218000: [0]: (PC:0x0042FA78) : .3 Kilo-In... 22: [0]: (PC:0x0042FA74) : .3 Kilo-In... 222000: [0]: (PC:0x004224B8) : .3 Kilo-In... 224000: [0]: (PC:0x004224AC) : .3 Kilo-In... 226000: [0]: (PC:0x0042FA78) : 2500.1 Kilo-In... 228000: [0]: (PC:0x004224B0) : .3 Kilo-In... 23: [0]: (PC:0x004224B8) : .3 Kilo-In... 2311211643: (2311063340): Bridge firewalling registered 231122: (2311076116): Ebtables v2.0 registered 232000: [0]: (PC:0x004224B8) : .3 Kilo-In... 23
Re: [XenPPC] systemsim-gpul problems
On Jan 4, 2007, at 4:15 PM, Yoder Stuart-B08248 wrote: -Original Message- From: Hollis Blanchard [mailto:[EMAIL PROTECTED] Sent: Thursday, January 04, 2007 2:12 PM Have you tried attaching GDB to systemsim to figure out what's going on? I haven't used gdb w/ the simulator yet-- I need to figure out how to do that. I have turned on debug prints in arch/powerpc/boot_of.c. One thing I'm puzzling over is why boot_of_alloc_init() is marking overlapping regions of memory. Does that make sense?? Overlapping is fine since its a bitmap we just end up setting the bit redundantly. The first thing we do is mark memory "outside of" the avail properties, you can find that out from systemsim with: systemsim % mysim of getprop [ mysim of find_device /[EMAIL PROTECTED] ] available 1000. which says that AFA FW is concerned we can do whatever we want from 0 to 0x1000 (which is 256M that I have mine configed with) You should have also seen this in output: 69254: (66466): avail: 73075: (69743): 0x00, 0x01000 I see now I should pretty that up a bit :) [snip] so first we mark from 0 to the bigining of "available" which is 0 also boot_of_alloc_init: marking 0x0 - 0x0^M This one does not make sense, is there garage that matches this in your available property? boot_of_alloc_init: marking 0x0 - 0x400^M We then explicitly mark the xen image itself, because we cannot depend on FW to claim it (tho it should). boot_of_alloc_init: marking 0x40 - 0x88b000^M Then we explicitly remove the exception area (the first 4 pages) boot_of_alloc_init: marking 0x0 - 0x3000^M [snip] Also, why is there a region w/ zero length? see above Here is all my debug output, which _does_ make sense, but does may be from a different sim version. 67604: (64816): boot_of_mem_init: number of bytes in property 'reg' 16 69254: (66466): avail: 73075: (69743): 0x00, 0x01000 76663: (73331): boot_of_alloc_init: marking 0x0 - 0x0 81799: (77651): boot_of_alloc_init: marking 0x40 - 0x771000 141392: (136972): boot_of_alloc_init: marking 0x0 - 0x3000 ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems: reserved memory areas
On Thu, 2007-01-04 at 14:15 -0700, Yoder Stuart-B08248 wrote: > I have turned on debug prints in arch/powerpc/boot_of.c. > One thing I'm puzzling over is why boot_of_alloc_init() > is marking overlapping regions of memory. Does that > make sense?? It doesn't exactly make sense, but in this case it's not a real problem... > [snip] > boot_of_alloc_init: marking 0x0 - 0x0^M > boot_of_alloc_init: marking 0x0 - 0x400^M These two come from your Open Firmware device tree. They don't make sense IMHO. systemsim may have a bug in its device tree. We could also have a bug in the way we parse the "available" property, but I believe our code has been tested on PowerMac and SLOF, and it looks correct to me. > boot_of_alloc_init: marking 0x40 - 0x88b000^M This is our code reserving Xen's text, _start to _end. > boot_of_alloc_init: marking 0x0 - 0x3000^M This is our code reserving the exception handlers. -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
Re: [XenPPC] systemsim-gpul problems
On Jan 4, 2007, at 7:19 PM, Hollis Blanchard wrote: On Thu, 2007-01-04 at 16:48 -0700, Yoder Stuart-B08248 wrote: Whad'ya know...if you wait long enough it works! So simulator performance is acceptable until mid-way through dom0 boot? It would be good to figure out the source of the slowdown. the last time we had this problem it was because Linux takes to long figuring out instructions-per-jiffy and then the result is pain. can try the workaround described in: http://lists.xensource.com/archives/html/xen-ppc-devel/2006-03/ msg00119.html We may have a systemsym regression here, because this was no longer necessary. -JX ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
Could be that midway through a Linux boot all code is sequential, but at a later point Linux tries to initialize its timing facilities. This involves linux spinning on the timebase register which forces systemsim to simulate many, many useless cycles. Beyond that Linux initialization sees a lot of cache initialization which requires memory zeroing, which also is not simulator friendly. I see this sort of behavior simply by running linux on the simulator wit or without a hypervisor. On Thu, 2007-01-04 at 18:19 -0600, Hollis Blanchard wrote: > On Thu, 2007-01-04 at 16:48 -0700, Yoder Stuart-B08248 wrote: > > Whad'ya know...if you wait long enough it works! > > So simulator performance is acceptable until mid-way through dom0 boot? > It would be good to figure out the source of the slowdown. > -- Michal Ostrowski <[EMAIL PROTECTED]> ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
I believe it's got to be related to clock and/or external interrupt delivery, but I admit that's not very specific. I don't have time to work on it now. Hope someone else does. [EMAIL PROTECTED] 01/04/2007 07:19 PM Please respond to [EMAIL PROTECTED] To Yoder Stuart-B08248 <[EMAIL PROTECTED]> cc Mark F Mergen/Watson/[EMAIL PROTECTED], xen-ppc-devel@lists.xensource.com Subject RE: [XenPPC] systemsim-gpul problems On Thu, 2007-01-04 at 16:48 -0700, Yoder Stuart-B08248 wrote: > Whad'ya know...if you wait long enough it works! So simulator performance is acceptable until mid-way through dom0 boot? It would be good to figure out the source of the slowdown. -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
On Thu, 2007-01-04 at 16:48 -0700, Yoder Stuart-B08248 wrote: > Whad'ya know...if you wait long enough it works! So simulator performance is acceptable until mid-way through dom0 boot? It would be good to figure out the source of the slowdown. -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
Whad'ya know...if you wait long enough it works! Thanks Mark. From: Mark F Mergen [mailto:[EMAIL PROTECTED] Sent: Thursday, January 04, 2007 4:30 PM To: Yoder Stuart-B08248; xen-ppc-devel@lists.xensource.com Subject: Fw: [XenPPC] systemsim-gpul problems Sorry, I should have read your note more carefully. Yes, my latest builds do boot all the way into Linux, but the timing is drastically altered from the past. There are places in the boot sequence (and I think what you mention: after protocol family 17 is one of them) where you must wait a VERY long time for something to happen. I can't tell you exactly how long, but it took me a while and accidental inattention to what was happening before I discovered this. I have not had time to work on why this is so. Also, if you do get all the way to a Linux command prompt, you may conclude it is not responding to commands. Just type a command that you think should work (such as pwd), hit Enter, and then WAIT A LONG TIME. If your experience turns out to be like mine, eventually the characters you typed will be echoed and the command will be executed with proper output, however slowly. Obviously, more debugging is needed. Maybe it's something simple, like some simulator option for how real time is reflected to the simulated machine. Maybe you will be rewarded by a little (or a LOT of) patience in your next try. Mark Mergen - Forwarded by Mark F Mergen/Watson/IBM on 01/04/2007 05:15 PM - Mark F Mergen/Watson/IBM 01/04/2007 05:12 PM To [EMAIL PROTECTED] cc Yoder Stuart-B08248 <[EMAIL PROTECTED]>, xen-ppc-devel@lists.xensource.com Subject Re: [XenPPC] systemsim-gpul problemsLink I pull only from xenppc-unstable.hg and linux-ppc-2.6.hg. I'm up to date with all latest that was pushed to them, and this combo runs on systemsim-gpul. There was a bug(s?) that caused symptoms like you mention, but they were fixed by Amos Waterland about 2nd week in December, and pushed to aformentioned repositories by the maintainers. I can't quote or vouch for specific changesets. Mark Mergen [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 01/04/2007 03:11 PM Please respond to [EMAIL PROTECTED] To Yoder Stuart-B08248 <[EMAIL PROTECTED]> cc xen-ppc-devel@lists.xensource.com Subject Re: [XenPPC] systemsim-gpul problems On Thu, 2007-01-04 at 10:32 -0700, Yoder Stuart-B08248 wrote: > I've been trying to get systemsim-gpul working with the > latest Xen-PPC. > > The last changeset that worked was 7ad4645e7a54 (11/22/06). > > Changeset ce8c1e26b2ae (Early boot memory avoidance improvemnts) > broke the simulator with the 'Could not allocate RTAS tree' / HANG > error. > > With changeset 878ce1f78ad3 (Fix systemsim-gpul failure to boot) > and later changesets the RTAS allocation / HANG problem is fixed > but the simulator still won't boot into Linux. > > If I compare logs of the working and non-working Xen/Linux runs > in the simulator, with the current Xen Linux hangs near the end > of its boot. Last messages from Linux are: > > i2c /dev entries driver IPv4 over IPv4 tunneling driver > TCP bic registered > NET: Registered protocol family 1 > NET: Registered protocol family 17 > > ...then nothing. > > Shortly after this point in the boot is where the RAMDISK is > decompressed and accessed. I'm wondering if the boot related memory > improvements have affected a RAMDISK built into Linux and Xen. Have you tried attaching GDB to systemsim to figure out what's going on? > Has anyone else had recent changesets working on the simulator? I haven't tried simulator in quite some time... -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
Re: [XenPPC] systemsim-gpul problems
I pull only from xenppc-unstable.hg and linux-ppc-2.6.hg. I'm up to date with all latest that was pushed to them, and this combo runs on systemsim-gpul. There was a bug(s?) that caused symptoms like you mention, but they were fixed by Amos Waterland about 2nd week in December, and pushed to aformentioned repositories by the maintainers. I can't quote or vouch for specific changesets. Mark Mergen [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 01/04/2007 03:11 PM Please respond to [EMAIL PROTECTED] To Yoder Stuart-B08248 <[EMAIL PROTECTED]> cc xen-ppc-devel@lists.xensource.com Subject Re: [XenPPC] systemsim-gpul problems On Thu, 2007-01-04 at 10:32 -0700, Yoder Stuart-B08248 wrote: > I've been trying to get systemsim-gpul working with the > latest Xen-PPC. > > The last changeset that worked was 7ad4645e7a54 (11/22/06). > > Changeset ce8c1e26b2ae (Early boot memory avoidance improvemnts) > broke the simulator with the 'Could not allocate RTAS tree' / HANG > error. > > With changeset 878ce1f78ad3 (Fix systemsim-gpul failure to boot) > and later changesets the RTAS allocation / HANG problem is fixed > but the simulator still won't boot into Linux. > > If I compare logs of the working and non-working Xen/Linux runs > in the simulator, with the current Xen Linux hangs near the end > of its boot. Last messages from Linux are: > > i2c /dev entries driver IPv4 over IPv4 tunneling driver > TCP bic registered > NET: Registered protocol family 1 > NET: Registered protocol family 17 > > ...then nothing. > > Shortly after this point in the boot is where the RAMDISK is > decompressed and accessed. I'm wondering if the boot related memory > improvements have affected a RAMDISK built into Linux and Xen. Have you tried attaching GDB to systemsim to figure out what's going on? > Has anyone else had recent changesets working on the simulator? I haven't tried simulator in quite some time... -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
RE: [XenPPC] systemsim-gpul problems
> -Original Message- > From: Hollis Blanchard [mailto:[EMAIL PROTECTED] > Sent: Thursday, January 04, 2007 2:12 PM > > Have you tried attaching GDB to systemsim to figure out > what's going on? > I haven't used gdb w/ the simulator yet-- I need to figure out how to do that. I have turned on debug prints in arch/powerpc/boot_of.c. One thing I'm puzzling over is why boot_of_alloc_init() is marking overlapping regions of memory. Does that make sense?? [snip] boot_of_alloc_init: marking 0x0 - 0x0^M boot_of_alloc_init: marking 0x0 - 0x400^M boot_of_alloc_init: marking 0x40 - 0x88b000^M boot_of_alloc_init: marking 0x0 - 0x3000^M [snip] Also, why is there a region w/ zero length? Stuart ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
Re: [XenPPC] systemsim-gpul problems
On Thu, 2007-01-04 at 10:32 -0700, Yoder Stuart-B08248 wrote: > I've been trying to get systemsim-gpul working with the > latest Xen-PPC. > > The last changeset that worked was 7ad4645e7a54 (11/22/06). > > Changeset ce8c1e26b2ae (Early boot memory avoidance improvemnts) > broke the simulator with the 'Could not allocate RTAS tree' / HANG > error. > > With changeset 878ce1f78ad3 (Fix systemsim-gpul failure to boot) > and later changesets the RTAS allocation / HANG problem is fixed > but the simulator still won't boot into Linux. > > If I compare logs of the working and non-working Xen/Linux runs > in the simulator, with the current Xen Linux hangs near the end > of its boot. Last messages from Linux are: > > i2c /dev entries driver IPv4 over IPv4 tunneling driver > TCP bic registered > NET: Registered protocol family 1 > NET: Registered protocol family 17 > > ...then nothing. > > Shortly after this point in the boot is where the RAMDISK is > decompressed and accessed. I'm wondering if the boot related memory > improvements have affected a RAMDISK built into Linux and Xen. Have you tried attaching GDB to systemsim to figure out what's going on? > Has anyone else had recent changesets working on the simulator? I haven't tried simulator in quite some time... -- Hollis Blanchard IBM Linux Technology Center ___ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
