Hi Korey, Yes, let's move this conversation back to m5-dev, since I think others may be interested and could help.
I don't know what the problem is exactly, but at some point of time (probably back in the early GEMS days) I seem to remember the Set code included an assertion check about the 31st bit in 32-bit mode. Therefore, I think we knew about this problem and made sure that never happened. I believe that is why we used to have a restriction that Ruby could only support 16 processors. I'm really fuzzy on the details...maybe someone else can elaborate. In the end, I just want to make sure we add something in the code that makes sure we don't encounter this problem again. This is one of those bugs that can take a while to track down, if you don't catch it right when it happens with an assertion. Brad From: koreylsew...@gmail.com [mailto:koreylsew...@gmail.com] On Behalf Of Korey Sewell Sent: Tuesday, April 05, 2011 7:14 AM To: Beckmann, Brad Subject: Re: [m5-dev] Running Ruby w/32 Cores Hi again Brad, I looked this over again and although my 32-bit patch "fixes" things, now that I look at it again, I'm not convinced that I actually fixed the symptom of the bug but rather the cause of the bug. Do you happen to know what are the problems with the 32-bit Set counts? Sorry for prolonging the issue, but I thought I had put this to bed but maybe not. Finally, it may not matter that this works on 32-bit machines but it'd be nice if it did. (Let me know if I should move this convo to the m5-dev list) I end up checking the last bit in the count function manually (the code as follows): int Set::count() const { int counter = 0; long mask; for (int i = 0; i < m_nArrayLen; i++) { mask = (long)0x01; for (int j = 0; j < LONG_BITS; j++) { // FIXME - significant performance loss when array // population << LONG_BITS if ((m_p_nArray[i] & mask) != 0) { counter++; } mask = mask << 1; } #ifndef _LP64 long msb_mask = 0x80000000; if ((m_p_nArray[i] & msb_mask) != 0) { counter++; } #endif } return counter; } On Tue, Apr 5, 2011 at 1:30 AM, Korey Sewell <ksew...@umich.edu<mailto:ksew...@umich.edu>> wrote: Brad, it looks like you were right on the money here. I found the spot where it was returning the wrong value via a SLICC function to "count sharers for everyone except the owner". I realized that the machine that I use for testing is just a 32-bit machine, and like you warned there look to be issues with the Set type there. I ran the Fft-32 cores on a 64-bit machine and it seems to work correctly. I'll be running on the full splash/parsec suites soon and that should stress Ruby a good bit :). I have a patch that checks to see if _LP64 is defined, and if not check that last bit when doing the set count function. Thanks for being helpful in debugging. It was a "relatively" easy bug, but as always going through code and becoming more proficient at getting around while trying to solve a bug is really helpful. On Fri, Apr 1, 2011 at 7:28 PM, Beckmann, Brad <brad.beckm...@amd.com<mailto:brad.beckm...@amd.com>> wrote: Ok for the first trace, the critical line is the following: 348523 0 L2Cache L1_GETX ILOSX>IFLXO [0x16180, line 0x16180] [NetDest (4) 0 - 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 - 0 0 - 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 - ]30 L2Cache identifies that 31 caches have a shared copy and that L1 cache 9 (L1-9) is the owner. When L1Cache 0 (L1-0) issues a GETX, the L2Cache issues 30 Inv probes, forwards the GETX to L1-9, and sends an ack to L1-0 itself. However, the L2 cache tells L1-0 to expect only 30 acks instead of 31. It could be something wrong with the NetDest::count() function, or the Set::count() function? I slightly modified my previous patch to isolate on what value the NetDest::count() function is returning. If it is returning 30, instead of 31, then it must be a problem with NetDest. You are compiling gem5 as a 64-bit binary, right? The second problem is essentially the same issue. L2Cache 31 (L2-31) is the owner of the block, but I suspect NetDest is not counting bit 31 and thus it is returning a count of 0...causing the error. Overall, concentrate on that NetDest::count function, or more importantly the Set::count() function. Once you find out the problem, please let me know. Thanks, Brad From: koreylsew...@gmail.com<mailto:koreylsew...@gmail.com> [mailto:koreylsew...@gmail.com<mailto:koreylsew...@gmail.com>] On Behalf Of Korey Sewell Sent: Friday, April 01, 2011 12:00 PM To: Beckmann, Brad Subject: Re: [m5-dev] Running Ruby w/32 Cores Brad, attached are the protocol traces grep'd for the offending addresses. I'm going to spend the weekend digging through Ruby code so hopefully I'm pretty close to generating the fixes myself. The first trace, fft.l2-1.trace, the offending address is 0x16b80 and the second trace the address is 0x1fc0. Please let me know if you anything catches your eye when you get some time. On Thu, Mar 31, 2011 at 5:01 PM, Beckmann, Brad <brad.beckm...@amd.com<mailto:brad.beckm...@amd.com>> wrote: Yes, there should have been an attached patch. Here it is again. Brad > -----Original Message----- > From: m5-dev-boun...@m5sim.org<mailto:m5-dev-boun...@m5sim.org> > [mailto:m5-dev-boun...@m5sim.org<mailto:m5-dev-boun...@m5sim.org>] > On Behalf Of Korey Sewell > Sent: Thursday, March 31, 2011 1:10 PM > To: M5 Developer List > Subject: Re: [m5-dev] Running Ruby w/32 Cores > > Is there an attached patch I should be running or did it get bounced by m5- > dev? If so, can you send it directly to me rather through m5-dev? > > On Wed, Mar 30, 2011 at 8:26 PM, Beckmann, Brad > <brad.beckm...@amd.com<mailto:brad.beckm...@amd.com>> wrote: > > Hi Korey, > > > > For the first trace, it looks like the L2 cache is either miscounting the > number of valid L1 copies, or there is an error with the ack arithmetic. We > are going to need a bit more information to figure out where the exact > problem is. Could you apply the attached patch and reply with the new > protocol trace? Thanks. > > > > For the second trace, you should be able to get the offending address by > simply attaching GDB to the aborted process. Without knowing which > address to zero in on, it is the proverbial "finding a needle in a haystack". > > > > Thanks, > > > > Brad > > > > > > > >> -----Original Message----- > >> From: m5-dev-boun...@m5sim.org<mailto:m5-dev-boun...@m5sim.org> > >> [mailto:m5-dev-<mailto:m5-dev-> > boun...@m5sim.org<mailto:boun...@m5sim.org>] On > >> Behalf Of Korey Sewell > >> Sent: Tuesday, March 29, 2011 10:15 AM > >> To: M5 Developer List > >> Subject: Re: [m5-dev] Running Ruby w/32 Cores > >> > >> Thanks for the response Brad. > >> > >> The 1st trace has 1 L2 and the 2nd has 1 L2 (i had a typo in the original > email). > >> > >> For each trace, I attach the stdout/stderr (*.out) and then the > >> protocol trace (*.prottrace). > >> > >> Also, in the 1st trace, the offending address is clear and I isolate > >> that in the protocol trace file provided. However, in the 2nd trace, > >> it's unclear (currently) which access caused it to fail so I took the > >> whole protocol trace file and gzip'd it. > >> > >> My current lack of expertise in SLICC limits me a bit, but I'd like > >> to be more helpful in debugging so if there is anything that I can > >> look into (or run) on my end to expedite the process, please advise. > >> In the interim, I'll try to locate the exact address that's breaking trace > >> 2 > and then hopefully repost that. > >> > >> Thanks! > >> > >> -Korey > >> > >> On Tue, Mar 29, 2011 at 12:02 PM, Beckmann, Brad > >> <brad.beckm...@amd.com<mailto:brad.beckm...@amd.com>> wrote: > >> > Hi Korey, > >> > > >> > I believe both of these issues should be easy to solve once we have > >> > a > >> protocol trace leading up to the error. If you could create such a > >> trace and send it to the list, that would be great. Just zero in on the > offending address. > >> > > >> > Thanks, > >> > > >> > Brad > >> > > >> > > >> >> -----Original Message----- > >> >> From: m5-dev-boun...@m5sim.org<mailto:m5-dev-boun...@m5sim.org> > >> >> [mailto:m5-dev-<mailto:m5-dev-> > >> boun...@m5sim.org<mailto:boun...@m5sim.org>] On > >> >> Behalf Of Korey Sewell > >> >> Sent: Tuesday, March 29, 2011 8:11 AM > >> >> To: M5 Developer List > >> >> Subject: [m5-dev] Running Ruby w/32 Cores > >> >> > >> >> Hi All, > >> >> I'm still having a bit of trouble running Ruby with 32+ cores. I > >> >> am experimenting w/configs varying the l2-caches. The runs seems > >> >> to generate various errors in the SLICC. > >> >> > >> >> Has anybody seen these or have any insight to how to start solving > >> >> these type of issues (posted below)? > >> >> ========= > >> >> The command line and errors are as follows: > >> >> (1) 32 Cores and 32 L2s > >> >> build/ALPHA_FS_MOESI_CMP_directory/m5.opt > >> >> configs/example/ruby_fs.py -b FftBase32 -n 32 --num-dirs=32 --num- > >> >> l2caches=32 ... > >> >> info: Entering event queue @ 0. Starting simulation... > >> >> Runtime Error at MOESI_CMP_directory-dir.sm:155, Ruby Time: 38279: > >> >> assert failure, PID: 5990 > >> >> press return to continue. > >> >> > >> >> Program aborted at cycle 19139500 > >> >> Aborted > >> >> > >> >> (2) 32 Cores and 1 L2 > >> >> build/ALPHA_FS_MOESI_CMP_directory/m5.opt > >> >> configs/example/ruby_fs.py -b FftBase32 -n 32 --num-dirs=32 --num- > >> >> l2caches=32 ... > >> >> fatal: Invalid transition > >> >> system.l1_cntrl0 time: 349075 addr: [0x16180, line 0x16180] event: > >> >> Ack > >> state: > >> >> MM @ cycle 174537500 > >> >> > >> > [doTransitionWorker:build/ALPHA_FS_MOESI_CMP_directory/mem/protoc > >> >> ol/L1Cache_Transitions.cc, > >> >> line 477] > >> >> Memory Usage: 2316756 KBytes > >> >> For more information see: http://www.m5sim.org/fatal/23f196b2 > >> >> ======== > >> >> > >> >> Please let me know if you do...Thanks! > >> >> > >> >> -- > >> >> - Korey > >> >> _______________________________________________ > >> >> m5-dev mailing list > >> >> m5-dev@m5sim.org<mailto:m5-dev@m5sim.org> > >> >> http://m5sim.org/mailman/listinfo/m5-dev > >> > > >> > > >> > _______________________________________________ > >> > m5-dev mailing list > >> > m5-dev@m5sim.org<mailto:m5-dev@m5sim.org> > >> > http://m5sim.org/mailman/listinfo/m5-dev > >> > > >> > >> > >> > >> -- > >> - Korey > > > > _______________________________________________ > > m5-dev mailing list > > m5-dev@m5sim.org<mailto:m5-dev@m5sim.org> > > http://m5sim.org/mailman/listinfo/m5-dev > > > > > > > > -- > - Korey > _______________________________________________ > m5-dev mailing list > m5-dev@m5sim.org<mailto:m5-dev@m5sim.org> > http://m5sim.org/mailman/listinfo/m5-dev -- - Korey -- - Korey -- - Korey _______________________________________________ m5-dev mailing list m5-dev@m5sim.org http://m5sim.org/mailman/listinfo/m5-dev
