Hi Joel, Thanks again for the details.
I am not sure I grok the latency and queue argument still. Adding a larger response queue does not increase the latency unless there is also a bunch of transactions queued up. Am I missing something? We can easily set the static pipeline latency to a high value, but 100 ns sounds much too large. What is that value based on? Do you have details on the addressing scheme you mention? In GDDR5, the command bus is DDR and the data bus QDR. The part you refer to is either running at 5.5 Gbps or 6 Gbps according to the data sheet, so I do find the 4 Gbps surprising. Are you sure? The burst length I am talking about is the DRAM burst length, which for GDDR5 is 8 beats (so with a x32 mode that would be 32 bytes per burst. I do not see this specified anywhere in the line you sent, and therefore I am curious if it is omitted completely (DRAMSim2 for example makes dangerous assumptions here). Thanks, Andreas On 14/10/2014 00:22, "Joel Hestness via gem5-dev" <[email protected]> wrote: >Hi Andreas, > > >> Thanks. I really do not understand the return queue argument. Why on >>earth >> would you need such a large return queue? Surely the agent making the >> requests (the GPU in this case) should have allocated space for the >> response, no? >> > >Good question. I believe that buffer is just to get the appropriate >overall >minimum latency for memory accesses (i.e. GPGPU-Sim does some level of >functional controller modeling, and the buffer is just for timing). If >there a way to add arbitrary latency to the DRAMCtrl, it would be nice to >aim for no-load latency of roughly 100ns (from when the controller >receives >the access to when a response is returned to the Ruby network). Also, if I >have a chance to test out Nilay's patch, I could tune on this setting. > >Concerning the configuration, what is the assumed clock speed (tCK), and >> is it operated as a x16 or x32 part? Is the burst length implicit in >>your >> configuration (or is 8 the default)? >> > >The memory modeled in GPGPU-Sim is x32 based on the addressing scheme. >That's where I'd start the mode. > >In GDDR5, the core frequency is 1/4 of command frequency (and 1/2 in >GDDR3?). A common channel frequency is 4GHz, resulting in effective >channel >bandwidth of 32GT/s. So, a starting baseline of tCK = 1ns should be >sufficient. Does that sound right? > >I believe burst length is typically 8 and can be much longer. In practice, >this depend highly on address hashing/interleaving across many controllers >in various different GPUs, so 8 should be a sufficient baseline. > > > Joel > > > >On 10/13/14, 10:28 PM, "Joel Hestness via gem5-dev" <[email protected]> >> wrote: >> >> >Hi Andreas, >> > Sure thing. We try to closely replicate the parameters used in >>GPGPU-Sim >> >v3.2.2, which are specified in the Hynix datasheet here: >> >http://www.hynix.com/datasheet/pdf/graphics/H5GQ1H24AFR(Rev1.0).pdf. >> > >> > Here are relevant excerpts from the GPGPU-Sim GTX480 config file >> >(attached): >> > >> ># The DRAM return queue and the scheduler queue together should provide >> >buffer >> ># to sustain the memory level parallelism to tolerate DRAM latency >> ># To allow 100% DRAM utility, there should at least be enough buffer to >> >sustain >> ># the minimum DRAM latency (100 core cycles). I.e. >> ># Total buffer space required = 100 x 924MHz / 700MHz = 132 >> >-gpgpu_frfcfs_dram_sched_queue_size 16 >> >-gpgpu_dram_return_queue_size 116 >> > >> >-dram_data_command_freq_ratio 4 # GDDR5 is QDR >> >-gpgpu_dram_timing_opt "nbk=16:CCD=2:RRD=6:RCD=12:RAS=28:RP=12:RC=40: >> > CL=12:WL=4:CDLR=5:WR=12:nbkgrp=4:CCDL=3:RTPL=2" >> > >> > If anything needs clarification, I'm happy to help sort it out. Just >>let >> >me know. >> > >> > Thanks! >> > Joel >> > >> > >> > >> >On Mon, Oct 13, 2014 at 4:11 PM, Andreas Hansson via gem5-dev < >> >[email protected]> wrote: >> > >> >> Hi Joel, >> >> >> >> I am happy to spend the 5 minutes creating a GDDR5 configuration. Do >>you >> >> have any specific data sheet you would like to capture? >> >> >> >> Andreas >> >> >> >> On 10/13/14, 10:09 PM, "Joel Hestness via gem5-dev" >><[email protected]> >> >> wrote: >> >> >> >> >Hi guys, >> >> > >> >> > >> >> >> Thanks for the clarification. I believe the RubyMemoryController >>is >> >> >> completely Pareto dominated by the vanilla DRAMCtrl module, but if >> >>there >> >> >> is any specific feature/setting missing I would be keen to know. >> >> >> >> >> >> If possible I would like to make sure we use the same controller >>as a >> >> >> default for all timing simulations (even if the other one would be >> >> >> maintained as a fallback). >> >> > >> >> > >> >> >I'd like to second the desire to have a simple replacement baseline >> >>that >> >> >performs at least as well as the RubyMemoryController in most/all >> >>cases: >> >> >gem5-gpu now has more than 100 users, and as far as I know, we are >>all >> >> >using Ruby and thus the RubyMemoryController. The >>RubyMemoryController >> >>is >> >> >pretty simple to configure similarly to DDR3 or GDDR5 and to >>interpret >> >> >results. It performs surprisingly close to some GPU hardware. If >>this >> >> >controller goes away, I (and I'm sure other gem5-gpu users) would >> >>prefer >> >> >to >> >> >have something that is known to perform as well and is also easy to >> >> >configure. >> >> > >> >> >I think we (the gem5-gpu crew) are fine with the >>RubyMemoryController >> >> >going >> >> >away eventually. However, given that there isn't currently a >>GDDR-like >> >> >DRAMCtrl configuration in gem5, I'd like to second Nilay and Brad >>that >> >>we >> >> >offer users sufficient time to prepare for RubyMemoryController >> >>removal. >> >> >We >> >> >will need to adapt our heterogeneous Ruby coherence protocols, and >> >>other >> >> >users have their own protocols they'd need to adapt as well. >> >> > >> >> > >> >> > Joel >> >> > >> >> > >> >> > >> >> > >> >> > >> >> >> On 10/13/14, 9:01 PM, "Nilay Vaish via gem5-dev" >><[email protected]> >> >> >> wrote: >> >> >> >> >> >> >On Mon, 13 Oct 2014, Andreas Hansson via gem5-dev wrote: >> >> >> > >> >> >> >> Hi all, >> >> >> >> >> >> >> >> With Nilay?s recent improvements to Ruby I would like to >> >>understand >> >> >>if >> >> >> >> there is any point in still having the RubyMemoryControl, or >>if we >> >> >> >> should just clean things up a bit and remove it. I would think >>the >> >> >>best >> >> >> >> way forward is to clean up the integration of Ruby and classic >>and >> >> >> >> ensure that there is no duplicated functionality beyond what is >> >> >> >>strictly >> >> >> >> necessary. >> >> >> >> >> >> >> >> Nilay, do you think this would make sense? Is there anyone else >> >>with >> >> >> >>any >> >> >> >> opinions in this matter? >> >> >> >> >> >> >> > >> >> >> > >> >> >> >I was in favor of dropping RubyMemoryControl. But I had some >> >> >>discussion >> >> >> >with Brad Beckmann from AMD. Since AMD has some infrastructure >>in >> >> >>place >> >> >> >already, they would like to retain RubyMemoryControl for the time >> >> >>being. >> >> >> > >> >> >> >I suggest that we retain the memory controller code in ruby for >> >>another >> >> >> >six months or so, and then we will drop it. In the mean time, >>we >> >> >> >will update the interface so that ruby protocols can use classic >> >>memory >> >> >> >controller. The code for this is already on the reviewboard. >>Over >> >> >>this >> >> >> >six month period, I hope, most users would have switched to using >> >> >>classic >> >> >> >controller. >> >> >> > >> >> >> >Thanks >> >> >> >Nilay >> >> >> >_______________________________________________ >> >> >> >gem5-dev mailing list >> >> >> >[email protected] >> >> >> >http://m5sim.org/mailman/listinfo/gem5-dev >> >> >> >> >> >> >> >> >> -- IMPORTANT NOTICE: The contents of this email and any >>attachments >> >>are >> >> >> confidential and may also be privileged. If you are not the >>intended >> >> >> recipient, please notify the sender immediately and do not >>disclose >> >>the >> >> >> contents to any other person, use it for any purpose, or store or >> >>copy >> >> >>the >> >> >> information in any medium. Thank you. >> >> >> >> >> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 >>9NJ, >> >> >> Registered in England & Wales, Company No: 2557590 >> >> >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge >>CB1 >> >> >>9NJ, >> >> >> Registered in England & Wales, Company No: 2548782 >> >> >> _______________________________________________ >> >> >> gem5-dev mailing list >> >> >> [email protected] >> >> >> http://m5sim.org/mailman/listinfo/gem5-dev >> >> >> >> >> > >> >> > >> >> > >> >> >-- >> >> > Joel Hestness >> >> > PhD Student, Computer Architecture >> >> > Dept. of Computer Science, University of Wisconsin - Madison >> >> > http://pages.cs.wisc.edu/~hestness/ >> >> >_______________________________________________ >> >> >gem5-dev mailing list >> >> >[email protected] >> >> >http://m5sim.org/mailman/listinfo/gem5-dev >> >> > >> >> >> >> >> >> -- IMPORTANT NOTICE: The contents of this email and any attachments >>are >> >> confidential and may also be privileged. If you are not the intended >> >> recipient, please notify the sender immediately and do not disclose >>the >> >> contents to any other person, use it for any purpose, or store or >>copy >> >>the >> >> information in any medium. Thank you. >> >> >> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >> >> Registered in England & Wales, Company No: 2557590 >> >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >> >>9NJ, >> >> Registered in England & Wales, Company No: 2548782 >> >> >> >> _______________________________________________ >> >> gem5-dev mailing list >> >> [email protected] >> >> http://m5sim.org/mailman/listinfo/gem5-dev >> >> >> > >> > >> > >> >-- >> > Joel Hestness >> > PhD Student, Computer Architecture >> > Dept. of Computer Science, University of Wisconsin - Madison >> > http://pages.cs.wisc.edu/~hestness/ >> >_______________________________________________ >> >gem5-dev mailing list >> >[email protected] >> >http://m5sim.org/mailman/listinfo/gem5-dev >> > >> >> >> -- IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy >>the >> information in any medium. Thank you. >> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >> Registered in England & Wales, Company No: 2557590 >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >>9NJ, >> Registered in England & Wales, Company No: 2548782 >> >> _______________________________________________ >> gem5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/listinfo/gem5-dev >> > > >-- > Joel Hestness > PhD Student, Computer Architecture > Dept. of Computer Science, University of Wisconsin - Madison > http://pages.cs.wisc.edu/~hestness/ >_______________________________________________ >gem5-dev mailing list >[email protected] >http://m5sim.org/mailman/listinfo/gem5-dev > -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782 _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
