I've figured it out.
To measure the time it took to get to main() I put a *return 0; *at the
beginning of the function so I wouldn't have to babysit it.
I didn't consider that it would also take some time for the simulator to
exit, which is where the extra few minutes comes from.
Side-note: *m5_exit(0);* instead of a return exits immediately.
5 min is a bit more reasonable of a slowdown for the difference between the
two clocks.
Two incidental things:
1. Is there a way to have gem5 spit out (real wall-clock) timestamps while
it's printing stuff?
2. A while ago I asked about hipDeviceSynchronize(); causing crashes
(panic: Tried to read unmapped address 0xff0000c29f48.). Has this been
fixed since?
I'm going to update to the head of this branch soon, and eventually to the
main branch. If it hasn't been fixed I've created a workaround by stealing
the completion signal of the kernel based on its launch id, and manually
waiting for it using the HSA interface.
Happy to help out and implement this as a m5op (or something) if that would
be helpful for you guys.
Best,
Dan
On Thu, Jun 11, 2020 at 12:40 PM Matt Sinclair <mattdsincl...@gmail.com>
wrote:
> I don't see anything amazingly amiss in your output, but the number of
> times the open/etc. fail is interesting -- Kyle do we see the same thing?
> If not, it could be that you should update your apu_se.py to point to the
> "correct" place to search for the libraries first?
>
> Also, based on Kyle's reply, Dan how long does it take you to boot up
> square? Certainly a slower machine might take longer, but it does seem
> even slower than expected. But if we're trying the same application, maybe
> it will be easier to spot differences.
>
> I would also recommend updating to the latest commit on the staging branch
> -- I don't believe it should break anything with those patches.
>
> Yes, looks like you are using the release version of ROCm -- no issues
> there.
>
> Matt
>
>
>
> On Thu, Jun 11, 2020 at 9:38 AM Daniel Gerzhoy <daniel.gerz...@gmail.com>
> wrote:
>
>> I am using the docker, yeah.
>> It's running on our server cluster which is a Xeon Gold 5117 @ (2.0 - 2.8
>> GHz) which might make up some of the difference, the r5 3600 has a faster
>> clock (3.6-4.2 GHz).
>>
>> I've hesitated to update my branch because in the Dockerfile it
>> specifically checks this branch out and applies a patch, though the patch
>> isn't very extensive.
>> This was from a while back (November maybe?) and I know you guys have
>> been integrating things into the main branch (thanks!)
>> I was thinking I would wait until it's fully merged into the mainline
>> gem5 branch and rebase onto that and try to merge my changes in.
>>
>> Last I checked the GCN3 stuff is in the dev branch not the master right?
>>
>> But if it will help maybe I should update to the head of this branch.
>> Will I need to update the docker as well?
>>
>> As for the debug vs release rocm I think I'm using the release version.
>> This is what the dockerfile built:
>>
>> ARG rocm_ver=1.6.2
>> RUN wget -qO- repo.radeon.com/rocm/archive/apt_${rocm_ver}.tar.bz2
>> <http://repo.radeon.com/rocm/archive/apt_$%7Brocm_ver%7D.tar.bz2> \
>> | tar -xjv \
>> && cd apt_${rocm_ver}/pool/main/ \
>> && dpkg -i h/hsakmt-roct-dev/* \
>> && dpkg -i h/hsa-ext-rocr-dev/* \
>> && dpkg -i h/hsa-rocr-dev/* \
>> && dpkg -i r/rocm-utils/* \
>> && dpkg -i h/hcc/* \
>> && dpkg -i h/hip_base/* \
>> && dpkg -i h/hip_hcc/* \
>> && dpkg -i h/hip_samples/*
>>
>>
>> I ran a benchmark that prints that it entered main and returns
>> immediately, this took 9 minutes.
>> I've attached a debug trace with debug flags = "GPUDriver,SyscallVerbose"
>> There's a lot of weird things going on, "syscall open: failed", "syscall
>> brk: break point changed to [...]", and lots of ignored system calls.
>>
>> head of Stats for reference:
>> ---------- Begin Simulation Statistics ----------
>> sim_seconds 0.096192
>> # Number of seconds simulated
>> sim_ticks 96192368500
>> # Number of ticks simulated
>> final_tick 96192368500
>> # Number of ticks from beginning of simulation (restored from checkpoints
>> and never reset)
>> sim_freq 1000000000000
>> # Frequency of simulated ticks
>> host_inst_rate 175209
>> # Simulator instruction rate (inst/s)
>> host_op_rate 338409
>> # Simulator op (including micro ops) rate (op/s)
>> host_tick_rate 175362515
>> # Simulator tick rate (ticks/s)
>> host_mem_usage 1628608
>> # Number of bytes of host memory used
>> host_seconds 548.53
>> # Real time elapsed on the host
>> sim_insts 96108256
>> # Number of instructions simulated
>> sim_ops 185628785
>> # Number of ops (including micro ops) simulated
>> system.voltage_domain.voltage 1
>> # Voltage in Volts
>> system.clk_domain.clock 1000
>> # Clock period in ticks
>>
>> Maybe something in the attached file explains it better than I can
>> express.
>>
>> Many thanks for your help and hard work!
>>
>> Dan
>>
>>
>>
>>
>>
>> On Thu, Jun 11, 2020 at 3:32 AM Kyle Roarty <kroa...@wisc.edu> wrote:
>>
>>> Running through a few applications, it took me about 2.5 minutes or less
>>> each time using docker to start executing the program on an r5 3600.
>>>
>>> I ran square, dynamic_shared, and MatrixTranspose (All from HIP) which
>>> took about 1-1.5 mins.
>>>
>>> I ran conv_bench and rnn_bench from DeepBench which took just about 2
>>> minutes.
>>>
>>> Because of that, it's possible the size of the app has an effect on
>>> setup time, as the HIP apps are extremely small.
>>>
>>> Also, the commit Dan is checked out on is d0945dc
>>> <https://gem5.googlesource.com/amd/gem5/+/d0945dc285cf146de160808d7e6d4c1fd3f73639>
>>> mem-ruby:
>>> add cache hit/miss statistics for TCP and TCC
>>> <https://gem5.googlesource.com/amd/gem5/+/d0945dc285cf146de160808d7e6d4c1fd3f73639>,
>>> which isn't the most recent commit. I don't believe that that would account
>>> for such a large slowdown, but it doesn't hurt to try the newest commit
>>> unless it breaks something.
>>>
>>> Kyle
>>> ------------------------------
>>> *From:* mattdsincl...@gmail.com <mattdsincl...@gmail.com>
>>> *Sent:* Thursday, June 11, 2020 1:15 AM
>>> *To:* gem5 users mailing list <gem5-users@gem5.org>
>>> *Cc:* Daniel Gerzhoy <daniel.gerz...@gmail.com>; GAURAV JAIN <
>>> gja...@wisc.edu>; Kyle Roarty <kroa...@wisc.edu>
>>> *Subject:* Re: [gem5-users] GCN3 GPU Simulation Start-Up Time
>>>
>>> Gaurav & Kyle, do you know if this is the case?
>>>
>>> Dan, I believe the short answer is yes although 7-8 minutes seems a
>>> little long. Are you running this in Kyle's Docker, or separately? If in
>>> the Docker, that does increase the overhead somewhat, so running it
>>> directly on a system would likely reduce the overhead somewhat. Also, are
>>> you running with the release or debug version of the ROCm drivers? Again,
>>> debug version will likely add some time to this.
>>>
>>> Matt
>>>
>>> On Wed, Jun 10, 2020 at 2:00 PM Daniel Gerzhoy via gem5-users <
>>> gem5-users@gem5.org> wrote:
>>>
>>> I've been running simulations using the GCN3 branch:
>>>
>>> rocm_ver=1.6.2
>>> $git branch
>>> * (HEAD detached at d0945dc)
>>> agutierr/master-gcn3-staging
>>>
>>> And I've noticed that it takes roughly 7-8 minutes to get to main()
>>>
>>> I'm guessing that this is the simulator setting up drivers?
>>> Is that correct? Is there other stuff going on?
>>>
>>> *Has anyone found a way to speed this up? *
>>>
>>> I am trying to get some of the rodinia benchmarks from the HIP-Examples
>>> running and debugging takes a long time as a result.
>>>
>>> I suspect that this is unavoidable but I won't know if I don't ask!
>>>
>>> Cheers,
>>>
>>> Dan Gerzhoy
>>> _______________________________________________
>>> gem5-users mailing list -- gem5-users@gem5.org
>>> To unsubscribe send an email to gem5-users-le...@gem5.org
>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
>>>
>>>
_______________________________________________
gem5-users mailing list -- gem5-users@gem5.org
To unsubscribe send an email to gem5-users-le...@gem5.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
