Hi Gabe, Thanks for uploading the changes. Typically it'll take less than an hour for RISC-V.
Regards, Hoa Nguyen On Wed, Dec 15, 2021 at 10:38 PM Gabe Black <[email protected]> wrote: > I have it running, although it's taking a while. How long should it take > to finish? Should it finish immediately, 15 minutes, a few hours, > tomorrow...? I'll upload my changes in the mean time. > > Gabe > > On Wed, Dec 15, 2021 at 9:13 PM Gabe Black <[email protected]> wrote: > >> Ok, I did/am doing a little more looking, and part of the problem seems >> to be that some of the python blobs executed with executePython assume >> prior blobs have done imports for them already. That was where the modules >> or their contents not being found was coming from, and then after that >> something about the error reporting was blowing up, but that wasn't the >> primary problem. >> >> With that fixed, it's still upset, but now it's because "fatal: Need to >> instantiate Root() before calling instantiate()". Also, gem5 is printing >> the help blob which seems wrong, so maybe the command line arguments aren't >> making it through like they should? It could be that no config script is >> run, and that's why m5.instantiate() doesn't work. >> >> It's also a little weird that there's an ordering dependency between the >> execPythonCommands invocations, considering the same set of those should >> still be happening in the same order. I suppose I'm not explicitly >> importing the m5 module into __main__.__dict__ to run m5.main(), so that >> may be where that's coming from. >> >> Anyway, I'm still chopping away at this, but I think it's pretty close. >> >> Gabe >> >> On Wed, Dec 15, 2021 at 8:29 PM Gabe Black <[email protected]> wrote: >> >>> Also, I have to say, while I'm grateful that there was a push to get the >>> SCons changes reviewed for the release, we wouldn't be in this crunch if >>> they had been reviewed a month ago. It would be best not to let things >>> bunch up and then make a herculean push right at the deadline, where >>> problems have a very limited time to get resolved. >>> >>> Gabe >>> >>> On Wed, Dec 15, 2021 at 4:38 PM Gabe Black <[email protected]> wrote: >>> >>>> That's actually very helpful, since I was pretty sure there was a >>>> memory problem but couldn't figure out where. I think there's a good chance >>>> this will be an easy fix. >>>> >>>> Gabe >>>> >>>> On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> >>>> wrote: >>>> >>>>> Ok, I was able to get a debugger to work and I dug in a bit more... >>>>> >>>>> The problem is that the new code does not initialize the __main__.py >>>>> module. The original SST code had the following line: >>>>> `pythonMain = PyImport_AddModule(PyCC("__main__"));` >>>>> >>>>> This was executed before `execPythonCommands`. >>>>> >>>>> In `execPythonCommands` we use the `pythonMain` variable. However, in >>>>> the *new* code, that variable is NULL (or unitialized, actually). >>>>> >>>>> As far as I can tell, all of the new python changes didn't come with >>>>> any documentation on how you are supposed to interact with Python from >>>>> gem5. So, there's no good way for us to update this code. >>>>> >>>>> Gabe, if you can update the `execPythonCommands` function to work >>>>> however you expected it to, that would be great. >>>>> >>>>> Thanks, >>>>> Jason >>>>> >>>>> On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power <[email protected]> >>>>> wrote: >>>>> >>>>>> Here's my understanding of where we are on this: >>>>>> >>>>>> As of >>>>>> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >>>>>> SST was working. (Committed Dec. 7th) >>>>>> >>>>>> Then, this large relation chain was pushed: >>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49425 ending >>>>>> with >>>>>> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >>>>>> This relation chain broke SST. This was committed on Dec. 8th. >>>>>> >>>>>> We have two options: Either we fix this bug ASAP (as in within 6 days >>>>>> so we can release on 12/22) or we revert the changesets that were >>>>>> committed >>>>>> on the 8th. >>>>>> >>>>>> SST is a *must have* feature of gem5 21.2. It is quite frustrating to >>>>>> see it broken after all of the hard work that Hoa and Giacomo put into >>>>>> it. >>>>>> >>>>>> Let me know if I'm missing something. >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> >>>>>> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>>>>>> >>>>>>> At least one problem I ran into was that the order of static >>>>>>> initializers was not determinstic, so the structures which said what >>>>>>> embedded python modules existed might be statically constructed after >>>>>>> the >>>>>>> thing that consumes and sets them up. That could work by coincidence if >>>>>>> the >>>>>>> order is favorable, but it wasn't for me and so lots of things didn't >>>>>>> exist >>>>>>> as far as the python was concerned. >>>>>>> >>>>>>> To be clear, there is some mysterious deeper issue here which is >>>>>>> probably separate from that. As far as I can tell running gem5 itself is >>>>>>> working fine, but when running as a library in sst there is some sort of >>>>>>> memory corruption or premature garbage collection or something, and it >>>>>>> leads to a segfault or incorrect behavior (failure to import modules, >>>>>>> failure to find things inside them, etc). Things were even more >>>>>>> tempermental until I added what feels like an unnecessary increment of >>>>>>> the >>>>>>> reference count for the modules in the new pybind_init.hh, which seemed >>>>>>> to >>>>>>> tame things in gem5 proper at least. I'm too tired to keep looking for >>>>>>> it >>>>>>> right now, but hopefully this will give you a good leg up. >>>>>>> >>>>>>> Gabe >>>>>>> >>>>>>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Everything was working as of last week-ish, if I remember >>>>>>>> correctly. We were booting both Arm and RISCV full system with gem5 >>>>>>>> cores >>>>>>>> and SST caches/memory. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Jason >>>>>>>> >>>>>>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks, that was really helpful. I've fixed the compilation >>>>>>>>> problems, and then a few more that were hiding behind that, including >>>>>>>>> a few >>>>>>>>> ones that were intrinsic to the way sst was initializing the gem5 >>>>>>>>> library >>>>>>>>> (with dlopen). Was this working to begin with and all this breakage >>>>>>>>> is new, >>>>>>>>> or was it just that the compilation broke and there were these other >>>>>>>>> problems already? I want to make sure I don't leave things in a worse >>>>>>>>> state >>>>>>>>> than when I got here, but I also don't want to go chasing down things >>>>>>>>> that >>>>>>>>> weren't working to begin with. >>>>>>>>> >>>>>>>>> I'm going to upload what I have so far, and hopefully someone else >>>>>>>>> can take a look at it too. The current problem seems to be that the >>>>>>>>> modules >>>>>>>>> in _m5 aren't working right, either because they've been garbage >>>>>>>>> collected, >>>>>>>>> or because they're not being set up properly, or... >>>>>>>>> >>>>>>>>> Gabe >>>>>>>>> >>>>>>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hey gabe. >>>>>>>>>> >>>>>>>>>> No idea if this is the _best_ solution to your problem, but my >>>>>>>>>> solution would be to rebuild the image with this installed. Modify >>>>>>>>>> the >>>>>>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you >>>>>>>>>> want. Then >>>>>>>>>> run `docker build -t <some name here> util/dockerfiles/sst-11.1.0` >>>>>>>>>> to build >>>>>>>>>> an image with the name "<some name here>". >>>>>>>>>> >>>>>>>>>> Then you can execute `docker run -u $UID:$GID --volume >>>>>>>>>> $(pwd):$(pwd) -w $(pwd) -it <same name here>` within the gem5 >>>>>>>>>> directory to >>>>>>>>>> spin up and enter a container from the image you just built. I think >>>>>>>>>> you'll >>>>>>>>>> be able to do what you want inside the container. >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> Bobby >>>>>>>>>> -- >>>>>>>>>> Dr. Bobby R. Bruce >>>>>>>>>> Room 3050, >>>>>>>>>> Kemper Hall, UC Davis >>>>>>>>>> Davis, >>>>>>>>>> CA, 95616 >>>>>>>>>> >>>>>>>>>> web: https://www.bobbybruce.net >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > Dumb question: I'm trying to run gdb inside this container on >>>>>>>>>> the sst thing. How do I do that? It's not installed in the container >>>>>>>>>> now, >>>>>>>>>> and I can't (easily) figure out how to get it installed. I can tell >>>>>>>>>> docker >>>>>>>>>> to install it, but then it seems to throw that away as soon as the >>>>>>>>>> command >>>>>>>>>> ends. >>>>>>>>>> > >>>>>>>>>> > Gabe >>>>>>>>>> > >>>>>>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >> >>>>>>>>>> >> Thanks Gabe, >>>>>>>>>> >> >>>>>>>>>> >> This is very much appreciated. I'm going to create the release >>>>>>>>>> staging once a couple more things get in. Feel free to push any >>>>>>>>>> patches >>>>>>>>>> related to these bugs to the release staging branch. >>>>>>>>>> >> >>>>>>>>>> >> If there is an order of priority I'd say the bug affecting SST >>>>>>>>>> is of higher importance than that affecting the Weeklies (as far as >>>>>>>>>> I can >>>>>>>>>> see the latter is hard to trigger). That being said, we'll apply >>>>>>>>>> both to >>>>>>>>>> the new release one way or another. >>>>>>>>>> >> >>>>>>>>>> >> -- >>>>>>>>>> >> Dr. Bobby R. Bruce >>>>>>>>>> >> Room 3050, >>>>>>>>>> >> Kemper Hall, UC Davis >>>>>>>>>> >> Davis, >>>>>>>>>> >> CA, 95616 >>>>>>>>>> >> >>>>>>>>>> >> web: https://www.bobbybruce.net >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>> >>>>>>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last >>>>>>>>>> couple days but kept running out of time. I'm sitting down to work >>>>>>>>>> on it >>>>>>>>>> right now. >>>>>>>>>> >>> >>>>>>>>>> >>> Gabe >>>>>>>>>> >>> >>>>>>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>> >>>>>>>>>> >>>> Hey Gabe, >>>>>>>>>> >>>> >>>>>>>>>> >>>> Is there any update on this? >>>>>>>>>> >>>> >>>>>>>>>> >>>> Kind regards, >>>>>>>>>> >>>> Bobby >>>>>>>>>> >>>> -- >>>>>>>>>> >>>> Dr. Bobby R. Bruce >>>>>>>>>> >>>> Room 3050, >>>>>>>>>> >>>> Kemper Hall, UC Davis >>>>>>>>>> >>>> Davis, >>>>>>>>>> >>>> CA, 95616 >>>>>>>>>> >>>> >>>>>>>>>> >>>> web: https://www.bobbybruce.net >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Hi Gabe, >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> I have more details about this. In this use case, SST >>>>>>>>>> initialized the >>>>>>>>>> >>>>> Python environment before adding the "gem5 object". This >>>>>>>>>> gem5 object >>>>>>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> The function that does that is initPython() >>>>>>>>>> >>>>> >>>>>>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> The following commands will pull the docker image for SST >>>>>>>>>> testing >>>>>>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root must >>>>>>>>>> be >>>>>>>>>> >>>>> specified), >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> ``` >>>>>>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>>>>>> >>>>> bash -c "\ >>>>>>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} >>>>>>>>>> --without-tcmalloc; \ >>>>>>>>>> >>>>> cd ext/sst; \ >>>>>>>>>> >>>>> make clean; make; \ >>>>>>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>>>>>> >>>>> ``` >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> We appreciate your help! >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> Regards, >>>>>>>>>> >>>>> Hoa Nguyen >>>>>>>>>> >>>>> >>>>>>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> wrote: >>>>>>>>>> >>>>> > Hey Gabe, >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>>>>>> integration python is >>>>>>>>>> >>>>> > initialized from the SST module, not from init.cc (this >>>>>>>>>> is because SST has >>>>>>>>>> >>>>> > their own python interpreter). >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and >>>>>>>>>> Giacomo can give you >>>>>>>>>> >>>>> > an example that's breaking to help you fix it, I believe. >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > >>>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > There's strong interest in having the SST integration >>>>>>>>>> working in this >>>>>>>>>> >>>>> > current release. We've spent a lot of time figuring out >>>>>>>>>> all of the >>>>>>>>>> >>>>> > intricacies and would appreciate any help you can provide >>>>>>>>>> in these last few >>>>>>>>>> >>>>> > days before the release! >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > Thanks! >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> > Jason >>>>>>>>>> >>>>> > >>>>>>>>>> >>>>> _______________________________________________ >>>>>>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>>>>>> >>>>>>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
