Also, I have to say, while I'm grateful that there was a push to get the SCons changes reviewed for the release, we wouldn't be in this crunch if they had been reviewed a month ago. It would be best not to let things bunch up and then make a herculean push right at the deadline, where problems have a very limited time to get resolved.
Gabe On Wed, Dec 15, 2021 at 4:38 PM Gabe Black <[email protected]> wrote: > That's actually very helpful, since I was pretty sure there was a memory > problem but couldn't figure out where. I think there's a good chance this > will be an easy fix. > > Gabe > > On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> > wrote: > >> Ok, I was able to get a debugger to work and I dug in a bit more... >> >> The problem is that the new code does not initialize the __main__.py >> module. The original SST code had the following line: >> `pythonMain = PyImport_AddModule(PyCC("__main__"));` >> >> This was executed before `execPythonCommands`. >> >> In `execPythonCommands` we use the `pythonMain` variable. However, in the >> *new* code, that variable is NULL (or unitialized, actually). >> >> As far as I can tell, all of the new python changes didn't come with any >> documentation on how you are supposed to interact with Python from gem5. >> So, there's no good way for us to update this code. >> >> Gabe, if you can update the `execPythonCommands` function to work however >> you expected it to, that would be great. >> >> Thanks, >> Jason >> >> On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power <[email protected]> >> wrote: >> >>> Here's my understanding of where we are on this: >>> >>> As of >>> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >>> SST was working. (Committed Dec. 7th) >>> >>> Then, this large relation chain was pushed: >>> https://gem5-review.googlesource.com/c/public/gem5/+/49425 ending with >>> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >>> This relation chain broke SST. This was committed on Dec. 8th. >>> >>> We have two options: Either we fix this bug ASAP (as in within 6 days so >>> we can release on 12/22) or we revert the changesets that were committed on >>> the 8th. >>> >>> SST is a *must have* feature of gem5 21.2. It is quite frustrating to >>> see it broken after all of the hard work that Hoa and Giacomo put into it. >>> >>> Let me know if I'm missing something. >>> >>> Thanks, >>> Jason >>> >>> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> >>> wrote: >>> >>>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>>> >>>> At least one problem I ran into was that the order of static >>>> initializers was not determinstic, so the structures which said what >>>> embedded python modules existed might be statically constructed after the >>>> thing that consumes and sets them up. That could work by coincidence if the >>>> order is favorable, but it wasn't for me and so lots of things didn't exist >>>> as far as the python was concerned. >>>> >>>> To be clear, there is some mysterious deeper issue here which is >>>> probably separate from that. As far as I can tell running gem5 itself is >>>> working fine, but when running as a library in sst there is some sort of >>>> memory corruption or premature garbage collection or something, and it >>>> leads to a segfault or incorrect behavior (failure to import modules, >>>> failure to find things inside them, etc). Things were even more >>>> tempermental until I added what feels like an unnecessary increment of the >>>> reference count for the modules in the new pybind_init.hh, which seemed to >>>> tame things in gem5 proper at least. I'm too tired to keep looking for it >>>> right now, but hopefully this will give you a good leg up. >>>> >>>> Gabe >>>> >>>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power <[email protected]> >>>> wrote: >>>> >>>>> Everything was working as of last week-ish, if I remember correctly. >>>>> We were booting both Arm and RISCV full system with gem5 cores and SST >>>>> caches/memory. >>>>> >>>>> Cheers, >>>>> Jason >>>>> >>>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> wrote: >>>>> >>>>>> Thanks, that was really helpful. I've fixed the compilation problems, >>>>>> and then a few more that were hiding behind that, including a few ones >>>>>> that >>>>>> were intrinsic to the way sst was initializing the gem5 library (with >>>>>> dlopen). Was this working to begin with and all this breakage is new, or >>>>>> was it just that the compilation broke and there were these other >>>>>> problems >>>>>> already? I want to make sure I don't leave things in a worse state than >>>>>> when I got here, but I also don't want to go chasing down things that >>>>>> weren't working to begin with. >>>>>> >>>>>> I'm going to upload what I have so far, and hopefully someone else >>>>>> can take a look at it too. The current problem seems to be that the >>>>>> modules >>>>>> in _m5 aren't working right, either because they've been garbage >>>>>> collected, >>>>>> or because they're not being set up properly, or... >>>>>> >>>>>> Gabe >>>>>> >>>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hey gabe. >>>>>>> >>>>>>> No idea if this is the _best_ solution to your problem, but my >>>>>>> solution would be to rebuild the image with this installed. Modify the >>>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you want. >>>>>>> Then >>>>>>> run `docker build -t <some name here> util/dockerfiles/sst-11.1.0` to >>>>>>> build >>>>>>> an image with the name "<some name here>". >>>>>>> >>>>>>> Then you can execute `docker run -u $UID:$GID --volume $(pwd):$(pwd) >>>>>>> -w $(pwd) -it <same name here>` within the gem5 directory to spin up and >>>>>>> enter a container from the image you just built. I think you'll be able >>>>>>> to >>>>>>> do what you want inside the container. >>>>>>> >>>>>>> Kind regards, >>>>>>> Bobby >>>>>>> -- >>>>>>> Dr. Bobby R. Bruce >>>>>>> Room 3050, >>>>>>> Kemper Hall, UC Davis >>>>>>> Davis, >>>>>>> CA, 95616 >>>>>>> >>>>>>> web: https://www.bobbybruce.net >>>>>>> >>>>>>> >>>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black <[email protected]> >>>>>>> wrote: >>>>>>> > >>>>>>> > Dumb question: I'm trying to run gdb inside this container on the >>>>>>> sst thing. How do I do that? It's not installed in the container now, >>>>>>> and I >>>>>>> can't (easily) figure out how to get it installed. I can tell docker to >>>>>>> install it, but then it seems to throw that away as soon as the command >>>>>>> ends. >>>>>>> > >>>>>>> > Gabe >>>>>>> > >>>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce <[email protected]> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> Thanks Gabe, >>>>>>> >> >>>>>>> >> This is very much appreciated. I'm going to create the release >>>>>>> staging once a couple more things get in. Feel free to push any patches >>>>>>> related to these bugs to the release staging branch. >>>>>>> >> >>>>>>> >> If there is an order of priority I'd say the bug affecting SST is >>>>>>> of higher importance than that affecting the Weeklies (as far as I can >>>>>>> see >>>>>>> the latter is hard to trigger). That being said, we'll apply both to the >>>>>>> new release one way or another. >>>>>>> >> >>>>>>> >> -- >>>>>>> >> Dr. Bobby R. Bruce >>>>>>> >> Room 3050, >>>>>>> >> Kemper Hall, UC Davis >>>>>>> >> Davis, >>>>>>> >> CA, 95616 >>>>>>> >> >>>>>>> >> web: https://www.bobbybruce.net >>>>>>> >> >>>>>>> >> >>>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black <[email protected]> >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last couple >>>>>>> days but kept running out of time. I'm sitting down to work on it right >>>>>>> now. >>>>>>> >>> >>>>>>> >>> Gabe >>>>>>> >>> >>>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce <[email protected]> >>>>>>> wrote: >>>>>>> >>>> >>>>>>> >>>> Hey Gabe, >>>>>>> >>>> >>>>>>> >>>> Is there any update on this? >>>>>>> >>>> >>>>>>> >>>> Kind regards, >>>>>>> >>>> Bobby >>>>>>> >>>> -- >>>>>>> >>>> Dr. Bobby R. Bruce >>>>>>> >>>> Room 3050, >>>>>>> >>>> Kemper Hall, UC Davis >>>>>>> >>>> Davis, >>>>>>> >>>> CA, 95616 >>>>>>> >>>> >>>>>>> >>>> web: https://www.bobbybruce.net >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>>> [email protected]> wrote: >>>>>>> >>>>> >>>>>>> >>>>> Hi Gabe, >>>>>>> >>>>> >>>>>>> >>>>> I have more details about this. In this use case, SST >>>>>>> initialized the >>>>>>> >>>>> Python environment before adding the "gem5 object". This gem5 >>>>>>> object >>>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>>> >>>>> >>>>>>> >>>>> The function that does that is initPython() >>>>>>> >>>>> >>>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>>> >>>>> >>>>>>> >>>>> The following commands will pull the docker image for SST >>>>>>> testing >>>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root must be >>>>>>> >>>>> specified), >>>>>>> >>>>> >>>>>>> >>>>> ``` >>>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>>> >>>>> bash -c "\ >>>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} >>>>>>> --without-tcmalloc; \ >>>>>>> >>>>> cd ext/sst; \ >>>>>>> >>>>> make clean; make; \ >>>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>>> >>>>> ``` >>>>>>> >>>>> >>>>>>> >>>>> We appreciate your help! >>>>>>> >>>>> >>>>>>> >>>>> Regards, >>>>>>> >>>>> Hoa Nguyen >>>>>>> >>>>> >>>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> wrote: >>>>>>> >>>>> > Hey Gabe, >>>>>>> >>>>> > >>>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>>> integration python is >>>>>>> >>>>> > initialized from the SST module, not from init.cc (this is >>>>>>> because SST has >>>>>>> >>>>> > their own python interpreter). >>>>>>> >>>>> > >>>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and >>>>>>> Giacomo can give you >>>>>>> >>>>> > an example that's breaking to help you fix it, I believe. >>>>>>> >>>>> > >>>>>>> >>>>> > https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>>> >>>>> > >>>>>>> >>>>> > There's strong interest in having the SST integration >>>>>>> working in this >>>>>>> >>>>> > current release. We've spent a lot of time figuring out all >>>>>>> of the >>>>>>> >>>>> > intricacies and would appreciate any help you can provide in >>>>>>> these last few >>>>>>> >>>>> > days before the release! >>>>>>> >>>>> > >>>>>>> >>>>> > Thanks! >>>>>>> >>>>> > >>>>>>> >>>>> > Jason >>>>>>> >>>>> > >>>>>>> >>>>> _______________________________________________ >>>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>>> >>>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
