That's actually very helpful, since I was pretty sure there was a memory problem but couldn't figure out where. I think there's a good chance this will be an easy fix.
Gabe On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> wrote: > Ok, I was able to get a debugger to work and I dug in a bit more... > > The problem is that the new code does not initialize the __main__.py > module. The original SST code had the following line: > `pythonMain = PyImport_AddModule(PyCC("__main__"));` > > This was executed before `execPythonCommands`. > > In `execPythonCommands` we use the `pythonMain` variable. However, in the > *new* code, that variable is NULL (or unitialized, actually). > > As far as I can tell, all of the new python changes didn't come with any > documentation on how you are supposed to interact with Python from gem5. > So, there's no good way for us to update this code. > > Gabe, if you can update the `execPythonCommands` function to work however > you expected it to, that would be great. > > Thanks, > Jason > > On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power <[email protected]> > wrote: > >> Here's my understanding of where we are on this: >> >> As of >> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >> SST was working. (Committed Dec. 7th) >> >> Then, this large relation chain was pushed: >> https://gem5-review.googlesource.com/c/public/gem5/+/49425 ending with >> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >> This relation chain broke SST. This was committed on Dec. 8th. >> >> We have two options: Either we fix this bug ASAP (as in within 6 days so >> we can release on 12/22) or we revert the changesets that were committed on >> the 8th. >> >> SST is a *must have* feature of gem5 21.2. It is quite frustrating to see >> it broken after all of the hard work that Hoa and Giacomo put into it. >> >> Let me know if I'm missing something. >> >> Thanks, >> Jason >> >> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> wrote: >> >>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>> >>> At least one problem I ran into was that the order of static >>> initializers was not determinstic, so the structures which said what >>> embedded python modules existed might be statically constructed after the >>> thing that consumes and sets them up. That could work by coincidence if the >>> order is favorable, but it wasn't for me and so lots of things didn't exist >>> as far as the python was concerned. >>> >>> To be clear, there is some mysterious deeper issue here which is >>> probably separate from that. As far as I can tell running gem5 itself is >>> working fine, but when running as a library in sst there is some sort of >>> memory corruption or premature garbage collection or something, and it >>> leads to a segfault or incorrect behavior (failure to import modules, >>> failure to find things inside them, etc). Things were even more >>> tempermental until I added what feels like an unnecessary increment of the >>> reference count for the modules in the new pybind_init.hh, which seemed to >>> tame things in gem5 proper at least. I'm too tired to keep looking for it >>> right now, but hopefully this will give you a good leg up. >>> >>> Gabe >>> >>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power <[email protected]> >>> wrote: >>> >>>> Everything was working as of last week-ish, if I remember correctly. We >>>> were booting both Arm and RISCV full system with gem5 cores and SST >>>> caches/memory. >>>> >>>> Cheers, >>>> Jason >>>> >>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> wrote: >>>> >>>>> Thanks, that was really helpful. I've fixed the compilation problems, >>>>> and then a few more that were hiding behind that, including a few ones >>>>> that >>>>> were intrinsic to the way sst was initializing the gem5 library (with >>>>> dlopen). Was this working to begin with and all this breakage is new, or >>>>> was it just that the compilation broke and there were these other problems >>>>> already? I want to make sure I don't leave things in a worse state than >>>>> when I got here, but I also don't want to go chasing down things that >>>>> weren't working to begin with. >>>>> >>>>> I'm going to upload what I have so far, and hopefully someone else can >>>>> take a look at it too. The current problem seems to be that the modules in >>>>> _m5 aren't working right, either because they've been garbage collected, >>>>> or >>>>> because they're not being set up properly, or... >>>>> >>>>> Gabe >>>>> >>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey gabe. >>>>>> >>>>>> No idea if this is the _best_ solution to your problem, but my >>>>>> solution would be to rebuild the image with this installed. Modify the >>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you want. >>>>>> Then >>>>>> run `docker build -t <some name here> util/dockerfiles/sst-11.1.0` to >>>>>> build >>>>>> an image with the name "<some name here>". >>>>>> >>>>>> Then you can execute `docker run -u $UID:$GID --volume $(pwd):$(pwd) >>>>>> -w $(pwd) -it <same name here>` within the gem5 directory to spin up and >>>>>> enter a container from the image you just built. I think you'll be able >>>>>> to >>>>>> do what you want inside the container. >>>>>> >>>>>> Kind regards, >>>>>> Bobby >>>>>> -- >>>>>> Dr. Bobby R. Bruce >>>>>> Room 3050, >>>>>> Kemper Hall, UC Davis >>>>>> Davis, >>>>>> CA, 95616 >>>>>> >>>>>> web: https://www.bobbybruce.net >>>>>> >>>>>> >>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black <[email protected]> >>>>>> wrote: >>>>>> > >>>>>> > Dumb question: I'm trying to run gdb inside this container on the >>>>>> sst thing. How do I do that? It's not installed in the container now, >>>>>> and I >>>>>> can't (easily) figure out how to get it installed. I can tell docker to >>>>>> install it, but then it seems to throw that away as soon as the command >>>>>> ends. >>>>>> > >>>>>> > Gabe >>>>>> > >>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce <[email protected]> >>>>>> wrote: >>>>>> >> >>>>>> >> Thanks Gabe, >>>>>> >> >>>>>> >> This is very much appreciated. I'm going to create the release >>>>>> staging once a couple more things get in. Feel free to push any patches >>>>>> related to these bugs to the release staging branch. >>>>>> >> >>>>>> >> If there is an order of priority I'd say the bug affecting SST is >>>>>> of higher importance than that affecting the Weeklies (as far as I can >>>>>> see >>>>>> the latter is hard to trigger). That being said, we'll apply both to the >>>>>> new release one way or another. >>>>>> >> >>>>>> >> -- >>>>>> >> Dr. Bobby R. Bruce >>>>>> >> Room 3050, >>>>>> >> Kemper Hall, UC Davis >>>>>> >> Davis, >>>>>> >> CA, 95616 >>>>>> >> >>>>>> >> web: https://www.bobbybruce.net >>>>>> >> >>>>>> >> >>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black <[email protected]> >>>>>> wrote: >>>>>> >>> >>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last couple >>>>>> days but kept running out of time. I'm sitting down to work on it right >>>>>> now. >>>>>> >>> >>>>>> >>> Gabe >>>>>> >>> >>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce <[email protected]> >>>>>> wrote: >>>>>> >>>> >>>>>> >>>> Hey Gabe, >>>>>> >>>> >>>>>> >>>> Is there any update on this? >>>>>> >>>> >>>>>> >>>> Kind regards, >>>>>> >>>> Bobby >>>>>> >>>> -- >>>>>> >>>> Dr. Bobby R. Bruce >>>>>> >>>> Room 3050, >>>>>> >>>> Kemper Hall, UC Davis >>>>>> >>>> Davis, >>>>>> >>>> CA, 95616 >>>>>> >>>> >>>>>> >>>> web: https://www.bobbybruce.net >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>> [email protected]> wrote: >>>>>> >>>>> >>>>>> >>>>> Hi Gabe, >>>>>> >>>>> >>>>>> >>>>> I have more details about this. In this use case, SST >>>>>> initialized the >>>>>> >>>>> Python environment before adding the "gem5 object". This gem5 >>>>>> object >>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>> >>>>> >>>>>> >>>>> The function that does that is initPython() >>>>>> >>>>> >>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>> >>>>> >>>>>> >>>>> The following commands will pull the docker image for SST >>>>>> testing >>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root must be >>>>>> >>>>> specified), >>>>>> >>>>> >>>>>> >>>>> ``` >>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>> >>>>> bash -c "\ >>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} --without-tcmalloc; >>>>>> \ >>>>>> >>>>> cd ext/sst; \ >>>>>> >>>>> make clean; make; \ >>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>> >>>>> ``` >>>>>> >>>>> >>>>>> >>>>> We appreciate your help! >>>>>> >>>>> >>>>>> >>>>> Regards, >>>>>> >>>>> Hoa Nguyen >>>>>> >>>>> >>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> wrote: >>>>>> >>>>> > Hey Gabe, >>>>>> >>>>> > >>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>> integration python is >>>>>> >>>>> > initialized from the SST module, not from init.cc (this is >>>>>> because SST has >>>>>> >>>>> > their own python interpreter). >>>>>> >>>>> > >>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and Giacomo >>>>>> can give you >>>>>> >>>>> > an example that's breaking to help you fix it, I believe. >>>>>> >>>>> > >>>>>> >>>>> > https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>> >>>>> > >>>>>> >>>>> > There's strong interest in having the SST integration working >>>>>> in this >>>>>> >>>>> > current release. We've spent a lot of time figuring out all >>>>>> of the >>>>>> >>>>> > intricacies and would appreciate any help you can provide in >>>>>> these last few >>>>>> >>>>> > days before the release! >>>>>> >>>>> > >>>>>> >>>>> > Thanks! >>>>>> >>>>> > >>>>>> >>>>> > Jason >>>>>> >>>>> > >>>>>> >>>>> _______________________________________________ >>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>> >>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
