Ok, I did/am doing a little more looking, and part of the problem seems to be that some of the python blobs executed with executePython assume prior blobs have done imports for them already. That was where the modules or their contents not being found was coming from, and then after that something about the error reporting was blowing up, but that wasn't the primary problem.
With that fixed, it's still upset, but now it's because "fatal: Need to instantiate Root() before calling instantiate()". Also, gem5 is printing the help blob which seems wrong, so maybe the command line arguments aren't making it through like they should? It could be that no config script is run, and that's why m5.instantiate() doesn't work. It's also a little weird that there's an ordering dependency between the execPythonCommands invocations, considering the same set of those should still be happening in the same order. I suppose I'm not explicitly importing the m5 module into __main__.__dict__ to run m5.main(), so that may be where that's coming from. Anyway, I'm still chopping away at this, but I think it's pretty close. Gabe On Wed, Dec 15, 2021 at 8:29 PM Gabe Black <[email protected]> wrote: > Also, I have to say, while I'm grateful that there was a push to get the > SCons changes reviewed for the release, we wouldn't be in this crunch if > they had been reviewed a month ago. It would be best not to let things > bunch up and then make a herculean push right at the deadline, where > problems have a very limited time to get resolved. > > Gabe > > On Wed, Dec 15, 2021 at 4:38 PM Gabe Black <[email protected]> wrote: > >> That's actually very helpful, since I was pretty sure there was a memory >> problem but couldn't figure out where. I think there's a good chance this >> will be an easy fix. >> >> Gabe >> >> On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> >> wrote: >> >>> Ok, I was able to get a debugger to work and I dug in a bit more... >>> >>> The problem is that the new code does not initialize the __main__.py >>> module. The original SST code had the following line: >>> `pythonMain = PyImport_AddModule(PyCC("__main__"));` >>> >>> This was executed before `execPythonCommands`. >>> >>> In `execPythonCommands` we use the `pythonMain` variable. However, in >>> the *new* code, that variable is NULL (or unitialized, actually). >>> >>> As far as I can tell, all of the new python changes didn't come with any >>> documentation on how you are supposed to interact with Python from gem5. >>> So, there's no good way for us to update this code. >>> >>> Gabe, if you can update the `execPythonCommands` function to work >>> however you expected it to, that would be great. >>> >>> Thanks, >>> Jason >>> >>> On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power <[email protected]> >>> wrote: >>> >>>> Here's my understanding of where we are on this: >>>> >>>> As of >>>> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >>>> SST was working. (Committed Dec. 7th) >>>> >>>> Then, this large relation chain was pushed: >>>> https://gem5-review.googlesource.com/c/public/gem5/+/49425 ending with >>>> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >>>> This relation chain broke SST. This was committed on Dec. 8th. >>>> >>>> We have two options: Either we fix this bug ASAP (as in within 6 days >>>> so we can release on 12/22) or we revert the changesets that were committed >>>> on the 8th. >>>> >>>> SST is a *must have* feature of gem5 21.2. It is quite frustrating to >>>> see it broken after all of the hard work that Hoa and Giacomo put into it. >>>> >>>> Let me know if I'm missing something. >>>> >>>> Thanks, >>>> Jason >>>> >>>> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> >>>> wrote: >>>> >>>>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>>>> >>>>> At least one problem I ran into was that the order of static >>>>> initializers was not determinstic, so the structures which said what >>>>> embedded python modules existed might be statically constructed after the >>>>> thing that consumes and sets them up. That could work by coincidence if >>>>> the >>>>> order is favorable, but it wasn't for me and so lots of things didn't >>>>> exist >>>>> as far as the python was concerned. >>>>> >>>>> To be clear, there is some mysterious deeper issue here which is >>>>> probably separate from that. As far as I can tell running gem5 itself is >>>>> working fine, but when running as a library in sst there is some sort of >>>>> memory corruption or premature garbage collection or something, and it >>>>> leads to a segfault or incorrect behavior (failure to import modules, >>>>> failure to find things inside them, etc). Things were even more >>>>> tempermental until I added what feels like an unnecessary increment of the >>>>> reference count for the modules in the new pybind_init.hh, which seemed to >>>>> tame things in gem5 proper at least. I'm too tired to keep looking for it >>>>> right now, but hopefully this will give you a good leg up. >>>>> >>>>> Gabe >>>>> >>>>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power <[email protected]> >>>>> wrote: >>>>> >>>>>> Everything was working as of last week-ish, if I remember correctly. >>>>>> We were booting both Arm and RISCV full system with gem5 cores and SST >>>>>> caches/memory. >>>>>> >>>>>> Cheers, >>>>>> Jason >>>>>> >>>>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks, that was really helpful. I've fixed the compilation >>>>>>> problems, and then a few more that were hiding behind that, including a >>>>>>> few >>>>>>> ones that were intrinsic to the way sst was initializing the gem5 >>>>>>> library >>>>>>> (with dlopen). Was this working to begin with and all this breakage is >>>>>>> new, >>>>>>> or was it just that the compilation broke and there were these other >>>>>>> problems already? I want to make sure I don't leave things in a worse >>>>>>> state >>>>>>> than when I got here, but I also don't want to go chasing down things >>>>>>> that >>>>>>> weren't working to begin with. >>>>>>> >>>>>>> I'm going to upload what I have so far, and hopefully someone else >>>>>>> can take a look at it too. The current problem seems to be that the >>>>>>> modules >>>>>>> in _m5 aren't working right, either because they've been garbage >>>>>>> collected, >>>>>>> or because they're not being set up properly, or... >>>>>>> >>>>>>> Gabe >>>>>>> >>>>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hey gabe. >>>>>>>> >>>>>>>> No idea if this is the _best_ solution to your problem, but my >>>>>>>> solution would be to rebuild the image with this installed. Modify the >>>>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you want. >>>>>>>> Then >>>>>>>> run `docker build -t <some name here> util/dockerfiles/sst-11.1.0` to >>>>>>>> build >>>>>>>> an image with the name "<some name here>". >>>>>>>> >>>>>>>> Then you can execute `docker run -u $UID:$GID --volume >>>>>>>> $(pwd):$(pwd) -w $(pwd) -it <same name here>` within the gem5 >>>>>>>> directory to >>>>>>>> spin up and enter a container from the image you just built. I think >>>>>>>> you'll >>>>>>>> be able to do what you want inside the container. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> Bobby >>>>>>>> -- >>>>>>>> Dr. Bobby R. Bruce >>>>>>>> Room 3050, >>>>>>>> Kemper Hall, UC Davis >>>>>>>> Davis, >>>>>>>> CA, 95616 >>>>>>>> >>>>>>>> web: https://www.bobbybruce.net >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > Dumb question: I'm trying to run gdb inside this container on the >>>>>>>> sst thing. How do I do that? It's not installed in the container now, >>>>>>>> and I >>>>>>>> can't (easily) figure out how to get it installed. I can tell docker to >>>>>>>> install it, but then it seems to throw that away as soon as the command >>>>>>>> ends. >>>>>>>> > >>>>>>>> > Gabe >>>>>>>> > >>>>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce <[email protected]> >>>>>>>> wrote: >>>>>>>> >> >>>>>>>> >> Thanks Gabe, >>>>>>>> >> >>>>>>>> >> This is very much appreciated. I'm going to create the release >>>>>>>> staging once a couple more things get in. Feel free to push any patches >>>>>>>> related to these bugs to the release staging branch. >>>>>>>> >> >>>>>>>> >> If there is an order of priority I'd say the bug affecting SST >>>>>>>> is of higher importance than that affecting the Weeklies (as far as I >>>>>>>> can >>>>>>>> see the latter is hard to trigger). That being said, we'll apply both >>>>>>>> to >>>>>>>> the new release one way or another. >>>>>>>> >> >>>>>>>> >> -- >>>>>>>> >> Dr. Bobby R. Bruce >>>>>>>> >> Room 3050, >>>>>>>> >> Kemper Hall, UC Davis >>>>>>>> >> Davis, >>>>>>>> >> CA, 95616 >>>>>>>> >> >>>>>>>> >> web: https://www.bobbybruce.net >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black <[email protected]> >>>>>>>> wrote: >>>>>>>> >>> >>>>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last >>>>>>>> couple days but kept running out of time. I'm sitting down to work on >>>>>>>> it >>>>>>>> right now. >>>>>>>> >>> >>>>>>>> >>> Gabe >>>>>>>> >>> >>>>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> Hey Gabe, >>>>>>>> >>>> >>>>>>>> >>>> Is there any update on this? >>>>>>>> >>>> >>>>>>>> >>>> Kind regards, >>>>>>>> >>>> Bobby >>>>>>>> >>>> -- >>>>>>>> >>>> Dr. Bobby R. Bruce >>>>>>>> >>>> Room 3050, >>>>>>>> >>>> Kemper Hall, UC Davis >>>>>>>> >>>> Davis, >>>>>>>> >>>> CA, 95616 >>>>>>>> >>>> >>>>>>>> >>>> web: https://www.bobbybruce.net >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>> >>>>>>>> >>>>> Hi Gabe, >>>>>>>> >>>>> >>>>>>>> >>>>> I have more details about this. In this use case, SST >>>>>>>> initialized the >>>>>>>> >>>>> Python environment before adding the "gem5 object". This gem5 >>>>>>>> object >>>>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>>>> >>>>> >>>>>>>> >>>>> The function that does that is initPython() >>>>>>>> >>>>> >>>>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>>>> >>>>> >>>>>>>> >>>>> The following commands will pull the docker image for SST >>>>>>>> testing >>>>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root must be >>>>>>>> >>>>> specified), >>>>>>>> >>>>> >>>>>>>> >>>>> ``` >>>>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>>>> >>>>> bash -c "\ >>>>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} >>>>>>>> --without-tcmalloc; \ >>>>>>>> >>>>> cd ext/sst; \ >>>>>>>> >>>>> make clean; make; \ >>>>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>>>> >>>>> ``` >>>>>>>> >>>>> >>>>>>>> >>>>> We appreciate your help! >>>>>>>> >>>>> >>>>>>>> >>>>> Regards, >>>>>>>> >>>>> Hoa Nguyen >>>>>>>> >>>>> >>>>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> wrote: >>>>>>>> >>>>> > Hey Gabe, >>>>>>>> >>>>> > >>>>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>>>> integration python is >>>>>>>> >>>>> > initialized from the SST module, not from init.cc (this is >>>>>>>> because SST has >>>>>>>> >>>>> > their own python interpreter). >>>>>>>> >>>>> > >>>>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and >>>>>>>> Giacomo can give you >>>>>>>> >>>>> > an example that's breaking to help you fix it, I believe. >>>>>>>> >>>>> > >>>>>>>> >>>>> > https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>>>> >>>>> > >>>>>>>> >>>>> > There's strong interest in having the SST integration >>>>>>>> working in this >>>>>>>> >>>>> > current release. We've spent a lot of time figuring out all >>>>>>>> of the >>>>>>>> >>>>> > intricacies and would appreciate any help you can provide >>>>>>>> in these last few >>>>>>>> >>>>> > days before the release! >>>>>>>> >>>>> > >>>>>>>> >>>>> > Thanks! >>>>>>>> >>>>> > >>>>>>>> >>>>> > Jason >>>>>>>> >>>>> > >>>>>>>> >>>>> _______________________________________________ >>>>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>>>> >>>>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
