https://gem5-review.googlesource.com/c/public/gem5/+/54325/1
On Wed, Dec 15, 2021 at 11:12 PM Gabe Black <[email protected]> wrote: > There will be a couple more patches coming, since the systemc subsystem > has problems when python is already running when static initializers run. I > had been building without systemc enabled to simplify things earlier, but I > need to get it running too. That should be a fairly quick fix. > > Gabe > > On Wed, Dec 15, 2021 at 10:40 PM Hoa Nguyen <[email protected]> wrote: > >> Hi Gabe, >> >> Thanks for uploading the changes. Typically it'll take less than an hour >> for RISC-V. >> >> Regards, >> Hoa Nguyen >> >> On Wed, Dec 15, 2021 at 10:38 PM Gabe Black <[email protected]> wrote: >> >>> I have it running, although it's taking a while. How long should it take >>> to finish? Should it finish immediately, 15 minutes, a few hours, >>> tomorrow...? I'll upload my changes in the mean time. >>> >>> Gabe >>> >>> On Wed, Dec 15, 2021 at 9:13 PM Gabe Black <[email protected]> wrote: >>> >>>> Ok, I did/am doing a little more looking, and part of the problem seems >>>> to be that some of the python blobs executed with executePython assume >>>> prior blobs have done imports for them already. That was where the modules >>>> or their contents not being found was coming from, and then after that >>>> something about the error reporting was blowing up, but that wasn't the >>>> primary problem. >>>> >>>> With that fixed, it's still upset, but now it's because "fatal: Need to >>>> instantiate Root() before calling instantiate()". Also, gem5 is printing >>>> the help blob which seems wrong, so maybe the command line arguments aren't >>>> making it through like they should? It could be that no config script is >>>> run, and that's why m5.instantiate() doesn't work. >>>> >>>> It's also a little weird that there's an ordering dependency between >>>> the execPythonCommands invocations, considering the same set of those >>>> should still be happening in the same order. I suppose I'm not explicitly >>>> importing the m5 module into __main__.__dict__ to run m5.main(), so that >>>> may be where that's coming from. >>>> >>>> Anyway, I'm still chopping away at this, but I think it's pretty close. >>>> >>>> Gabe >>>> >>>> On Wed, Dec 15, 2021 at 8:29 PM Gabe Black <[email protected]> >>>> wrote: >>>> >>>>> Also, I have to say, while I'm grateful that there was a push to get >>>>> the SCons changes reviewed for the release, we wouldn't be in this crunch >>>>> if they had been reviewed a month ago. It would be best not to let things >>>>> bunch up and then make a herculean push right at the deadline, where >>>>> problems have a very limited time to get resolved. >>>>> >>>>> Gabe >>>>> >>>>> On Wed, Dec 15, 2021 at 4:38 PM Gabe Black <[email protected]> >>>>> wrote: >>>>> >>>>>> That's actually very helpful, since I was pretty sure there was a >>>>>> memory problem but couldn't figure out where. I think there's a good >>>>>> chance >>>>>> this will be an easy fix. >>>>>> >>>>>> Gabe >>>>>> >>>>>> On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Ok, I was able to get a debugger to work and I dug in a bit more... >>>>>>> >>>>>>> The problem is that the new code does not initialize the __main__.py >>>>>>> module. The original SST code had the following line: >>>>>>> `pythonMain = PyImport_AddModule(PyCC("__main__"));` >>>>>>> >>>>>>> This was executed before `execPythonCommands`. >>>>>>> >>>>>>> In `execPythonCommands` we use the `pythonMain` variable. However, >>>>>>> in the *new* code, that variable is NULL (or unitialized, actually). >>>>>>> >>>>>>> As far as I can tell, all of the new python changes didn't come with >>>>>>> any documentation on how you are supposed to interact with Python from >>>>>>> gem5. So, there's no good way for us to update this code. >>>>>>> >>>>>>> Gabe, if you can update the `execPythonCommands` function to work >>>>>>> however you expected it to, that would be great. >>>>>>> >>>>>>> Thanks, >>>>>>> Jason >>>>>>> >>>>>>> On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Here's my understanding of where we are on this: >>>>>>>> >>>>>>>> As of >>>>>>>> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >>>>>>>> SST was working. (Committed Dec. 7th) >>>>>>>> >>>>>>>> Then, this large relation chain was pushed: >>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49425 ending >>>>>>>> with >>>>>>>> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >>>>>>>> This relation chain broke SST. This was committed on Dec. 8th. >>>>>>>> >>>>>>>> We have two options: Either we fix this bug ASAP (as in within 6 >>>>>>>> days so we can release on 12/22) or we revert the changesets that were >>>>>>>> committed on the 8th. >>>>>>>> >>>>>>>> SST is a *must have* feature of gem5 21.2. It is quite frustrating >>>>>>>> to see it broken after all of the hard work that Hoa and Giacomo put >>>>>>>> into >>>>>>>> it. >>>>>>>> >>>>>>>> Let me know if I'm missing something. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Jason >>>>>>>> >>>>>>>> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>>>>>>>> >>>>>>>>> At least one problem I ran into was that the order of static >>>>>>>>> initializers was not determinstic, so the structures which said what >>>>>>>>> embedded python modules existed might be statically constructed after >>>>>>>>> the >>>>>>>>> thing that consumes and sets them up. That could work by coincidence >>>>>>>>> if the >>>>>>>>> order is favorable, but it wasn't for me and so lots of things didn't >>>>>>>>> exist >>>>>>>>> as far as the python was concerned. >>>>>>>>> >>>>>>>>> To be clear, there is some mysterious deeper issue here which is >>>>>>>>> probably separate from that. As far as I can tell running gem5 itself >>>>>>>>> is >>>>>>>>> working fine, but when running as a library in sst there is some sort >>>>>>>>> of >>>>>>>>> memory corruption or premature garbage collection or something, and it >>>>>>>>> leads to a segfault or incorrect behavior (failure to import modules, >>>>>>>>> failure to find things inside them, etc). Things were even more >>>>>>>>> tempermental until I added what feels like an unnecessary increment >>>>>>>>> of the >>>>>>>>> reference count for the modules in the new pybind_init.hh, which >>>>>>>>> seemed to >>>>>>>>> tame things in gem5 proper at least. I'm too tired to keep looking >>>>>>>>> for it >>>>>>>>> right now, but hopefully this will give you a good leg up. >>>>>>>>> >>>>>>>>> Gabe >>>>>>>>> >>>>>>>>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Everything was working as of last week-ish, if I remember >>>>>>>>>> correctly. We were booting both Arm and RISCV full system with gem5 >>>>>>>>>> cores >>>>>>>>>> and SST caches/memory. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Jason >>>>>>>>>> >>>>>>>>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks, that was really helpful. I've fixed the compilation >>>>>>>>>>> problems, and then a few more that were hiding behind that, >>>>>>>>>>> including a few >>>>>>>>>>> ones that were intrinsic to the way sst was initializing the gem5 >>>>>>>>>>> library >>>>>>>>>>> (with dlopen). Was this working to begin with and all this breakage >>>>>>>>>>> is new, >>>>>>>>>>> or was it just that the compilation broke and there were these other >>>>>>>>>>> problems already? I want to make sure I don't leave things in a >>>>>>>>>>> worse state >>>>>>>>>>> than when I got here, but I also don't want to go chasing down >>>>>>>>>>> things that >>>>>>>>>>> weren't working to begin with. >>>>>>>>>>> >>>>>>>>>>> I'm going to upload what I have so far, and hopefully someone >>>>>>>>>>> else can take a look at it too. The current problem seems to be >>>>>>>>>>> that the >>>>>>>>>>> modules in _m5 aren't working right, either because they've been >>>>>>>>>>> garbage >>>>>>>>>>> collected, or because they're not being set up properly, or... >>>>>>>>>>> >>>>>>>>>>> Gabe >>>>>>>>>>> >>>>>>>>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hey gabe. >>>>>>>>>>>> >>>>>>>>>>>> No idea if this is the _best_ solution to your problem, but my >>>>>>>>>>>> solution would be to rebuild the image with this installed. Modify >>>>>>>>>>>> the >>>>>>>>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you >>>>>>>>>>>> want. Then >>>>>>>>>>>> run `docker build -t <some name here> util/dockerfiles/sst-11.1.0` >>>>>>>>>>>> to build >>>>>>>>>>>> an image with the name "<some name here>". >>>>>>>>>>>> >>>>>>>>>>>> Then you can execute `docker run -u $UID:$GID --volume >>>>>>>>>>>> $(pwd):$(pwd) -w $(pwd) -it <same name here>` within the gem5 >>>>>>>>>>>> directory to >>>>>>>>>>>> spin up and enter a container from the image you just built. I >>>>>>>>>>>> think you'll >>>>>>>>>>>> be able to do what you want inside the container. >>>>>>>>>>>> >>>>>>>>>>>> Kind regards, >>>>>>>>>>>> Bobby >>>>>>>>>>>> -- >>>>>>>>>>>> Dr. Bobby R. Bruce >>>>>>>>>>>> Room 3050, >>>>>>>>>>>> Kemper Hall, UC Davis >>>>>>>>>>>> Davis, >>>>>>>>>>>> CA, 95616 >>>>>>>>>>>> >>>>>>>>>>>> web: https://www.bobbybruce.net >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > Dumb question: I'm trying to run gdb inside this container on >>>>>>>>>>>> the sst thing. How do I do that? It's not installed in the >>>>>>>>>>>> container now, >>>>>>>>>>>> and I can't (easily) figure out how to get it installed. I can >>>>>>>>>>>> tell docker >>>>>>>>>>>> to install it, but then it seems to throw that away as soon as the >>>>>>>>>>>> command >>>>>>>>>>>> ends. >>>>>>>>>>>> > >>>>>>>>>>>> > Gabe >>>>>>>>>>>> > >>>>>>>>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >> >>>>>>>>>>>> >> Thanks Gabe, >>>>>>>>>>>> >> >>>>>>>>>>>> >> This is very much appreciated. I'm going to create the >>>>>>>>>>>> release staging once a couple more things get in. Feel free to >>>>>>>>>>>> push any >>>>>>>>>>>> patches related to these bugs to the release staging branch. >>>>>>>>>>>> >> >>>>>>>>>>>> >> If there is an order of priority I'd say the bug affecting >>>>>>>>>>>> SST is of higher importance than that affecting the Weeklies (as >>>>>>>>>>>> far as I >>>>>>>>>>>> can see the latter is hard to trigger). That being said, we'll >>>>>>>>>>>> apply both >>>>>>>>>>>> to the new release one way or another. >>>>>>>>>>>> >> >>>>>>>>>>>> >> -- >>>>>>>>>>>> >> Dr. Bobby R. Bruce >>>>>>>>>>>> >> Room 3050, >>>>>>>>>>>> >> Kemper Hall, UC Davis >>>>>>>>>>>> >> Davis, >>>>>>>>>>>> >> CA, 95616 >>>>>>>>>>>> >> >>>>>>>>>>>> >> web: https://www.bobbybruce.net >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last >>>>>>>>>>>> couple days but kept running out of time. I'm sitting down to work >>>>>>>>>>>> on it >>>>>>>>>>>> right now. >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> Gabe >>>>>>>>>>>> >>> >>>>>>>>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Hey Gabe, >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Is there any update on this? >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> Kind regards, >>>>>>>>>>>> >>>> Bobby >>>>>>>>>>>> >>>> -- >>>>>>>>>>>> >>>> Dr. Bobby R. Bruce >>>>>>>>>>>> >>>> Room 3050, >>>>>>>>>>>> >>>> Kemper Hall, UC Davis >>>>>>>>>>>> >>>> Davis, >>>>>>>>>>>> >>>> CA, 95616 >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> web: https://www.bobbybruce.net >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> Hi Gabe, >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> I have more details about this. In this use case, SST >>>>>>>>>>>> initialized the >>>>>>>>>>>> >>>>> Python environment before adding the "gem5 object". This >>>>>>>>>>>> gem5 object >>>>>>>>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> The function that does that is initPython() >>>>>>>>>>>> >>>>> >>>>>>>>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> The following commands will pull the docker image for SST >>>>>>>>>>>> testing >>>>>>>>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root >>>>>>>>>>>> must be >>>>>>>>>>>> >>>>> specified), >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> ``` >>>>>>>>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>>>>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>>>>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>>>>>>>> >>>>> bash -c "\ >>>>>>>>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} >>>>>>>>>>>> --without-tcmalloc; \ >>>>>>>>>>>> >>>>> cd ext/sst; \ >>>>>>>>>>>> >>>>> make clean; make; \ >>>>>>>>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>>>>>>>> >>>>> ``` >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> We appreciate your help! >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>> >>>>> Hoa Nguyen >>>>>>>>>>>> >>>>> >>>>>>>>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> wrote: >>>>>>>>>>>> >>>>> > Hey Gabe, >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>>>>>>>> integration python is >>>>>>>>>>>> >>>>> > initialized from the SST module, not from init.cc (this >>>>>>>>>>>> is because SST has >>>>>>>>>>>> >>>>> > their own python interpreter). >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and >>>>>>>>>>>> Giacomo can give you >>>>>>>>>>>> >>>>> > an example that's breaking to help you fix it, I >>>>>>>>>>>> believe. >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > There's strong interest in having the SST integration >>>>>>>>>>>> working in this >>>>>>>>>>>> >>>>> > current release. We've spent a lot of time figuring out >>>>>>>>>>>> all of the >>>>>>>>>>>> >>>>> > intricacies and would appreciate any help you can >>>>>>>>>>>> provide in these last few >>>>>>>>>>>> >>>>> > days before the release! >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > Thanks! >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> > Jason >>>>>>>>>>>> >>>>> > >>>>>>>>>>>> >>>>> _______________________________________________ >>>>>>>>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>>>>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>>>>>>>> >>>>>>>>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
