Thanks, Gabe! We'll run the nightly tests and double check everything is working as expected.
Jason On Thu, Dec 16, 2021 at 6:49 AM Gabe Black <[email protected]> wrote: > I just realized I turned off my computer since I'll be traveling, and my > test run of sst can't finish if the computer is off :-P. I have not > completed a run of your example command, but it starts and I have no reason > to believe it won't finish. I would still suggest running it yourself (with > my changes) to verify that things are fixed. I need to wake up to go to the > airport in 4 hours, so I won't be able to do anything non-trivial with this > until Friday at the earliest. > > Gabe > > On Thu, Dec 16, 2021 at 3:29 AM Gabe Black <[email protected]> wrote: > >> https://gem5-review.googlesource.com/c/public/gem5/+/54325/1 >> >> On Wed, Dec 15, 2021 at 11:12 PM Gabe Black <[email protected]> wrote: >> >>> There will be a couple more patches coming, since the systemc subsystem >>> has problems when python is already running when static initializers run. I >>> had been building without systemc enabled to simplify things earlier, but I >>> need to get it running too. That should be a fairly quick fix. >>> >>> Gabe >>> >>> On Wed, Dec 15, 2021 at 10:40 PM Hoa Nguyen <[email protected]> >>> wrote: >>> >>>> Hi Gabe, >>>> >>>> Thanks for uploading the changes. Typically it'll take less than an >>>> hour for RISC-V. >>>> >>>> Regards, >>>> Hoa Nguyen >>>> >>>> On Wed, Dec 15, 2021 at 10:38 PM Gabe Black <[email protected]> >>>> wrote: >>>> >>>>> I have it running, although it's taking a while. How long should it >>>>> take to finish? Should it finish immediately, 15 minutes, a few hours, >>>>> tomorrow...? I'll upload my changes in the mean time. >>>>> >>>>> Gabe >>>>> >>>>> On Wed, Dec 15, 2021 at 9:13 PM Gabe Black <[email protected]> >>>>> wrote: >>>>> >>>>>> Ok, I did/am doing a little more looking, and part of the problem >>>>>> seems to be that some of the python blobs executed with executePython >>>>>> assume prior blobs have done imports for them already. That was where the >>>>>> modules or their contents not being found was coming from, and then after >>>>>> that something about the error reporting was blowing up, but that wasn't >>>>>> the primary problem. >>>>>> >>>>>> With that fixed, it's still upset, but now it's because "fatal: Need >>>>>> to instantiate Root() before calling instantiate()". Also, gem5 is >>>>>> printing >>>>>> the help blob which seems wrong, so maybe the command line arguments >>>>>> aren't >>>>>> making it through like they should? It could be that no config script is >>>>>> run, and that's why m5.instantiate() doesn't work. >>>>>> >>>>>> It's also a little weird that there's an ordering dependency between >>>>>> the execPythonCommands invocations, considering the same set of those >>>>>> should still be happening in the same order. I suppose I'm not explicitly >>>>>> importing the m5 module into __main__.__dict__ to run m5.main(), so that >>>>>> may be where that's coming from. >>>>>> >>>>>> Anyway, I'm still chopping away at this, but I think it's pretty >>>>>> close. >>>>>> >>>>>> Gabe >>>>>> >>>>>> On Wed, Dec 15, 2021 at 8:29 PM Gabe Black <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Also, I have to say, while I'm grateful that there was a push to get >>>>>>> the SCons changes reviewed for the release, we wouldn't be in this >>>>>>> crunch >>>>>>> if they had been reviewed a month ago. It would be best not to let >>>>>>> things >>>>>>> bunch up and then make a herculean push right at the deadline, where >>>>>>> problems have a very limited time to get resolved. >>>>>>> >>>>>>> Gabe >>>>>>> >>>>>>> On Wed, Dec 15, 2021 at 4:38 PM Gabe Black <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> That's actually very helpful, since I was pretty sure there was a >>>>>>>> memory problem but couldn't figure out where. I think there's a good >>>>>>>> chance >>>>>>>> this will be an easy fix. >>>>>>>> >>>>>>>> Gabe >>>>>>>> >>>>>>>> On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Ok, I was able to get a debugger to work and I dug in a bit more... >>>>>>>>> >>>>>>>>> The problem is that the new code does not initialize the >>>>>>>>> __main__.py module. The original SST code had the following line: >>>>>>>>> `pythonMain = PyImport_AddModule(PyCC("__main__"));` >>>>>>>>> >>>>>>>>> This was executed before `execPythonCommands`. >>>>>>>>> >>>>>>>>> In `execPythonCommands` we use the `pythonMain` variable. However, >>>>>>>>> in the *new* code, that variable is NULL (or unitialized, actually). >>>>>>>>> >>>>>>>>> As far as I can tell, all of the new python changes didn't come >>>>>>>>> with any documentation on how you are supposed to interact with >>>>>>>>> Python from >>>>>>>>> gem5. So, there's no good way for us to update this code. >>>>>>>>> >>>>>>>>> Gabe, if you can update the `execPythonCommands` function to work >>>>>>>>> however you expected it to, that would be great. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jason >>>>>>>>> >>>>>>>>> On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Here's my understanding of where we are on this: >>>>>>>>>> >>>>>>>>>> As of >>>>>>>>>> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >>>>>>>>>> SST was working. (Committed Dec. 7th) >>>>>>>>>> >>>>>>>>>> Then, this large relation chain was pushed: >>>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49425 >>>>>>>>>> ending with >>>>>>>>>> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >>>>>>>>>> This relation chain broke SST. This was committed on Dec. 8th. >>>>>>>>>> >>>>>>>>>> We have two options: Either we fix this bug ASAP (as in within 6 >>>>>>>>>> days so we can release on 12/22) or we revert the changesets that >>>>>>>>>> were >>>>>>>>>> committed on the 8th. >>>>>>>>>> >>>>>>>>>> SST is a *must have* feature of gem5 21.2. It is quite >>>>>>>>>> frustrating to see it broken after all of the hard work that Hoa and >>>>>>>>>> Giacomo put into it. >>>>>>>>>> >>>>>>>>>> Let me know if I'm missing something. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Jason >>>>>>>>>> >>>>>>>>>> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>>>>>>>>>> >>>>>>>>>>> At least one problem I ran into was that the order of static >>>>>>>>>>> initializers was not determinstic, so the structures which said what >>>>>>>>>>> embedded python modules existed might be statically constructed >>>>>>>>>>> after the >>>>>>>>>>> thing that consumes and sets them up. That could work by >>>>>>>>>>> coincidence if the >>>>>>>>>>> order is favorable, but it wasn't for me and so lots of things >>>>>>>>>>> didn't exist >>>>>>>>>>> as far as the python was concerned. >>>>>>>>>>> >>>>>>>>>>> To be clear, there is some mysterious deeper issue here which is >>>>>>>>>>> probably separate from that. As far as I can tell running gem5 >>>>>>>>>>> itself is >>>>>>>>>>> working fine, but when running as a library in sst there is some >>>>>>>>>>> sort of >>>>>>>>>>> memory corruption or premature garbage collection or something, and >>>>>>>>>>> it >>>>>>>>>>> leads to a segfault or incorrect behavior (failure to import >>>>>>>>>>> modules, >>>>>>>>>>> failure to find things inside them, etc). Things were even more >>>>>>>>>>> tempermental until I added what feels like an unnecessary increment >>>>>>>>>>> of the >>>>>>>>>>> reference count for the modules in the new pybind_init.hh, which >>>>>>>>>>> seemed to >>>>>>>>>>> tame things in gem5 proper at least. I'm too tired to keep looking >>>>>>>>>>> for it >>>>>>>>>>> right now, but hopefully this will give you a good leg up. >>>>>>>>>>> >>>>>>>>>>> Gabe >>>>>>>>>>> >>>>>>>>>>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Everything was working as of last week-ish, if I remember >>>>>>>>>>>> correctly. We were booting both Arm and RISCV full system with >>>>>>>>>>>> gem5 cores >>>>>>>>>>>> and SST caches/memory. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Jason >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks, that was really helpful. I've fixed the compilation >>>>>>>>>>>>> problems, and then a few more that were hiding behind that, >>>>>>>>>>>>> including a few >>>>>>>>>>>>> ones that were intrinsic to the way sst was initializing the gem5 >>>>>>>>>>>>> library >>>>>>>>>>>>> (with dlopen). Was this working to begin with and all this >>>>>>>>>>>>> breakage is new, >>>>>>>>>>>>> or was it just that the compilation broke and there were these >>>>>>>>>>>>> other >>>>>>>>>>>>> problems already? I want to make sure I don't leave things in a >>>>>>>>>>>>> worse state >>>>>>>>>>>>> than when I got here, but I also don't want to go chasing down >>>>>>>>>>>>> things that >>>>>>>>>>>>> weren't working to begin with. >>>>>>>>>>>>> >>>>>>>>>>>>> I'm going to upload what I have so far, and hopefully someone >>>>>>>>>>>>> else can take a look at it too. The current problem seems to be >>>>>>>>>>>>> that the >>>>>>>>>>>>> modules in _m5 aren't working right, either because they've been >>>>>>>>>>>>> garbage >>>>>>>>>>>>> collected, or because they're not being set up properly, or... >>>>>>>>>>>>> >>>>>>>>>>>>> Gabe >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hey gabe. >>>>>>>>>>>>>> >>>>>>>>>>>>>> No idea if this is the _best_ solution to your problem, but >>>>>>>>>>>>>> my solution would be to rebuild the image with this installed. >>>>>>>>>>>>>> Modify the >>>>>>>>>>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you >>>>>>>>>>>>>> want. Then >>>>>>>>>>>>>> run `docker build -t <some name here> >>>>>>>>>>>>>> util/dockerfiles/sst-11.1.0` to build >>>>>>>>>>>>>> an image with the name "<some name here>". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then you can execute `docker run -u $UID:$GID --volume >>>>>>>>>>>>>> $(pwd):$(pwd) -w $(pwd) -it <same name here>` within the gem5 >>>>>>>>>>>>>> directory to >>>>>>>>>>>>>> spin up and enter a container from the image you just built. I >>>>>>>>>>>>>> think you'll >>>>>>>>>>>>>> be able to do what you want inside the container. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>> Bobby >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Dr. Bobby R. Bruce >>>>>>>>>>>>>> Room 3050, >>>>>>>>>>>>>> Kemper Hall, UC Davis >>>>>>>>>>>>>> Davis, >>>>>>>>>>>>>> CA, 95616 >>>>>>>>>>>>>> >>>>>>>>>>>>>> web: https://www.bobbybruce.net >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Dumb question: I'm trying to run gdb inside this container >>>>>>>>>>>>>> on the sst thing. How do I do that? It's not installed in the >>>>>>>>>>>>>> container >>>>>>>>>>>>>> now, and I can't (easily) figure out how to get it installed. I >>>>>>>>>>>>>> can tell >>>>>>>>>>>>>> docker to install it, but then it seems to throw that away as >>>>>>>>>>>>>> soon as the >>>>>>>>>>>>>> command ends. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Gabe >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Thanks Gabe, >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> This is very much appreciated. I'm going to create the >>>>>>>>>>>>>> release staging once a couple more things get in. Feel free to >>>>>>>>>>>>>> push any >>>>>>>>>>>>>> patches related to these bugs to the release staging branch. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> If there is an order of priority I'd say the bug affecting >>>>>>>>>>>>>> SST is of higher importance than that affecting the Weeklies (as >>>>>>>>>>>>>> far as I >>>>>>>>>>>>>> can see the latter is hard to trigger). That being said, we'll >>>>>>>>>>>>>> apply both >>>>>>>>>>>>>> to the new release one way or another. >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> -- >>>>>>>>>>>>>> >> Dr. Bobby R. Bruce >>>>>>>>>>>>>> >> Room 3050, >>>>>>>>>>>>>> >> Kemper Hall, UC Davis >>>>>>>>>>>>>> >> Davis, >>>>>>>>>>>>>> >> CA, 95616 >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> web: https://www.bobbybruce.net >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last >>>>>>>>>>>>>> couple days but kept running out of time. I'm sitting down to >>>>>>>>>>>>>> work on it >>>>>>>>>>>>>> right now. >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> Gabe >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Hey Gabe, >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Is there any update on this? >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> Kind regards, >>>>>>>>>>>>>> >>>> Bobby >>>>>>>>>>>>>> >>>> -- >>>>>>>>>>>>>> >>>> Dr. Bobby R. Bruce >>>>>>>>>>>>>> >>>> Room 3050, >>>>>>>>>>>>>> >>>> Kemper Hall, UC Davis >>>>>>>>>>>>>> >>>> Davis, >>>>>>>>>>>>>> >>>> CA, 95616 >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> web: https://www.bobbybruce.net >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Hi Gabe, >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> I have more details about this. In this use case, SST >>>>>>>>>>>>>> initialized the >>>>>>>>>>>>>> >>>>> Python environment before adding the "gem5 object". >>>>>>>>>>>>>> This gem5 object >>>>>>>>>>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> The function that does that is initPython() >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> The following commands will pull the docker image for >>>>>>>>>>>>>> SST testing >>>>>>>>>>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root >>>>>>>>>>>>>> must be >>>>>>>>>>>>>> >>>>> specified), >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> ``` >>>>>>>>>>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>>>>>>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>>>>>>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>>>>>>>>>> >>>>> bash -c "\ >>>>>>>>>>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} >>>>>>>>>>>>>> --without-tcmalloc; \ >>>>>>>>>>>>>> >>>>> cd ext/sst; \ >>>>>>>>>>>>>> >>>>> make clean; make; \ >>>>>>>>>>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>>>>>>>>>> >>>>> ``` >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> We appreciate your help! >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> Regards, >>>>>>>>>>>>>> >>>>> Hoa Nguyen >>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>> > Hey Gabe, >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>>>>>>>>>> integration python is >>>>>>>>>>>>>> >>>>> > initialized from the SST module, not from init.cc >>>>>>>>>>>>>> (this is because SST has >>>>>>>>>>>>>> >>>>> > their own python interpreter). >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and >>>>>>>>>>>>>> Giacomo can give you >>>>>>>>>>>>>> >>>>> > an example that's breaking to help you fix it, I >>>>>>>>>>>>>> believe. >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > There's strong interest in having the SST integration >>>>>>>>>>>>>> working in this >>>>>>>>>>>>>> >>>>> > current release. We've spent a lot of time figuring >>>>>>>>>>>>>> out all of the >>>>>>>>>>>>>> >>>>> > intricacies and would appreciate any help you can >>>>>>>>>>>>>> provide in these last few >>>>>>>>>>>>>> >>>>> > days before the release! >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > Thanks! >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> > Jason >>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>> >>>>> _______________________________________________ >>>>>>>>>>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>>>>>>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>>>>>>>>>> >>>>>>>>>>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
