There will be a couple more patches coming, since the systemc subsystem has problems when python is already running when static initializers run. I had been building without systemc enabled to simplify things earlier, but I need to get it running too. That should be a fairly quick fix.
Gabe On Wed, Dec 15, 2021 at 10:40 PM Hoa Nguyen <[email protected]> wrote: > Hi Gabe, > > Thanks for uploading the changes. Typically it'll take less than an hour > for RISC-V. > > Regards, > Hoa Nguyen > > On Wed, Dec 15, 2021 at 10:38 PM Gabe Black <[email protected]> wrote: > >> I have it running, although it's taking a while. How long should it take >> to finish? Should it finish immediately, 15 minutes, a few hours, >> tomorrow...? I'll upload my changes in the mean time. >> >> Gabe >> >> On Wed, Dec 15, 2021 at 9:13 PM Gabe Black <[email protected]> wrote: >> >>> Ok, I did/am doing a little more looking, and part of the problem seems >>> to be that some of the python blobs executed with executePython assume >>> prior blobs have done imports for them already. That was where the modules >>> or their contents not being found was coming from, and then after that >>> something about the error reporting was blowing up, but that wasn't the >>> primary problem. >>> >>> With that fixed, it's still upset, but now it's because "fatal: Need to >>> instantiate Root() before calling instantiate()". Also, gem5 is printing >>> the help blob which seems wrong, so maybe the command line arguments aren't >>> making it through like they should? It could be that no config script is >>> run, and that's why m5.instantiate() doesn't work. >>> >>> It's also a little weird that there's an ordering dependency between the >>> execPythonCommands invocations, considering the same set of those should >>> still be happening in the same order. I suppose I'm not explicitly >>> importing the m5 module into __main__.__dict__ to run m5.main(), so that >>> may be where that's coming from. >>> >>> Anyway, I'm still chopping away at this, but I think it's pretty close. >>> >>> Gabe >>> >>> On Wed, Dec 15, 2021 at 8:29 PM Gabe Black <[email protected]> wrote: >>> >>>> Also, I have to say, while I'm grateful that there was a push to get >>>> the SCons changes reviewed for the release, we wouldn't be in this crunch >>>> if they had been reviewed a month ago. It would be best not to let things >>>> bunch up and then make a herculean push right at the deadline, where >>>> problems have a very limited time to get resolved. >>>> >>>> Gabe >>>> >>>> On Wed, Dec 15, 2021 at 4:38 PM Gabe Black <[email protected]> >>>> wrote: >>>> >>>>> That's actually very helpful, since I was pretty sure there was a >>>>> memory problem but couldn't figure out where. I think there's a good >>>>> chance >>>>> this will be an easy fix. >>>>> >>>>> Gabe >>>>> >>>>> On Wed, Dec 15, 2021, 8:59 AM Jason Lowe-Power <[email protected]> >>>>> wrote: >>>>> >>>>>> Ok, I was able to get a debugger to work and I dug in a bit more... >>>>>> >>>>>> The problem is that the new code does not initialize the __main__.py >>>>>> module. The original SST code had the following line: >>>>>> `pythonMain = PyImport_AddModule(PyCC("__main__"));` >>>>>> >>>>>> This was executed before `execPythonCommands`. >>>>>> >>>>>> In `execPythonCommands` we use the `pythonMain` variable. However, in >>>>>> the *new* code, that variable is NULL (or unitialized, actually). >>>>>> >>>>>> As far as I can tell, all of the new python changes didn't come with >>>>>> any documentation on how you are supposed to interact with Python from >>>>>> gem5. So, there's no good way for us to update this code. >>>>>> >>>>>> Gabe, if you can update the `execPythonCommands` function to work >>>>>> however you expected it to, that would be great. >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> >>>>>> On Wed, Dec 15, 2021 at 10:45 AM Jason Lowe-Power < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Here's my understanding of where we are on this: >>>>>>> >>>>>>> As of >>>>>>> https://gem5.googlesource.com/public/gem5/+/cf7ce21848ea4aeee28737823e6e768f9a14ceaf >>>>>>> SST was working. (Committed Dec. 7th) >>>>>>> >>>>>>> Then, this large relation chain was pushed: >>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49425 ending >>>>>>> with >>>>>>> https://gem5.googlesource.com/public/gem5/+/4c1422e3ba780a5f152426e214b4be3d45b751e4. >>>>>>> This relation chain broke SST. This was committed on Dec. 8th. >>>>>>> >>>>>>> We have two options: Either we fix this bug ASAP (as in within 6 >>>>>>> days so we can release on 12/22) or we revert the changesets that were >>>>>>> committed on the 8th. >>>>>>> >>>>>>> SST is a *must have* feature of gem5 21.2. It is quite frustrating >>>>>>> to see it broken after all of the hard work that Hoa and Giacomo put >>>>>>> into >>>>>>> it. >>>>>>> >>>>>>> Let me know if I'm missing something. >>>>>>> >>>>>>> Thanks, >>>>>>> Jason >>>>>>> >>>>>>> On Sat, Dec 11, 2021 at 10:35 AM Gabe Black <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/54006 >>>>>>>> >>>>>>>> At least one problem I ran into was that the order of static >>>>>>>> initializers was not determinstic, so the structures which said what >>>>>>>> embedded python modules existed might be statically constructed after >>>>>>>> the >>>>>>>> thing that consumes and sets them up. That could work by coincidence >>>>>>>> if the >>>>>>>> order is favorable, but it wasn't for me and so lots of things didn't >>>>>>>> exist >>>>>>>> as far as the python was concerned. >>>>>>>> >>>>>>>> To be clear, there is some mysterious deeper issue here which is >>>>>>>> probably separate from that. As far as I can tell running gem5 itself >>>>>>>> is >>>>>>>> working fine, but when running as a library in sst there is some sort >>>>>>>> of >>>>>>>> memory corruption or premature garbage collection or something, and it >>>>>>>> leads to a segfault or incorrect behavior (failure to import modules, >>>>>>>> failure to find things inside them, etc). Things were even more >>>>>>>> tempermental until I added what feels like an unnecessary increment of >>>>>>>> the >>>>>>>> reference count for the modules in the new pybind_init.hh, which >>>>>>>> seemed to >>>>>>>> tame things in gem5 proper at least. I'm too tired to keep looking for >>>>>>>> it >>>>>>>> right now, but hopefully this will give you a good leg up. >>>>>>>> >>>>>>>> Gabe >>>>>>>> >>>>>>>> On Sat, Dec 11, 2021 at 8:25 AM Jason Lowe-Power < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Everything was working as of last week-ish, if I remember >>>>>>>>> correctly. We were booting both Arm and RISCV full system with gem5 >>>>>>>>> cores >>>>>>>>> and SST caches/memory. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Jason >>>>>>>>> >>>>>>>>> On Sat, Dec 11, 2021, 8:23 AM Gabe Black <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks, that was really helpful. I've fixed the compilation >>>>>>>>>> problems, and then a few more that were hiding behind that, >>>>>>>>>> including a few >>>>>>>>>> ones that were intrinsic to the way sst was initializing the gem5 >>>>>>>>>> library >>>>>>>>>> (with dlopen). Was this working to begin with and all this breakage >>>>>>>>>> is new, >>>>>>>>>> or was it just that the compilation broke and there were these other >>>>>>>>>> problems already? I want to make sure I don't leave things in a >>>>>>>>>> worse state >>>>>>>>>> than when I got here, but I also don't want to go chasing down >>>>>>>>>> things that >>>>>>>>>> weren't working to begin with. >>>>>>>>>> >>>>>>>>>> I'm going to upload what I have so far, and hopefully someone >>>>>>>>>> else can take a look at it too. The current problem seems to be that >>>>>>>>>> the >>>>>>>>>> modules in _m5 aren't working right, either because they've been >>>>>>>>>> garbage >>>>>>>>>> collected, or because they're not being set up properly, or... >>>>>>>>>> >>>>>>>>>> Gabe >>>>>>>>>> >>>>>>>>>> On Fri, Dec 10, 2021 at 8:17 PM Bobby Bruce <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hey gabe. >>>>>>>>>>> >>>>>>>>>>> No idea if this is the _best_ solution to your problem, but my >>>>>>>>>>> solution would be to rebuild the image with this installed. Modify >>>>>>>>>>> the >>>>>>>>>>> `util/dockerfiles/sst-11.1.0/Dockerfile` to the environment you >>>>>>>>>>> want. Then >>>>>>>>>>> run `docker build -t <some name here> util/dockerfiles/sst-11.1.0` >>>>>>>>>>> to build >>>>>>>>>>> an image with the name "<some name here>". >>>>>>>>>>> >>>>>>>>>>> Then you can execute `docker run -u $UID:$GID --volume >>>>>>>>>>> $(pwd):$(pwd) -w $(pwd) -it <same name here>` within the gem5 >>>>>>>>>>> directory to >>>>>>>>>>> spin up and enter a container from the image you just built. I >>>>>>>>>>> think you'll >>>>>>>>>>> be able to do what you want inside the container. >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> Bobby >>>>>>>>>>> -- >>>>>>>>>>> Dr. Bobby R. Bruce >>>>>>>>>>> Room 3050, >>>>>>>>>>> Kemper Hall, UC Davis >>>>>>>>>>> Davis, >>>>>>>>>>> CA, 95616 >>>>>>>>>>> >>>>>>>>>>> web: https://www.bobbybruce.net >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Dec 10, 2021 at 7:24 PM Gabe Black <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Dumb question: I'm trying to run gdb inside this container on >>>>>>>>>>> the sst thing. How do I do that? It's not installed in the >>>>>>>>>>> container now, >>>>>>>>>>> and I can't (easily) figure out how to get it installed. I can tell >>>>>>>>>>> docker >>>>>>>>>>> to install it, but then it seems to throw that away as soon as the >>>>>>>>>>> command >>>>>>>>>>> ends. >>>>>>>>>>> > >>>>>>>>>>> > Gabe >>>>>>>>>>> > >>>>>>>>>>> > On Fri, Dec 10, 2021 at 5:09 PM Bobby Bruce < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >> >>>>>>>>>>> >> Thanks Gabe, >>>>>>>>>>> >> >>>>>>>>>>> >> This is very much appreciated. I'm going to create the >>>>>>>>>>> release staging once a couple more things get in. Feel free to push >>>>>>>>>>> any >>>>>>>>>>> patches related to these bugs to the release staging branch. >>>>>>>>>>> >> >>>>>>>>>>> >> If there is an order of priority I'd say the bug affecting >>>>>>>>>>> SST is of higher importance than that affecting the Weeklies (as >>>>>>>>>>> far as I >>>>>>>>>>> can see the latter is hard to trigger). That being said, we'll >>>>>>>>>>> apply both >>>>>>>>>>> to the new release one way or another. >>>>>>>>>>> >> >>>>>>>>>>> >> -- >>>>>>>>>>> >> Dr. Bobby R. Bruce >>>>>>>>>>> >> Room 3050, >>>>>>>>>>> >> Kemper Hall, UC Davis >>>>>>>>>>> >> Davis, >>>>>>>>>>> >> CA, 95616 >>>>>>>>>>> >> >>>>>>>>>>> >> web: https://www.bobbybruce.net >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> On Fri, Dec 10, 2021 at 5:02 PM Gabe Black < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>> >>>>>>>>>>> >>> Hi Bobby, not yet, I meant to look into this for the last >>>>>>>>>>> couple days but kept running out of time. I'm sitting down to work >>>>>>>>>>> on it >>>>>>>>>>> right now. >>>>>>>>>>> >>> >>>>>>>>>>> >>> Gabe >>>>>>>>>>> >>> >>>>>>>>>>> >>> On Fri, Dec 10, 2021 at 1:21 PM Bobby Bruce < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Hey Gabe, >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Is there any update on this? >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Kind regards, >>>>>>>>>>> >>>> Bobby >>>>>>>>>>> >>>> -- >>>>>>>>>>> >>>> Dr. Bobby R. Bruce >>>>>>>>>>> >>>> Room 3050, >>>>>>>>>>> >>>> Kemper Hall, UC Davis >>>>>>>>>>> >>>> Davis, >>>>>>>>>>> >>>> CA, 95616 >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> web: https://www.bobbybruce.net >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> On Wed, Dec 8, 2021 at 5:51 PM Hoa Nguyen via gem5-dev < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Hi Gabe, >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> I have more details about this. In this use case, SST >>>>>>>>>>> initialized the >>>>>>>>>>> >>>>> Python environment before adding the "gem5 object". This >>>>>>>>>>> gem5 object >>>>>>>>>>> >>>>> will add more Python stuff from gem5 to the environment. >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> The function that does that is initPython() >>>>>>>>>>> >>>>> >>>>>>>>>>> https://gem5.googlesource.com/public/gem5/+/refs/heads/develop/ext/sst/gem5.cc#415 >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> The following commands will pull the docker image for SST >>>>>>>>>>> testing >>>>>>>>>>> >>>>> purposes (note that host_gem5_root and guest_gem5_root >>>>>>>>>>> must be >>>>>>>>>>> >>>>> specified), >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> ``` >>>>>>>>>>> >>>>> docker run -u $UID:$GID --volume >>>>>>>>>>> "${host_gem5_root}":"${guest_gem5_root}" -w \ >>>>>>>>>>> >>>>> "${guest_gem5_root}" --rm gcr.io/gem5-test/sst-env \ >>>>>>>>>>> >>>>> bash -c "\ >>>>>>>>>>> >>>>> scons build/RISCV/libgem5_opt.so -j${nproc} >>>>>>>>>>> --without-tcmalloc; \ >>>>>>>>>>> >>>>> cd ext/sst; \ >>>>>>>>>>> >>>>> make clean; make; \ >>>>>>>>>>> >>>>> sst --add-lib-path=./ sst/example.py; >>>>>>>>>>> >>>>> ``` >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> We appreciate your help! >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Regards, >>>>>>>>>>> >>>>> Hoa Nguyen >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> On 12/8/21, Jason Lowe-Power <[email protected]> wrote: >>>>>>>>>>> >>>>> > Hey Gabe, >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > This change breaks the SST integration. In the SST >>>>>>>>>>> integration python is >>>>>>>>>>> >>>>> > initialized from the SST module, not from init.cc (this >>>>>>>>>>> is because SST has >>>>>>>>>>> >>>>> > their own python interpreter). >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > We would appreciate some help in fixing this. Hoa and >>>>>>>>>>> Giacomo can give you >>>>>>>>>>> >>>>> > an example that's breaking to help you fix it, I believe. >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> https://gem5-review.googlesource.com/c/public/gem5/+/49413 >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > There's strong interest in having the SST integration >>>>>>>>>>> working in this >>>>>>>>>>> >>>>> > current release. We've spent a lot of time figuring out >>>>>>>>>>> all of the >>>>>>>>>>> >>>>> > intricacies and would appreciate any help you can >>>>>>>>>>> provide in these last few >>>>>>>>>>> >>>>> > days before the release! >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > Thanks! >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > Jason >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> _______________________________________________ >>>>>>>>>>> >>>>> gem5-dev mailing list -- [email protected] >>>>>>>>>>> >>>>> To unsubscribe send an email to [email protected] >>>>>>>>>>> >>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s >>>>>>>>>>> >>>>>>>>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
