I'm no Docker expert, but I believe that's pretty much what the path redirection scheme I described does, though probably in a more limited way. We resisted the temptation to make it fancier because I think the right approach would be to combine our gem5-based path redirection with something like overlayfs if you really want to get fancy.
Steve On Thu, Sep 10, 2015 at 12:50 AM Andreas Hansson <[email protected]> wrote: > Hi all, > > Merely a reflection on what Tony wrote in his mail: For SE regressions > runs the binaries are _not_ always in the same path. In fact we probably > have 3-4 different permutations in ARM alone. I do not know if the path > difference will impact the behaviour, but it is definitely a concern. > > A completely random thought, and I am out of my depth here, but is there > any chance we could use something like Docker images as the SE filesystem? > We could use a base image just like we do in FS, and then have overlays > for any experiment-specific files. Again, I do not know what is > possible/desirable here...merely a thought. > > Andreas > > On 09/09/2015 19:47, "gem5-dev on behalf of Steve Reinhardt" > <[email protected] on behalf of [email protected]> wrote: > > >Hi Joel, > > > >We haven't run into this specific issue internally; I think our bugs must > >be robust enough that they're reproducible even in the face of minor > >changes from system to system. (And of course once you start debugging on > >a particular system, the behavior is deterministic.) > > > >This is really a symptom of a larger problem with SE mode, which is that > >too much of the underlying host filesystem leaks into the simulated > >environment. We've run into this in a bigger way with dynamic linking > >(and > >yes, Mike Upton, those were our patches that were described at the > >workshop > >which we haven't posted yet), where you can end up linking in different > >versions of shared libraries when you run on different platforms and ld.so > >just grabs whatever happens to be in the local /lib directory. Another > >thing David did this past summer is set up a path redirection scheme that > >lets you build a fake filesystem for your SE-mode binary. This lets you > >do > >things like set up a directory tree with a particular set of shared libs > >in > >it, make that look like "/" to the simulated binary, then reuse that tree > >on multiple systems. It also allows you to do some handy things like > >auto-generate a file that shows up as /proc/cpuinfo to the simulated > >system > >with information determined by your gem5 config. This doesn't directly > >solve the readlink() issue, but I think you could use this feature to put > >your executable at a reproducible host-independent path. Of course, this > >is yet another set of patches we have yet to post, as far as I can tell. > >Unfortunately we have some big deadlines this month, but I'm hoping we can > >get back to clearing up some of our patch backlog next month. In the > >meantime, as Tony says, I'm not sure how else to resolve the tension > >between reproducibility and correctness here. > > > >Steve > > > > > >On Wed, Sep 9, 2015 at 11:14 AM Joel Hestness <[email protected]> > >wrote: > > > >> > If any codes use readlink() in this way, they need the correct path, > >>and > >> > not gem5’s path. This isn’t non-determinism, the runs will always be > >>the > >> > same given the binaries are in the same path. I’m not sure how this > >>can > >> be > >> > solved other than to say we won’t allow codes to use readlink() this > >>way. > >> > > >> > >> It's not a viable solution to just disallow this kind of use, though. > >>GLIBC > >> uses readlink, which gets compiled into some static binaries, so a gem5 > >> user may not have control over the use of readlink. A simple example > >>where > >> this fails is if I run simulations on a cluster with a different > >>directory > >> structure than machines I can use to debug. The difference in directory > >> structure is enough so that the cluster runs cannot be replicated on > >>other > >> machines. > >> > >> > >> > >> > Using readlink() on /proc/self/exe would return the path to the gem5 > >> > binary previously, as opposed to the binary running inside of gem5. We > >> > noticed this because some of our codes used readlink() this way to > >>figure > >> > out their paths. > >> > > >> > >> Sure. Can you explain how you are running simulations such that you > >>don't > >> need to worry about simulations being replicable in different settings? > >>Are > >> you always able to debug simulations in the exact same setting where you > >> discovered a bug? > >> > >> Joel > >> > >> > >> > >> > > >> > *From:* Joel Hestness [mailto:[email protected]] > >> > *Sent:* Wednesday, September 09, 2015 10:32 AM > >> > *To:* gem5 Developer List; Hashe, David; Che, Shuai; Gutierrez, > >>Anthony > >> > *Subject:* Re: [gem5-dev] changeset in gem5: syscall: Add readlink to > >>x86 > >> > with special cas... > >> > > >> > > >> > > >> > Hey David, Shuai, and Tony, > >> > > >> > > >> > > >> > It appears that this patch can introduce simulator non-determinism > >>due > >> > to the use of Process::progName(). Basically, if benchmark paths are > >> > changed (e.g. running on a different system), then the string copy > >> > operations in the "/proc/self/exe" codepath can create strings of > >> different > >> > lengths. As a result, readlink can return different values depending > >>on > >> the > >> > progName() and cause the simulated benchmark to behave differently. > >>This > >> is > >> > problematic when trying to replicate sims for regressions or > >>debugging. > >> > > >> > > >> > > >> > It looks like this patch aims to address changes in GLIBC, which now > >> > uses readlink to find the binary before handling environment variables > >> (in > >> > _dl_non_dynamic_init > >><http://osxr.org/glibc/source/elf/dl-support.c#0312 > >> >). > >> > That said, I'm not clear how this patch is supposed to fix problems or > >> how > >> > we can avoid the nondeterminism it introduces. Can you guys elaborate > >>on > >> > how this change is intended to work (e.g. what is the use case you > >>were > >> > aiming for)? > >> > > >> > > >> > > >> > Thank you, > >> > > >> > Joel > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Sat, Aug 1, 2015 at 11:07 AM, David Hashe <[email protected]> > >> wrote: > >> > > >> > changeset 9abf6a7c14ab in /z/repo/gem5 > >> > details: http://repo.gem5.org/gem5?cmd=changeset;node=9abf6a7c14ab > >> > description: > >> > syscall: Add readlink to x86 with special case /proc/self/exe > >> > > >> > This patch implements the correct behavior. > >> > > >> > diffstat: > >> > > >> > src/arch/x86/linux/process.cc | 6 +++--- > >> > src/sim/syscall_emul.cc | 31 ++++++++++++++++++++++++++----- > >> > 2 files changed, 29 insertions(+), 8 deletions(-) > >> > > >> > diffs (80 lines): > >> > > >> > diff -r 255ebb0b32b4 -r 9abf6a7c14ab src/arch/x86/linux/process.cc > >> > --- a/src/arch/x86/linux/process.cc Mon Jul 20 09:15:18 2015 -0500 > >> > +++ b/src/arch/x86/linux/process.cc Mon Jul 20 09:15:18 2015 -0500 > >> > @@ -485,7 +485,7 @@ > >> > /* 264 */ SyscallDesc("renameat", unimplementedFunc), > >> > /* 265 */ SyscallDesc("linkat", unimplementedFunc), > >> > /* 266 */ SyscallDesc("symlinkat", unimplementedFunc), > >> > - /* 267 */ SyscallDesc("readlinkat", unimplementedFunc), > >> > + /* 267 */ SyscallDesc("readlinkat", readlinkFunc), > >> > /* 268 */ SyscallDesc("fchmodat", unimplementedFunc), > >> > /* 269 */ SyscallDesc("faccessat", unimplementedFunc), > >> > /* 270 */ SyscallDesc("pselect6", unimplementedFunc), > >> > @@ -626,7 +626,7 @@ > >> > /* 82 */ SyscallDesc("select", unimplementedFunc), > >> > /* 83 */ SyscallDesc("symlink", unimplementedFunc), > >> > /* 84 */ SyscallDesc("oldlstat", unimplementedFunc), > >> > - /* 85 */ SyscallDesc("readlink", unimplementedFunc), > >> > + /* 85 */ SyscallDesc("readlink", readlinkFunc), > >> > /* 86 */ SyscallDesc("uselib", unimplementedFunc), > >> > /* 87 */ SyscallDesc("swapon", unimplementedFunc), > >> > /* 88 */ SyscallDesc("reboot", unimplementedFunc), > >> > @@ -846,7 +846,7 @@ > >> > /* 302 */ SyscallDesc("renameat", unimplementedFunc), > >> > /* 303 */ SyscallDesc("linkat", unimplementedFunc), > >> > /* 304 */ SyscallDesc("symlinkat", unimplementedFunc), > >> > - /* 305 */ SyscallDesc("readlinkat", unimplementedFunc), > >> > + /* 305 */ SyscallDesc("readlinkat", readlinkFunc), > >> > /* 306 */ SyscallDesc("fchmodat", unimplementedFunc), > >> > /* 307 */ SyscallDesc("faccessat", unimplementedFunc), > >> > /* 308 */ SyscallDesc("pselect6", unimplementedFunc), > >> > diff -r 255ebb0b32b4 -r 9abf6a7c14ab src/sim/syscall_emul.cc > >> > --- a/src/sim/syscall_emul.cc Mon Jul 20 09:15:18 2015 -0500 > >> > +++ b/src/sim/syscall_emul.cc Mon Jul 20 09:15:18 2015 -0500 > >> > @@ -365,12 +365,10 @@ > >> > } > >> > strncpy((char *)buf.bufferPtr(), cwd.c_str(), size); > >> > result = cwd.length(); > >> > - } > >> > - else { > >> > + } else { > >> > if (getcwd((char *)buf.bufferPtr(), size) != NULL) { > >> > result = strlen((char *)buf.bufferPtr()); > >> > - } > >> > - else { > >> > + } else { > >> > result = -1; > >> > } > >> > } > >> > @@ -405,7 +403,30 @@ > >> > > >> > BufferArg buf(bufPtr, bufsiz); > >> > > >> > - int result = readlink(path.c_str(), (char *)buf.bufferPtr(), > >> bufsiz); > >> > + int result = -1; > >> > + if (path != "/proc/self/exe") { > >> > + result = readlink(path.c_str(), (char *)buf.bufferPtr(), > >> bufsiz); > >> > + } else { > >> > + // readlink() will return the path of the binary given > >> > + // with the -c option, however it is possible that this > >> > + // will still result in incorrect behavior if one binary > >> > + // runs another, e.g., -c time -o "my_binary" where > >> > + // my_binary calls readlink(). this is a very unlikely case, > >> > + // so we issue a warning. > >> > + warn_once("readlink may yield unexpected results if multiple > >>" > >> > + "binaries are used\n"); > >> > + if (strlen(p->progName()) > bufsiz) { > >> > + // readlink will truncate the contents of the > >> > + // path to ensure it is no more than bufsiz > >> > + strncpy((char*)buf.bufferPtr(), p->progName(), bufsiz); > >> > + result = bufsiz; > >> > + } else { > >> > + // return the program's working path rather > >> > + // than the one for the gem5 binary itself. > >> > + strcpy((char*)buf.bufferPtr(), p->progName()); > >> > + result = strlen((char*)buf.bufferPtr()); > >> > + } > >> > + } > >> > > >> > buf.copyOut(tc->getMemProxy()); > >> > > >> > _______________________________________________ > >> > gem5-dev mailing list > >> > [email protected] > >> > http://m5sim.org/mailman/listinfo/gem5-dev > >> > > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > Joel Hestness > >> > PhD Candidate, Computer Architecture > >> > Dept. of Computer Science, University of Wisconsin - Madison > >> > http://pages.cs.wisc.edu/~hestness/ > >> > > >> > >> > >> > >> -- > >> Joel Hestness > >> PhD Candidate, Computer Architecture > >> Dept. of Computer Science, University of Wisconsin - Madison > >> http://pages.cs.wisc.edu/~hestness/ > >> _______________________________________________ > >> gem5-dev mailing list > >> [email protected] > >> http://m5sim.org/mailman/listinfo/gem5-dev > >> > >_______________________________________________ > >gem5-dev mailing list > >[email protected] > >http://m5sim.org/mailman/listinfo/gem5-dev > > > -- IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > Registered in England & Wales, Company No: 2557590 > ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > Registered in England & Wales, Company No: 2548782 > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
