thanks for the NFS remarks On Wed, 5 Sep 2012 18:38:05 +0000 Haller, John H (John) wrote: > I'm thinking this is more likely to be a network filesystem problem than > a mktemp problem. The OP reported that this was done in a large shared > filesystem. I would also guess that it is a NFS filesystem. If it is using > NFS v2, O_EXCL doesn't work properly, and mktemp won't be able to > properly check for collisions. O_EXCL should work for long-lived files, > though, as it is emulated by looking up whether the file exists. This is > a race-prone check, which can fail if two systems are both trying to > create the same filename. Given that AST mktemp already checks for > duplicates using O_EXCL, I don't see how AST mktemp could cause the > failure. While increasing the universe of filenames will decrease the > collision rate, it won't eliminate them. But, for NFS v2 to have caused > the problem, two programs would have had to created the same temp > file nearly simultaneously, although it's possible one created the file > long before it was used for writing anything and causing the corruption.
> A related problem would be if a program created a temp file with mktemp > long before it needed it, a temp cleaner removed the temp file, another > client created the temp file with the same name and used it, then the > first program came in and used the temp file it thought it had exclusively > opened long ago. This doesn't require a network filesystem problem, but > is an unusual use case such that a file is created long before it is actually > needed. But, given that a "temp" file needed to be kept in a directory for > over > 96 hours, this doesn't seem that unlikely. > Regards, > John Haller > > -----Original Message----- > > From: [email protected] [mailto:ast-users- > > [email protected]] On Behalf Of Glenn Fowler > > Sent: Wednesday, September 05, 2012 12:42 PM > > To: [email protected]; [email protected] > > Cc: [email protected]; [email protected] > > Subject: Re: [ast-users] [ast-developers] RFE: longer temp names in mktemp > > builtin and libast API > > > > > > On Wed, 5 Sep 2012 18:48:00 +0200 Lionel Cons wrote: > > > On 5 September 2012 17:00, Glenn Fowler <[email protected]> wrote: > > > > > > > > (1) are we talking about libast mktemp(3) or ksh mktemp(1) > > > > > AST mktemp as plain command. We replaced the machine's native > > > /usr/bin/mktemp with AST mktemp since GNU coreutils and (especially!) > > > the Solaris /usr/bin/mktemp are prone to even more collisions (Solaris > > > mktemp in Solaris 2.6-10 and 11 (Opensolaris didn't have the problem > > > since it used the ksh93 mktemp) suffers from printing random garbage > > > in rare occasions, too). > > > > > > (2) did the original temp file exist when the dup name was generated > > > > > I don't know. I have to ask. I'm just the messenger. > > > > roland is correct that the low level ast routine is pathtemp() > > > > the reason for question (2) is that pathtemp() has a collision detection > > loop > > and mktemp(1) calls pathtemp() with an fd pointer that instructs it to > > create the temp file with > > open(path, O_CREAT|O_RDWR|O_EXCL, mode) > > if this fails then another pseudo-random tmp path is generated > > until there is no collision > > > > even if the range of the generated paths were limited pathtemp(3) as called > > by mktemp(1) should never return success on a path that already exists > > unless mktemp were called with --unsafe or with --directory > > > > pathtemp() in this mode could fail (by not returning) if the entire range > > were covered by existing files -- it would loop in a bad fashion, attempting > > random paths attempting to hit unused names > > > > since the collision in your case was after 96 hours I don't think its > > any kind of weird filesystem timing problem > > > > can you send the mktemp command line used? > > > > I looked at the pathtemp() code and the range can be improved by switching > > from base 32 (should have been 36! : [0-9]+[a-z]) to base 62 > > ([0-9]+[a-z]+[A-Z]) > > numeric representation of the pesudorandom hash > > and fixing the mktemp(1) user supplied prefix logic to use more of the hash > > when the prefix length is less than the max 5 chars > > > > I ran this test with the old and new pathtemp() > > > > ksh -c ' > > integer n > > typeset f > > typeset -A seen > > builtin mktemp > > for p in _____ ____ ___ __ _ "" > > do while : > > do ((n++)) > > f=$(mktemp -u "$p") > > if [[ ${seen[$f]} ]] > > then printf "%11s %9d %9d\n" $f ${seen[$f]} $n > > break > > fi > > seen[$f]=$n > > done > > done > > ' > > > > the test uses -u (--unsafe) so it doesn't generate any temp files > > it does check for collisions using access(2), but that's not a factor for > > the test > > the results show that the old alg collides around ~10^4 calls regardless of > > prefix > > size > > and the new alg collides around ~10^6 calls with prefix length 0 > > > > for avoiding predictability the new results are better > > and they also make the collision detection loop more efficient > > > > with mktemp prefix "": > > the old alg is limited to 32^5 = ~10^7 different names > > the new alg is limited to 32^10 = ~10^15 different names > > > > using the max #X's (14) in the template notation > > mktemp ${prefix}XXXXXXXXXXXXXX > > the old and new alg are limited to 32^14 = ~10^21 different names > > > > again, the test calls mktemp with -u (collision detection disabled), > > so the results show collisions in name generation only > > > > old > > > > _____3e.3vj 2657 3178 > > ____2c.vks 3828 4723 > > ___0c.7td 4814 5268 > > __1i.2q7 8004 10667 > > _26.6kb 13720 14311 > > 3n.dc5 16113 16578 > > > > new > > > > _____03.44w 76 359 > > ____041.2Nu 2460 2890 > > ___02Vu.1De 7965 23963 > > __02eD1.3p1 144094 246462 > > _04p47i.3E5 316344 855852 > > 04yrNdG.0NY 1375224 2357065 > > > > I looked further into the code and also modified pathtemp() to > > to call mkdir() atomically rahter than open(O_CREAT) if the prefix > > arg ends with "/" > > > > this will move the mktemp --directory collision detection from > > mktemp(1) to pathtemp(3) > > > > _______________________________________________ > > ast-users mailing list > > [email protected] > > https://mailman.research.att.com/mailman/listinfo/ast-users _______________________________________________ ast-users mailing list [email protected] https://mailman.research.att.com/mailman/listinfo/ast-users
