Wouldn't this be a much smaller effort?
Steve
Hani Suleiman wrote:
Aha, yeah, System.currentTimeMillis would definitely be of much higher resolution.
I know the problem also intermittently affects linux, as I've seen it happen on there too (but a lot more rarely).
I think it's a feature of the antiquated HFS mac filesystem that OSX yses by default, rather than the OS itself. Adding a long timestamp to the segments file would fix this everywhere I think, since the worst resolution timers out there are in the region of 10ms, which isn't so bad.
I can work on that (pretty big, I suspect) patch if everyone else is too busy, since I'm quite looking forward to all tests passing on all platforms. I can verify on windows, linux, solaris, and OSX. Of course it'd be better if someone a bit more experienced with the codebase were to handle it ;)
On Tuesday, September 30, 2003, at 06:37 PM, Dmitry Serebrennikov wrote:
Hani Suleiman wrote:
The count for FSDirectory on OSX is around 700-800. The problem is that the filesystem timestamps on OSX has abysmal resolution (nothing finer than 1 whole second!)
Wow! Write once, debug everywhere...
Is this an OSX "feature" or is this a feature of a particular disk / filesystem / driver / whatnot that is in use in the current run?
If this is unavoidable on OSX, perhaps other platforms are also affected? Does anyone know the timestamp resolution on Wintel, Lintel, Solaris, etc?
Finally, what if we allocate a "long" in the beginning of the files that need to be touched (like the segments and other files) and use it to store the full System.currentTimeMillis timestamp? Does this method return a more granular number on OSX? This would be a big change, but it would work the same on all platforms and will not have a slowdown.
So the touch method will spin its wheels for anything up to a second before we've registered an actual 'touch'.
This isn't so bad as files aren't touch that often, but it's a lot more problematic in the case of SegmentInfos.write(), where a new file is created, then copied over the old one. The testDeleteReaderWriteConflict testcase fails due to this. The old file and the new have the same timestamp, and so the staleness check fails. How should this be fixed?
Also, won't the fix for FSDirectory mean there's a window where other threads could end up with bad info, if we're taking up to a second to touch a file?
I don't know off hand, but I believe there is file locking in place to prevent other threads from getting into this code until the update operation is completed, and it will not be completed until the touch finishes. This still sucks with a 1-second blocking...
Dmitry.
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
-- Steve Rowe Software Engineer Center for Natural Language Processing School of Information Studies Syracuse University www.cnlp.org
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
