Ah -- I (finally) understand... I must have missed the detail re:
serialization in message #30 out of #60 or so
So, this clarifies to me:
-- cache by filename key is correct and good for most cases and should
be on by default
-- the grace period is a clever solution for the
Jim,
Can I ask why the filename is important for the cache key? With the
cache delay, the inode/dev + *time + size should do it all.
In fact, I finally understood the difference between mtime and ctime, if
any change is made, it should be the change to ctime.
Why ctime? ctime is unique in that
On Aug 21, 2008, at 8:14 AM, Dossy Shiobara wrote:
I've remained silent on this issue because I didn't want to be
accused of stifling the community, etc.
...
End of discussion.
Accused. Guilty.
Don Baccus
http://donb.photo.net
http://birdnotes.net
http://openacs.org
--
ReplyTo: AOLserver Discussion
Sent: Aug 21, 2008 12:25 PM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
On Aug 21, 2008, at 8:14 AM, Dossy Shiobara wrote:
I've remained silent on this issue because I didn't want to be
accused of stifling the community, etc.
...
End
On Thursday 08:14 AM 8/21/2008, Dossy Shiobara wrote:
4) I see the simplest (best?) solution here being a configurable
parameter that controls fastpath's cache key generation. As Jim points
out, one can quickly test whether this would solve the problem at hand by
temporarily #define'ing
Dossy Shiobara wrote:
However, to let it continue to spin over and over is unproductive.
I'd love to hear other solutions other than configurabe cache key
strategies or a time-based delay caching strategy, but the time to
debate whether this is a defect or not is officially over: it is a
On Thu, 2008-08-21 at 11:14 -0400, Dossy Shiobara wrote:
4) I see the simplest (best?) solution here being a configurable
parameter that controls fastpath's cache key generation. As Jim points
out, one can quickly test whether this would solve the problem at hand
by temporarily #define'ing
On Thursday 09:25 AM 8/21/2008, Tom Jackson wrote:
Why put newly created
files into a cache, if the point of the cache is to handle static files?
We can wait for evidence that it is static. In this case, we can wait
until it is a few seconds old, at least.
This is a very good point, actually.
, August 21, 2008 12:25 PM
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
On Thu, 2008-08-21 at 11:14 -0400, Dossy Shiobara wrote:
4) I see the simplest (best?) solution here being a configurable
parameter that controls fastpath's cache key
] On
Behalf Of Tom Jackson
Sent: Thursday, August 21, 2008 12:25 PM
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
On Thu, 2008-08-21 at 11:14 -0400, Dossy Shiobara wrote:
4) I see the simplest (best?) solution here being a configurable
parameter that controls
Titi Alailima wrote:
what you were looking for. Even with the mtime fix there's no
guarantee that systems which muck around with mtime (such as tar)
won't cause separate files to collide. For a contrived example:
I think the best you can do is to use ctime instead of mtime, or maybe
btime
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
I don't have any opinion on the fix, but I think the actual objection
to
using the filename in the fix is that this would cause hard links to
files, which are for all intents and purposes The Same File
I'm lost. If you are interested in serving the same content, mtime tells
you the last time the content was modified. ctime changes for reasons
all unrelated to the content.
But this is a cache, which is a copy. There is never any way to
guarantee that the content is the same as what is currently
Of Tom Jackson
Sent: Thursday, August 21, 2008 12:25 PM
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
On Thu, 2008-08-21 at 11:14 -0400, Dossy Shiobara wrote:
4) I see the simplest (best?) solution here being a configurable
parameter that controls
On Thursday 12:08 PM 8/21/2008, Titi Alailima wrote:
It's perfectly acceptable in dealing with caching systems that the cached
value could be out of sync, but not that the cached value could be for
something entirely different from what you were looking for.
Yep. I think that aspect of the
On Thu, 2008-08-21 at 17:34 -0400, Jim Davidson wrote:
So, technically this was a case where
we dynamically created code which was later read by ADP (which had the
same dev/inode cache stuff as fastpath). However, this was done
carefully:
-- Tcl-level mutex/condition variables to
On Thu, 2008-08-21 at 13:52 -0700, John Caruso wrote:
On Thursday 12:08 PM 8/21/2008, Titi Alailima wrote:
It's perfectly acceptable in dealing with caching systems that the cached
value could be out of sync, but not that the cached value could be for
something entirely different from what
On Tue, 2008-08-19 at 18:25 -0700, John Caruso wrote:
On Tuesday 05:59 PM 8/19/2008, Juan José del Río wrote:
If you don't want to deactivate it, and have some C skills, I would
recommend you to make the needed changes to fastpath code to enable it
to use the kernel facilities of the operating
, August 19, 2008 8:39 PM
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
Your right, the code snippet below could trip over a race condition as
you've described. But, that's not reason enough to change the
fastpath, it's reason to better document
Titi Alailima schrieb:
This sounds like the problem. Not a bug with fastpath,
Come on folks, the discussion wether or the behavior shown by John Caruso
is a bug or not is completely fruitless. Most aolserver users are not
novices,
and if it takes some of us several weeks to find, what
On Wednesday 08:45 AM 8/20/2008, Jim Davidson wrote:
Overall, it seems one thing to do would be to switch to filename-based
cache keys by default, leaving the dev/inode pair as an option for
folks who run sites with large symlinks and want to benefit from
caching objects just once. I think that
On Aug 20, 2008, at 11:54 AM, John Caruso wrote:
Actually that wouldn't have fixed the problem in the code that led
us to find out about this in the first place. The change that I
suggested does fix that problem, though, and it directly addresses
the limitation of mtime's one-second
John Caruso wrote:
BTW, Jeff, the scenario you'd outlined that you thought would trip this
up...:
13:50:21 - create file
13:50:21 - serve file (gets cached)
13:50:21 - delete file
13:50:21 - create file again (reuses inode)
... time passes ...
13:55:11 - serve file
Hmm... I may be confused and have to re-read all the past messages
(except I've deleted them) but my understanding was it was rapid re-
use of inodes within the 1-second resolution of mtime and the same
file size, all confusing the cache code that caused the problem.
With the fix below with
Jim Davidson wrote:
BTW: Which OS is re-using inodes so quickly? I can't get my Mac OS/X
laptop to do that -- figured the inode re-use/prediction thing was
plugged years ago, e.g., when fsirand was introduced for scrambling NFS
vnodes.
Linux. This tcl page:
set fn /tmp/tmpfile[expr
On Wednesday 10:58 AM 8/20/2008, Jim Davidson wrote:
With the fix below with resolution for mtime at 1 second and the
grace period at 2 seconds I can see how it would work but it would
make me a bit queasy -- fixes which have assumptions of timing can be
fragile.
Perhaps, but not in this
By the way, Jeff, regarding this...:
On Wednesday 10:56 AM 8/20/2008, Jeff Rogers wrote:
And it still hurts the optimization of using a 404 adp page to generate a
heavyweight file only once that gets cached.
...which you'd explained elsewhere as...:
There is also at least one clever
On Aug 20, 2008, at 1:29 PM, John Caruso wrote:
Whether or not that's so, the fact is that everyone on this list
appeared to share the same utterly natural assumption that
ns_returnfile X really will return file X--which turns out to be
untrue solely because of fastpath caching's design
John,
The patch below is not at all unreasonable as far as your stated goal of
not caching a newly modified file.
BTW, I think that fastpath is not enabled by default, your config has to
have the fastpath config section with the cache parameter set to true.
Also, ns_returnfile is really just an
On 8/20/08 11:29 AM, John Caruso [EMAIL PROTECTED] wrote:
Whether or not that's so, the fact is that everyone on this list appeared
to share the same utterly natural assumption that ns_returnfile X really
will return file X
All, I've been on vacation or I would have chimed in earlier, but as
John Caruso wrote:
No, you're still not understanding how the patch works.
Ok, I'll admit that I misread it at first, but you're also not
understanding why I'm saying why it will still break.
I'm surprised you're taking such an all-or-nothing view now,
I don't think I'm taking an
Derek wrote:
how is this thread still alive.
I think this bikeshed should be painted blue.
See http://www.bikeshed.com/ if you don't understand this.
-J
--
AOLserver - http://www.aolserver.com/
To Remove yourself from this list, simply send an email to [EMAIL PROTECTED]
with the
body of
On Wed, 2008-08-20 at 12:23 -0700, Eric Larkin wrote:
On 8/20/08 11:29 AM, John Caruso [EMAIL PROTECTED] wrote:
Whether or not that's so, the fact is that everyone on this list appeared
to share the same utterly natural assumption that ns_returnfile X really
will return file X
All, I've
On Wednesday 12:30 PM 8/20/2008, Jeff Rogers wrote:
I can very easily come up with a scenario that breaks your patched
fastpath just as easily as the original, to which you can rightly say,
but why would you do it that way?. And you would be right.
Do it, then. This is the simplest example
Hi Folks,
I agree with Eric, even though I wrote the original code and was one
of the first to suggest is wasn't a bug. This thread has surprised me
in a few ways:
-- The bug was indeed subtle and curious
-- The debate on dynamic vs. static and underlying assumptions and
performance was
Hello John,
I just think that you're introducing complexity to a common and widely
used infraestructure (with a hack), just to hide the flaws on your side.
I don't like this hack. It's not the correct thing to do, not even if
you make some parameters (like the timeout) changeable via the
On Wednesday 01:45 PM 8/20/2008, Juan José del Río wrote:
But, since Linux version 2.6.13, inotify is into the kernel, and
aolserver can subscribe to a path, so know if that file has been
deleted, modified, or anything else. That's the way a cache can know if
the file has been altered in any
On Wed, Aug 20, 2008 at 12:22 PM, Jeff Rogers [EMAIL PROTECTED] wrote:
Linux. This tcl page:
set fn /tmp/tmpfile[expr rand()]
set f [open $fn w]
puts $f [ns_queryget data]
close $f
after 2
ns_returnfile 200 text/plain $fn
ns_unlink $fn
[after 2] would wait 2 milliseconds. [after 2000]
Hello again John,
Oops, sorry. I thought RedHat Linux had everything an Enterprise
needed ;-)
(Take the above line with a lot of salt. I don't want to discuss about
linux distros... yet ;-)
Then what about adding a -nocache parameter to the function? That way
you won't modify the original
John,
Your last patch suggestion seems good, not caching something that looks
like a new file is fully in line with the intent of fastpath and
ns_returnfile.
I'm not sure everyone is commenting on this new patch idea, maybe a
previous idea?
Anyway if the cache is for serving static (and
On 21/08/2008, at 1:45 AM, Jim Davidson wrote:
I looked at the code a bit closer. The ns_returnfile and
ns_respond commands both call Ns_ConnReturnFile, the public API to
the underlying FastPath. It does more than just blast the content
-- it handles:
...
-- caches, mmap's, or
On Mon, 2008-08-18 at 19:20 -0700, John Caruso wrote:
I'd say it's still better, because it requires explicit action on the
user's part to enable the flawed caching mechanism in that case. And
actually I don't think fastpath in its default configuration would be of
much help in
On Mon, Aug 18, 2008 at 06:06:23PM -0700, John Caruso wrote:
That'd be an improvement over the current situation, but it's still the
case that AOLserver as currently shipped has a file cache mechanism built
into it which 1) may return incorrect data and 2) is enabled by
default. Given the
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
On Tue, 2008-08-19 at 12:24 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 11:59 AM, Tom Jackson wrote:
On Tue, 2008-08-19 at 11:37 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 10:56 AM, Tom Jackson wrote
:18 AM
To: AOLSERVER@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Data corruption with fastpath caching
On Tue, 2008-08-19 at 12:24 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 11:59 AM, Tom Jackson wrote:
On Tue, 2008-08-19 at 11:37 +1000, russell muetzelfeldt wrote:
On 19/08
Andrew,
This is not a corner case. The exact same thing could happen without
fastpath.
What is that thing? That the contents of a file changes after a request
is made and before the file is returned. In fact, there is no guarantee
that it won't change mid-return. This is a fact of life with
What about using epoll (or equivalent) in Linux, and kqueue in FreeBSD
to tell the kernel to notify AOLServer in change a file has changed?
That'd be a pretty easy and efficient way to discard fastpath items in
case they have been deleted and/or modified.
Just my two cents ;-)
-
Juan José
Tom Jackson wrote:
If you want to avoid use of fastpath, just set the configuration lower
than your dynamic content:
#
# Fastpath
#
ns_section ns/server/${server}/fastpath
ns_param cache[set cache 10] ;# max entries ??
ns_param cachemaxsize [set cachemaxsize [expr 5 *
Hi folks,
I wrote the code. The explanation below is correct -- I chose inode/
dev combination to cache the same file even with multiple names which
was the case at AOL -- hundreds of symlinks and hard links to the same
file. The same strategy is used for ADP templates. I think the
On Tuesday 10:40 AM 8/19/2008, Jim Davidson wrote:
I would suggest
the code snippet of create temp file and use fastpath to return
contents is not a use case I was solving for or recommend.
It's also not the use case in question--just a simple illustration of the
problem. Here's a more
John,
It is not a bug in ns_returnfile.
tom jackson
On Tue, 2008-08-19 at 11:52 -0700, John Caruso wrote:
On Tuesday 10:40 AM 8/19/2008, Jim Davidson wrote:
I would suggest
the code snippet of create temp file and use fastpath to return
contents is not a use case I was solving for or
On Monday 05:53 PM 8/18/2008, Jeff Rogers wrote:
russell muetzelfeldt wrote:
fastpath is making assumptions about what means something is the same
file, and those assumptions are not consistent with unix filesystem
semantics - how is this not a bug?
It's not a bug because no one ever said
John Caruso wrote:
Think of it as a if-modified-since or if-none-match
conditional GET.
Actually that's not analogous, ...
I didn't mean to say it was exactly the same, just similar in that given
a particular system that makes particular assumptions it is possible to
construct a
On Tuesday 02:10 PM 8/19/2008, Jeff Rogers wrote:
A default configuration change was suggested which seems generally viewed
as undesirable.
My impression was that support was split about evenly, actually. I take
it that means you're against changing the default? I'm a bit surprised,
since
John,
This isn't a democracy. You have to demonstrate some understanding of
how things work.
The only real security issue is your misuse/abuse of ns_returnfile to
serve dynamic data.
Nobody is going to guarantee that you can't shoot yourself in the foot
due to your lack of understanding of
Personally I can't
imagine any persuasive argument that a caching mechanism that can easily
confuse /usr/local/private/var/rootpass and
/var/tmp/verisign/certs/webcert.txt should be enabled by default in a
web server.
Oh, come on. Only if you're rapidly creating and deleting these files.
On Tuesday 04:57 PM 8/19/2008, Rusty Brooks wrote:
Personally I can't
imagine any persuasive argument that a caching mechanism that can easily
confuse /usr/local/private/var/rootpass and
/var/tmp/verisign/certs/webcert.txt should be enabled by default in a
web server.
Oh, come on. Only
Hi,
I haven't looked at a directory change notification type scheme in a
long time but that could be very clever. Aside from addressing issues
discussed here, the key benefit would be to avoid the repeated stat
syscalls. Those stat calls always bothered me conceptually but the
On 20/08/2008, at 9:57 AM, Rusty Brooks wrote:
I think it's interesting, at least, that this topic has created
more traffic than the list usually sees in a whole year.
most of which isn't actually about the issue at hand, but rather
whether John is an idiot for expecting ns_returnfile to
On Tuesday 05:39 PM 8/19/2008, Jim Davidson wrote:
Your right, the code snippet below could trip
over a race condition as you've described.
It's not a race condition, actually; the code in
that example was serialized, so there's no race involved.
...fastpath isn't broken.
It's designed
One other idea could be to modify the code so that the Win32 behavior
(cache by filename) could be made a configurable option (perhaps
default on) instead of compile-time. Would be easy to fiddle with
code snippets like this to make that happen:
#ifdef _WIN32
key = file;
#else
Consider the following pseudocode snippet:
...generate file $myfile in some way...
ns_returnfile 200 text/plain $myfile
ns_unlink $myfile
If this snippet is executed in a tight loop on a Linux system, the chances
of returning the wrong results are very high due to AOLserver's
There is probably someone here that can directly address a better way to
do what you want, with ns_cache or some other scheme, but it looks like
your basic problem is saving rapidly changing data to disk or serving it
from cache. Why do this? If you data is changing faster than once per
second,
On Monday 01:33 PM 8/18/2008, Tom Jackson wrote:
It's not be a data corruption issue
because you are choosing to overwrite the old data with new data using
the exact same file name. If the data is important, don't overwrite it,
thus no corruption.
No, you've misunderstood the scenario. The
John,
Just to be clear: fastpath is for serving static content. This is not
what you are doing here, you are creating a temporary file to store
dynamic content. For your bug to work you must delete the old file and
create a new one within the same second, etc.
Also, your code sequence below
While I'd agree this is a bug in fastpath, the real problem is that
fastpath is being used at all in this case. The intent of fastpath is
to avoid reading a seldom-changed file from disk. It happens to be used
in ns_returnfile since that is the normal use case. On unix the
fastpath cache is
I would call that a security issue then. Leaking the wrong data to the wrong
connection is pretty serious.
Jade
Jade Rubick
Director of Development
Truist
120 Wall Street, 4th Floor
New York, NY USA
[EMAIL PROTECTED]
+1 503 285 4963
+1 707 671 1333 fax
The information contained in this
Jade,
It is a security issue mostly because the code sequence is incorrect.
(which also means that ns_returnfile should not be used for temp file
return)
The safe way to do this is to open the temp file, then immediately
unlink it! Then write to the fd.
BTW, this same bug exists in the
On Monday 03:38 PM 8/18/2008, Jeff Rogers wrote:
While I'd agree this is a bug in fastpath, the real problem is that
fastpath is being used at all in this case. The intent of fastpath is to
avoid reading a seldom-changed file from disk.
I'd agree that that's the intent, but the caching is
On Mon, 2008-08-18 at 15:38 -0700, Jeff Rogers wrote:
While I'd agree this is a bug in fastpath, the real problem is that
fastpath is being used at all in this case.
I don't think it is a bug in fastpath.
Think about the case where multiple logical files are actually the same
physical
On Mon, 2008-08-18 at 16:20 -0700, John Caruso wrote:
It's not difficult to come up with examples where it might happen,
BTW...say, a web service that returns the result of an operating system
command to a user.
The command is named ns_returnfile.
The expectation is that you are returning a
Consider this use case:
- You use git or another version control system to store for a bunch of
static html files you serve with Aolserver.
- You check out all of your static html files. Because they're all
checked out at the same time, many of them have identical timestamps.
Could
John Caruso wrote:
I'd agree that that's the intent, but the caching is hidden within
ns_returnfile and it's not clear at all from the user's perspective that
this alligator is lurking in the swamp. Using ns_returnfile in this way
may not be the best approach in any particular situation, but
Tom Jackson wrote:
Think about the case where multiple logical files are actually the same
physical file. Using the name would result in caching the same object
under different names. This is a much more likely situation than this so
called bug.
Huh, hard links - I sometimes forget about
On 19/08/2008, at 9:37 AM, Tom Jackson wrote:
On Mon, 2008-08-18 at 15:38 -0700, Jeff Rogers wrote:
While I'd agree this is a bug in fastpath, the real problem is that
fastpath is being used at all in this case.
I don't think it is a bug in fastpath.
fastpath is making assumptions about
On Mon, 2008-08-18 at 16:56 -0700, Jade Rubick wrote:
Consider this use case:
* You use git or another version control system to store for a
bunch of static html files you serve with Aolserver.
* You check out all of your static html files. Because they're
all
On Monday 04:56 PM 8/18/2008, Jade Rubick wrote:
Consider this use case:
* You use git or another version control system to store for a bunch
of static html files you serve with Aolserver.
* You check out all of your static html files. Because they're all
checked out at the same time,
On 19/08/2008, at 10:13 AM, Jeff Rogers wrote:
John Caruso wrote:
The system needs to be free to do some things to improve
performance with the understanding that the user needs to be aware
of those things or risk bad behaviour. I wouldn't call it an
unreasonable assumption that a file
On Tue, 2008-08-19 at 10:01 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 9:37 AM, Tom Jackson wrote:
On Mon, 2008-08-18 at 15:38 -0700, Jeff Rogers wrote:
While I'd agree this is a bug in fastpath, the real problem is that
fastpath is being used at all in this case.
I don't
russell muetzelfeldt wrote:
On 19/08/2008, at 9:37 AM, Tom Jackson wrote:
On Mon, 2008-08-18 at 15:38 -0700, Jeff Rogers wrote:
While I'd agree this is a bug in fastpath, the real problem is that
fastpath is being used at all in this case.
I don't think it is a bug in fastpath.
fastpath is
On Tue, 2008-08-19 at 10:39 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 10:13 AM, Jeff Rogers wrote:
John Caruso wrote:
The system needs to be free to do some things to improve
performance with the understanding that the user needs to be aware
of those things or risk bad
On 19/08/2008, at 10:14 AM, Tom Jackson wrote:
No, fastpath is making the exact same assumptions that any archive
program would make, which is to record certain attributes at the time
something is cached and then compare them with the same attributes
at a
Could the file name (just the name,
On 19/08/2008, at 10:44 AM, Tom Jackson wrote:
On Tue, 2008-08-19 at 10:01 +1000, russell muetzelfeldt wrote:
sure, the original use case that triggered this seems non-optimal,
and could be done in other ways that don't trigger the bug, but that
doesn't mean fastpath is behaving correctly...
Tom Jackson wrote:
No, because each file has a different inode. The bug requires that you
create and destroy one file and create another one within one second (so
they have the same timestamp) also required that the same inode is used
and that the file is the same exact size.
But beyond that,
On Monday 05:13 PM 8/18/2008, Jeff Rogers wrote:
Simply put, fastpath caching is inherently broken because it's not
possible to guarantee that the file in question really should be served
from cache (again, short of a cache-defeating checksum).
The same can be said about nearly any caching
On 19/08/2008, at 11:06 AM, John Caruso wrote:
That'd be an improvement over the current situation, but it's still
the case that AOLserver as currently shipped has a file cache
mechanism built into it which 1) may return incorrect data and 2)
is enabled by default. Given the risk, I'd say
On Tue, 2008-08-19 at 11:04 +1000, russell muetzelfeldt wrote:
snark aside, if I say ns_returnfile /tmp/foo-abcd but nsd sends the
contents of the now-deleted /tmp/bar-wxyz to the client then it's not
doing what I've explicitly asked, and it's a bug.
just because the correct (imo)
On 19/08/2008, at 10:56 AM, Tom Jackson wrote:
You want a transactional database but you are using a filesystem.
Grow up.
and
If your application wasn't the responsible party which violated the
expectation you state, I would agree (maybe).
please go and re-read this thread, and get
On Tue, 2008-08-19 at 11:37 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 10:56 AM, Tom Jackson wrote:
You want a transactional database but you are using a filesystem.
Grow up.
and
If your application wasn't the responsible party which violated the
expectation you
On 19/08/2008, at 11:59 AM, Tom Jackson wrote:
On Tue, 2008-08-19 at 11:37 +1000, russell muetzelfeldt wrote:
On 19/08/2008, at 10:56 AM, Tom Jackson wrote:
You want a transactional database but you are using a filesystem.
Grow up.
and
If your application wasn't the responsible party
On Monday 06:21 PM 8/18/2008, russell muetzelfeldt wrote:
On 19/08/2008, at 11:06 AM, John Caruso wrote:
That'd be an improvement over the current situation, but it's still
the case that AOLserver as currently shipped has a file cache
mechanism built into it which 1) may return incorrect data
91 matches
Mail list logo
