Re: mod_ftp, status and progress?
On Wed, 2 May 2007, Jim Jagielski wrote:
In fact, to be honest, it would be easier still to just
update ftp_direntry_get() to use apr_fnmatch(), since we
always want to support globing. ftp_direntry_get already
does most of what makes apr_match_glob attractive in
the 1st place.
Should have a patch to commit later on tomorrow,
after some more tests :)
I suspect that you're fixing the large file issues while you're at it?
Another thing I noticed when we started to look at mod_ftp (looking at
strace/truss-output trying to figure out why things didn't work) was
that it stats all entries in a directory twice, first explicitly and
then via the subreq. Wouldn't the subreq be enough? It's no biggie for
now, but it would be nice to get rid of unneccessary stats as a bonus
;)
/Nikke - eager to give it a spin :)
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
You wanted to make it law. Make it a good one. - Picard
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_ftp, status and progress?
Niklas Edmundsson wrote: On Wed, 2 May 2007, Jim Jagielski wrote: In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place. Should have a patch to commit later on tomorrow, after some more tests :) I suspect that you're fixing the large file issues while you're at it? Another thing I noticed when we started to look at mod_ftp (looking at strace/truss-output trying to figure out why things didn't work) was that it stats all entries in a directory twice, first explicitly and then via the subreq. Wouldn't the subreq be enough? It's no biggie for now, but it would be nice to get rid of unneccessary stats as a bonus ;) This is a separate issue; we need to refactor out 90% of the subrequests and treat these at top level requests. I discovered while trying to accomodate named virtual hosts (the hack to let [EMAIL PROTECTED] resolve to the host.com vhost) it's simply not worth hacking one without the other. WIP. Bill
Re: mod_ftp, status and progress?
On Thu, 3 May 2007, William A. Rowe, Jr. wrote:
Another thing I noticed when we started to look at mod_ftp (looking at
strace/truss-output trying to figure out why things didn't work) was
that it stats all entries in a directory twice, first explicitly and
then via the subreq. Wouldn't the subreq be enough? It's no biggie for
now, but it would be nice to get rid of unneccessary stats as a bonus ;)
This is a separate issue; we need to refactor out 90% of the subrequests
and treat these at top level requests.
OK. I was under the impression that those subrequests were made to
filter out stuff you don't have access to from the directory listings,
but I stand corrected.
I discovered while trying to accomodate named virtual hosts (the hack to
let [EMAIL PROTECTED] resolve to the host.com vhost) it's simply not worth
hacking one without the other.
Ah. Am I right in guessing that making it play well with mod_cache
would come more or less for free after the request-refactoring is
done?
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
He who laughs last is probably your boss!
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_ftp, status and progress?
Niklas Edmundsson wrote: This is a separate issue; we need to refactor out 90% of the subrequests and treat these at top level requests. OK. I was under the impression that those subrequests were made to filter out stuff you don't have access to from the directory listings, but I stand corrected. Oh - they still need to fail - and we still need to (for example) validate both the RNFR and RNTO target paths. But for trivial requests such as RETR the double-test is silly. I discovered while trying to accomodate named virtual hosts (the hack to let [EMAIL PROTECTED] resolve to the host.com vhost) it's simply not worth hacking one without the other. Ah. Am I right in guessing that making it play well with mod_cache would come more or less for free after the request-refactoring is done? :)
Re: mod_ftp, status and progress?
On May 3, 2007, at 4:37 AM, William A. Rowe, Jr. wrote: Niklas Edmundsson wrote: On Wed, 2 May 2007, Jim Jagielski wrote: In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place. Should have a patch to commit later on tomorrow, after some more tests :) I suspect that you're fixing the large file issues while you're at it? Another thing I noticed when we started to look at mod_ftp (looking at strace/truss-output trying to figure out why things didn't work) was that it stats all entries in a directory twice, first explicitly and then via the subreq. Wouldn't the subreq be enough? It's no biggie for now, but it would be nice to get rid of unneccessary stats as a bonus ;) This is a separate issue; we need to refactor out 90% of the subrequests and treat these at top level requests. Agreed... Some of them, of course, we can't get rid of, and the current design was basically for ease of development.
Re: mod_ftp, status and progress?
On Apr 28, 2007, at 1:40 PM, Jim Jagielski wrote: In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place. Should have a patch to commit later on tomorrow, after some more tests :)
Re: mod_ftp, status and progress?
On Apr 27, 2007, at 5:18 PM, Niklas Edmundsson wrote: On Fri, 27 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... Why? Really, it's no big deal to ensure it runs on both. I'm not against keeping compatibility. However I feel that the right way to do it would be to design stuff for current httpd and then add glue for the backward compat stuff (and not doing it the #ifdef-mess-way). So, going for regexes just because apr_match_glob() doesn't exist in 2.0.x seems a bit sub-optimal... The idea was either to redo ftp_glob via regexes or simply use apr_fnmatch... The emulate was to allow for extensions to simple globing, which could make the FTP module more useful, but, of course, client support would be lacking. But really, again, I repeat that it's no big deal to ensure it runs on both. Even if we simply just create ftp_match_glob() from apr_match_glob(). And, fwiw, mod_ftp was originally written *for* 2.0.x, not 2.2.x. So most of the work had been in getting it to work with the current httpd. But if anyone does anything to prevent mod_ftp from building on 2.0.x, expect a big -1.
Re: mod_ftp, status and progress?
On Apr 28, 2007, at 1:34 PM, Jim Jagielski wrote: On Apr 27, 2007, at 5:18 PM, Niklas Edmundsson wrote: On Fri, 27 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... Why? Really, it's no big deal to ensure it runs on both. I'm not against keeping compatibility. However I feel that the right way to do it would be to design stuff for current httpd and then add glue for the backward compat stuff (and not doing it the #ifdef-mess-way). So, going for regexes just because apr_match_glob() doesn't exist in 2.0.x seems a bit sub-optimal... The idea was either to redo ftp_glob via regexes or simply use apr_fnmatch... The emulate was to allow for extensions to simple globing, which could make the FTP module more useful, but, of course, client support would be lacking. But really, again, I repeat that it's no big deal to ensure it runs on both. Even if we simply just create ftp_match_glob() from apr_match_glob(). And, fwiw, mod_ftp was originally written *for* 2.0.x, not 2.2.x. So most of the work had been in getting it to work with the current httpd. But if anyone does anything to prevent mod_ftp from building on 2.0.x, expect a big -1. In fact, to be honest, it would be easier still to just update ftp_direntry_get() to use apr_fnmatch(), since we always want to support globing. ftp_direntry_get already does most of what makes apr_match_glob attractive in the 1st place.
Re: mod_ftp, status and progress?
On Apr 26, 2007, at 5:39 PM, Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... Why? Really, it's no big deal to ensure it runs on both.
Re: mod_ftp, status and progress?
On Fri, 27 Apr 2007, Jim Jagielski wrote:
I'm actually looking at removing the whole glob stuff
and emulating it as regexes...
Wouldn't apr_match_glob() be a better starting point? I don't really see
the point of going via regexes...
I was thinking for 2.0.x compatibility...
Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of
people still running 1.3 and 2.0, but that doesn't mean that we have to
make it run on all of them...
Why? Really, it's no big deal to ensure it runs on both.
I'm not against keeping compatibility. However I feel that the right
way to do it would be to design stuff for current httpd and then add
glue for the backward compat stuff (and not doing it the
#ifdef-mess-way).
So, going for regexes just because apr_match_glob() doesn't exist in
2.0.x seems a bit sub-optimal...
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
You're the security chief-shouldn't you be out securing something?
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_ftp, status and progress?
On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; I'm actually looking at removing the whole glob stuff and emulating it as regexes...
Re: mod_ftp, status and progress?
On Thu, 26 Apr 2007, Jim Jagielski wrote:
On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote:
Hi,
the current code fails to build for Win32 target.
This is because ftp_glob.c seems not APR-ised yet;
I'm actually looking at removing the whole glob stuff
and emulating it as regexes...
Wouldn't apr_match_glob() be a better starting point? I don't really
see the point of going via regexes...
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
Luckily, I'm out of hairs to split!
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_ftp, status and progress?
On Apr 26, 2007, at 8:55 AM, Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility...
Re: mod_ftp, status and progress?
On Apr 26, 2007, at 9:22 AM, Jim Jagielski wrote: On Apr 26, 2007, at 8:55 AM, Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: On Apr 18, 2007, at 1:22 PM, Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... which, of course, implies apr_fnmatch and not apr_match_glob...
Re: mod_ftp, status and progress?
On Thu, 26 Apr 2007, Jim Jagielski wrote:
I'm actually looking at removing the whole glob stuff
and emulating it as regexes...
Wouldn't apr_match_glob() be a better starting point? I don't really see
the point of going via regexes...
I was thinking for 2.0.x compatibility...
Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot
of people still running 1.3 and 2.0, but that doesn't mean that we
have to make it run on all of them...
I'm all for focusing on getting it usable for 2.2+, and if people
really want the httpd-tree mod_ftp that bad they can see it as yet
another good reason to upgrade. There's a lot of work that needs to be
done in order to have mod_ftp usable, and making the code more complex
in order to support the previous stable httpd version doesn't really
sound that appealing.
Not that I'm against backward compatibility, but I'd prefer seeing a
clean design for the current/future httpd version and the compat stuff
handled by wrapper functions stashed in a mod_ftp_20compat.c or
something like that.
In any case, as long as the code is readable and works. The current
mod_ftp globbing stuff is simply a mess.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
Reality is for people who can't handle Star Trek.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_ftp, status and progress?
Hi, Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... I'm all for focusing on getting it usable for 2.2+, and if people it compile and runs fine with 2.2.x at least on Linux and Win32, so there's nothing to focus on. It does however not compile with 2.3.x trunk due to removal of ap_requires()... Guenter.
Re: mod_ftp, status and progress?
On 4/26/2007 at 4:16 PM, in message [EMAIL PROTECTED], Guenter Knauf [EMAIL PROTECTED] wrote: Hi, Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... I'm all for focusing on getting it usable for 2.2+, and if people it compile and runs fine with 2.2.x at least on Linux and Win32, so there's nothing to focus on. It does however not compile with 2.3.x trunk due to removal of ap_requires()... Guenter. Sounds like it need to be ported to use the new provider based authz. Brad
Re: mod_ftp, status and progress?
Niklas Edmundsson wrote: On Thu, 26 Apr 2007, Jim Jagielski wrote: I'm actually looking at removing the whole glob stuff and emulating it as regexes... Wouldn't apr_match_glob() be a better starting point? I don't really see the point of going via regexes... I was thinking for 2.0.x compatibility... Wouldn't it be better to focus on 2.2.x and onwards? OK, there's a lot of people still running 1.3 and 2.0, but that doesn't mean that we have to make it run on all of them... There's a well established group of 2.0 users out there (pre-ASF). We would all love to see them migrate entirely to 2.2 but that never happens overnight. I think the best solution is to either add apr_match_glob to apr 0.9 (no rule against it) or to clone it as match_glob.c which would devolve to nothing. Once this is merged in-tree (if that happens) - all of the version-specific stubs can go away. But I don't think we are quite at that point. Bill
Re: mod_ftp, status and progress?
Hi Bill, You built from modules/ftp/mod_ftp.dsp, or built with modules/ftp/Makefile.apxs? Apologies! Was entirely my fault; I did self add the ftp_glob.c to mod_ftp.dsp cause I thought that was forgotten to add; sorry. Compiles now fine when I use original mod_ftp.dsp, at least for 2.2; for compiling with trunk seems we need a hack to overcome the missing ap_requires(r)... For modules/ftp/mod_ftp.dsp, the project should wipe out the configuration header include/ftp_config.h with 'empty' as every detectable feature should not be detected (they are all unix-ish). yep, works. I'd love to get those detections into httpd proper rather than go heavy with a ftp-specific autoconf; they are (so far) for HAVE_FCHMOD, HAVE_GLOB and HAVE_ARPA_FTP_H (for practical purposes not really worthwhile). +1. greets, Guenter.
Re: mod_ftp, status and progress?
Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; g_opendir() uses the DIR struct which is not available on Win32 AFAICT. Guenter.
Re: mod_ftp, status and progress?
Guenter Knauf wrote: Hi, the current code fails to build for Win32 target. This is because ftp_glob.c seems not APR-ised yet; g_opendir() uses the DIR struct which is not available on Win32 AFAICT. You built from modules/ftp/mod_ftp.dsp, or built with modules/ftp/Makefile.apxs? For modules/ftp/mod_ftp.dsp, the project should wipe out the configuration header include/ftp_config.h with 'empty' as every detectable feature should not be detected (they are all unix-ish). I'd love to get those detections into httpd proper rather than go heavy with a ftp-specific autoconf; they are (so far) for HAVE_FCHMOD, HAVE_GLOB and HAVE_ARPA_FTP_H (for practical purposes not really worthwhile). Bill
Re: mod_ftp, status and progress?
Hi, is it possible that with your last commit changes you forgot to update mod_ftp.h too? I tried to compile for NetWare, and needed this in order to get it compiled: --- mod_ftp.h.orig Tue Mar 27 13:27:22 2007 +++ mod_ftp.h Tue Mar 27 13:49:04 2007 @@ -250,6 +250,8 @@ const char *readme; const char *path; int readme_isfile; +apr_fileperms_t fileperms; +apr_fileperms_t dirperms; }; /* also want to ask if it is ok when I add NetWare makefiles? Guenter.
Re: mod_ftp, status and progress?
Guenter Knauf wrote: Hi, is it possible that with your last commit changes you forgot to update mod_ftp.h too? Feh - yup. My bad - thanks! also want to ask if it is ok when I add NetWare makefiles? Be our guest :) Keep it mind it's ment to be built against installed httpd, or built in tree (within modules/ftp/) as desired.
Re: mod_ftp, status and progress?
Hi, also want to ask if it is ok when I add NetWare makefiles? Be our guest :) Keep it mind it's ment to be built against installed httpd, or built in tree (within modules/ftp/) as desired. sorry, but for now only inside tree; anyway all NetWare compile is sort of cross-compile, and way more complex than anything other I came over. No paths are really fix, nor does our installed Apache ship with the headers / libs by default; and the NetWare build system isnt really prepared for such. Although we can create an 'Apache SDK' this lacks currently of the needed files which all makefiles include, and so a makefile which works with that looks completely different. I will though look if I can fix that over the time with the SDK, and then update the makefiles. Guenter.
Re: mod_ftp, status and progress?
On Fri, 23 Mar 2007, William A. Rowe, Jr. wrote:
* Play well with mod_cache, if a file has been requested with HTTP a
FTP request should reuse the cached copy. Last time I checked
mod_ftp only did subrequests which mod_cache didn't act on.
In terms of using 'top level' requests in lieu of subrequests, it's
not low hanging fruit but definitely worth the refactoring. Doing this
against httpd trunk/ will show up the API's that httpd is missing for
providers of resource-based servers such as ftp.
OK. This will need more investigation then, the easiest solution
would seem to be to get the subrequest interaction with mod_cache
right. Bright insights are welcome.
* Both passive and active mode MUST work. I think there was some
issues causing it to always use 0.0.0.0 in passive mode last time I
checked.
fixed afaik, unless you are on win2000 and didn't DisableWin32AcceptEx
in 2.2.4 release (apr 1.2.8). The fix is in trunk and will percolate
out as apr 1.2.9 (or later) with 2.2.5.
Nice.
* Large file support MUST work (we serve DVD images). Last time I
checked there was a whole slew of LFS issues with the mod_ftp
globbing code which was simply broken since it didn't use the information
gathered by configure and didn't use APR - it should be ripped out
and replaced with APR stuff instead IMHO. It makes no sense to keep
that mess just o keep httpd 2.0 compatibility...
Patches welcome, yes this needs some refactoring.
Any thoughts on how to do this? My mind tend to be focused on what
needs to work for anonftp in this regard, and that means naively
listing a directory without any thoughts on file permissions and such.
If a file/directory is within the anonftp tree it's OK to include it
in the listing.
However, if there's some special care that needs to be done when
supporting file uploads (only listing directories which the auth:ed
user have access to and other special stuff) I will probably miss this
unless someone clued does the high level design.
* IPv6 MUST work. I think this is being addressed.
I'd fixed the traditional interfaces (PORT/PASV) but we need to hack
together EPRT/EPSV support, yet.
OK. This shouldn't be too hard, given that EPRT/EPSV doesn't differ
too much from PORT/PASV.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | [EMAIL PROTECTED]
---
It is always darkest before it goes totally black.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_ftp, status and progress?
Niklas Edmundsson wrote: * Play well with mod_cache, if a file has been requested with HTTP a FTP request should reuse the cached copy. Last time I checked mod_ftp only did subrequests which mod_cache didn't act on. In terms of using 'top level' requests in lieu of subrequests, it's not low hanging fruit but definitely worth the refactoring. Doing this against httpd trunk/ will show up the API's that httpd is missing for providers of resource-based servers such as ftp. * Only anonyomous read-only-access is required. I think this is working today. it works * Download-related items like file listings, continuation, etc MUST work. they work afaik, the Server: 8L hack makes things more interoperable. * Both passive and active mode MUST work. I think there was some issues causing it to always use 0.0.0.0 in passive mode last time I checked. fixed afaik, unless you are on win2000 and didn't DisableWin32AcceptEx in 2.2.4 release (apr 1.2.8). The fix is in trunk and will percolate out as apr 1.2.9 (or later) with 2.2.5. * Large file support MUST work (we serve DVD images). Last time I checked there was a whole slew of LFS issues with the mod_ftp globbing code which was simply broken since it didn't use the information gathered by configure and didn't use APR - it should be ripped out and replaced with APR stuff instead IMHO. It makes no sense to keep that mess just o keep httpd 2.0 compatibility... Patches welcome, yes this needs some refactoring. * IPv6 MUST work. I think this is being addressed. I'd fixed the traditional interfaces (PORT/PASV) but we need to hack together EPRT/EPSV support, yet.
Re: mod_ftp, status and progress?
On 03/23/2007 04:59 PM, Niklas Edmundsson wrote: * Play well with mod_cache, if a file has been requested with HTTP a FTP request should reuse the cached copy. Last time I checked mod_ftp only did subrequests which mod_cache didn't act on. Of mod_cache works with subrequests as well. It maybe the case that some types of subrequests do not run the quick handler hook (see comments in ap_run_sub_req in request.c). In this case of course mod_cache does not work. Regards Rüdiger
