Stop the title from changing
Hello, I was wondering if there was a way to prevent the title changing... wget is currently nested in another script, and would probally confuse the user as to why the title says wget file location is it possible to retain its former title? (at the top of the script is title my title and i would like that to remain.) Thanks ~Cody
About Automated Unit Test for Wget
Hello, I want to ask about Unit test of Wget in the future. I want to ask about unit test. Now unit test of Wget is written only for following .c files. -- http.c init.c main.c res.c url.c utils.c (test.c) So as written in Wiki, new unit test suite is necessary. (ref. http://wget.addictivecode.org/FeatureSpecifications/Testing) In order to make new unit test suite, I think following work is necessary. 1) Select functions which can be tested in unit test. But How can I select them? is difficult. Basically the less dependency the function has, more easy to include in unit test, but about boundary line, I'm not sure. 2) In order to do above 1), How about Making a list of all functions in Wget? and maintain? The advantage of 2) is ... * make clear which function is included in Unit test * make clear which function is _not_ in Unit test * make easy to manage testing * make easy to devide testing work So once this list is done, it would become easier to maintain testing schedule, progress, etc.. And when Unit test suite is done, this list should be able to be generated automatically... and to do regression test, all we do is just run Unit test again :) 3) Contents of list I come up is following: * Wget version num * Filename * function name * Included in Unit Test or not * Simple Call graph of the function So let me ask your opinions. And is there any suggestion about unit test of Wget? (test tools, other preliminary work for unit test, how manage it ...) Thank you for your time. -- Yoshihiro TANAKA
Re: Stop the title from changing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 My Name? wrote: Hello, I was wondering if there was a way to prevent the title changing... wget is currently nested in another script, and would probally confuse the user as to why the title says wget file location is it possible to retain its former title? (at the top of the script is title my title and i would like that to remain.) Is this on Windows? Here's a copy-paste from an answer Christopher Lewis gave to someone asking a similar question: Depends on the scripting language. VBScript has the Shell.RUN command, first parm is the command WGET.EXE http://www.google.com; Second parm is window state. 6, I believe, hides the window. Dim oShell Set oShell = WScript.CreateObject (WSCript.shell) oShell.run WGET.EXE http://www.google.com;, 6, true Set oShell = Nothing - -- HTH, Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer, and GNU Wget Project Maintainer. http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH9dWA7M8hyUobTrERAgx6AJ9z79feQFB4UfE+anj97slZzkkmUACeJkrb zG+zXK0rchdSY6BrtQbjCJo= =2JMO -END PGP SIGNATURE-
Re: Stop the title from changing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Micah Cowan wrote: My Name? wrote: Hello, I was wondering if there was a way to prevent the title changing... wget is currently nested in another script, and would probally confuse the user as to why the title says wget file location is it possible to retain its former title? (at the top of the script is title my title and i would like that to remain.) Is this on Windows? Here's a copy-paste from an answer Christopher Lewis gave to someone asking a similar question: (Similar, but not the same: the question was how to hide Wget from the task bar; I guess yours is how to prevent it from changing some title somewhere; more context for your question would be nice.) - -Micah Depends on the scripting language. VBScript has the Shell.RUN command, first parm is the command WGET.EXE http://www.google.com; Second parm is window state. 6, I believe, hides the window. Dim oShell Set oShell = WScript.CreateObject (WSCript.shell) oShell.run WGET.EXE http://www.google.com;, 6, true Set oShell = Nothing -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH9dhC7M8hyUobTrERAh8yAJ0XaQ3l5uNjFWQV1yDQMpQw6YWwcQCdHt8q EdWrwq4l3CKo4CXABUazTcQ= =T/Cn -END PGP SIGNATURE-
Re: About Automated Unit Test for Wget
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yoshihiro Tanaka wrote: Hello, I want to ask about Unit test of Wget in the future. I want to ask about unit test. Now unit test of Wget is written only for following .c files. -- http.c init.c main.c res.c url.c utils.c (test.c) So as written in Wiki, new unit test suite is necessary. (ref. http://wget.addictivecode.org/FeatureSpecifications/Testing) Well, or expansions to the existing one. However, it's my desire (as expressed on that page) that the test code be separated from the .c files containing the code-to-test. This may mean making some functions that are currently static to be externally linked. IMO, if it's worth testing, it's probably better to have external linkage anyway. In order to make new unit test suite, I think following work is necessary. 1) Select functions which can be tested in unit test. But How can I select them? is difficult. Basically the less dependency the function has, more easy to include in unit test, but about boundary line, I'm not sure. This is precisely the problem, and one reason I've been thinking that this might not make an ideal topic for a GSoC proposal, unless you want to include refactoring existing functions like gethttp and http_loop into more logically discreet sets of functions. Essentially, to get better coverage of the code that needs it the most, that code will need to be rewritten. I believe this can be an iterative process (find one function to factor out, write a unit test for it, make it work...). 2) In order to do above 1), How about Making a list of all functions in Wget? and maintain? The advantage of 2) is ... * make clear which function is included in Unit test * make clear which function is _not_ in Unit test * make easy to manage testing * make easy to devide testing work Hm... I'm not sure that the benefits are worth the effort. If we _really_ wanted this, I'd propose that we use a naming convention (or processed comment, etc) for the unit test functions so that the list of functions that are covered can be determined automatically by a program; the ones that aren't covered would be any functions remaining. However, I personally wouldn't find this useful. I don't intend that every function in existence has to have a unit test covering it. Some functions will have already been tested through the exercise of higher-level calling functions (in which case they should probably have internal linkage); others may have been tested through the exercise of the functions it calls. What I'm most keenly interested in, is the ability to verify the logic of how follow/don't-follow is decided (that actually may not be too hard to write tests against now), how Wget handles various protocol-level situations, how it chooses the filename and deals with the local filesystem, etc. I will be very, _very_ happy when everything that's in http_loop and gethttp is verified by unit tests. But a lot of getting to where we can test that may mean abstracting out things like the Berkeley Sockets API and filesystem interactions, so that we can drop in fake replacements for testing. (test tools, other preliminary work for unit test, how manage it ...) There is an incredibly basic test framework, completely defined in src/test.h. See src/test.c for how it is being used. I'm familiar with a framework called (simply) Check, which might be worth considering. It forks a new process for each test, which isolates it from interfering with the other tests, and also provides a clean way to handle things like segmentation violations or aborts. However, it's intended for Unix, and probably doesn't compile on other systems. http://check.sourceforge.net/ - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer, and GNU Wget Project Maintainer. http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH9d0Q7M8hyUobTrERApdnAJ905n4j0oglUHekP6MJaE4dBCEw+QCeL4RE 0lnwnZgHQjSEom4f1MfiviM= =UejZ -END PGP SIGNATURE-
Re: [GSoC] Testing for Wget
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 [ I've Cc'd the list to invite comment on the idea of overriding Berkeley Sockets and I/O functions for testing. ] Yoshihiro Tanaka wrote: Actually, the .px test files already spawn HTTP and/or FTP servers to do their testing. Part of the problem with doing testing with a real local server, though, is that there are a lot of cases I want to handle that involve things like multiple hostnames/servers. With spawning _real_ servers, the only name we can really count on is localhost. Ideally, I'd like to completely encapsulate the Berkeley Sockets API, so we can simply swap in a fake one for testing purposes. Then we could do all the protocol-level tests, without having to involve real servers and their shortcomings. Sorry I'm not sure about it, if we use fake one, how do you test without connecting HTTP/FTP server ? Here's the problem with using real servers for core testing (I do think we should still use real tests against servers, functionality tests to ensure that Wget is basically functional: I just don't think we should rely heavily on it). With a real remote server, I can't assume that a test failure means that the test actually failed: it might have been a network failure. With a real locally-spawned server, I can't test arbitrary hostnames. So I can't do testing of issues related to --span-hosts, for instance, or ensure that Wget only sends authorization tokens to the correct server, and no others, or that Wget never puts the user:password@ part of URLs in the Referer header. This can be done with a remote server, but requires that I set up multiple servers (or at the least, multiple names for a given server); and as already mentioned I don't like relying on the availability of remote servers for Wget to run tests. We _could_ spawn a local proxy server, and have it pretend to Wget that it's forwarding requests to different servers, when it's actually only pretending; however, ideally we should be explicitly testing these things both with and without a proxy: some behaviors may change depending on that (such as the recently reported bug against Wget's HTTP authentication). With real servers, it's hard to test how Wget handles rare error codes from some of the Berkeley Sockets functions or I/O functions. Overriding socket I/O operations also lets us write tests against cases when there's a specific number of bytes available for reading, which actually does correspond to a bug that was fixed not too long ago. It also lets us dictate exact sequences of server behaviors across multiple hosts (I can't think of any uses for that for current Wget, though there probably are some; but there will be plenty once Wget supports multiple simultaneous connections). Emulating the networking layer also means that we can write tests for how Wget handles buggy or quirky servers (especially useful for FTP), without actually having to have those real servers set up somewhere for testing (that couldn't be done with locally-spawned servers, either). For example, file retrieval is not possible, I guess ? File retrieval should still be possible, by having the fake, emulated server produce content we specify. Emulating interfaces to the filesystem here might also be advantageous, so we can emulate systems that have varying support for large files, without actually having to require that test-runners have space for them; or to simulate rare I/O error conditions. what do you mean by protocol-level test ? I mean, checking how Wget handles certain protocol-level situations. Buggy headers, missing headers, disconnects at specific points in the conversation, etc. . If, in order to support emulation of the networking API, we abstract it out so we can swap out real sockets with our testing layer, we also gain the advantage that we could later modify it to be swapped out for other things as well: for instance, true SOCKS support. This sort of abstraction is something I wanted eventually anyway. Performing the same abstractions for filesystem emulation would also provide a point for retrieve-to-tarball (or .mht) functionality. In fact, the abstraction itself might make a worthy GSoC proposal - though it's probably not enough to fill a summer. Perhaps using the abstraction to then emulate an HTTP server for a few tests would be a good way to fill out the rest. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer, and GNU Wget Project Maintainer. http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH9eda7M8hyUobTrERAjOKAJ9+mifWOi3Gc7LWdVIeakzt2cNcSACfSQFk /dff3ajYY7sCLyM6SfglR9M= =SRcY -END PGP SIGNATURE-
Re: wget 1.11.1 make test fails
Alain Guibert [EMAIL PROTECTED] writes: On Wednesday, April 2, 2008 at 23:09:52 +0200, Hrvoje Niksic wrote: Micah Cowan [EMAIL PROTECTED] writes: It's hard for me to imagine an fnmatch that ignores FNM_PATHNAME The libc 5.4.33 fnmatch() supports FNM_PATHNAME, and there is code apparently intending to return FNM_NOMATCH on a slash. But this code seems to be rather broken. Or it could be that you're picking up a different fnmatch.h that sets up a different value for FNM_PATHNAME. Do you have more than one fnmatch.h installed on your system?
Re: wget 1.11.1 make test fails
Alain Guibert [EMAIL PROTECTED] writes:
Maybe you could put a breakpoint in fnmatch and see what goes wrong?
The for loop intended to eat several characters from the string also
advances the pattern pointer. This one reaches the end of the pattern,
and points to a NUL. It is not a '*' anymore, so the loop exits
prematurely. Just below, a test for NUL returns 0.
Thanks for the analysis. Looking at the current fnmatch code in
gnulib, it seems that the fix is to change that NUL test to something
like:
if (c == '\0')
{
/* The wildcard(s) is/are the last element of the pattern.
If the name is a file name and contains another slash
this means it cannot match. */
int result = (flags FNM_PATHNAME) == 0 ? 0 : FNM_NOMATCH;
if (flags FNM_PATHNAME)
{
if (!strchr (n, '/'))
result = 0;
}
return result;
}
But I'm not at all sure that it covers all the needed cases. Maybe we
should simply switch to gnulib-provided fnmatch? Unfortunately that
one is quite complex and quite hard for the '**' extension Micah
envisions. There might be other fnmatch implementations out there in
GNU which are debugged but still simpler than the gnulib/glibc one.
It's kind of ironic that while the various system fnmatches were
considered broken, the one Wget was using (for many years
unconditionally!) was also broken.
Re: Stop the title from changing
Micah Cowan [EMAIL PROTECTED] writes: My Name? wrote: Hello, I was wondering if there was a way to prevent the title changing... wget is currently nested in another script, and would probally confuse the user as to why the title says wget file location is it possible to retain its former title? (at the top of the script is title my title and i would like that to remain.) Is this on Windows? Here's a copy-paste from an answer Christopher Lewis gave to someone asking a similar question: [...] Christopher describes how to hide the console window opened by Wget, while the poster would like to prevent Wget from changing the title of the existing console. Maybe we should make the title-changing behavior optional. After all, we don't do anything of the sort on Unix, nor (IMHO) should we.
Re: Problem with wget 1.11. Please help
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Amit Patel wrote: Ya. I have checked it properly. It checks certificates. If i don't specify /etc/ca-bundle.crt with my working version of 1.10.2, it provides Self-signed certificate encountered error and fails. Er, I'm not sure, but I think I would expect that to be a problem, regardless of whether /etc/ca-bundle.crt. Could you please verify whether a stock wget-1.10.2, configured and installed from our source tarballs (not RedHat's), available at ftp.gnu.org/gnu/wget/wget-1.10.2.tar.gz ? - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer, and GNU Wget Project Maintainer. http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH9isf7M8hyUobTrERApgfAJ4801HnL9/5W4QdcLMF87KNnutHJgCeP2bm 8rU8dsRxa4s76i/a3pnaeZ4= =MLQZ -END PGP SIGNATURE-
Re: wget 1.11.1 make test fails
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hrvoje Niksic wrote:
Alain Guibert [EMAIL PROTECTED] writes:
Maybe you could put a breakpoint in fnmatch and see what goes wrong?
The for loop intended to eat several characters from the string also
advances the pattern pointer. This one reaches the end of the pattern,
and points to a NUL. It is not a '*' anymore, so the loop exits
prematurely. Just below, a test for NUL returns 0.
Thanks for the analysis. Looking at the current fnmatch code in
gnulib, it seems that the fix is to change that NUL test to something
like:
if (c == '\0')
{
/* The wildcard(s) is/are the last element of the pattern.
If the name is a file name and contains another slash
this means it cannot match. */
int result = (flags FNM_PATHNAME) == 0 ? 0 : FNM_NOMATCH;
if (flags FNM_PATHNAME)
{
if (!strchr (n, '/'))
result = 0;
}
return result;
}
But I'm not at all sure that it covers all the needed cases.
I'm thinking not: the loop still shouldn't be incrementing n, since that
forces each additional * to match at least one character, doesn't it?
Gnulib's version seems to handle that better.
Maybe we
should simply switch to gnulib-provided fnmatch? Unfortunately that
one is quite complex and quite hard for the '**' extension Micah
envisions. There might be other fnmatch implementations out there in
GNU which are debugged but still simpler than the gnulib/glibc one.
Maybe. I'm not sure ** would be too hard to add to gnulib's fnmatch,
just have to toggle with the FNM_FILE_NAME tests within the '*' case, if
we see an immediate second '*'. But maybe ** as part of a *?**? sequence
is more complex. I don't think so, though.
The main thing is that we need it to support the invalid sequence stuff.
Hm; I'm not sure we'll ever want fnmatch() to be locale-aware, though.
User-specified match patterns should interpret characters based on the
locale; but the source strings may be in different encodings altogether.
If we solve this by transcoding to the current locale, we may find that
the user's locale doesn't support all of the characters that the
original string's encoding does. Probably we'll need to transcode both
to Unicode before comparison.
In the meantime, though, I think we want a simple byte-by-byte match.
Perhaps it's best to (a) use our custom matcher, ignoring the system's
(so we don't get locale specialness), and (b) fix it, providing as
thorough test coverage as possible.
- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer,
and GNU Wget Project Maintainer.
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFH9jWi7M8hyUobTrERAglwAKCDnpnDjr44Ovgh/oBuzkM4mu/gKACeNnN8
arvFSrCEBatNeO29fzHxuU4=
=QDMp
-END PGP SIGNATURE-
Re: wget 1.11.1 make test fails
On Thursday, April 3, 2008 at 22:37:41 +0200, Hrvoje Niksic wrote: Or it could be that you're picking up a different fnmatch.h that sets up a different value for FNM_PATHNAME. Do you have more than one fnmatch.h installed on your system? I have only /usr/include/fnmatch.h installed, identical to the file in the libc-5.4.33 tarball, and defining the same values as wget's src/sysdep.h (even comments are identical). Just my fnmatch.h defines two more flags, FNM_LEADING_DIR=8 and FNM_CASEFOLD=16, and defines an FNM_FILE_NAME alias (commented as Preferred GNU name) to FNM_PATHNAME=1 (the libc code uses only this alias). Anyway I had noticed your comment about incompatible headers, and double-checked your little test program also with explicit value 1: same results. BTW everybody should be able to reproduce the make test failure, on any system, just by #undefining SYSTEM_FNMATCH in src/sysdep.h Alain.
Re: wget 1.11.1 make test fails
On Thursday, April 3, 2008 at 9:14:52 -0700, Micah Cowan wrote: Are you certain you rebuilt cmpt.o? This seems pretty unlikely, to me. Certain: make test after touching src/sysdep.h rebuilds both cmpt.o, the normal in src/ and the one in tests/. And both those cmpt.o become 784 bytes bigger without SYSTEM_FNMATCH. Alain.
