It would be a good idea, in the general case, for us all to learn how to properly override config parameters in the code, so that a server block or URL block definition doesn't override an internal override
Maybe I am missing something. I am not aware of a way that allows us to override blocks definitions through the Configuration classes. Can you please point it out to me? Sorry.
doing an initial dig, then it will only ever take effect when doing an update (or incremental) dig with persistent connections turned on.
No no ... wait. I have never talked about turning off head before get. Let me try and give an explanation about this parameter.
I remember we issued the 'head_before_get' attribute because of this: when requesting a non-parsable document we generally had 3 options:
1) persistent connections on:
a) head before get on: we perform a HEAD call and notice that the document's content-type is not what we want so we simply avoid doing the GET call
b) head before get off: we perform a GET call but in this case we must receive all the content returned by the server, otherwise we have to close the connection - that's not what we want in general.
2) persistent connections off: we perform a GET call and if the document is not what we want we simply close the connection (we anticipate it).
IMHO the 'head_before_get' could make the difference in some cases with persistent connections on and only the webmaster can see the difference in performances between turning it on or off. If we don't have many multimedia files we could simply turn it off (avoiding a 'double' call), whereas if we have big files to be downloaded (especially from the Internet) this attribute could make the difference, as a pre-emptive HEAD call would let us know about the type of document we are being requested and eventually save us a big download.
not as versed in HTTP/1.1 as you are. It seems to me that htdig should always be doing a HEAD before a GET when doing incremental digs through persistent connections.
Yes. And not only there. Even when performing an initial dig, if the user wants it, we must enable it.
I can think of this possible solution. The scenario above is still valid (IMHO) for the initial dig case; I would modify it for the incremental dig as mentioned yesterday, as follows:
if "persistent_connections" (on a server basis) is set to on:
enable persistent connections
else
disable themif incremental or ("head_before_get" and "persistent_connections" are both set to on) - I have to modify yesterday's patch a bit
enable head before get
else
disable head before get
In this way, for initial dig the user can choose whether activate persistent connections and head before get, whether for incremental digs the users' settings get overridden.
For me this sounds good. There can be issues regarding the way of doing it; I thought that adding some object variables in the Retriever and Document class would be fine. Unless there is a way of overriding specific settings through the Configuration classes.
Please let me know.
By the way, Gabriele, good call on the Accept-Encoding header. It's a simple, elegant fix to a troublesome bug. You're right that adding support for gzip encoding is a feature request, and not a bug fix, and should be done after the upcoming release (not before). Good work.
Thank you. However, by following Neal's directives, could someone of you try it and let me know so I can close the bug?
Ciao ciao -Gabriele
-- Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 3J7 (Canada)
------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. SourceForge.net hosts over 70,000 Open Source Projects. See the people who have HELPED US provide better services: Click here: http://sourceforge.net/supporters.php _______________________________________________ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
--
Gabriele Bartolini: Web Programmer, ht://Dig & IWA/HWG Member, ht://Check maintainer
Current Location: Melbourne, Victoria, Australia
[EMAIL PROTECTED] | http://www.prato.linux.it/~gbartolini | ICQ#129221447
> "Leave every hope, ye who enter!", Dante Alighieri, Divine Comedy, The Inferno
------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. SourceForge.net hosts over 70,000 Open Source Projects. See the people who have HELPED US provide better services: Click here: http://sourceforge.net/supporters.php _______________________________________________ ht://Dig Developer mailing list: [EMAIL PROTECTED] List information (subscribe/unsubscribe, etc.) https://lists.sourceforge.net/lists/listinfo/htdig-dev
