Re: HTTP::Headers - remove_header not removing header?
Gisle, First, let me say thanks for some great modules! There is actually no easy way to do that. They are inserted by the protocol module when it feels they are appropriate. Can you explain why you need them to to go away? I need them to go away because I want Perl to look just like an application that normally requests the data I'm looking for. The app doesn't write Connection or TA headers, so I don't want Perl to write them either. After all, why bother spoofing a user-agent string if you proceed to send some other headers that immediately fingerprint you as a perl script? The Connection: close will go away of you pass the keep_alive option to the LWP::UserAgent. If you insist hard on the TE header to be disabled you you try something like: push(@LWP::Protocol::http::EXTRA_SOCK_OPTS, SendTE = 0); before you send your first request. Thanks! That succeeded in nuking the TA header. But, I'm still getting the Connection header when I use: my $ua = LWP::UserAgent-new(keep_alive = 1); I've simply managed to change the Connection: close into Connection: Keep-Alive. Any advice, or am I stuck? Thanks, Phil ps. I also looked at Net::HTTP as a lower-level option for what I'm doing, but I need the proxy feature from LWP::UserAgent.
Re: HTTP::Headers - remove_header not removing header?
Phil Thompson [EMAIL PROTECTED] writes: There is actually no easy way to do that. They are inserted by the protocol module when it feels they are appropriate. Can you explain why you need them to to go away? I need them to go away because I want Perl to look just like an application that normally requests the data I'm looking for. The app doesn't write Connection or TA headers, so I don't want Perl to write them either. After all, why bother spoofing a user-agent string if you proceed to send some other headers that immediately fingerprint you as a perl script? So you are not on friendly terms with the server you are trying to access then. Oh well,... The Connection: close will go away of you pass the keep_alive option to the LWP::UserAgent. If you insist hard on the TE header to be disabled you you try something like: push(@LWP::Protocol::http::EXTRA_SOCK_OPTS, SendTE = 0); before you send your first request. Thanks! That succeeded in nuking the TA header. But, I'm still getting the Connection header when I use: my $ua = LWP::UserAgent-new(keep_alive = 1); I've simply managed to change the Connection: close into Connection: Keep-Alive. Any advice, or am I stuck? Seems like you can try to push (PeerHTTPVersion = 1.1) onto the @EXTRA_SOCK_OPTS to supress this one as well. ps. I also looked at Net::HTTP as a lower-level option for what I'm doing, but I need the proxy feature from LWP::UserAgent. It's no problem taking to a proxy with Net::HTTP, but it is certainly easier to use LWP::UserAgent. Regards, Gisle
Re: HTTP::Headers - remove_header not removing header?
Seems like you can try to push (PeerHTTPVersion = 1.1) onto the @EXTRA_SOCK_OPTS to supress this one as well. That did the trick! thanks! So you are not on friendly terms with the server you are trying to access then. Oh well,... It isn't really a matter of being friendly. I can use a crappy proprietary app to grab the data in question once a day, in a manual tedious fashion, or I can use Perl to do it with elegance, hands free. I'm pretty sure they wouldn't object to what I'm doing, but let's just say I'd rather not cause any alarm :-) Alert! Alert! We have a customer with half a brain! Lock down the webservers ASAP!
Re: HTTP::Headers - remove_header not removing header?
Phil Thompson [EMAIL PROTECTED] writes: From the code below, I'm expecting the following headers: Host: www.yahoo.com User-Agent: Squid 3.92 But I'm getting the following (when I use a sniffer): TE: deflate,gzip;q=0.3 Connection: TE, close Host: www.yahoo.com User-Agent: Squid 3.92 Any idea how I can kill the TE and Connection headers? My use of remove_header doesn't seem to do any good... There is actually no easy way to do that. They are inserted by the protocol module when it feels they are appropriate. Can you explain why you need them to to go away? The Connection: close will go away of you pass the keep_alive option to the LWP::UserAgent. If you insist hard on the TE header to be disabled you you try something like: push(@LWP::Protocol::http::EXTRA_SOCK_OPTS, SendTE = 0); before you send your first request. Regards, Gisle
