Hello All,
I'm not sure if this is the correct place to post this question but I've dug as
deep as I can from my end and this seems like the best place to solicit help
from the wget gurus.
I'm running into some weird behavior where the same request sent from the same
version of wget from different machines is handling cookie processing
differently.
This is the version that I'm running (on both machines):
$ wget -V
GNU Wget 1.17.1 built on linux-gnu.
+digest -gpgme +https +ipv6 +iri +large-file -metalink +nls +ntlm
+opie -psl +ssl/openssl
Wgetrc:
/etc/wgetrc (system)
Locale:
/usr/share/locale
Compile:
gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
-DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
-I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include
-DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
-Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
-g -Wall
Link:
gcc -DHAVE_LIBSSL -DNDEBUG -g -O2 -fPIE -fstack-protector-strong
-Wformat -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64
-g -Wall -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro
-Wl,-z,now -L/usr/lib -lpcre -luuid -lssl -lcrypto -lz -lidn
ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a
This is the command I'm running:
$ wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies
--no-check-certificate https://docs.google.com/uc?export=download&id=$DOC_ID
This is the response I'm getting on both machines (replaced cookie value with
sample text for brevity). Note the three cookies being set:
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Fri, 17 Aug 2018 19:45:05 GMT
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Set-Cookie: download_warning_random_text=SOME_COOKIE_VALUE;
Domain=.docs.google.com; Expires=Fri, 17-Aug-2018 19:50:05 GMT; Path=/uc;
Secure; HttpOnly
Set-Cookie: NID=SOME_COOKIE_VALUE;Domain=.google.com;Path=/;Expires=Sat,
16-Feb-2019 19:45:05 GMT;HttpOnly
Set-Cookie: NID= SOME_COOKIE_VALUE;Domain=.google.com;Path=/;Expires=Sat,
16-Feb-2019 19:45:05 GMT;HttpOnly
Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
When running from machine 1 (Ubuntu - 16.04 LTS) I see the following debug
statements (note all three cookies are stored):
cdm: 2
Stored cookie docs.google.com -1 (ANY) /uc <permanent> <secure> [expiry
2018-08-16 16:23:03] download_warning_random_text SOME_COOKIE_VALUE
cdm: 2 3 4 5 6 7 8
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-15
15:18:03] NID SOME_COOKIE_VALUE
cdm: 2 3 4 5 6 7 8Deleted old cookie (to be replaced.)
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-15
15:18:03] NID SOME_COOKIE_VALUE
Registered socket 3 for persistent reuse.
URI content encoding = 'utf-8'
Saving cookies to /tmp/cookies.txt.
Done saving cookies.
When running from machine 2 (Ubuntu - 16.04.4 LTS) I see the following debug
statements (note only two of the three cookies are stored):
cdm: 2cdm: 2 3 4 5 6 7 8
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-16
19:45:05] NID SOME_COOKIE_VALUE
cdm: 2 3 4 5 6 7 8Deleted old cookie (to be replaced.)
Stored cookie google.com -1 (ANY) / <permanent> <insecure> [expiry 2019-02-16
19:45:05] NID SOME_COOKIE_VALUE
Registered socket 3 for persistent reuse.
URI content encoding = 'utf-8'
Saving cookies to /tmp/cookies.txt.
Done saving cookies.
For some reason the "download_warning" cookie is not being stored when running
from machine 2.
I'm not using a custom ~/.wgetrc file from either machine and both machines are
using the stock /etc/wgetrc file (no difference between files).
For the life of me, I can't figure out why that third cookie isn't being stored
from machine 2? The only thing I noticed that is different about that cookie
is that it's marked as "secure" while the other two are not.
I looked through the wget man pages and didn't see any other options that
impact cookie processing aside from the ones I've used. Any help would be
greatly appreciated.
Thanks
Sean
-This e-mail and any attachments may contain CONFIDENTIAL information,
including PROTECTED HEALTH INFORMATION. If you are not the intended recipient,
any use or disclosure of this information is STRICTLY PROHIBITED; you are
requested to delete this e-mail and any attachments, notify the sender
immediately, and notify the LabCorp Privacy Officer at
privacyoffi...@labcorp.com or call (877) 23-HIPAA / (877) 234-4722.
