Re: [Bug-wget] Query about correcting for DST with Wget

[Eli Zaretskii]

> From: Tim Ruehsen 
> Date: Tue, 15 Nov 2016 10:41:40 +0100
> 
> > If we care about this, we could have a private implementation of
> > utimes in mswindows.c, which would DTRT in the use cases needed by
> > Wget.
> 
> Wget uses utime() if available. This function is not covered by gnulib and it 
> is obsoleted by POSIX 2008.
> 
> Instead we should use utimens which is covered by gnulib and circumvents 
> several issues. Currently, I can see no special Windows code in gnulib - but 
> if the issue persists, it should IMO be fixed in gnulib.
> 
> WDYT ?

I won't hold my breath for this to solve the issue.  At best, gnulib
will probably call utimes in its utimens implementation, which will
reset us back to square one.  At worse, they will ask us to provide
the missing implementation for MS-Windows.

This issue is not solvable without calling win32 APIs directly,
because the MS C runtime function all behave consistently -- and
wrongly -- in this case.

Re: [Bug-wget] Query about correcting for DST with Wget

[Tim Ruehsen]

On Monday, November 14, 2016 5:47:36 PM CET Eli Zaretskii wrote:
> > Date: Sun, 13 Nov 2016 21:39:32 +0100
> > From: Jernej Simončič 
> > 
> > On Sunday, November 13, 2016, 19:53:06, Eli Zaretskii wrote:
> > > Does "while DST is in effect" mean that you download the file when DST
> > > is in effect, or you examine the timestamp of the file when the DST is
> > > in effect?
> > 
> > I download the file when DST is(n't) in effect (I download that
> > specific URL quite often, on different computers).
> 
> Then yes, it's a known problem with how the MS-Windows implementation
> of the utimes function works: it converts (internally) from local time
> to UTC using the current setting of the DST flag, not its setting at
> the time being converted.  The irony of this is that there should be
> no need to go through local time in this case, because the timestamp
> provided by Wget is in UTC to begin with, and the low-level Windows
> APIs that timestamp files accept UTC values.
> 
> If we care about this, we could have a private implementation of
> utimes in mswindows.c, which would DTRT in the use cases needed by
> Wget.

Wget uses utime() if available. This function is not covered by gnulib and it 
is obsoleted by POSIX 2008.

Instead we should use utimens which is covered by gnulib and circumvents 
several issues. Currently, I can see no special Windows code in gnulib - but 
if the issue persists, it should IMO be fixed in gnulib.

WDYT ?

Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Query about correcting for DST with Wget

[Tim Ruehsen]

On Sunday, November 13, 2016 7:10:57 PM CET Jernej Simončič wrote:
> On Friday, November 11, 2016, 8:52:49, Eli Zaretskii wrote:
> > Can you tell more details, like the exact URL you downloaded and how
> > you see the 1-hour difference?  I'd like to try to reproduce this
> > here.
> 
> I'm not sure if this is a problem with wget, Windows or the server
> hosting the file, but I observed this happening with
>  - while DST is in effect,
> the file gets timestamp of 22:19, and when it's not it's 23:19 (I'm in
> the CET timezone).

Just tested on Debian, the file time on the filesystem after download is 23:19, 
no matter is DST or not. Since I am on CET, that is the correct time (server 
says "Thu, 04 Feb 2016 22:19:14 GMT".

Now, looking at the code in src/http.c/http_atotm()...

struct tm t;
setlocale (LC_TIME, "C");
strptime ("Thu, 04 Feb 2016 22:19:14 GMT", "%a, %d %b %Y %T", );
return timegm ();


strptime and timegm are covered by gnulib (please check, that your 
bootstrap.conf contains these two modules). But it lacks 'setlocale'.

What you could try is (in DST where the timeshift occurs):

1. add 'setlocale' to the bootstrap.conf (and ./bootstrap + ./configure + make 
clean + make). And see if the problem persists.

 If it does... 
2. add a printf to src/http.c/http_atotm() to print out the time_t value - it 
should be the same in DST and outside DST.

3. If the time_t value differs, try without the setlocale code (comment the 
line with setlocale (LC_TIME, "C");).

4. if the time_t value still differs, maybe the struct tm t becomes filled with 
wrong values, e.g. tm_isdst field is set in DST. Try setting t.tm_isdst to 0 
before calling timegm().

Somewhere here you should be able to pinpoint the problem. Maybe searching the 
web helps, the problem should be known. Once we know it, we can fix it in wget 
- but in the long term it should go into gnulib (if it is not the missing 
'setlocale' module in bootstrap.conf).

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Query about correcting for DST with Wget

[Tim Ruehsen]

On Monday, November 14, 2016 5:47:36 PM CET Eli Zaretskii wrote:
> > Date: Sun, 13 Nov 2016 21:39:32 +0100
> > From: Jernej Simončič 
> > 
> > On Sunday, November 13, 2016, 19:53:06, Eli Zaretskii wrote:
> > > Does "while DST is in effect" mean that you download the file when DST
> > > is in effect, or you examine the timestamp of the file when the DST is
> > > in effect?
> > 
> > I download the file when DST is(n't) in effect (I download that
> > specific URL quite often, on different computers).
> 
> Then yes, it's a known problem with how the MS-Windows implementation
> of the utimes function works: it converts (internally) from local time
> to UTC using the current setting of the DST flag, not its setting at
> the time being converted.  The irony of this is that there should be
> no need to go through local time in this case, because the timestamp
> provided by Wget is in UTC to begin with, and the low-level Windows
> APIs that timestamp files accept UTC values.
> 
> If we care about this, we could have a private implementation of
> utimes in mswindows.c, which would DTRT in the use cases needed by
> Wget.
> 
> > I just remembered that there may be a 3rd explanation: some msvcrt
> > functions return different timestamps depending on whether DST is in
> > effect or not - at least with GIMP you can observe that it'll rescan
> > all fonts the first time it's run after DST change (I'm not sure if
> > this applies only to msvcrt.dll [which MinGW uses by default], or also
> > to the runtimes shipped with newer Visual Studio versions).
> 
> You are talking about 'stat', I believe.  Yes, they, too, have a
> similar bug, but 'stat' is not involved in the code in Wget that sets
> the timestamps of downloaded files.

Ah, Eli :-) Just got this mail after sending...

Then all please forget my email that I just sent...

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: [Bug-wget] Query about correcting for DST with Wget

[Eli Zaretskii]

> Date: Sun, 13 Nov 2016 21:39:32 +0100
> From: Jernej Simončič 
> 
> On Sunday, November 13, 2016, 19:53:06, Eli Zaretskii wrote:
> 
> > Does "while DST is in effect" mean that you download the file when DST
> > is in effect, or you examine the timestamp of the file when the DST is
> > in effect?
> 
> I download the file when DST is(n't) in effect (I download that
> specific URL quite often, on different computers).

Then yes, it's a known problem with how the MS-Windows implementation
of the utimes function works: it converts (internally) from local time
to UTC using the current setting of the DST flag, not its setting at
the time being converted.  The irony of this is that there should be
no need to go through local time in this case, because the timestamp
provided by Wget is in UTC to begin with, and the low-level Windows
APIs that timestamp files accept UTC values.

If we care about this, we could have a private implementation of
utimes in mswindows.c, which would DTRT in the use cases needed by
Wget.

> I just remembered that there may be a 3rd explanation: some msvcrt
> functions return different timestamps depending on whether DST is in
> effect or not - at least with GIMP you can observe that it'll rescan
> all fonts the first time it's run after DST change (I'm not sure if
> this applies only to msvcrt.dll [which MinGW uses by default], or also
> to the runtimes shipped with newer Visual Studio versions).

You are talking about 'stat', I believe.  Yes, they, too, have a
similar bug, but 'stat' is not involved in the code in Wget that sets
the timestamps of downloaded files.

Re: [Bug-wget] Query about correcting for DST with Wget

[Jernej Simončič]

On Sunday, November 13, 2016, 19:53:06, Eli Zaretskii wrote:

> Does "while DST is in effect" mean that you download the file when DST
> is in effect, or you examine the timestamp of the file when the DST is
> in effect?

I download the file when DST is(n't) in effect (I download that
specific URL quite often, on different computers).

> Also, how do you display the timestamp of the file? with what program?

Windows Explorer, or dir in command prompt.

I just remembered that there may be a 3rd explanation: some msvcrt
functions return different timestamps depending on whether DST is in
effect or not - at least with GIMP you can observe that it'll rescan
all fonts the first time it's run after DST change (I'm not sure if
this applies only to msvcrt.dll [which MinGW uses by default], or also
to the runtimes shipped with newer Visual Studio versions).

-- 
< Jernej Simončič ><><><><>< http://eternallybored.org/ >

If you need n items of anything, you will have n-1 in stock.
   -- Sueker's Note

Re: [Bug-wget] Query about correcting for DST with Wget

[gerdd]

One explanation I often found in the old days is this: People mess with the 
clock settings until they get the time they expect rather than setting the time 
zone and DST correctly. Should be hapening less frequently nowadays - my guess. 
But amateur admins of platforms they are not familiar with could still be doing 
it, I guess.

I used to have a lot of missed conference calls because of that - people 
invited me and the meeting ended up being an hour before or after the time in 
my invite. It gets worse if you mix different DST parameters with northern and 
southern hemisphere. 



Sent from Samsung Mobile

 Original message 
From: Jernej Simončič <jernej|s-w...@eternallybored.org> 
Date:2016/11/13  20:10  (GMT+02:00) 
To: "Eli Zaretskii on [bug-wget]" <bug-wget@gnu.org> 
Subject: Re: [Bug-wget] Query about correcting for DST with Wget 

On Friday, November 11, 2016, 8:52:49, Eli Zaretskii wrote:

> Can you tell more details, like the exact URL you downloaded and how
> you see the 1-hour difference?  I'd like to try to reproduce this
> here.

I'm not sure if this is a problem with wget, Windows or the server
hosting the file, but I observed this happening with
<https://live.sysinternals.com/procexp.exe> - while DST is in effect,
the file gets timestamp of 22:19, and when it's not it's 23:19 (I'm in
the CET timezone).

-- 
< Jernej Simončič ><><><><>< http://eternallybored.org/ >

Being sure mistakes will occur is a good frame of mind for catching them.
   -- Berkeley's Thirteenth Law

Re: [Bug-wget] Query about correcting for DST with Wget

[Eli Zaretskii]

> Date: Sun, 13 Nov 2016 19:10:57 +0100
> From: Jernej Simončič 
> 
> I'm not sure if this is a problem with wget, Windows or the server
> hosting the file, but I observed this happening with
>  - while DST is in effect,
> the file gets timestamp of 22:19, and when it's not it's 23:19 (I'm in
> the CET timezone).

Does "while DST is in effect" mean that you download the file when DST
is in effect, or you examine the timestamp of the file when the DST is
in effect?

Also, how do you display the timestamp of the file? with what program?

Re: [Bug-wget] Query about correcting for DST with Wget

[Jernej Simončič]

On Friday, November 11, 2016, 8:52:49, Eli Zaretskii wrote:

> Can you tell more details, like the exact URL you downloaded and how
> you see the 1-hour difference?  I'd like to try to reproduce this
> here.

I'm not sure if this is a problem with wget, Windows or the server
hosting the file, but I observed this happening with
 - while DST is in effect,
the file gets timestamp of 22:19, and when it's not it's 23:19 (I'm in
the CET timezone).

-- 
< Jernej Simončič ><><><><>< http://eternallybored.org/ >

Being sure mistakes will occur is a good frame of mind for catching them.
   -- Berkeley's Thirteenth Law

Re: [Bug-wget] Query about correcting for DST with Wget

[Eli Zaretskii]

> From: "Tim" 
> Date: Thu, 10 Nov 2016 19:26:45 -
> 
> I would be very grateful for any help with an issue I am having with 
> downloading files from a website using Version 1.11.4.3287 of Wget on a 
> Windows XP computer.

That's a very old version of Wget.

> When I use Wget to download a file from a website, the timestamp is out by an 
> hour. I think this is because of Daylight Saving Time. Do any of you know how 
> I can correct this?

Can you tell more details, like the exact URL you downloaded and how
you see the 1-hour difference?  I'd like to try to reproduce this
here.

In general, Windows XP has a database of DST offsets and should use it
to avoid such problems, but maybe Wget doesn't DTRT in this matter,
somehow, at least in your case.

[Bug-wget] Query about correcting for DST with Wget

[Tim]

Hi Everyone

I would be very grateful for any help with an issue I am having with 
downloading files from a website using Version 1.11.4.3287 of Wget on a Windows 
XP computer.

When I use Wget to download a file from a website, the timestamp is out by an 
hour. I think this is because of Daylight Saving Time. Do any of you know how I 
can correct this?

It is important that I can get a solution to this because the timestamps on the 
files I am downloading need to be correct for synchronization purposes.

Please note that I have access to both the web server and the client computer 
so I can make any necessary changes at either end.

Thank you very much.

Kind regards

Tim