bug#37207: guix.gnu.org Last Modified at epoch

[Ludovic Courtès]

Hi Tobias,

Tobias Geerinckx-Rice  skribis:

> Turning these off will of course prevent all caching.  I don't know if
> that would add measurable load to guix.gnu.org (it would be more
> problematic if we used a CDN, but it might still make a difference).
>
> Nix does something both interesting and icky — as always: patch[0]
> nginx to look up the realpath() instead, so clients can still cache
> using If-None-Match.

> [0]: https://github.com/NixOS/nixpkgs/pull/48337

(See
.)

I had overlooked this patch but it looks like the right approach
overall.  Calling ‘realpath’ each time seems a bit expensive as it
creates an ‘lstat’ storm, but I can’t think of a better solution.

I also found this post whose main interest is in showing how to write a
plugin to generate custom etags:

  https://mikewest.org/2008/11/generating-etags-for-static-content-using-nginx/

Thoughts?

Ludo’.

bug#37207: guix.gnu.org Last Modified at epoch

[Ludovic Courtès]

Hello!

Did one of you have chance to come up with a trick to emit the right
‘Last-Modified’?  We seemed to be close to having something.  :-)

Thanks,
Ludo’.

bug#37207: guix.gnu.org Last Modified at epoch

[Gábor Boskovits]

Hello Ludo,

Ludovic Courtès  ezt írta (időpont: 2019. aug. 28., Sze,
22:32):

> Hello,
>
> Gábor Boskovits  skribis:
>
> > we should create a file with the git last modification time of the files,
> > updated when there is a new commit in the repo => last-modified
> > we should create a file with some hash of the files, updated when there
> is
> > a new commit in the repo => etag
> > we could restrict these operations to the files modified since the last
> > checkout.
> >
> > Retrieve these with embededd perl.
> > Wdyt?
>
> What would the config look like?  AFAICS our ‘nginx’ package doesn’t
> embed Perl, and I think it’s better this way.  :-)  Can we do that with
> pure nginx directives?
>
> We create /srv/guix.gnu.org (as a symlink) with the correct mtime¹.  If
> we can tell nginx to use it as the ‘Last-Modified’ date, that’s perfect.
>
>
I was thinking about this. Yes, we can solve that with pure nginx. There is
an issue however.
It invalidates all cached entries on update, so files not modified will
also need to be downloaded again.

The easiest way to do that would be to simply generate an nginx config
snippet at a configurable location,
setting up the mtime and etags variable, and include that from the main
config.

If this would be ok, then I will have a look at implementing this.

Ludo’.
>
> ¹
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin.scm#n212
>

Best regards,
g_bor

-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

bug#37207: guix.gnu.org Last Modified at epoch

[Ludovic Courtès]

Hello,

Gábor Boskovits  skribis:

> we should create a file with the git last modification time of the files,
> updated when there is a new commit in the repo => last-modified
> we should create a file with some hash of the files, updated when there is
> a new commit in the repo => etag
> we could restrict these operations to the files modified since the last
> checkout.
>
> Retrieve these with embededd perl.
> Wdyt?

What would the config look like?  AFAICS our ‘nginx’ package doesn’t
embed Perl, and I think it’s better this way.  :-)  Can we do that with
pure nginx directives?

We create /srv/guix.gnu.org (as a symlink) with the correct mtime¹.  If
we can tell nginx to use it as the ‘Last-Modified’ date, that’s perfect.

Ludo’.

¹ 
https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin.scm#n212

bug#37207: guix.gnu.org Last Modified at epoch

[Gábor Boskovits]

Hello Tobias,

Tobias Geerinckx-Rice via Bug reports for GNU Guix  ezt
írta (időpont: 2019. aug. 28., Sze, 16:38):

> Gábor, Ludo',
>
> Gábor Boskovits 写道：
> > Supressing the last modified header is just an
> > add_header Last-Modified "";
> > away.
>
> You'll also need:
>
> # Don't honour client If-Modified-Since constraints.
> if_modified_since off;
> # Nginx's etags are hashes of file timestamp & file length.
> etag off;
>
>
You really have a point here.

Based on my reseach, I came up with the following:

we need
etag off;

we should create a file with the git last modification time of the files,
updated when there is a new commit in the repo => last-modified
we should create a file with some hash of the files, updated when there is
a new commit in the repo => etag
we could restrict these operations to the files modified since the last
checkout.

Retrieve these with embededd perl.
Wdyt?


> Turning these off will of course prevent all caching.  I don't
> know if that would add measurable load to guix.gnu.org (it would
> be more problematic if we used a CDN, but it might still make a
> difference).
>
> Nix does something both interesting and icky — as always: patch[0]
> nginx to look up the realpath() instead, so clients can still
> cache using If-None-Match.
>
> Kind regards,
>
> T G-R
>
> [0]: https://github.com/NixOS/nixpkgs/pull/48337
>

Best regards,
g_bor

-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

bug#37207: guix.gnu.org Last Modified at epoch

[Gábor Boskovits]

Hello Danny,

Danny Milosavljevic  ezt írta (időpont: 2019. aug.
28., Sze, 17:05):

> Hi Gabor,
>
> On Wed, 28 Aug 2019 12:40:37 +0200
> Gábor Boskovits  wrote:
>
> > Supressing the last modified header is just an
> > add_header Last-Modified "";
> > away.
> >
> > To get the info from the symlink seems to be much trickier, i would do
> with
> > either embedded perl or embedded lua. I am not sure if we should bother
> > with it, though. Wdyt?
>
> Since we already emit ETag, I don't think we need to bother with
> Last-Modified.
>
> Why is the ETag so short, though?
>
>
The ETag we emit is also bad. Nginx calculates this from mtime and
content-lenght,
so in our case it's just content length.


> >wget --debug -O /dev/null   https://guix.gnu.org/packages.json 2>&1 |
> grep -i etag
> >ETag: "1-2f38b1"
>
>
Best regards,
g_bor

-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

bug#37207: guix.gnu.org Last Modified at epoch

[Danny Milosavljevic]

Hi Gabor,

On Wed, 28 Aug 2019 12:40:37 +0200
Gábor Boskovits  wrote:

> Supressing the last modified header is just an
> add_header Last-Modified "";
> away.
> 
> To get the info from the symlink seems to be much trickier, i would do with
> either embedded perl or embedded lua. I am not sure if we should bother
> with it, though. Wdyt?

Since we already emit ETag, I don't think we need to bother with Last-Modified.

Why is the ETag so short, though?

>wget --debug -O /dev/null   https://guix.gnu.org/packages.json 2>&1 | grep -i 
>etag
>ETag: "1-2f38b1"



pgpYU_7jEciPx.pgp
Description: OpenPGP digital signature

bug#37207: guix.gnu.org Last Modified at epoch

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

Gábor, Ludo',

Gábor Boskovits 写道：

Supressing the last modified header is just an
add_header Last-Modified "";
away.


You'll also need:

# Don't honour client If-Modified-Since constraints.
if_modified_since off;
# Nginx's etags are hashes of file timestamp & file length.
etag off;

Turning these off will of course prevent all caching.  I don't 
know if that would add measurable load to guix.gnu.org (it would 
be more problematic if we used a CDN, but it might still make a 
difference).


Nix does something both interesting and icky — as always: patch[0] 
nginx to look up the realpath() instead, so clients can still 
cache using If-None-Match.


Kind regards,

T G-R

[0]: https://github.com/NixOS/nixpkgs/pull/48337


signature.asc
Description: PGP signature

bug#37207: guix.gnu.org Last Modified at epoch

[Gábor Boskovits]

Hello,

Supressing the last modified header is just an
add_header Last-Modified "";
away.

To get the info from the symlink seems to be much trickier, i would do with
either embedded perl or embedded lua. I am not sure if we should bother
with it, though. Wdyt?