Re: Stuck module upload for Date::Parse::Modern

[Ask Bjørn Hansen]

`curl` on the box can validate the certificate okay (right now), for what it’s 
worth.

Ask

> On Jan 14, 2023, at 13:49, Ricardo Signes  
> wrote:
> 
> I looked at the PAUSE logs:
> 2023-01-13 22:50:39 $$3234 v1049: Alert: nosuccesscount[0] error[Can't 
> connect to www.perturb.org:443  (certificate 
> verify failed)] (paused:701)
> 
> My guess is that PAUSE's CAs are too old "or something".

Re: Move dev release tarballs to GitHub?

[Ask Bjørn Hansen]

That’d make sense to me, deleting the old images and letting them just be on 
backpan. That should allow whatever tooling depends on recent releases being in 
the regular CPAN mirrors to still work.

Looking at 3 days of www.cpan.org  logs only the most 
recent development image seems to have been downloaded more than the background 
noise of everything getting a request now and then because web crawlers. (And 
the most recent development image by just number of downloads recently is less 
popular than about a dozen very old not-development releases).

I put a CSV of the counts of “perl5-“ URLs temporarily at 
https://tmp.askask.com/os/cpan/cpan-perl-downloads-2022-04-05-02_13_21.csv 



Ask

Re: Mirror update - cpan.perl.pt

[Ask Bjørn Hansen]

> On Mar 5, 2019, at 15:54, Pedro Melo  wrote:
> 
> … every couple of hours to sync our CPAN mirror to backpan, but we would like 
> to mirror backpan once a week from a master repo with —delete. Any 
> recommendation on which backpan mirror to use?

I don’t think that’s how backpan is supposed to work. It’s not much of a backup 
if you trust another “backup” to not lose data, if that makes sense.

(We have a backpan at backpan.perl.org, but I don’t think it has rsync enabled).

Ask

Re: What happens to CPAN clients when TLS 1.2 is required?

[Ask Bjørn Hansen]

> On Apr 24, 2018, at 18:11 , David Golden  wrote:
> 
> But when they do opt into TLS, it's 1.2 required, right?

Sure, but … TLS 1.2 is almost ten years old. 1.1 is only barely older.  What 
operating systems don’t support TLS 1.2, but are otherwise functional / 
reasonable enough that you’d be installing anything new?  (And if you are 
running something that old, downloading over TLS shouldn’t be your top priority 
problem).


Ask

Re: Making www.cpan.org TLS-only

[Ask Bjørn Hansen]

> On Sep 5, 2017, at 11:22 , Leo Lapworth  wrote:
> 
> Would (at least for the short term) just adding the HSTS header to every 
> request be the best solution? Then browsers get told to switch to secure and 
> other clients can do either.

HSTS only works on TLS requests, so you have to get the browser to use that 
first and then it’ll pay attention to the header (and use TLS across all 
requests).

Ask

Re: Making www.cpan.org TLS-only

[Ask Bjørn Hansen]

The Google change was the impetus to get around to it.

Clients should use TLS to request content. It limits the trust for downloading 
CPAN content roughly to:

- The author
- PAUSE system maintainers
- perl.org infrastructure maintainers
- Fastly
- Global CA infrastructure

Without TLS you basically trust anyone with any sort of access to your internet 
connection to not muck with the code you receive.

Obviously the real fix here is that clients need to request via TLS (since I 
doubt any clients other than regular browsers support HSTS).


Ask

Re: Making www.cpan.org TLS-only

[Ask Bjørn Hansen]

Uh, there’s no “SSL” anymore. The newer versions of SSL have been “TLS” since 
the end of the nineties.  https://en.wikipedia.org/wiki/Transport_Layer_Security

That being said, the suggested change here is to require HTTPS for www.cpan.org 
by redirecting all plain-text HTTP requests to the HTTPS version.


Ask

Re: Making www.cpan.org TLS-only

[Ask Bjørn Hansen]

> On Aug 31, 2017, at 19:44, James E Keenan  wrote:
> 
> To be honest, I had no idea what 'TLS' meant when I first read this message.  
> So I can't say anything one way or the other about your proposal.
> 
> I suspect I'm not alone in this.  I would encourage you to post in a location 
> like blogs.perl.org as to what 'TLS' is, so that the census count of the 
> ignorant can be reduced.

I posted on http://log.perl.org/ earlier. Feel free to link to that from 
blogs.perl.org.


Ask

Making www.cpan.org TLS-only

[Ask Bjørn Hansen]

Hi everyone,

We’re considering how/how-much we can make www.cpan.org TLS-only.
http://log.perl.org/2017/08/tls-only-for-wwwcpanorg.html

I expect that we can’t make the whole site TLS-only without breaking some CPAN 
clients, so the conservative version is to force TLS for

- any url ending in *.html
- any url not in matching some variation of
 (/authors/ | /MIRRORED.BY | ^/modules/[^/]+ )

Does that sound about right? Maybe /src/, too?

(Also - we will support TLS for www.cpan.org permanently now, so please update 
URLs where possible and appropriate).


Ask

Re: pause.perl.org seems down - can anyone fix?

[Ask Bjørn Hansen]

On Jan 8, 2014, at 9:05 AM, Cosimo Streppone cos...@cpan.org wrote:

http://log.perl.org/2014/01/multiple-hard-drive-faliure.html

 On 01/08/2014 01:08 PM, Leo Lapworth wrote:
 
 Pause seems to be down, can anyone help out?
 
 I wish I could. Sadly still down.
 No news on log.perl.org either.
 
 Tracing the route to pause.perl.org, I get
 up to gw-a.develooper.com.solfo.net.
 
 Perl.org MX is mx.develooper.com and that appears
 to be unreachable, so Robert and Ask are probably
 not getting these emails either?
 
 Hope I got Andreas email right ...
 
 -- 
 Cosimo



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: CPAN map

[Ask Bjørn Hansen]

On Mar 22, 2012, at 3:38, Henk P. Penning wrote:

 Hi,
 
  The CPAN mirrors map
 
http://mirrors.cpan.org/map.html

Neat. 

What do the blue markers mean?

Re: rrr-client using 100% CPU and 300Mb+ memory?

[Ask Bjørn Hansen]

On Sep 3, 2011, at 2:08, Pedro Melo m...@simplicidade.org wrote:

 :) Ok, I'll try and debug it during the weekend.

Try to make the batch size (much) larger. (I forget what the parameter is 
called)

Re: new instant mirroring client

[Ask Bjørn Hansen]

On Jun 16, 2011, at 1:03, Henk P. Penning wrote:

  I'm very curious to know what you think ; so, please try it out ;
  installing and testing should be very easy.

It's really great to get another client for the 'instant mirroring' system; 
it'll suit different people and more importantly it will help make the whole 
system more robust!

A couple of comments/questions:

- Why skip some of the files from the (instant) mirror?  Or am I reading the 
code wrong?
- The fake DBI.pm is a little dangerous in case someone tries to install it to 
the standard paths.
- At a glance it looks like you just mirror one file at a time.  If there are 
multiple new events, can't rsync mirror all of them in one connection/execution?



 - ask

Re: new instant mirroring client

[Ask Bjørn Hansen]

On Jun 16, 2011, at 3:47, David Precious wrote:

 Incidentally, I've not seen any reports on the progress of instant mirroring 
 for a bit - have I missed anything?


No - it's working nicely.  

You want to upgrade to 0.2.1 if you haven't already.  It's fixing deletion of 
symlinks that weren't before (Henk noticed!).

The outstanding todos are:

1) Get the rest of the mirrors using funet to stop doing that (to help out the 
funet people).
2) Encourage people to use instant mirroring (this can wait until it's been 
tested even more).
3) Make cpan-rsync.perl.org point to a geographically nearby (and up-to-date) 
mirror.  I thought this would be needed sooner; but the new master server is 
doing fine so far (Yay for SSD!)


 - ask

Re: Adjusting CPAN::Mirrors to make it more useful

[Ask Bjørn Hansen]

On May 6, 2011, at 23:33, p...@0ne.us wrote:

 (And everyone mirroring from www.cpan.org would sorta defeat the purpose of 
 having a bunch of mirrors -- unless we put more mirrors behind the 
 www.cpan.org name.).
 
 Hmm, while reading this I was reminded of the pool.ntp.org concept, is
 it applicable here? :)


Yes, search.cpan.org is using the same software as the NTP Pool for DNS.

However - http://www.cpan.org/ doesn't get enough requests that bandwidth is a 
concern so it's really just about making it (very slightly) faster or maybe 
trading complexity to get it a little bit more reliable.   The benefits are not 
really obvious enough to make it to the top of my todo.

Optimizing/distributing cpan-rsync.perl.org will probably come first, but that 
also got down the priority list a bit.

Just looking at the logs for today there are about 60 mirrors using rrr and 
about the same number doing an occasional full rsync (between once a day and 
every few hours) and the load is basically completely negligible so far.

'rrr' drastically cuts down the IO required and the SSD that's serving the CPAN 
data can just do a crazy amount of rsync type I/O.


   - ask

-- 
Ask Bjørn Hansen, http://askask.com/

Re: Adjusting CPAN::Mirrors to make it more useful

[Ask Bjørn Hansen]

On May 2, 2011, at 12:54, David Golden wrote:

 Doesn't a lot of this problem go away with www.cpan.org resolving to
 tier 1 mirrors?  Did I see that Robert/Ask were using some sort of
 GeoIP-aware DNS?

Yeah, though for now only with mirrors in Los Angeles and one in Europe.  
Somewhat similar to the search.cpan.org setup.

(And everyone mirroring from www.cpan.org would sorta defeat the purpose of 
having a bunch of mirrors -- unless we put more mirrors behind the 
www.cpan.org name.).


 - ask

Re: Adjusting CPAN::Mirrors to make it more useful

[Ask Bjørn Hansen]

On Apr 28, 2011, at 9:19, brian d foy wrote:

 I think I may have implemented what you're looking for several years
 ago for JSAN, which has a client that auto-detected appropriate
 mirrors in a few seconds each time it starts.
 
 http://search.cpan.org/~adamk/Mirror-URI-0.90/lib/Mirror/YAML.pm
 
 I was looking at this, but it seems like the idea of downloading a
 small file from several mirrors isn't a good way to figure out which
 mirrors to use, especially with a large number of mirrors.

What's the goal here?

Faster is sorta dumb, really.  There are few files on CPAN that are 
significantly bigger that the checking for a faster mirror won't take longer 
than just getting the file from a slower mirror.

If it's to find a good/up-to-date mirror, then there are a couple of json files 
available (on CPAN and the mirrors.cpan.org server).

I'll talk to Henk about getting the mirrors.json file - 
http://www.cpan.org/indices/mirrors.json - to include a is this mirror good? 
flag of sorts.


 - ask

cpan.org updated

[Ask Bjørn Hansen]

Hi everyone,

CPAN got its first big update in a while:
http://log.perl.org/2011/03/big-cpanorg-update.html


 - ask

-- 
http://log.perl.org/ - http://askask.com/

CPAN master mirror update / rrr testing

[Ask Bjørn Hansen]

Hi everyone,

Monday afternoon I swapped the temporary CPAN master mirror for the new server.

The testing of the 'rrr' stuff is progressing; there are some more bits about 
how to test it on the wiki page now: 
https://github.com/perlorg/cpanorg/wiki/Instant-update-mirroring (and I think 
it's ready to be somewhat more widely tested after some bug fixing this 
evening).

Leo Lapworth has been doing a lot of work cleaning up some of the pages (FAQ, 
ports page, and others).  We're getting a few more bits in place to properly 
integrate some of the changing data bits (number of modules, recent uploads, 
...); but I hope we'll put up the updated pages within the next few days.


  - ask

-- 
Ask Bjørn Hansen, http://askask.com/

CPAN mirrors list in JSON format

[Ask Bjørn Hansen]

The list of mirrors is now available in JSON format:

http://log.perl.org/2011/02/cpan-mirrors-list-now-also-in-json-format.html


 - ask

-- 
Ask Bjørn Hansen, http://askask.com/

Re: update faq

[Ask Bjørn Hansen]

On Feb 25, 2011, at 5:54, Henk P. Penning wrote:

  .. to see if I got it right ; I forked, cloned, committed, pushed
  and set a 'pull request' ; please consider the change below.

That worked. :-)  I think you had some other changes to the how to mirror 
section, too...

Re: Ports list - is it being maintained?

[Ask Bjørn Hansen]

On Feb 17, 2011, at 11:43, David Golden wrote:

 That raises a broader question.
 
 Who maintains all of the non tarball content on CPAN and what is the
 process for getting it changed?

We at perl.org took it over recently but haven't gotten this sort of thing 
setup yet.  (First order of business was getting the mirror list maintained 
again, we setup a process for that and Henk Penning is now maintaining the 
list).

Next on the list is getting a proper tiered mirroring setup going so we can get 
some load off FUnet.  FUnet is already mirroring from here, but nobody else is 
yet (doh!).  It will use Andreas' File::Rsync::Mirror::Recent system for 
instant updates.  I bought some new hardware for this but the ethernet 
controller went bad so it's back at the vendor.

Lastly I'm going to setup a git repository and a system to maintain the static 
pages; probably building them with ttree (from template toolkit).Since 
there seems to be interest in helping with this, I'll try to get it done soon.  
:-)



 - ask

Re: Adjusting CPAN::Mirrors to make it more useful

[Ask Bjørn Hansen]

On Feb 7, 2011, at 22:55, brian d foy wrote:

 And, who makes the MIRRORED.BY file? I imagine that's something from a
 script that Jarkko makes, but how does it get the data?

Henk Penning (aka the mirror list master since a few months ago) maintains a 
master mirrors.json file that a script on the perl.org servers converts to 
MIRRORED.BY.

 I'd like to see about exporting it as JSON or something.

Henk and I were just talking about adding a mirrors.json file a few days ago 
actually...  So yes, coming soon.

 Also, is it something that noc.perl.org has to handle as the master CPAN 
 moves off of FUNET?

Already done; just not announced yet.


 - ask

Re: Can anyone give me access to an authenticating proxy server for testing?

[Ask Bjørn Hansen]

On Nov 2, 2010, at 20:12, David Golden wrote:

 I want to experiment with CPAN.pm and authenticating proxies.  Can
 anyone give me access to a proxy server?

Install squid on your laptop?


 - ask

Re: How to get the Pause ID

[Ask Bjørn Hansen]

On Oct 3, 2010, at 20:31, Andreas J. Koenig wrote:

 http://www.cpan.org/scripts/submitting.html
 
 Yes, I'm not sure that even worked in 2002 when the page was last updated.
 
 /scripts/.* should just be redirected somewhere else.
 
 I must correct you, Ask, scripts/ is indeed alive, it is updated by a
 cronjob that Kurt Starsinic contributed.

We have different definitions for alive, then.  :-)

There have been no new scripts included since the old submission instructions 
were renamed to legacy.html ~12 years ago.  Indeed I have to go *15* years 
back to find any files that were touched recently  (see below).

 We never deprecated it.

That's what I was suggesting we do.  Just make a directory on backpan and move 
it there.  

 
  - ask

# 14 years
[r...@mirrors.la scripts]# find . -type f -mtime -5110  | grep -v index.html
./new/submitting.html
./submitting.html
./legacy.html

# 15 years
[r...@mirrors.la scripts]# find . -type f -mtime -5475  | grep -v index.html
./new/submitting.html
./dbase/cisamperl-0.95
./dbase/rdb.info
./dbase/menu
./submitting.html
./admin/menu
./date_and_time/menu
./date_and_time/itimers.pl
./interfaces/curseperl/menu.pl.announce
./interfaces/curseperl/menu
./interfaces/curseperl/perlmenu.v3.3.tar.Z
./ftpstuff/mirror-2.8.tar.gz
./ftpstuff/menu
./ftpstuff/ftplib-1.1.patch
./mailstuff/menu
./mailstuff/mailsort.tar.gz
./mailstuff/mailsort
./legacy.html
./infoserv/WWW/sgmlstripper
./infoserv/WWW/rjsemail.cgi
./infoserv/WWW/mhttpd-0.1.tar.gz
./infoserv/WWW/menu
./math.and.stat.stuff/call_graph.pro.shar
./math.and.stat.stuff/menu
./text-processing/fortran_scripts.uu

Re: How to get the Pause ID

[Ask Bjørn Hansen]

On Oct 3, 2010, at 21:16, Andreas J. Koenig wrote:

 There's a mirroring bug somewhere then.

Ah, I'm sorry -- I completely missed that the index files were pointing to 
proper PAUSE files!

Anyway, I still think we should deprecate it and recommend people create 
App::foobar modules with a script runner.



 - ask

Re: How to get the Pause ID

[Ask Bjørn Hansen]

On Oct 2, 2010, at 20:52, Parag Kalra wrote:

 I have been trying to get Pause ID but seems like either the request is 
 getting rejected or something is wrong in the way I am raising the request.

You mention sharing scripts in the request -- PAUSE doesn't actually do that; 
it's only for modules.  If you meant modules by scripts, then maybe that was 
the misunderstanding.


 - ask

Re: Leo is skinning www.cpan.org

[Ask Bjørn Hansen]

On Sep 27, 2010, at 8:17, David Cantrell wrote:

 While we're on the subject of RECENT*, what is RECENT-Z.yaml?  And will
 it ever be updated?  At a glance, it appears to be a log of all uploads
 and deletes from the beginning of time up to some point in recent
 history, at which point it stops.

It continues in the RECENT-1Y file.   It's used by File::Rsync::Mirror::Recent.


  - ask


[r...@mirrors.la authors]# ls -lt RECENT*yaml
-rw-r--r--+ 1 mirrors mirrors 4614 Sep 27 08:56 RECENT-1h.yaml
-rw-r--r--+ 1 mirrors mirrors 7793 Sep 27 08:56 RECENT-6h.yaml
-rw-r--r--+ 1 mirrors mirrors   763623 Sep 27 08:41 RECENT-1M.yaml
-rw-r--r--+ 1 mirrors mirrors   207896 Sep 27 08:41 RECENT-1W.yaml
-rw-r--r--+ 1 mirrors mirrors32697 Sep 27 08:41 RECENT-1d.yaml
-rw-r--r--+ 1 mirrors mirrors  1984910 Sep 24 07:41 RECENT-1Q.yaml
-rw-r--r--+ 1 mirrors mirrors  9833609 Sep  3 00:50 RECENT-1Y.yaml
-rw-r--r--+ 1 mirrors mirrors 15545888 Jun 24 01:41 RECENT-Z.yaml

Re: so long, CPAN

[Ask Bjørn Hansen]

On Sep 26, 2010, at 5:52, Jarkko Hietaniemi wrote:

 On a more urgent note: could you and Elaine coordinate on moving/copying
 stuff out of gargoyle where e.g. the mirrors.cpan.org runs?

Yes, of course.  I'll send a mail off-list to get started.


 - ask

Re: Trimming the CPAN - Automatic Purging

[Ask Bjørn Hansen]

On Apr 2, 2010, at 1:50, Arthur Corliss wrote:

 And my assertion has been that the excessive stats by the server are a bigger
 impediment to synchronization than the inode count.

Well, then one of us don't understand how file systems etc work.  :-)


  - ask

Re: Distributing the CPAN

[Ask Bjørn Hansen]

On Apr 2, 2010, at 14:03, Tim Bunce wrote:

 Imagine a cpan-all 'superproject' repro that has all the distros as
 submodules.  This repro would be tiny when cloned because it only
 contains empty directories for the distos plus the metadata for where
 the upstream distro repro lives and what the current commit it.
 When a distro is updated the cpan-all repro would be updated
 to reference the latest version of the distro.

That's a really good idea actually.   That'd mean, too, that it's possible to 
reset a distribution (to get rid of excessive size etc).

It'd be fun to try on the gitpan data...


 - ask (on a sketchy 3g connection out in the country)

Re: Distributing the CPAN

[Ask Bjørn Hansen]

On Apr 1, 2010, at 16:50, Tim Bunce wrote:

 * The need for widespread mirroring is less significant than it was in
 years past. (Also using git as the inter-mirror transport of source files
 means there'll be much less traffic between mirrors. Effectively only
 the diffs between releases.)

The bandwidth isn't an issue -- the disk IO is.

Maybe there'd be less disk IO with git if all of CPAN was in one big 
repository; but there are many good reasons for it not to be.

If we had a repository per distribution we're back to square one; more or less.


  - ask

Re: Trimming the CPAN - Automatic Purging

[Ask Bjørn Hansen]

On Mar 31, 2010, at 6:52, David Nicol wrote:

 new proposal: Make modules pay rent in order to remain on a mirror.
 Rent could be in the form of actual user interest, or good reviews.

How you are proposing purging useless stuff from CPAN -- that's a lot more 
radical than Tim's proposal of  just purging _old_ useless stuff.


 - ask

Re: Trimming the CPAN - Automatic Purging

[Ask Bjørn Hansen]

On Mar 26, 2010, at 16:02, Arthur Corliss wrote:

 Why use rsync, then?  Why not have checkpointed logs on cpan with
 additions/removals logged by date so you can roll forward on the client,
 processing only those files?  It would be trivial to set up and a lot more
 efficient.


I find it curious that everyone who's actually involved in syncing the files or 
running mirror servers seem to think it generally sounds like a good idea and 
everyone who doesn't say it's not worth the effort.

Anyway -- we have some other ideas for cutting down the number of files that we 
already agreed on but just needs announcement (which I promised to write up, 
oops).  No, I'm not going to make Tim's mistake and suggest it here first.

Tim: Next time just get the paint in your preferred color.  :-)


 - ask

Re: Trimming the CPAN - Automatic Purging

[Ask Bjørn Hansen]

On Mar 25, 2010, at 8:38, Andy Armstrong wrote:

 I like that solution better
 
 
 [snip]
 
 But solution to what? Are we convinced there's actually a problem here?

CPAN has almost 200k files.  www.cpan.org says there are 17627 modules.  
rsyncing a gazillion files doesn't work that well (on the server).  Helping 
authors remember to delete things that are now irrelevant from the main CPAN 
system will make it easier to run mirrors and keep them fresh.


 - ask

Re: Trimming the CPAN - Automatic Purging

[Ask Bjørn Hansen]

On Mar 25, 2010, at 13:23, Eric Wilhelm wrote:

 Maybe CPAN mirrors are more easily updated than via a generic rsync?  Is 
 the burden only network/cpu for checking whether a bunch of old 
 archives have changed, or does disk matter?

Most CPAN mirrors use rsync.  It's not realistic to make them change that 
(Hello all mirror operators -- so that tool that you use for ALL YOUR MIRRORS; 
well ... maybe you can use something else for us?).

rsync is all disk i/o -- relatively negligible network and CPU.


 - ask

Re: RFC: PGAN

[Ask Bjørn Hansen]

On Jan 7, 2010, at 21:29, David E. Wheeler wrote:

 See File::Rsync::Mirror::Recent
 
 Interesting. I especially like the  PRE-ALPHA ALERT .

www.cpan.org and the search.cpan.org mirrors are all using it, too (some of 
them indirectly, but still).


 - ask

Re: CPAN vs Perl 6

[Ask Bjørn Hansen]

As much as I like JSON I agree that for CPAN.pm like programs a variation of 
the current index really seems like the best match (5 lines of perl for the 
parser; backwards compatible; easy-ish to make as memory efficient as possible).

For other tools having a more complete data structure would be nice; but 
there's no reason that PAUSE couldn't produce indexes in different formats.  If 
so then the optional fancy index could be any combination of SQLite, JSON, 
buzzword-bingo-of-the-day that we like.


 - ask

Re: META.json: duty now for the future

[Ask Bjørn Hansen]

0) Yay for JSON.

1) Is it really necessary to use ascii instead of just regular utf-8  
for the JSON output?


2) The documentation is a little confusing saying If it can be  
loaded, any META.yml file produced will contain JSON. -- but it looks  
like the code (correctly) makes the output file be META.json when all  
is well.



 - ask