Re: [Savannah-hackers-public] Git email hook?

2022-09-22 Thread Bob Proulx
Hi Paul,

Paul Smith wrote:
> > I pushed a change this morning and didn't see any email about it.
>
> Got the email about 4 hours later; apparently there's some lag in the
> system (I just got this email, about 4.5 hours later...)

There isn't any intentional mailing list delays.  But on this last
Tuesday I saw mail delayed by 11 hours which is very excessive!  No
one knows why.  FSF sysadmin looked at the issue but by that time the
mail had been flowing so there wasn't any problem found.  There was
some possibility that backup had browned out the I/O and slowed things
down but no hard evidence yet.  I am also suspicious that it could
cause that long of a delay.  So...  Don't know!

I say please keep reporting any odd problems and delays like this.
Just because we say, "I don't know." doesn't mean that we won't keep
looking until we find a problem.  And by we here I mean others since I
don't personally do anything with exim on the lists or eggs systems.

Bob



Re: [Savannah-hackers-public] Sending email via fencepost fails

2022-08-24 Thread Bob Proulx
Hello Eli,

Eli Zaretskii wrote:
> Since about half an hour ago, I cannot send email via fencepost port
> 587: I get "connection refused".
> Logging into fencepost via SSH does work.
>
> What is the problem with the SMTP service on fencepost?  Can this be
> fixed, please?
>
> P.S. And if this is due to some maintenance, why don't I see anything
> on the https://hostux.social/@fsfstatus page?

I know you CC:'d sysadmin and that is the right place to ask about
fencepost problems.  But regarding the TO: to this group none of the
Savannah Hackers have any access to fencepost.  As far as the Savannah
group goes we can only suffer along with you and comiserate along with
you concerning any fencepost issue.

There is a fencepost-us...@gnu.org mailing list presumably for
fencepost users to discuss all things fencepost.  However it has been
converted to be a notification of aliases changes list almost
exclusively.  I haven't seen any other posting there in the last year.
At least not since April.  I don't know if anyone actively reads that
mailing list anymore.

Your email to sysadmin will create an RT ticket.  They always have a
couple of hundred RT tickets in their queue though.  It might take
some time for them to process down to that ticket.

Honestly on fencepost something probably happened and then got fixed
comletely asynchronously to the problem reports about it.  I don't see
any chatter on the IRC channels about fencepost but as a general
comment I often see exim get snarled up because of some problem with
spamassassin or other getting broken by a spam message and then a
restart is needed to clear the problem.  AFAICT all of that noticing
and restarting is fully manual.  I expect that or something like it is
what happened on fencepost.

Bob


signature.asc
Description: PGP signature


[Savannah-hackers-public] Savannah non-gnu download area policy for scp, sftp, rsync

2022-08-18 Thread Bob Proulx
Savannah Hackers,

bill-auger helped me debug the upload issue reported in
https://savannah.nongnu.org/support/?110690 today.

The root cause of this problem is users who have upgraded their system
to OpenSSH 9.0 and the upstream OpenSSH at 9.0 has switched the
internal protocol of scp from the legacy (and problematic) scp/rcp
protocol to using sftp internally instead.  This is actually a very
good upgrade and I welcome it but the result was that it switched the
protocol to one that has been blocked due to a security concern.

https://www.openssh.com/txt/release-9.0

This release switches scp(1) from using the legacy scp/rcp protocol
to using the SFTP protocol by default.
...
In case of incompatibility, the scp(1) client may be instructed to use
the legacy scp/rcp using the -O flag.

Just for completeness I will say that OpenSSH provides the scp -O
option to use the previous now obsolete scp protocol.  That would have
forced use of the old scp protocol and worked.  Until even that became
obsolete and removed.  But it is not needed because I reviewed the
issue with sftp and believe it is not a problem for Savannah due to
the special nature of Savannah only hosting Free Software and nothing
more.  Therefore I enabled sftp access.

In 2016 Sylvain Beucler reported a security issue with use of the sftp
protocol such that members may access as themselves the raw file
system.  On other systems that would be a problem of information
leakage.  But on Savannah all of the files are accessible from the
public side of things already.  There isn't anything visible from this
information leakage hack that isn't already visible by other means.
This is not an anonymous access but an access possible by project
members with commit access only.  And note that standard Unix file
system permissions prevent access to sensitive files such as to the
local encoded passwords in /etc/shadow which are not accessible to
non-root non-superuser users regardless.  They would only have access
as themselves and their project groups.  All accessible information is
already publicly accessible.

Therefore I have enabled the sftp protocol again regardless of the
possible leakage of already public information to members.

download: /etc/membersh-conf.pl
-$use_sftp = '0';
+$use_sftp = '1';

I have updated the documentation for the scp and rsync commands.

https://savannah.nongnu.org/maintenance/DownloadArea/

Note that when using rsync do not use the -a option, for other reasons
unrelated to this report that attempts a chown operation which is
blocked and results in a hang and failure.  Using -t and letting owner
and group default is sufficient.

Does anyone have any reason to do something different from the above?
Although I took this action without discussion that doesn't mean that
I might be wrong.  In which case we will need to react and do
something different.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Savannah bug->email gateway problems

2022-08-17 Thread Bob Proulx
Bob Proulx wrote:
> It appears that systemd is setting NoNewPrivileges=yes for apache and
> if I read the documentation correctly this will definitely break
> things in the way we are seeing.  I have removed that setting and am
> trying things again.

That seems to have been the problem.  I upgraded all of the packages
that I had downgraded during testing and restarted all.  I just tested
this with a different ticket and it sent the mail okay.

The root cause of the problem appears to have been over exuberant
hardening from someone setting NoNewPrivileges=yes in systemd for the
apache processes and as that prevents all suid in child processes it
basically breaks anything and everything that calls out to
subprocesses such as sending email with /usr/sbin/sendmail and other
things.

Bob



Re: [Savannah-hackers-public] Savannah bug->email gateway problems

2022-08-17 Thread Bob Proulx
Bob Proulx wrote:
> I continue to poke at things.

It appears that systemd is setting NoNewPrivileges=yes for apache and
if I read the documentation correctly this will definitely break
things in the way we are seeing.  I have removed that setting and am
trying things again.

Bob



Re: [Savannah-hackers-public] Savannah bug->email gateway problems

2022-08-17 Thread Bob Proulx
Bob Proulx wrote:
> I have downgraded all of apache and php packages to a version prior to
> the latest point release.
...
> I'll try to test the outbound mail from the web UI and verify that it
> is functioning now or not.

Unfortunately that has not resolved the problem.  Which means it is
something else. :-(

I continue to poke at things.

Bob



Re: [Savannah-hackers-public] Savannah bug->email gateway problems

2022-08-17 Thread Bob Proulx
Bob Proulx wrote:
> G. Branden Robinson wrote:
> > The bug-groff list hasn't gotten any updates on applicable Savannah
> > tickets for the groff group since sometime on 12 August.
> >
> > https://lists.gnu.org/archive/html/bug-groff/2022-08/index.html

> Your date above is an important clue.  Convergent with the 12th on
> that day I applied the pending Trisquel security patches which were
> rather extensive in that point release and included both apache and
> php and a lot more.  It's possible that these security patches broke
> something.  I might try backing out the apache and/or php ones to see
> if that improves things.

I have downgraded all of apache and php packages to a version prior to
the latest point release.  Unfortunately I was unable to locate the
immediately previous versions from the Trisquel archive but have
reverted to the original release versions from Triquel main which are
significantly older.  They should function though.

On the good news it has been a few minutes already and I have not seen
the bad behavior that was always immedaitely apparent previously.
Always before almost immediately there would be mail errors reported
in the apache logs.  And then apache processes would start stacking
up.  It's been 15 minutes now and I am not seeing that happening.

I'll try to test the outbound mail from the web UI and verify that it
is functioning now or not.

Bob



Re: [Savannah-hackers-public] Savannah bug->email gateway problems

2022-08-16 Thread Bob Proulx
G. Branden Robinson wrote:
> Possibly related to the recent DoS-driven outage...

Possibly.  The abuse continues currently.  And it is hard to identify.
Though this seems like this problem would be an independent problem.

> The bug-groff list hasn't gotten any updates on applicable Savannah
> tickets for the groff group since sometime on 12 August.
>
> https://lists.gnu.org/archive/html/bug-groff/2022-08/index.html

That's not good.  Thanks for reporting it.

> And when I update such a ticket, as with
> , the page reload hangs forever.
>
> If this is happening for multiple projects, this might be related to the
> giant mail backlog that was observed.

Your date above is an important clue.  Convergent with the 12th on
that day I applied the pending Trisquel security patches which were
rather extensive in that point release and included both apache and
php and a lot more.  It's possible that these security patches broke
something.  I might try backing out the apache and/or php ones to see
if that improves things.

> (I see that bug-gnu-emacs, by contrast, seems to have been churning
> along this whole time without trouble, so the problem may be restricted
> to a subset of Savannah projects.)
>
> https://lists.gnu.org/archive/html/bug-gnu-emacs/2022-08/index.html

Emacs uses the debbugs BTS which is completely separate and unrelated.

> Can this be fixed, and the queued-up ticket updates (re)processed if
> necessary?  The availability of ticket history via email archives is
> extremely valuable for groff development, since it gives me multiple
> lines of attack when searching.

Unknown until the problem is understood and fixed.  And I am hoping
that Ineiev will see this and take a look as the resident expert on
that area of things.

Bob



Re: [Savannah-hackers-public] Anyone have any news on Savannah?

2022-08-15 Thread Bob Proulx
It looks like the automatic restart is working.  But the underlying
problem remains.

Afer a restart...

[Mon Aug 15 17:47:40.409406 2022] [mpm_prefork:notice] [pid 16147] AH00173: 
SIGHUP received.  Attempting to restart
[Mon Aug 15 17:47:41.241212 2022] [mpm_prefork:notice] [pid 16147] AH00163: 
Apache/2.4.29 (Trisquel_GNU/Linux) OpenSSL/1.1.1 configured -- resuming normal 
operations
[Mon Aug 15 17:47:41.241352 2022] [core:notice] [pid 16147] AH00094: 
Command line: '/usr/sbin/apache2'
postdrop: warning: mail_queue_enter: create file maildrop/338453.17005: 
Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/933049.17009: 
Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/339760.17005: 
Permission denied
postdrop: warning: mail_queue_enter: create file maildrop/934393.17009: 
Permission denied
...

Endless logs of mail failure.  Which I don't yet understand.  It would
not be normal for the system to be sending mail at the rate that
attempts are being logged.  Also testing shows that email is working
okay otherwise.  Something is trying to send a lot of mail but
fortunately failing.  The access logs don't immediately point to a
culprit.  But at least it has been automatically restarted
successfully and that is keeping the web site online.

I continue to poke at things...

Bob

P.S. I really wish this were nginx instead of apache.  Every time I am
forced to deal with apache problems I feel like I am being punished
for bad karma from a previous life.  But maybe I am...



Re: [Savannah-hackers-public] Anyone have any news on Savannah?

2022-08-15 Thread Bob Proulx
Paul Smith wrote:
> Unfortunately it seems to be down / under attack again this morning :(

I see that my attempts at mitigation of the current problem are
failing.  I am looking into things again now.  I'll type this in
stream of consciousness as I work the problem and then send it.

I may have fixed the insufficient mitigation for the apache web server
wedging up.  I had thought that "apachectl graceful" should be
sufficient to restart the apache server.  But no it was not.  The
apache logs were filled with attempts to restart using that command
and obviously it wasn't working.

It appears that what is happening is that external client agents are
connecting to the apache web server and hanging onto connections
forever.  This causes apache to spawn up to the MaxRequestWorkers
number and hold there.  And it doesn't matter the number.  Eventually
all server processes are consumed regardless of the number.  Processes
respond to other requests until the malicious agent eventually grabs
them and holds onto them.  That's when no one else can get response
from the site.

Meanwhile that number must be below the max memory available for use.
If it is larger than available then of course that's bad and we must
start swapping.  I reduced it from the large 255 value to 32 yesterday
on the previous pass at this tuning.

The graceful action politely waits for the client to finish
processing.  Which in this case never occurs because the abusive
agents are not polite and hold on forever.  Therefore using graceful
is insufficient.  That is why it requires the forced stop action to
have apache stop talking to the abusive agents forcibly and to drop
all connections.  And of course then a start works.

I changed that to be a full "apachectl stop" and then followed by a
"apachectl start" and that worked.  Rather than simply restarting I
changed the automatic mitigation test and let it work and so tested
that it would automatically detect and restart okay.

But I am going to configure the "restart" action.  It makes me nervous
that the automation triggers "stop;start" immediately with no delay.
I worry that it might take a moment for the stop to propagate and all
of the old apache server processes to actually exit.  And if they
don't exit then the restart action can't succeed.  It's a race
condition due to the asynchronous behavior of systemd.  Though the
automation would trigger again and that one might succeed if the
previous action failed.  Feels like the apachectl restart would handle
this action more reliably since that instructs the supervisor process
to perform the restart.  Will configure it while the current malicious
agents are active to help us test for it.

[[ In this investigation I see that on systemd systems there is an
"interesting" control loop.  apachectl calls systemd and systemd calls
apachectl.  This would be a loop except for a programmed in internal
variable that breaks the loop.  Isn't it a wonderful world that we now
live in?  Not! ]]

If the restart action proves insufficient (though I think it will
work) then I will script up a stronger more explicit restart action.
Let's hope that isn't needed.

Additionally I tuned the apache server to have a smaller number of
MaxRequestWorkers than the default 255 which is more than the memory
available.  I have it currently set to a more conservative 32.  Though
now that things are observable again it appears that each process uses
about 25MB of memory for about 800MB of memory consumed in this
configuration.  We could increase that number somewhat and use more
memory for server processes.  Increasing this would eat into available
file system buffer cache though.  Life is a tradeoff.  But this seems
pretty responsive right now.  I'll leave it this way for a while and
observe.

Additionally I configured MaxConnectionsPerChild to be 100 rather than
unlimited.  This will cause apache to restart server processes after
that number of client requests have been handled.  That's just a
useful "random bug" guard restarting processes ever so often rather
than never.

> Don't people have better things to do with their lives?  Sigh.

It's why we can't have nice things.  :-(

Bob



Re: [Savannah-hackers-public] Anyone have any news on Savannah?

2022-08-14 Thread Bob Proulx
Hi Paul,

Paul Smith wrote:
> I haven't been able to reach the Savannah website for most of the day.
> Things like the Git service are available but the website is not.

Thanks for the report.  It appears that some agent was pounding on the
web site.  There were max processes of apache2 web server running and
nothing making progress.  I killed all and restarted the web server
and things seem to be functional now.

It looks like the fail2ban dynamic rules were not transferred over to
the new system.  We have some custom rules there that help block
abusive agents.  I'll get those set up on the system.

Bob



[Savannah-hackers-public] IPv6 problems since recent VM movement

2022-07-17 Thread Bob Proulx
Savannah Hackers,

Since Thursday's VM movement there have been problems with IPv6
connectivity.  I mapped out the connectivity between systems.  Since
this hasn't been tracked it's possible that these are not all of the
same issue and for example nfs1 may have been this way for a while.

Bob

| ping|
+-+
|Host1  |Host2  | IPv4 | IPv6 |
+-+
|download0  |internal0  | Okay | FAIL |
|download1  |internal0  | Okay | FAIL |
|internal1  |internal0  | Okay | FAIL |
|mgt0   |internal0  | Okay | FAIL |
|nfs1   |internal0  | Okay | FAIL |
|vcs0   |internal0  | Okay | FAIL |
|vcs1   |internal0  | Okay | FAIL |
|vcs2   |internal0  | Okay | FAIL |
+-+
|mgt0   |download0  | Okay | FAIL |
|mgt0   |download1  | Okay | Okay |
|mgt0   |internal0  | Okay | FAIL |
|mgt0   |internal1  | Okay | Okay |
|mgt0   |nfs1   | Okay | Okay |
|mgt0   |vcs0   | Okay | FAIL |
|mgt0   |vcs1   | Okay | Okay |
|mgt0   |vcs2   | Okay | Okay |
+-+
|download0  |nfs1   | Okay | Okay |
|download1  |nfs1   | Okay | Okay |
|internal0  |nfs1   | Okay | Okay |
|internal1  |nfs1   | Okay | Okay |
|vcs0   |nfs1   | Okay | Okay |
|vcs1   |nfs1   | Okay | Okay |
|vcs2   |nfs1   | Okay | Okay |

| SSH port 22 |
+-+
|Host1  |Host2  | IPv4 | IPv6 |
+-+
|mgt0   |download0  | Okay | FAIL |
|mgt0   |download1  | Okay | Okay |
|mgt0   |internal0  | Okay | FAIL |
|mgt0   |internal1  | Okay | Okay |
|mgt0   |nfs1   | FAIL | Okay |
|mgt0   |vcs0   | Okay | FAIL |
|mgt0   |vcs1   | Okay | Okay |
|mgt0   |vcs2   | Okay | Okay |

| MySQL port 3306 |
+-+
|Host1  |Host2  | IPv4 | IPv6 |
+-+
|download0  |internal0  | Okay | FAIL |
|download1  |internal0  | Okay | FAIL |
|internal1  |internal0  | Okay | Okay |
|mgt0   |internal0  | Okay | FAIL |
|nfs1   |internal0  | Okay | Okay |
|vcs0   |internal0  | Okay | FAIL |
|vcs1   |internal0  | Okay | Okay |
|vcs2   |internal0  | Okay | Okay |

| HTTP port 80|
+-+
|Host1  |Host2  | IPv4 | IPv6 |
+-+
|mgt0   |download0  | Okay | FAIL |
|mgt0   |vcs0   | Okay | FAIL |
|mgt0   |vcs1   | Okay | FAIL |
+-+
|external   |download   | Okay | FAIL |
|external   |download0  | Okay | FAIL |
|external   |git| Okay | Okay |
|external   |vcs0   | Okay | Okay |
|external   |cvs| Okay | Okay |
|external   |vcs1   | Okay | Okay |
|external   |savannah   | Okay | Okay |
|external   |frontend2  | Okay | Okay |



Re: [Savannah-hackers-public] Install a post-receive hook in the GNU Emacs repository

2022-07-13 Thread Bob Proulx
Gregory Heytings wrote:
> Bob Proulx wrote:
> > I had written that first while loop some years ago for a different
> > multiple hook case.  I therefore just reached for it and grabbed it for
> > this task.
> >
> > But that doesn't mean that it can't do something different.
>
> The main reason was to minimize the network load on a (IIUC) already loaded
> server.

The CI server or the VCS server?  (Rhetorical question!  It was just
ambiguous though.)

> And I see that the git_multimail.py script also handles the
> multiple lines case by itself.  So I would suggest something like:

Ah!  That's convenient.

[[ Inconveniently I notice that git-multimail.py is a python2 script.
Making me hope that the upstream version of it has been ported to
python3 as we don't need yet another obstacle to upgrading. ]]

> #!/bin/sh
> TMP=$(mktemp -u)
> cat - > $TMP
> cat $TMP | hooks/post-receive-git_multimail-emacs
> cat $TMP | hooks/post-receive-ci.heytings.org
> rm -f $TMP

Note that the reason the option is -u is because it is an unsafe
option.  :-) It is definitely not desired in this type of context.

Also if we add temporary files then we would need to add signal
handling too.  Temporary files and signal handling is all boilerplate
stuff.  But that would make things significantly larger and over the
point where it would need a copyright statement added.

But let's crank up the tunes of Pink Floyd and We Don't Need No
Temporary Files, We Don't Need No Signal Handling here.  :-)

I installed this latest iteration to the post-receive hook script,
leaving the others as previously noted.

lines=$(cat)
printf "%s\n" "$lines" | ./hooks/post-receive-git_multimail-emacs
printf "%s\n" "$lines" | ./hooks/post-receive-ci.heytings.org

I think that should do it.  Once again please let me know how I
screwed things up with this iteration.  :-)

Bob

P.S. Notes.

"lines=...anything..." is internal to the shell and therefore doesn't
require quoting since it is never expanded.  So things like
lines=$(cat) are more efficient without quoting avoiding a temporary
string assignment.

Using $(...) always strips the trailing newline from the string.  But
internal newlines are preserved.

printf "%s\n" "$lines" will print the internal newlines in the string
verbatim and then add on a newline on the end.  One newline was
stripped and one added so things are back to zero sum total.

The size of this in memory is small.  For example pushing two refs in
my testing generates two lines.  Therefore reading that into memory
for that short script should be sufficiently small.

9e430f9fb95cc519d86b65d4118602687a2d06f1 
1faf76545d46c9b2c6ff35d3eb5460aebcd82194 refs/heads/branch1
331b8a7d65b9af08af6493e922a4492ed9be8ad4 
811b125eaddc26dada30c9abc8eda064328a7bee refs/heads/branch2



Re: [Savannah-hackers-public] Install a post-receive hook in the GNU Emacs repository

2022-07-12 Thread Bob Proulx
Gregory Heytings wrote:
> That looks good, except for two things:
>
> (1) the hooks/post-receive-ci.heytings.org must be made executable, and

Drat!  Sorry.  Fixed now.

> (2) given that only one oldrev/newrev/refname triplet is passed to the

I had written that first while loop some years ago for a different
multiple hook case.  I therefore just reached for it and grabbed it
for this task.

But that doesn't mean that it can't do something different.  I am
distracted right now be being in an in-real-life meetup.  I fixed the
executable mode now because that doesn't take any thought but I am
sure we can feed all of them to your CI script all at once.  And
actually I can imagine that you would want that because otherwise it
would get multiple events in a stream.  I think.

Bob



Re: [Savannah-hackers-public] Install a post-receive hook in the GNU Emacs repository

2022-07-12 Thread Bob Proulx
Gregory Heytings wrote:
> We're in the process of installing a CI system for GNU Emacs.
>
> Could you please add the following post-receive hook in the Emacs
> repository?

Sure.  Done.

> #!/bin/bash
> while read line; do lines="${lines}${line};;"; done
> { timeout -s 9 -k 5 10 wget -q -O- --post-data "$lines" 
> https://emacs-ci.heytings.org/ ; } &> /dev/null
>
> Of course, if the Emacs repository already contains a post-receive hook, it
> might be necessary to adapt the above lines.

Of course there is already a post-receive hook that sends out commit
diffs for emacs.  This needed adapting.

I moved the existing git-multimail hook to a new name.

mv post-receive post-receive-git_multimail-emacs

Then installed this script to drive multiple hooks.

#!/bin/sh

# 2022-07-12 -- rwp
# CI Build hook requested by Gregory Heytings.
# 
https://lists.gnu.org/archive/html/savannah-hackers-public/2022-07/msg1.html

while read oldrev newrev refname; do
echo $oldrev $newrev $refname | hooks/post-receive-git_multimail-emacs
echo $oldrev $newrev $refname | hooks/post-receive-ci.heytings.org
done

I installed this hook script to trigger the CI build.  We don't like
discarding errors to /dev/null though.  And there was no need to
depend upon the heavier bash process so I lightened things up somewhat
and converted it to portable idiomatic form.  Also the git server is
often a very loaded system.  Short timeouts can be counter productive.
I lengthened them somewhat to account for this.

#!/bin/sh

# 2022-07-12 -- rwp
# CI Build hook requested by Gregory Heytings.
# 
https://lists.gnu.org/archive/html/savannah-hackers-public/2022-07/msg1.html

lines=""
while IFS= read -r line; do
lines="${lines}${line};;"
done

timeout --kill-after=30s 60s wget -q -O/dev/null --post-data "$lines" 
https://emacs-ci.heytings.org/

Let me know if I missed getting something right.  It isn't trivial to
test these other than by live action use.

Note that one can always see the current state of the hooks directory
for any of the hosted projects.  Here is the current emacs state,
which has gathered up some significant lint over the years.

$ rsync git.savannah.gnu.org::git/emacs.git/hooks/
drwxrwsr-x  4,096 2022/07/12 15:08:11 .
-rwxrwxr-x452 2014/11/12 14:56:01 applypatch-msg.sample
-rwxrwxr-x896 2014/11/12 14:55:59 commit-msg.sample
-rwxr-xr-x 91,100 2014/12/01 00:48:54 git_multimail.py
-rwxr-xr-x 90,557 2014/11/25 01:10:36 git_multimail.py.20141125
-rwxr-xr-x 90,808 2014/11/25 17:12:29 git_multimail.py.20141127
-rwxr-xr-x 90,768 2014/11/18 00:52:40 git_multimail.py.ORIG
-rwxrwxr-x160 2014/11/12 14:55:59 post-commit.sample
-rwxr-xr-x350 2022/07/12 15:08:11 post-receive
-rw-r--r--328 2022/07/12 15:08:09 post-receive-ci.heytings.org
-rwxr-xr-x  1,677 2017/01/30 17:27:41 
post-receive-git_multimail-emacs
-rwxr-xr-x  1,567 2014/11/13 14:28:01 post-receive.20141116
-rwxr-xr-x  1,523 2014/11/18 00:25:37 post-receive.20141117
-rwxr-xr-x  1,682 2014/11/18 10:16:42 post-receive.20141125
-rwxrwxr-x552 2014/11/12 14:56:02 post-receive.sample
-rwxr-xr-x 38 2014/11/12 14:56:38 post-update
-rwxrwxr-x189 2014/11/12 14:56:03 post-update.sample
-rwxrwxr-x398 2014/11/12 14:56:00 pre-applypatch.sample
-rwxrwxr-x  1,578 2014/11/12 14:55:31 pre-commit.sample
-rwxrwxr-x  4,951 2014/11/12 14:56:03 pre-rebase.sample
-rwxrwxr-x  1,239 2014/11/12 14:56:05 prepare-commit-msg.sample
-rwxrwxr-x  3,611 2014/11/12 14:56:04 update.sample

Bob



Re: [Savannah-hackers-public] Add information about downloading Git snapshots

2022-05-06 Thread Bob Proulx
Eli Zaretskii wrote:
> Can this information please be added to the appropriate page of each
> project?

Honestly this worries me a because I recall that creating those
snapshots puts a high load on the server.  Right now a few people do
it and it isn't so bad.  But if it is advertised as a feature then it
is likely that 10 or more people will do it and that many in parallel
will brown out the system.

This is operating from my memory of having the system brown out before
and finding that it was cgit on the fly git tar bundle creation.  That
was a previous version of git and cgit.  We have upgraded since then.
Also I remember we were short on disk at that time and those snapshots
would sometimes run the disk to 100%.  Since then we have expanded the
amount of disk space available and space should not be a problem.

Before we advertise this feature for wide use I would like to try to
stress test doing this first.  At a time that someone can be looking
at the system load at the same time.  Because it worries me.  It has
browned out the system before.  But time has passed and I don't know
if it is a problem now or not.

The best thing is to do a stress test benchmark and understand how
much cpu, memory, disk I/O bandwidth, this uses on the system.  Make a
determination as to how many simultaneous git tar bundle creations can
be one.  Crank up the number of parallel creation tasks and see how
many we can do before we brown out the system.  And then decide if
that many is something that the system can handle or not.  Certainly
if the data shows it to be okay then it is not a problem.

Bob



[Savannah-hackers-public] OpenSSH updated on vcs0 and download0

2022-03-23 Thread Bob Proulx
Savannah Hackers,

I upgraded vcs0 today to the next newer OpenSSH version.  This means
that all of the client facing ssh member access systems have all been
upgraded past the SHA1 obsolescence point that previously required
either upgrading to ed25519 keys or using the following workaround.

HostkeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa

The above should no longer be needed for standard member ssh access to
the services.  It is still useful for mgt0 for the time being as it is
still running the older OpenSSH release.  But mgt0 is not member
facing.  Only Savannah Hackers, sysadmin, and backup, accesses it.

I still recommend upgrading ssh keys to ed25519 which offers better
security with faster performance using a more compact key.  That
avoids these problems and I think will eventually be required.

Bob



Re: [Savannah-hackers-public] scp asks for password on savannah nongnu.org

2022-02-23 Thread Bob Proulx
Joël Krähemann wrote:
> `scp` asks for password while connecting, I think this shouldn't happen.

Here it is two days later that I am seeing this email to the mailing
list.  But Joël and I chatted on IRC at the time and I referenced our
documentation here.

https://savannah.gnu.org/maintenance/SshAccess/

In particular I suggested mitigation number 2 from there.  Create and
register an ssh ed25519 key.  Which Joël did at that time.  And then
was able to access the download server and post a new release file
bundle there.

Bob



Re: [Savannah-hackers-public] git.savannah.gnu.org is not able to be connected with git protocol

2022-02-20 Thread Bob Proulx
antique refer wrote:
> git.savannah.gnu.org is not able to be connected with git protocol.
>
> # git clone git://git.savannah.gnu.org/gnulib.git
> Cloning into 'gnulib'...
> fatal: read error: Connection reset by peer

This works for me.  Which leads me to think that it is a firewall in
your environment which is blocking access to git port 9418 tcp.

Try using a different system in a different location (home, another
office, a coffee shop) and I am sure it will work.

> Can you help us to resolve this?

Easiest is Ineiev's suggestion of using https protocol.  Note that the
https path is slightly different and has a another "/git" path segment.

Bob



[Savannah-hackers-public] Network Outage Today

2022-02-01 Thread Bob Proulx
This morning there was a long network outage which separated the VMs
from their root file system holding the OS files.  This started at
8:18am US/Eastern time and seemed to mostly conclude about 9:38am
US/Eastern time.

That's a long time for a system to be without a root file system.  The
system times out trying to load code and data.  Some daemons die.
Others get killed.

This last happened 41 days ago.  The recovery including rebooting all
of the systems so that they all get reset to a known working state.

Thanks to Amin Bandali for handling the issue this morning with the
git server vcs2 and rebooting it.

I am walking through all of the other VMs, peeking at their state, and
rebooting all of them to make sure they are reset to a good known
working state.  (This includes debbugs which isn't Savannah but I
happen to be active there too.)

I suggest that all VM owners reboot their VMs in order to ensure that
they have no problems.

I note that right now 5pm US/Eastern time that Mailman on lists is now
sending out its daily housekeeping messages that normally are sent out
at 5am so things there are acting 12 hours delayed.  Seems like a wide
spread issue.

Just trying to communicate the news of the day.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Update on upgrades

2022-01-29 Thread Bob Proulx
Bob Proulx wrote:
> I have been banging my head against the https/http WebDAV
> configuration.  That's all Apache using libapache2-mod-svn and seems
> like it should be working but I haven't been able to make it happy
> yet.  I'll keep working on it.  The smallest of details can cause the
> biggest problems.  Once that is fixed then Subversion can move to the
> new OS and ssh for svn will also be good to go for SHA1 obsolescence.

I have Subversion working with the new OS version now.  And it was
basically the typical problem of small details causing big problems.
But have it all worked through now.

PASS: svn-http
PASS: svn-https
PASS: svn-rsync
PASS: svn-ssh
PASS: svn-svn
PASS: svn-web

I am not sure is the best time to switch the DNS over from the old to
the new.  I have reduced the TTL time for svn records so that these
can be switched around with a shorter cache timeout.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] frontend-dev?? Can it be shutdown?

2022-01-29 Thread Bob Proulx
Ineiev wrote:
> Bob Proulx wrote:
> > Can we shutdown frontend-dev and release those resources?  I don't
> > think we are using it for anything now.  Are you using it?
> 
> No; I test changes in frontend code either locally or on current
> frontend VM (it has a dedicated Savane instance for testing).

Sounds good.  I will have those resources reclaimed with the next new
VM request.  Thanks!

Bob


signature.asc
Description: PGP signature


[Savannah-hackers-public] frontend-dev?? Can it be shutdown?

2022-01-28 Thread Bob Proulx
Ineiev,

Can we shutdown frontend-dev and release those resources?  I don't
think we are using it for anything now.  Are you using it?  If not I
would like to shut it down and return the VM resources to the pool.

Thanks!
Bob


signature.asc
Description: PGP signature


[Savannah-hackers-public] Update on upgrades

2022-01-28 Thread Bob Proulx
Savannah Hackers,

Just a very short status update note.

I have been focusing on migrating services to the new vcs2 which is
the up to date Trisquel 9 image.  It includes the newer OpenSSH and
avoids the SHA1 deprecation problem with ssh that members committing
vcs possibly hit if they have the newest OpenSSH.

Git has been transferred to the new server late December.  A few
initial forgotten items but worked out asap and hopefully soon
enough.  All seems good.

One problem report outstanding from Paul Eggert about ssh connections
hanging around that is as yet not understood.

I have been subsequently working on getting Subversion over on the new
systems.  The regression test suite of external services is the
easiest summary.

FAIL: svn-http
FAIL: svn-https
PASS: svn-rsync
PASS: svn-ssh
PASS: svn-svn
PASS: svn-web

The svn:// protocol through xinetd for anonymous checkouts is working.
The authenticated ssh for member access (same as for git) and commits
is working.  The ViewVC web interface is working.  The raw repository
rsync access (trivial really) is of course working.

I have been banging my head against the https/http WebDAV
configuration.  That's all Apache using libapache2-mod-svn and seems
like it should be working but I haven't been able to make it happy
yet.  I'll keep working on it.  The smallest of details can cause the
biggest problems.  Once that is fixed then Subversion can move to the
new OS and ssh for svn will also be good to go for SHA1 obsolescence.

After that I would start on hg and bzr.  Also download which also uses
member ssh access.

I also should work on frontend too.  I actually don't think it would
be as much effort as the vcs systems.  It was the SHA1 obsolescence
that focused my efforts there.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] What is the size of the nongnu directory?

2022-01-24 Thread Bob Proulx
Thérèse Godefroy wrote:
> The sizes were last updated on Feb. 25, 2017. So the average growth rate of
> nongnu is around 2 GB/year.

Cool! :-)

Bob



Re: [Savannah-hackers-public] What is the size of the nongnu directory?

2022-01-24 Thread Bob Proulx
Hello Thérèse,

Thérèse Godefroy wrote:
> We are trying to update directory sizes in the mirroring guide [0].
> Thanks to Anton McClure and Ian Kelling (RT #1784571), we have current
> data for gnu and gnu-alpha. Could you tell us the size of the nongnu
> directory? Thanks in advance.

The Savannah download directory is 64,092,044 bytes (about 62 GB) in size.

That contains both the gnu and nongnu directories in the shared file
system.

If you know when that page was previously updated to say 51 GB then
knowing the size now has grown by about 11 GB then we could make a
prediction about the average rate of growth per year for it.

Bob



Re: [Savannah-hackers-public] Error en https://savannah.gnu.org/register/

2022-01-07 Thread Bob Proulx
This appears to have been BCC'd to savannah-hackers-public so I make a
reply to ensure that follow-ups go here instead of elsewhere.  And
also to offer a translation.

In the Spanish version of https://savannah.gnu.org/register/

In line 309 of the html it says:

"My balls include a copy of the license"

This is clearly a bug, it should say:

"My project includes a copy of the license"

This error was already reported by a colleague of mine a few months ago, 
but still
keep going.

It must be that someone is amused.

The English for that line was:

My tarball includes a copy of the license

I tend to think that "tarball" isn't a very good word to use in any
other than casual conversation.  My preference would likely be to say
"file bundle" as a generic statement instead.

If this was previously reported I do not know where that might have
been.  I think perhaps it wasn't actually reported but just mentioned
somewhere.

Bob

GNU Hacker wrote:
>
> En la version español de https://savannah.gnu.org/register/
>
> En la linea 309 del html dice:
>
> "Mis pelotas incluyen una copia de la licencia"
>
> Esto es claramente un error, deberia decir:
>
> "Mi proyecto incluye una copia de la licencia"
>
>
>
>
> Este error ya fue reportado por un colega mio hace unos meses, pero aun
> continua.
>
> Debe ser que a alguien le divierte.



Re: [Savannah-hackers-public] Git on Savannah rejects git:// protocol?

2021-12-31 Thread Bob Proulx
Eli Zaretskii wrote:
> I get this:
> 
>   ~/test-clone$ git clone git://git.savannah.gnu.org/git/emacs.git
>   Cloning into 'emacs'...
>   fatal: unable to connect to git.savannah.gnu.org:
>   git.savannah.gnu.org[0: 2001:470:142::168]: errno=Connection refused
>   git.savannah.gnu.org[1: 209.51.188.168]: errno=Connection refused

If connections are being refused then the git-daemon is unable to
service incoming on port 9418 for some reason.  Most likely the system
is overloaded.

> But:
> 
>   ~/test-clone$ git clone https://git.savannah.gnu.org/git/emacs.git
>   Cloning into 'emacs'...
>   remote: Counting objects: 968472, done.
>   remote: Compressing objects: 100% (177822/177822), done.
>   remote: Total 968472 (delta 795603), reused 960310 (delta 789344)
>   Receiving objects: 100% (968472/968472), 349.50 MiB | 7.58 MiB/s, done.
>   Resolving deltas: 100% (795603/795603), done.
>   Checking out files: 100% (4575/4575), done.

Nginx is a pre-forking daemon.  git-daemon is not.  (Or at least not
as configured if that is possible to do so.)

> Does this mean Savannah no longer supports the git:// protocol?

Definitely not.  And git:// is working great now, several hours later
after your problem report.

> Or is this an unintended consequence of some change in the
> configuration lately?

Possibly.  Or possibly it would have happened on the old server too.
Don't know.  Because it is a combination of the server and the
interaction with the Internet.  The first is at least knowable but the
second is always changing and often hostile.

Another thing that I note is that for whatever reason there seems to
be more Internet abuse happening in the hours from 3am to 6am
(local'ish time, the FSF admins are US/Eastern timezone and I am
US/Mountain timezone two hours later) than at other times of the day.
This is subjective but it always seems like problem reports of
problems will be occurring at that time of day more so than during US
daylight hours.

This has a more proportionate effect on our overseas free software
friends who operate mostly during that time.  Other than being awake
during that time myself it makes it hard to diagnose what happens then
when it recovers and is fine some hours later.  I have this belief
that more fail2ban automation to detect abuse and to automatically
mitigate it is needed.

Hmm...  Did I port over all of the fail2ban and other dynamic
mitigation from the old server to the new?  I did not!  That is almost
certainly a contributor to the problem seen last night.  It's running
in a default configuration and not our very customized one.

I'll read and respond to your next message.

Bob



[Savannah-hackers-public] [task #16101] Python warnings on push and groff-commit list not getting mails

2021-12-30 Thread Bob Proulx
Follow-up Comment #7, task #16101 (project administration):

Yay!  Thanks for helping! :-)

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




[Savannah-hackers-public] [task #16101] Python warnings on push and groff-commit list not getting mails

2021-12-29 Thread Bob Proulx
Follow-up Comment #5, task #16101 (project administration):

I think maybe it is fixed.  Maybe.  The problem is that the hook was a symlink
to a shared /usr/src/git_multimail/... directory and that directory did not
exist on the new server.  I had missed it in the setup.

I don't have an easy way to test this but once again please let me know if
this is still not working. :-(


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




Re: [Savannah-hackers-public] Request ability to push non-fast-forward commits

2021-12-29 Thread Bob Proulx
Michael Siegel wrote:
> I was able to clean things up for the repository's mirror at
> 
>   https://notabug.org/dragora/dragora-website
> 
> Now, unfortunately, I didn't know that non-fast-forward commits weren't
> allowed on Savannah.

That's the GNU hosting policy for community hosted software.

> So, I would like to request temporary ability to make such commits in
> order to clean up the log. Virtually no code would be lost in the
> process.

Against my better judgement I opened dragora-website.git up for
non-fastforward pushes.  So you can get them in sync again.

But I must advise you that regardless of other things that every time
a non-fastforward commit is made it breaks every downstream mirror of
the repository at that point.  That's one of the reasons its not good
for the main trunk.  I do realize there are some "next" and "pu" style
branching where rewinding is expected.

But this is the web site.  And you asked very nicely with the right
magic words saying it was not losing code.  :-)

Bob



[Savannah-hackers-public] [task #16101] Python warnings on push and groff-commit list not getting mails

2021-12-29 Thread Bob Proulx
Follow-up Comment #4, task #16101 (project administration):

Drat!  But that is at least a problem I understand and can fix.  I'll get it
sorted out.  Sorry for the trouble.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




[Savannah-hackers-public] [task #16101] Python warnings on push and groff-commit list not getting mails

2021-12-27 Thread Bob Proulx
Update of task #16101 (project administration):

  Status:None => Done   
 Assigned to:None => rwp
 Open/Closed:Open => Closed 

___

Follow-up Comment #2:

Thanks for the problem report.  Regardless of the bug tracker confusion.  I
think if we were to try to change it that it would only close this one and
open another one in the other tracker.

When you started noticing the problem was when the server handling the git
service was switched from the old server to the new one.  It's now running on
a fully updated Trisquel 9 LTS system.  This resolves other problems such as
the SHA1 obsolescence in newer OpenSSH releases.

But the new server has a newer version of Python.  That's the problem seen
here.

root@vcs2:~# file /net/vcs/git/groff.git/hooks/git_multimail.pyc
/net/vcs/git/groff.git/hooks/git_multimail.pyc: python 2.6 byte-compiled

root@vcs2:~# ll /net/vcs/git/groff.git/hooks/git_multimail.pyc
-rw-r--r-- 1 root root  79898 Jan 29  2014
/net/vcs/git/groff.git/hooks/git_multimail.pyc

I am not really a python person so it is a mystery to me why that file was
byte-compiled.  It sits in a hooks directory that has no write permissions
from anyone other than root.  And the file was owned by root.  Therefore
someone intentionally byte compiled that file.  Have no idea why.  As it is
definitely undesirable in such a situation.

I renamed it out of the way.  Won't Python work okay without having that file
byte-compiled?  Because then it won't depend upon the specific version of
Python that happens to be installed.  And certainly there will continue to be
upgrades in the future.

In addition to the above file I found the following files too.  I similarly
moved them all out of the way as well.  Because I assume that each of them
would have the same problem.

root@vcs2:/net/vcs/git# find . -name '*.pyc' -ls
-rw-r--r--   1 ksterker adonthell90447 Jun 22  2017
./adonthell/adonthell-wastesedge.git/hooks/git_multimail.pyc
-rw-r--r--   1 karl automake140163 Jan  6  2020
./automake.git/hooks/git_multimail.pyc
-rw-r--r--   1 root root 85815 Apr  6  2019
./emacs/elpa.git/hooks/git_multimail.pyc

Thanks again for reporting this problem.  It wasn't something I had known
about or even thought about.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




Re: [Savannah-hackers-public] Suspicious ‘451-You dont seem to have a reverse dns entry.’

2021-11-20 Thread Bob Proulx
Tobias Geerinckx-Rice wrote:
> First of all, thanks for the lengthy response.  I could only skim it. 

I do sometimes get that response.  :-)

> Bob Proulx 写道:
> > Since it is reported as being greylisted this should have been a
> > single one time delay for the greylist and then all subsequent
> > interactions would remember your address and avoid the delay.
> 
> This server has delivered a ludicrous number of messages to debbugs over the
> years :-) including several only yesterday (19 Nov).  Of course, the TTL
> might be something like 24h, without being itself unreasonable.

But you only showed exactly one of the messages.  You might follow-up
to sysadmin with three or five from different days to show that the
greylisting is not working.

And that greylisting is on top of the issue not resolved about FCrDNS
which would have avoided the greylist delay.

In my mind the above ordering of fixes is most desirable because it
fixes the most things.  First the greylisting which should work.  Then
the FCrDNS which should avoid it.

Just as a by-the-by comment too is that likely a workaround would be
to force IPv4 for this transport.  At a guess.  But better to get
those two problems above fixed.

Good luck!
Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Suspicious ‘451-You dont seem to have a reverse dns entry.’

2021-11-20 Thread Bob Proulx
I removed sysadmin from the recipient list so that each of these would
not create an additional ticket in their system.

Tobias Geerinckx-Rice wrote:
> I noticed the following response in my mail logs:

That word wrapped.  Let me reconstruct it for readability.

Nov 20 14:25:26 localhost smtpd[602]: 8be5d3679f9641dc mta delivery 
evpid=b7cb816f882ca1b8 from= to=<51...@debbugs.gnu.org> 
rcpt=<-> source="80.241.217.52" relay="209.51.188.43 (debbugs.gnu.org)" 
delay=3s result="Ok" stat="250 OK id=1moRJ1-0007Au-9p"

That message went to debbugs.  That does not happen to be a Savannah
system.  For debbugs help please write to help-debb...@gnu.org where
the sysadmins for debbugs hang out.  But the above shows that the mail
was delivered okay.  It's happy.  The mail went through.  I think it
is this one in the logs.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51512#20

But note that the IP address used for debbugs was 80.241.217.52 and
the below was 2a02:c205:2020:6054::1 which is a big difference.  For
debbugs this is because it only advertises an IPv4 address.

Nov 20 14:25:28 localhost smtpd[602]: 8be5d3680647 mta delivery 
evpid=b7cb816fe63e0547 from= to= rcpt=<-> 
source="[2a02:c205:2020:6054::1]" relay="[2001:470:142:3::10] (eggs.gnu.org)" 
delay=5s result="TempFail" stat="451-You dont seem to have a reverse dns entry. 
Come back later. You aregreylisted for 20 minutes. See 
http://www.fsf.org/about/systems/greylistingfor more information."

That message went to eggs the inbound MX relay for @gnu.org email.
That does not happen to be a Savannah system.  You wrote to sysadmin
in your message and they are the correct admins to look at that
problem.  They are the only ones who can.

Nov 20 15:25:30 localhost dovecot: imap-login: Disconnected: Connection 
closed (no auth attempts in 0 secs): user=<>, 
rip=2a01:4f8:251:53df:0:242:ac11:e, lip=2a02:c205:2020:6054::1:993, TLSv1.3 
with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
Nov 20 15:25:31 localhost last message repeated 2 times

Two unrelated messages of a seemingly normal nature.

> I might be missing an obvious, but that claim looks bogus from here:

I think you are referring to the "451-You dont seem to have a reverse
dns entry. Come back later. You aregreylisted for 20 minutes. See
http://www.fsf.org/about/systems/greylistingfor more information."
part.  That "aregreylisted" smash together seems like something bad
happened assembling the report.

It does look like something on eggs failed to resolve
2a02:c205:2020:6054::1 because I can resolve it here.  Looks okay.
And the forward-reverse loop through there to tobias.gr and back to
the address looks okay.  Personally I prefer to see real hostnames
there rather than the domain name but either are acceptable.

No idea here of course and the only ones that can look into the
problem are sysadmin.

> IPv4 seems equally in order, although the log mentions only IPv6.

The OS prefers IPv6 first and IPv4 second.  Which means that if an
IPv6 address is published and then if an IPv6 connection can be made
then that is what is done.  If either of those two things fail then
IPv4 is tried second.  This is configurable on a system by system
basis but IPv6 is the future and the future is now.

> I'd love to get rid of this >= 20-minute delay.  Please let me know if
> there's anything more I can do!

Since it is reported as being greylisted this should have been a
single one time delay for the greylist and then all subsequent
interactions would remember your address and avoid the delay.  If it
were me I would need to produce two log entries showing that different
messages more than 20 minutes apart with the same IP address were both
greylisted in order to show that the greylisting was not operating
correctly.

Good luck!
Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] List of available SSH cipher types?

2021-10-10 Thread Bob Proulx
Andrew Engelbrecht wrote:
> Someone said they were having trouble ssh'ing to Savannah, and course
> they're an Arch user, so likely using SSH 8.8. ; )

Agreed.  Very likely.

> They did apply the +ssh-rsa trick, but for some reason Savannah
> wasn't accepting their key that had been working for a while
> already.

Likely that was actually a different problem.  Since the workaround
for it did not work.  Or perhaps the workaround was not correctly
applied.  For example have had one case already where misunderstanding
of the "old-host" example name caused the user to use that placeholder
string literally instead of using the actual hostname.

> They said that once they created an ED25519 key, the could log in.

Though undocumented in the OpenSSH 8.8 release notes it seems likely
that using an ED25519 user key also enables using an ED25519 host key
and thereby avoiding the SHA-1 algorithm in the ssh-rsa host key which
is otherwise used by default.  There have been several reports that
upgrading to ED25519 user keys works.

Upgrading to an ED25519 user key is definitely a good upgrade all
around.  I think we should be recommending that for people who wish to
move forward.

[[ I still don't have an OpenSSH 8.8 client system of my own to try
experiments with and therefore am just working based upon reports from
others. ]]

> It's possible that their SSH authorized keys list on Savannah was
> changed at some point, and they forgot?

Historically users have had a variety of problems.  There are an
infinite number of ways for things to fail.  But there is only one way
for things to work correctly.  Trying to guess why something has
failed without any information is a gamble at best.

Other users have successfully applied the +ssh-rsa workaround and it
has worked.  The release notes document it.  If that did not work then
the problem must be something else.

> In any case, they requested that we update the following page with info
> about acceptable ciphers:
>
> https://savannah.gnu.org/maintenance/SshAccess/

Thanks for the nudge to do this.  I have updated that page with
information concerning this issue.

> I don't think it's super urgent, but it might be nice to add a list to that
> page. I hope that I sent this to the right list. I'm likely not subscribed,
> so please CC me on any replies.

OpenSSH does not make this information trivially available to the
user!  And I should just stop the email here but...  You asked!  And
so here is actually a way to get this information.  :-)

I would "ssh -vv git.savannah.gnu.org" and then look through the
verbose information provided there.  That's always going to be the
correct information about what is happening.  That going to be the
easier way to figure out what is happening.  And it is mostly
incomprehensible to mere mortals reading it.  For example.

rwp@angst:~$ ssh -vv git.savannah.gnu.org

debug2: local client KEXINIT proposal
debug2: KEX algorithms: 
curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-cdebug2:
 host key algorithms: 
ssh-rsa-cert-...@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521-cert-...@openssh.com,ssh-ed25519-cert-...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: 
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: 
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: 
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: 
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,z...@openssh.com,zlib
debug2: compression stoc: none,z...@openssh.com,zlib

debug2: peer server KEXINIT proposal
debug2: KEX algorithms: 
curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellma$-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: 

Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-27 Thread Bob Proulx
Bob Proulx wrote:
> Daniel Katz wrote:
> > I've created a sv account (with ssh and gpg) and submitted a new project
> > request. My user name is daniel_k and the project's short name is
> > fsftest. Please enable the project.
> 
> The reason we needed the account is so that we could set you up with a
> login on the Savannah systems using the registered ssh keys.  And the
> gpg keys is needed for account recovery.
> 
> I'll go create an account for you using those keys and then send you
> instructions on using the system.

I see that Ian has already created accounts for you.  He did it in a
different way from the way we normally set up Savannah admins.  That's
fine.  I'll just note that your account is uniquely configured
different from other admin accounts.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-27 Thread Bob Proulx
Ian Kelling wrote:
> Thanks. I'm going to create a user daniel-k on the savannah machines so
> daniel can setup development copies of things based on that link.

Hmm...  It's fine but I see that daniel-k has been set up completely
uniquely different from the way most of our admins are configured on
the systems.

For future reference normally we set up individual logins on the mgt0
system.  Then add users to appropriate privilege group from the
selection of adm, staff, svadm, svstaff, sudo.  This allows people the
associated access to the systems.  For example svadm gives full
read-only access everywhere, svstaff gives access to /usr/local
everywhere, sudo gives root superuser access everywhere.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-27 Thread Bob Proulx
Karl Berry wrote:
> Ok, seems like we should setup a development instance of Savane, at
> least the relevant parts. Does that sound right?
> 
> FWIW, I found it impossible in practice to set up a usable development
> instance of Savannah (or Savane). Thus, when I wanted to do experiments
> like this, I merely copied the php file in the live directory (on
> frontend) to a different name (e.g., editgroupkarl.php) and then worked
> on that.

This is exactly what I do as well.  I always develop "on the side" of
the existing scripts.  Then commit them into place once they are ready
to be committed to version control.

> Otherwise I could have spent my whole life trying to get the development
> savannah working, and never made any progress on the actual
> problem. Adding in the FSF connection (precisely one of the places where
> "Savane" and "Savannah" differ, BTW, as I understood it) and it sounds
> even more problematic to get a development instance working.

The database content problem is the main issue.

> On the other hand, if you want to try, Assaf (Gordon) put a lot of
> effort into setting up a development instance, and (afaik) it did work
> for him. He documented his process here:
> https://savannah.gnu.org/maintenance/FrontEndDevelopmentSite

Yep!

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-27 Thread Bob Proulx
Ian Kelling wrote:
> Ok, seems like we should setup a development instance of Savane, at
> least the relevant parts. Does that sound right?

Yes.  A while back rms and I were chatting about things and he had
suggested that we set up something pre-seeded with whatever is
needed.  Because setting up a local sandbox is definitely not
trivial.  Here are Assaf's instructions.

https://savannah.gnu.org/maintenance/RunningSavaneLocally/

The biggest problem is that the site requires the database to be
pre-seeded with a bunch of stuff that is not completely known what is
needed.  If you start with an empty database then the site is
basically not functional.  Therefore rms had put in a request that we
set up a development site.

Previously we had been using frontend-dev as a partial way along that
way.  But that system interacts with the live production database.
And that is where the problems and danger lies.

I couldn't bring myself to ask for yet another VM until some of the
previous work on the existing VMs were completed.  But in the long run
we would need a new development system.  The old frontend-dev is at
the end of its life.  Just need to confirm with Ineiev.  Then we could
drop it and create a new one for that purpose.

For the record I don't like the RunningSavaneLocally path completely
because that uses a sanitized version of the production database.  I
think that cleaning personal account information is always too risky.
Instead I think we need to create a new database from scratch without
using any personal account information.  Using fake data.  There is a
long trend to use faked data for web site development.  Setting that
up would be a great project to help out Savannah development.

Bob



Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-27 Thread Bob Proulx
Hi Daniel,

Daniel Katz wrote:
> I've created a sv account (with ssh and gpg) and submitted a new project
> request. My user name is daniel_k and the project's short name is
> fsftest. Please enable the project.

I don't understand the need for a new project request.  And I am not
trained up myself on approving new projects.  I'll leave that for
others.

The reason we needed the account is so that we could set you up with a
login on the Savannah systems using the registered ssh keys.  And the
gpg keys is needed for account recovery.

I'll go create an account for you using those keys and then send you
instructions on using the system.

> Right now, most of the work for enabling git and other non-CVS web
> updates is done on the gnu.org side of things so I want to have a repo
> on Savannah to do further testing and development.

Well...  As previously noted that is the easy part of things!  It's
the web UI part that is where most of the effort lies.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Hi Daniel,

Daniel Katz wrote:
> I'm currently a FSF tech intern.

I don't see an account for you on Savannah.  Please register an
account on Savannah.  Then please upload both an ssh rsa key and a gpg
key.

At this moment please use either an ssh rsa key or both an ssh rsa key
*and* an ssh ed25519 key.  Not everything fully supports ed25519 yet.

> My next project is likely going to be allowing the sub pages of
> gnu.org/software/ to be updated from a Git repo. Savannah has
> web.cvs.savannah.gnu.org/ for this functionality right now, but only
> from CVS based repos. How would Savannah maintainers set up
> something similar for Git?

I would start by adding a selectable feature to the web UI.  It should
allow selecting from the available list of supported version control
systems to include cvs, git, hg, svn.  The associated database table
will need to be altered to hold this new data.


https://git.savannah.nongnu.org/cgit/administration/savane.git/tree/frontend/php/project/admin/editgroupfeatures.php

Then the processes that create repositories will need to be updated to
recognize this new data field and to create the appropriate new
repository based upon the selection.

Then the process that configures the hook scripts will need to be
updated for the same so that the hook scripts are updated
appropriately to communicate to wildebeest's receiving end to update
from the correct vcs system.

> I was thinking something similar to
> what's in place for CVS (web.git.savannah.gnu.org), or each GNU
> project having a second repo with /web appended to the name where
> the files would be kept.

I worry about namespaces.  There might be (I didn't look and don't
know) projects with a web sub project already.  Or there might be a
project with such in the future.  This seems to be eating part of the
namespace that would be better to not consume.

But perhaps this could be just another selectable feature.  Then it
could either be namespaced inside the project's git namespace.  Or it
could be separate from it.  For git there is that choice.

For the other version control systems there isn't the concept of a
sub-repository.  Therefore it will need to be available in a separate
namespace for them.

[[ I somewhat recall that we worked through the details for hg sometime
back but would need to find the communications on it in order to
refresh my memory of the state of things there.  I don't use hg myself
and therefore don't really know how to drive it. ]]

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Ian Kelling wrote:
> Website cvs repos are under the domain web.cvs.savannah.gnu.org,
> whereas non-website cvs repos are at cvs.savannah.gnu.org. Should we
> create an analogous domain: web.git.savannah.gnu.org ?

There are a lot of legacy paths to the same place that never made a
lot of sense to me.  But we have maintained them because for any
particular thing there are dozens of people using that path.

Both web.cvs.savannah.gnu.org and cvs.savannah.gnu.org are the same
system and in practice interchangeable.  I think I accidentally
dropped web.cvs at one point because I didn't know anything about it
and didn't know to preserve it and immediately people who were using
it started to file bug tickets.  Which is basically to say that I
don't know the real distinction between them at this time.

On that system there are four paths.  Two that appear to be primary
and two that appear to be aliases.

lrwxrwxrwx   1 root root7 Jun 19  2019 cvsroot -> sources
lrwxrwxrwx   1 root root   15 Jun 17  2019 sources -> cvsdata/sources
lrwxrwxrwx   1 root root   11 Jun 17  2019 web -> cvsdata/web
lrwxrwxrwx   1 root root3 Jun 18  2019 webcvs -> web

I imagine that in the earlier days when everything was cvs and
everything was on one system that /sources held all of the sources and
/web held web pages.  I infer by that the symlinks for cvsroot points
to sources that it is an alias and that webcvs points to web that it
is also an alias.  That something uses a symlink makes no difference
to the user using it.  I have just been preserving them along.

The cvsdata consolidation is my creation however.  That allows the
data to be mounted from a separate partition from the OS and not
included in the root system partition.  Those used to be directories
on the root partition and the data there.  I relocated that data onto
its own partition.

So personally I don't see a need for a new domain name.  But neither
am I opposed to a new domain name either.  Personally I would put them
in a different directory in the same domain name.

I do have some concern about namespaces and I am thinking that putting
them in a different directory from sources would be most generic.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Eli Zaretskii wrote:
> More generally, why wouldn't the above be possible to do from the
> "Select Features" page of the project on Savannah?  For example, Emacs
> has this:
> 
>   https://savannah.gnu.org/project/admin/editgroupfeatures.php?group=emacs
> 
> Adding a checkbox for "Web Pages" there would allow the project
> maintainers to specify the URL for the webpages repository by
> themselves.

That's it exactly!  It should be a selectable feature.

> As to the "create a repo called web" part: I wasn't aware that every
> GNU maintainer has access rights for doing that on Savannah.  Is that
> really so?  And if so, where can I find the instructions for
> performing that job?

It's not really so.  There is currently no self-service way to create
git sub-repositories.  It's documented as please file a ticket and a
Savannah Hacker will be happy to create one for you.  See "Creating an
additional repository" here.

https://savannah.gnu.org/maintenance/Git/

Modifying the web UI to do this self-serve would also be wonderful.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Eli Zaretskii wrote:
> Daniel Katz wrote:
> > I'm currently a FSF tech intern. My next project is likely going to be
> > allowing the sub pages of gnu.org/software/ to be updated from a Git
> > repo.
> 
> Could you tell more about what this will entail?  In particular, does
> this mean the files in gnu.org/software/ could be updated _only_ via
> Git, or both via Git and via CVS?  I think RMS asked at some point to
> keep the CVS access.

I'll just come right out and say that keeping cvs for those who don't
want to migrate is required.  And also that if we support git that we
should also be able to support svn and hg too.  Because it is all the
same problem.  There are many in the community that would never
migrate to git but would migrate to svn or hg.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Bob Proulx wrote:
> If not then many things are done through cron on an hourly or every
> half hour basis.  In which case there will be a cron action on vcs1
> which will query the database and if a repository is desired but does
> not yet exist then it will create the repository at that time.
> 
> And for web pages with cvs it is the same but on vcs2.  Everything
> about cvs is on vcs2.

Since I had not looked it up I was shifted and off by one.  Because
git is on vcs0 and cvs is on vcs1.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Ian Kelling wrote:
> How exactly do you create new git repos? How exactly do you create a
> webpages cvs repo?

Project admins would log into Savannah web site UI.  Under the
administration menu there is a "Select Features" menu.  That page has
a long list of features that may be enabled or disabled for projects.
Each of the version control systems may be enabled/disabled
individually on that page.  Submit to save any changes.

I don't recall if repositories are created immediately or not.  If
they are created immediately then the action is triggered by an ssh
from frontend1 over to vcs1 to run a script there to create the
repository.

If not then many things are done through cron on an hourly or every
half hour basis.  In which case there will be a cron action on vcs1
which will query the database and if a repository is desired but does
not yet exist then it will create the repository at that time.

And for web pages with cvs it is the same but on vcs2.  Everything
about cvs is on vcs2.

The place to start if trying to add a git web page backend would be to
modify editgroupfeatures.php to add the feature selection.  This is
really just an interface to the database.  The appropriate table would
need to be modified to add a field to hold the project's web vcs
selection.


https://git.savannah.nongnu.org/cgit/administration/savane.git/tree/frontend/php/project/admin/editgroupfeatures.php

If that were in place then the process that sets up the hook scripts
could do the right thing and trigger the correct checkout on the
backend of things.

Bob



signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Allowing git repos to automatically push to gnu.org/software/$PROJECT

2021-07-23 Thread Bob Proulx
Hello Daniel,

I will apologize to everyone here for being mostly absent from the
mailing lists the past few months.  I have been overbooked and needed
to shed load and keeping up with the mailing lists was part of the
load that got shed.  Sorry.

Karl Berry wrote:
> Project maintainers on savannah cannot create their own git repos.  They
> have send email/submit a support request.  (This is a constant thorn in
> everyone's side.)

Yes.  It would be super awesome if the web UI had an interface for
project maintainers to create sub project git repositories.  The only
reason this hasn't been a serious problem is that there really are not
that many requests for this.  And once they get created then there is
no more maintenance needed.

> I guess your proposal is intended to avoid any work being necessary on
> the Savannah side. I can understand that goal.

Yes.  But...  I think it is starting from the wrong end of things.
Certainly checking out web pages with git is the easy part of the
problem.  And if that were the only problem then one might consider
that it would already have been done at some point.  Instead the hard
part of the problem is to modify the web UI and the associated
database tables to handle project maintainers selecting their version
control system choice for web pages.  That's the bit of work that is
needed to make this happen.

> All that said, let me just say that the interface that seems cleanest to
> me would be to change new.py to take an additional option to specify the
> web repository VC type, say "-R cvs" or "-R git".

Very small note.  For curl it would probably be a form entry option as
is currently used for the other options.  So probably "-F vcs=git" or
"-F vcs=git" or "-F vcs=svn" or default to "-F vcs=cvs" or some such
to pass through.

> Then, on the savannah side, git would have to be supported as a backend
> for web repos as well as cvs (database work, UI work). On the fsf side,
> it would be a matter of running git commands instead of cvs commands,
> but following exactly the same pattern.

There are only three magic numbers: Zero, One, and Many.  Currently
there is support for One.  To change that then we need to support
Many.  Therefore some web UI work will be needed to allow project
maintainers to select their web backend system of choice.  Because in
yesteryear it was cvs.  Right now it is git.  But svn and hg people
feel just as strongly about their vcs of choice.

As it turns out I am out on holiday away from my house and the
networking here has just turned from pretty bad to really terrible.
I will have to leave this here and send it off and hope that it
transmits.

Bob


signature.asc
Description: PGP signature


[Savannah-hackers-public] [task #15922] Savannah project memberlists do not load (HTTP 500)

2021-03-24 Thread Bob Proulx
Follow-up Comment #1, task #15922 (project administration):

Ineiev, I saw that it was a typo and hacked in a quick edit on frontend1 to
avoid the error.  I did not update the vcs for the fix however.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




[Savannah-hackers-public] [task #15902] GNU Guix - Cuirass - Email notifications

2021-02-08 Thread Bob Proulx
Follow-up Comment #1, task #15902 (project administration):

1) Fencepost is administered by "accounts" (FSF Admin) not Savannah. So
nothing we can do here about it.  Docs here.

https://www.gnu.org/software/README.accounts.html

2) I don't follow why there would be need for a fencepost account for guix-ci
for this?  Could you say some more about what you are trying to set up?  Such
as why doesn't the system that is doing the continuous integration builds send
email?  Note that Savannah systems send a lot of email but none of them have
matching accounts on fencepost.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




Re: [Savannah-hackers-public] Aim to be a real free software engineer

2021-02-03 Thread Bob Proulx
Hello Aisuko,

Aisuko LI wrote:
> Thanks for your sharing. I'd like to contribute to the community as I
> personally like to contribute to the operation system or microKernel. But I
> do not know which project I can take with my past experiences. I was
> tracking some issues these days, but may not be easy to jump in. So, would
> you mind sharing some advice with me?

First, Let me post the official reference page that everyone should
read.  It's better written than anything I might type in quickly.

https://www.gnu.org/help/help.html

And perhaps especially this one. 

https://www.gnu.org/software/tasklist/howto-volunteer.html

Second, Let me say that the best places to contribute are to projects
that you are using yourself.  Projects that you feel an attachment.
Projects for which you feel passion.  Those are the best.  Much better
than something that I might just point to and say, this bug here, go
fix that bug, in something you have never heard of before.

There are a lot of good ideas in the howto-volunteer page above.  I
don't want to repeat it.  But that is what I would like to say here.

And remember that contributing is about everything.  It's about code.
It's about documentation.  It's about translations.  It's about web
pages.  It's about mailing lists.  It's about IRC.  It's about
triaging bug reports.  It's about coordinating.  It's about helping
others.  Perhaps most importantly it is about helping others!

What are you interested in?  I know in your introduction you mentioned
many things that I am not familiar with myself.  That makes it hard
for me to suggest something since I know so little myself.  But I did
see that you listed several "cloud computing" topics.  (But see
 as to
the problem with that term.  It mostly means someone else's computer.)

Also note that there are only a very few people on this
savannah-hackers-public mailing list as it is primarily dedicated to
discussion about the Savannah software forge.  For example if you were
interested in working on GNU Guix then help-g...@gnu.org would be the
very active place for discussion about all things Guix.  So far you
haven't mentioned the Savannah software forge itself but here is where
we talk about it.  For example if you were interested in Emacs then
the help-gnu-em...@gnu.org mailing list is a good place to start.  For
example if you were interested in Jami then j...@gnu.org is the place
for it.  And so on and so on and so on for all of the many, many, many
packages of the GNU Project.

I suggest wandering around a little and seeing what interests you! :-)

Bob



Re: [Savannah-hackers-public] Aim to be a real free software engineer

2021-01-29 Thread Bob Proulx
Aisuko LI wrote:
> My name is Aisuko and the real name is BowenLi, I'm a open-source software
> engineer. And I read a few blogs on GNU.org. And I believe that people
> should know the operator system name should be `GNU/Linux`, and I have to
> consider my motivation. Why do I want to be an open source projects
> contributor? and are modern open source projects really for freedom?

Hello Aisuko!  And thank you for contributing to Free Software! :-)

> So, I want to be a geek, although the way is difficult and boring. But I'm
> a really boring guy too.And I keep trying to find the lightweight operating
> systems.

Hmm...  I have been a geek all of my life.  But I have never
considered it boring.  And actually by contrast I know people that
would not be considered a geek and they often say they are bored.
They just don't know what to do with themselves.  Meanwhile I am
always busy.  Always something interesting to do.  I find it hard to
know how people can be bored?  There is always so much fun stuff to
do!

> About my experiences, I'm working on many open-source projects, including
> my Gihtub address.  https://github.com/Aisuko. I have little development
> experience on rancherOS(python, golang) due to my job, and am familiar with
> community management and project maintenance, remote collaboration is easy
> to me. And the development experiences much more on python, golang, C#,
> industry in Cloud-native like Docker,Kubernetes and Service-mesh.

\o/  Yay!  You aren't asking anything so I will just applaud you for
your contributions to the community.  We are stronger together.  The
total of all of us working together is what gives us this great
environment.

> Finally, I'm going back to the college school this year, and I'd like to
> write more articles to spread out  GNU/Linux and freedom software.

Good luck!

Bob



Re: [Savannah-hackers-public] [Savannah-cvs] [433] drop dead link, Savannah sr #110274

2021-01-29 Thread Bob Proulx
ine...@gnu.org wrote:
> --- trunk/sviki/GettingHelp.mdwn  2021-01-27 04:06:57 UTC (rev 432)
> +++ trunk/sviki/GettingHelp.mdwn  2021-01-29 20:46:10 UTC (rev 433)
> @@ -5,7 +5,4 @@
>  Savannah administration support tracker
>  ().
>  
> -There is no mail interface to these trackers. The itsalltext browser
> -plug-in
> -() may
> -be useful for writing lengthy comments.
> +There is no mail interface to these trackers.

I was very sad when It's All Text stopped working with the changed API
in the web browsers.  Because it was really a useful plugin.

Since then I have discovered a new plugin that works very well for
me.  I have been using the "Edit With Emacs" plugin.  It's inspired by
the previous It's All Text plugin.  Of course I am using it with Emacs
but it just spawns $EDITOR and can be used with any editor.

It's not completely trivial to use however.  One needs to use a
separate script server to listen for requests from the browser and
react to them.  However I am using it every day now and would hate to
be with it as much as I hated to be without the previous.  But I know
it is only a matter of time before the browser API changes again and
once again breaks all plugins.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Mailman generates invalid From headers

2021-01-02 Thread Bob Proulx
Eli Zaretskii wrote:
> (If this is the wrong list to post this problem, please tell where to
> redirect this.)

I am probably one of the few that continues the distinction that the
mailing lists are in the "Not Savannah" category.  However I agree
that there are too many mailing lists to keep track of.

https://savannah.gnu.org/maintenance/NotSavannahAdmins/

Mailing lists (lists.gnu.org) - the host, mail setup, and Mailman
installation are managed by FSF sysadmins. Savannah hackers have
limited (non-root) access to mailing lists and archives. All
requests for deletion of archived email must go to FSF sysadmin.

The best place for Mailman issues is mailman AT gnu.org.  All of the
FSF staff are on that mailing list as well as the volunteer mailing
list team such as Ineiev and myself too.  So you get the entire group
there.  But this clearly seems like a bug problem so sending to
sysadmin and opening an RT ticket seems good too.

> Mailman's rewriting of 'From:' addresses in GNU mailing lists
> sometimes produces broken results.  Here's an example (from the
> bug-gnu-emacs mailing list):
> 
>   From: "gliao.tw--- via "Bug reports for GNU Emacs,
>the Swiss army knife of text editors" 
> 
> (These are 2 actual physical lines.)  As you see, the quotes don't
> pair up.  The original address seems to be valid, judging by the
> 'Resent-from:' header:
> 
>   Resent-From: "gliao...@pm.me" 
> 
> Why does this happen?  Can it be fixed?

Of course this rewriting is required when the sending site declares
strict DMARC for their domain.  pm.me does this.

$ host -t txt _dmarc.pm.me
_dmarc.pm.me descriptive text "v=DMARC1; p=quarantine;"

As far as I know this implemented within Exim on the GNU/FSF systems.
Ian is the person who will know the details of all of this.  Ian?
Help!  :-)

> These unbalanced quotes cause Rmail in Emacs to format the inbox
> summary badly.  I fixed Rmail, but the result is still sub-optimal,
> and it would be good to avoid generating such invalid addresses in the
> first place.

It does seem that if quotes were embedded in the comment string that
they would need to be escaped with a backslash.

Bob



Re: [Savannah-hackers-public] Something's wrong with GNU mailing lists?

2020-12-12 Thread Bob Proulx
Eli Zaretskii wrote:
> Looks like email delivery has stopped, at least for Emacs-related
> mailing lists?  Can someone tale a look?
> 
> On second thought, maybe all the email delivery of gnu.org stopped?
> My inbox on fencepost.gnu.org is empty for the last several hours, and
> stays empty even though I sent a test message to myself from a
> different account.

It seems things are resolved now.  I see this from Andrew in an IRC.
(11:14 is in my US/Mountain -0700 timezone from my irc client)

11:14  I restarted exim on eggs

And that is all I know.

Bob



[Savannah-hackers-public] Savannah News Items on Frontpage

2020-11-01 Thread Bob Proulx
Savannah Hackers,

On IRC some of the users objected to the very old News items on the
front page of the Savannah web UI.  And it does look a little stale.
But there hasn't been any new Savannah news to post there.

What do people think about opening up the front page to approving the
random project's News entries that they post to their projects?  That
would allow the very old news that is there now to roll off the bottom
naturally.  I think we should start doing this.

WDYT?

Bob


signature.asc
Description: PGP signature


[Savannah-hackers-public] [task #15795] remove pending account - email to me cannot get thru forwarding service

2020-10-20 Thread Bob Proulx
Update of task #15795 (project administration):

 Assigned to:None => rwp
 Open/Closed:Open => Closed 

___

Follow-up Comment #1:

Hi Alan,

I have updated your email address and activated your account.  If you
need to reset the password please use the lost password reset links on
the login page.

We are very happy to help with these things!  Thank you for submitting
a ticket.  Just for purposes of communication I want to say that after
a timeout of 3 days any unconfirmed accounts are automatically
expired.  That would have allowed you to resubmit the attempt again
with a different email address.  But...  There would have been that
expiration delay to wait through.  And you would need to know the
inner workings of the expiration.  So submitting a ticket and getting
help is better. :-)

Your account is now active.  However please note that if you are not a
member of any group and do not have any tickets associated with your
account then it will be an "idle" account and subject to idle account
removal as an anti-abuse measure.  Spammers and other malicious agents
have automated mechanisms that create thousands of accounts forcing us
to now prune those.  Please see this page for details.

https://savannah.nongnu.org/maintenance/IdleAccounts/

Note: Submitting a comment to this ticket would be sufficient to make
the account no longer idle.  Hint, hint.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




Re: [Savannah-hackers-public] filius

2020-10-20 Thread Bob Proulx
Hello!

Guillaume Sagette wrote:
> hi pls i want have a new link for go to that sit because i have some lag on
> the traditional sit and i need to practice for tuesday
> 
> cordially: sagette guillaume

I am sorry but I don't understand your message.  And no one else of
the team responded either therefore I think no one understood your
message.  We will need something more clear to understand what it is
that you are talking about.

You have reached the Savannah administrators for the Free(dom)
Software forge.

Perhaps you are talking about one of the many, many, many projects
that is hosted on the free software forge site?

Bob



Re: [Savannah-hackers-public] [task #15787] Remove last news item from 'Lastest News'

2020-10-09 Thread Bob Proulx
Ineiev wrote:
> > https://savannah.nongnu.org/news/approve.php?group=kasoverb
> > 
> > I don't see a reason not to remove that news item.  I just have no
> > idea how to do it without debugging it from first principles on the
> > database side of things.
> 
> I'm reluctant to remove any published announcements, it smells like
> rewriting history.  I believe when mistakes are found,
> a counter-announcement should be made instead;
> and I see nothing crucially wrong in that particular item.
> 
> Savannah web interface has no means to remove news,
> only a "quarantine" area where the new items can be checked
> before "approving" them.

Good points all.  I will follow up on that ticket with the suggestion
to simply add another news item with more current information.

Thanks!
Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] [task #15787] Remove last news item from 'Lastest News'

2020-10-07 Thread Bob Proulx
Hey Savannah Hackers, and specifically Ineiev!  :-)

Mark J. Olesen wrote:
> KasoVerb Project
> https://savannah.nongnu.org/projects/kasoverb/
> 
> Please remove last news item ('KasoVerb 3 release candidate...') from News.

I don't know how to do this.  I am sure I could dig around the
database and figure it out.  But looking at the newsKasoVerb - News:
Manage page I don't see any interface for doing this type of thing.

https://savannah.nongnu.org/news/approve.php?group=kasoverb

I don't see a reason not to remove that news item.  I just have no
idea how to do it without debugging it from first principles on the
database side of things.

Bob



Re: [Savannah-hackers-public] Anonymous HTTPS cloning of emacs git repository is failed

2020-10-02 Thread Bob Proulx
Bob Proulx wrote:
> I'll go ahead and send this update and keep looking into why this is failing.

I think I have this problem mostly resolved right now.  I can git
clone the repository now when previously I could not.

rwp@mgt0:~/src/emacs-stuff$ time git clone 
https://git.savannah.gnu.org/git/emacs.git
Cloning into 'emacs'...
remote: Counting objects: 903373, done.
remote: Compressing objects: 100% (160456/160456), done.
remote: Total 903373 (delta 742468), reused 902659 (delta 741755)
Receiving objects: 100% (903373/903373), 293.80 MiB | 15.14 MiB/s, done.
Resolving deltas: 100% (742468/742468), done.
Checking connectivity... done.
Checking out files: 100% (4126/4126), done.

real5m5.358s
user3m36.801s
sys 0m17.380s

What I did to debug...

I made an rsync copy of the emacs.git repository (not a recommended
method, not atomic, better to use a git method that will copy things
the correct way, but fine for my throwaway test, where I needed a bit
exact copy) to my own server which has the same git-http-backend
configuration.  And also copied grep.git too for a comparison.  And
had exactly the same reproducible behavior there!  Attempting to clone
emacs times out but cloning grep works perfectly.  Exact same problem
on my testing copy.

Therefore the problem was definitely something in the emacs.git
repository.

I performed some experiments on my copy offline.  It seemed that for
whatever reason git-pack-objects was always a heavy cpu use process at
the start of the the clone.  Interesting.  But there was a
difference between the working and non-working protocols.

git pack-objects --revs --thin --stdout --progress --delta-base-offset

By my stopwatch and thumb this ran for 1m22s at the start of
every git clone.  Using git-http-backend this would timeout due to
some 60s timeout that I have not yet identified.  However using the
git-daemon only 6 seconds would pass before seeing progress feedback
to the terminal.  The two programs handle this differently,
apparently, and git-http-backend is holding all output until after git
pack-objects completes.

Background: Once a month on the 4th all git repositories are garbage
collecte with "git gc".  Today is the 1st.  So it has been almost the
maximum length of time between explicit git gc runs.  I decided to
manually run this now.

root@vcs0:/net/vcs/git/emacs.git# git gc
Counting objects: 900046, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (161844/161844), done.
Writing objects: 100% (900046/900046), done.
Total 900046 (delta 739533), reused 896905 (delta 737041)
Removing duplicate objects: 100% (256/256), done.
Checking connectivity: 903487, done.

Before the storage size was 411MB and git gc reduced the size ot 333MB
after the run.  After doing the git gc then the git pack-objects takes
only 33 seconds to run at the start of the clone.  So only 33 seconds
before there is interaction.

And so it seems that for whatever reason when using the
git-http-backend that this feedback prior to the compression step
would take longer than 60 seconds, would take 1m22s, and would hit the
60 second timeout, and fail returning the HTTP 504 Timeout error.  Now
that git gc has been recently run it takes only 33 seconds, is below
the 60 second timeout, and therefore is allowed to continue.

At this time I can git clone the repository.

Since it appears that for emacs at least the once a month git gc is
not enough to keep the incremental git pack-objects happy I have
increased that to four times a month, about weekly.  That should keep
on top of this problem for the indefinite future.  I'll try to figure
out what the 60s timeout is since it is not proxy_read_timeout but
some other timeout.  Because if this gets really worse then we will
need to increase that value.

Bob



Re: [Savannah-hackers-public] Anonymous HTTPS cloning of emacs git repository is failed

2020-10-01 Thread Bob Proulx
Yasuhiro KIMURA wrote:
> I'm reporting here because I was recommended in emacs-devel ML.

Yes.  This is the right place for the report.  Thanks for reporting it.

> For about half a day anonymous HTTPS cloning of emacs git repository
> has been failing as following.
> 
> yasu@eastasia[3561]% git clone https://git.savannah.gnu.org/git/emacs.git
> Cloning into 'emacs'...
> error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
> fatal: the remote end hung up unexpectedly

It's a strange problem.  I am not immediately sure why this is failing.  It
had been working.  It works for other projects.  It works for git-daemon
protocol.  It works for ssh member protocol.  It works for the raw dumb http
protocol.  It is only the git-http-backend that this times out after exactly 1
minute.

Therefore this feels like something specific to the emacs.git repository such
as something that has been corrupted recently.  However once again the other
protocols succeed. Using a fast local system a full emacs clone takes 6
minutes.  Remote clones over the Internet can only take longer.

The 1 minute timeout on the surface seems like the default proxy_read_timeout
60s nginx setting.  However for the other protocols interaction begins within
2 seconds.  Also changing that timeout to 180s had no change and the http 504
timeout still occurred at exactly 1 minute. Therefore this seems to be a
different timeout.

Just by way of a workaround you might use the raw repository dumb http
protocol for access.  It is slower but it should work.  Note also that
the dumb http protocol gives little feedback until the very end when
suddenly it is reported as finished.  So for progress reporting it is
needed to do du and ls listings in a different terminal in order to
know that it is doing anything.

rwp@mgt0:~/src/emacs-stuff$ time git clone 
https://git.savannah.gnu.org/r/emacs.git
Cloning into 'emacs'...
Checking connectivity... done.
Checking out files: 100% (4126/4126), done.

real9m24.158s
user6m0.027s
sys 1m9.163s

However obviously the git:// protocol is faster and has better
feedback.  The raw repository dumb http protocol does have the
advantage of using https however.

Further debugging...  All of the files are readable.  I don't see any
obvious errors.  I made a copy for safety and I ran a git fsck on a
copy of the repository and had this result.

root@vcs0:~/rwp/emacs.git# git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (901650/901650), done.
Checking connectivity: 903357, done.
dangling commit 53d8419fb4d30e4537f714e2bc0facf2d93693cd
dangling commit 324683df526cfdceffde5936e27b074f4a091208
dangling commit 16bd454fa837f50fd5f1d159e8f152c597592415
dangling commit 0b3a68ed521bd526714fb261126805c94d30b828
dangling commit 56a64ec49a4693e965da158b8fa5a9858b9abf88
dangling commit 4409ef6deade193058ad74f80c41eb72265645d0
dangling commit d067d05eb06d613f5b1be3f558eb88c814eda9fc
dangling commit fb4c51743bcf33efad8718b0e8cd54dfadc0a6ae
dangling commit 3ae232d923608ca3561385dc587e619e0ac9fcc5
dangling commit 612df640cbcc800c14768f0722e5cd7001faa5f0
dangling commit 8e3df6bc291616858f7858d283a1b2da610398a2
dangling commit 0382368015bfcdb7b88dd68a739f2bc5cde77bce
dangling commit 0f28d7ae1c4a28a35a3ae7047e4c246cc2e1c846
dangling commit ad4879d9717cc116008f4083573bdbcceb782dd6
dangling commit d8e6b9ba8af329e009a2d675c43ba678412690df
dangling commit 666bda627c265cec7c2ab4de6bfa4b9a6c1c0568
dangling commit 4b8edaf3046d6ddfa50a7965dfab7ed6baaacf90
dangling commit ae921bb85a7a452554e72551783e859780b27e11
dangling commit 8d085d2f8d3a9deb82cdfd319a4206abb043951e
dangling commit 555efd91c9815849f241afbf97ea32ed8c577c1f

I'll go ahead and send this update and keep looking into why this is failing.

Bob



Re: [Savannah-hackers-public] can I get a tar copy of the ccvs repo?

2020-09-06 Thread Bob Proulx
Hello David,

David Cantrell wrote:
> I'm working on a side project where I am computing some stats of various
> version control systems and looking at performance from different points in
> historical releases.

Sounds interesting.

> I would like to include cvs in this comparison and it would really help me if
> I could get a tar copy of the ccvs project from the cvs server.  From here:
> 
> http://cvs.savannah.nongnu.org/viewvc/cvs/ccvs/
> 
> Would that be possible?

As I read this it reads like you would like your own local copy of the
cvs source repository.  Rsync the files out of the cvs repository to
your own location.  Then extract what you want from there.

Get a listing:

rsync rsync://cvs.savannah.nongnu.org/sources/cvs/ccvs/

Copy the files:

/var/tmp$ rsync -a --info=progress2 
rsync://cvs.savannah.nongnu.org/sources/cvs/ccvs .
 70,693,319 100%2.16MB/s0:00:31 (xfr#1578, to-chk=0/1688)  

$ du -sh ccvs
72M ccvs

If that isn't what you are asking then let us know.

Hope that helps!
Bob



Re: [Savannah-hackers-public] Email address no more hidden

2020-08-03 Thread Bob Proulx
Ian Kelling wrote:
> tldr: Addresses weren't fully hidden and hiding has significant
> downsides so we turned it off.

I am happy this mis-feature has been turned off.  There were so many
times when the usefulness of the email archive was broken because the
email address obfuscation broke it.  If I were thinking about it I
would do the manual < mailman AT gnu DOT org > way to say where to
write some place for help but often I would forget and then the result
in the archive would be a useless rendering.  Therefore I am happy not
to need to do that type of transliteration and usually forgetting.

Bob



Re: [Savannah-hackers-public] Unreachable https and rsync mirrors

2020-06-08 Thread Bob Proulx
Ian Kelling wrote:
> Awesome. Onto server upgrades! I just decommissioned a debian lenny
> machine this last weekend.

Lenny!  He was a good chap.  Always kept his eyes open!

:-)

Bob



Re: [Savannah-hackers-public] Unreachable https and rsync mirrors

2020-06-08 Thread Bob Proulx
Bob Proulx wrote:
> All of those seem to be the outdated CA list and openssl on
> download0.  All but one of the above were issued by Comodo.  Which is
> the mostly common thread among them.  They apparently have a newly
> issued trust anchor.

It turns out that this was a pretty widely felt expired certificate.
And tickles an openssl bug.  And therefore fixes have been rippling
through.

The way I have been reading the blogs on this the problem is one of
the certificate chains has expired.  Coupled with openssl prior to 1.1
which flagged as invalid the chain if either were invalid.  Requiring
both of them to be valid.  As opposed to validating it as okay if any
of the chains validated as it is supposed to have been working.

Over the weekend I realized that I could extract the expired
certificate and leave only the valid one and this would fix the
problem.  And I could even update the bundle.

But then the Debian Stretch LTS team prepared a package upgrade doing
all of the work very nicely packaged making this trivial to install
their package and not needing any work at all.  :-)

I have upgrade the CA certificate bundle on our three machines that
were needing it, download0, vcs0, mgt0.  Testing shows that the
previous certificates that were previously invalid are now validating.
I am going to wait and let the mirmon scripts run and hopefully that
will now validate those mirrors and they will come back online in the
redirector over the next couple of hours.

Bob



Re: [Savannah-hackers-public] Unreachable https and rsync mirrors

2020-06-05 Thread Bob Proulx
Ian Kelling wrote:
> Related, on https://savannah.gnu.org/maintenance/Mirmon/ there is a
> broken link, which also has a bad cert
> 
> $ curl https://dl.sv.gnu.org/releases-noredirect/00_MIRRORS.html
> curl: (51) SSL: no alternative certificate subject name matches target host 
> name 'dl.sv.gnu.org'

dl.sv is a shortcut DNS name.  There are so many of those!  Take all
of the combination of gnu and nongnu, of savannah and sv, of dl and
download, and then do that for all of the systems since most have
short typing aid names, and there are a lot of names!  I didn't have
them all on download.

I have updated the certificates to include these.  These have been
missing since February 10, 2020 when things were converted from
Certbot to Dehydrated.

  dl.savannah.gnu.org
  dl.savannah.nongnu.org
  dl.sv.gnu.org
  dl.sv.nongnu.org

Those now have been issued certificates and should be working for
https now.

> I manually curled 2 of the bad mirrors listed here
> https://download.savannah.gnu.org/mirmon/allgnu/, and there is a cert
> error. I'm pretty sure, the issue is that the ca-certs needs updating on
> the machine running mirmorn. The os itself could use an update too. Bob,
> you there?

You are correct.  Which becomes a motivation to get off the fallen out
of support OS on download0 and over to the newer OS on the new
download1 system soonest.

I tried simply upgrading the ca-certificates individally but there is
a dependency upon a newer openssl.  At which point I stopped because
that would open a can of worms of dependencies.  Time better spent
working on getting onto the newer system.

> Thérèse Godefroy  writes:
> > 8 have been reported off-line for 6 days:
> > https://www.singleboersen.com (http OK)
> > https://mirror.checkdomain.de (http OK)
> > https://www.gutscheinrausch.de (http OK)
> > https://ftp.wrz.de (http OK)
> > https://mirrors.nav.ro (http OK)
> > http://mirror.lihnidos.org
> > https://mirror.us-midwest-1.nexcess.net (http OK)
> > https://mirrors.syringanetworks.net (http OK)

All of those seem to be the outdated CA list and openssl on
download0.  All but one of the above were issued by Comodo.  Which is
the mostly common thread among them.  They apparently have a newly
issued trust anchor.

> > One for 6.8 days:
> > rsync://mirror2.evolution-host.com::gnu
> > One for nearly 99 days:
> > rsync://mirrors.syringanetworks.net/gnu

I don't know.  I didn't have time to look at the rsync mirrors.  I need to dig 
more.

> > What's strange is that I can reach all of them from France, except
> > http://mirror.lihnidos.org.

That system is simply down.  Can't ping it or get any other life from it.

> > Several rsync mirrors (not only these 2) have been wrongly reported
> > off-line since January 2019, occasionally or almost constantly. But this
> > is the first time I see so many https URLs being unreachable for Mirmon,
> > while they are fine for me.

The CA Certificate Authority trust anchors they are now using are
newer than the files available on download0 to validate.

> > Since these https URLs are supposedly off-line, they are not taken into
> > account by the multiplexer. So the load on the other mirrors increases.
> > Right?

There are many mirrors however.  The collection of all of them is
therefore resilient to a small number of them being offline.  At the
time I look there are 13 listed offline out of 223 total.  That is 6%
which leaves 94% of the mirrors online.

> > Is there a way to fix this?

The real answer is the OS upgrade to get the newer openssl and newer
ca-certificates package.  Ian and I started on this back at
LibrePlanet in March.  But then other prioritites distracted me.  I'll
get back on the task again.

Bob



Re: [Savannah-hackers-public] Emacs git repository clone limits

2020-06-01 Thread Bob Proulx
Philippe Vaucher wrote:
> >   git daemon --init-timeout=10 --timeout=28800 --max-connections=15 
> > --export-all --base-path=/srv/git --detach
> >
> > In this configuration git-daemon acts as the supervisor process
> > managing its children.  The limit values are tweaked and learned from
> > having too many connections causing failures and tuned lower in order
> > to prevent this.
> >
> > When connections exceed the max-connection limit then they will queue
> > to the kernel limit /proc/sys/net/core/somaxconn (defaults to 128) and
> > be serviced as able.  Any queued connections past the 128 default the
> > client will get a connection failure.  The behavior at that point is
> > client dependent.  It might retry.
> 
> Interesting. So you're saying the timeouts I had when using git://
> meant that there was too much queue.

Likely.  Yes.

Because SO_MAXCONN defaults to 128 this means that 128 connections
will connect.  The 129th connection will not be able to connect and
will get a failure to connect.  The client should report it as a
failure to connect error.  The git-daemon will have 15 concurrent
processes draining the queue continuously.  We currently have 8 cpu
cores processing.

> > The nginx configuration is this:
> >
> > location /git/ {
> > autoindex on;
> > root /srv;
> > location ~ ^/git(/.*/(info/refs|git-upload-pack)$) {
> > gzip off;
> > include fastcgi_params;
> > fastcgi_pass unix:/var/run/fcgiwrap.socket;
> > fastcgi_param SCRIPT_FILENAME 
> > /usr/local/sbin/git-http-backend;
> > fastcgi_param PATH_INFO $1;
> > fastcgi_param GIT_HTTP_EXPORT_ALL true;
> > fastcgi_param GIT_PROJECT_ROOT /srv/git;
> > client_max_body_size 0;
> > }
> > }
> >
> > Looking at this now I see there is no rate limit being applied to this
> > section.  Therefore what I mentioned previously applies to the cgit
> > and gitweb sections which have been more problematic.  With no rate
> > limits all clients will be attempted.  Hmm...  I think that may have
> > been a mistake.  It is possible that adding a rate limit will smooth
> > the resource use and actually improve the situation.  The cgit and
> > gitweb sections use a "limit_req zone=one burst=15;" limit.  cgit in
> > particular is resource intensive for various reasons.  I'll need to do
> > some testing.
> 
> So, where did my 502/504 errors come from? Each job was retried 3
> times, with a 5 seconds delay. I'd understand some of them failing but
> not all of them.

The http/https side comes under DDoS abusive much more often.
Probably because script kiddies target it more often.  And at the
times when it is being driving off the network then the result is that
it is pretty much driven off the network.

If I am looking at it when this is happening I can sometimes mitigate
the attach by fail2ban rules and other things.  But sometimes it just
happens for a few hours and stops.  Sometimes in the 4am timeframe
when I don't see them until 10am or so the next day at which time
things have reset back to normal.

And then on top of that is all of the sum total of everything that is
running there.

> > When you are seeing proxy gateway failures I think it most likely that
> > the system is under resource stress and is unable to launch a
> > git-http-backend process within the timeouts.  This resource stress
> > can occur as a sum total of everything that is happening on the server
> > at the same time.  It includes git://, http(s)://, and also svn and
> > bzr and hg.  (Notably all of the CVS operations are on a different VM,
> > though likely on the same host server.)  All of those are running on
> > this system and when all of them coincidentally spike use at the same
> > time then they will compete with themselves for resources.  The system
> > will run very slowly.  I/O is shared.  Memory is shared.
> 
> Ah, ignore my question above then :-) Interesting!

Plus a few of the services seem very heavy.  For example cgit really
seems to grind on the system.  And some features of cgit don't help.
One can generate a tar.gz file of any tag, branch, commit.  So if an
anti-social web crawler is crawling every link they can find
(regardless of robots.txt) then they have at times tried to download a
tar.gz of every historical version of every project that is hosted.
It would seem to me that doing this would need a lot of disk space on
the receiving end.  But that has been one of the repeating themes.

> > Among other things the current VM has Linux memory overcommit
> > enabled.  Which means that the OOM Out of Memory Killer is triggered
> > at times.  And when that happens there is no longer any guarentee that
> > the machine is in a happy state.  Pretty much it requires a reboot to
> > ensure that everything is 

Re: [Savannah-hackers-public] Emacs git repository clone limits

2020-05-28 Thread Bob Proulx
Philippe Vaucher wrote:
> Alfred M. Szmidt wrote:
> > Why not simply use a reference repository?
> >   git clone --mirror git://git.gnu.org/emacs.git ~/emacs-reference
> > And when cloning:
> >   git clone --reference ~/emacs-reference git://git.gnu.org/emacs.git
> > the ~/emacs-reference place can be accessible where ever you are using
> > it.  This will work incrementally, and not use up scares resources..
> 
> Thanks, good idea I didn't know about it. I don't know how well
> `--reference` will cope with diverged branches, but I'll run some test.

If you are bundling up a result then also look at the git clone option
--dissociate too.  That produces a result that is independent of the
reference.  I think that would be more suitable to you the way you
described it.

Bob



Re: [Savannah-hackers-public] Emacs git repository clone limits

2020-05-28 Thread Bob Proulx
Philippe Vaucher wrote:
> > > Is there a limit and/or maintainance going? Was I but in some sort of
> > > throttle-list?
> >
> > We do have rate limits in place.  Because otherwise the general
> > background radiation activity of the Internet will break things.
> >
> > However nothing has changed recently in regard to the rate limits for
> > a long time.  As I look at the logs the last rate limit change was Sat
> > Dec 7 06:31:50 2019 -0500 which seems long enough ago that it isn't
> > anything recent.  Meaning that this is probably simply just you
> > competing with other users on the Internet for resources.
> 
> Until recently I only did up to ~8 concurrent git clones, but recently
> with infrastructure changes I'm able to do much more.

For a high level of parallelism I would definitely try to fanout using
local resources.  It would be much more reliable.

In electrical design the fanout and load of a circuit is a design
consideration.  One doesn't want to place too much of a load on a
remote source.  And a source knows how much fanout it can drive.  This
is a similar problem to software mirroring.

I would limit the amount of load placed and then keep all of the
fanout to using local resources.  Then you get to see the effects of
both ends of things locally and have better visibility.  Also since
that is on the LAN side the network performance is unlikely to
bottleneck.

I will resist telling some anecdotal stories of various experiences
here and some managerial problems in this area. :-)

> > Cloning with the https transport that uses git-http-backend for the
> > backend.  We are using Nginx rate limiting.  Which you can read about
> > how the algorithm works here.  It is basically a smoothing process.
> 
> Until recently I was cloning git://git.sv.gnu.org/emacs.git, the
> switch to https is an attempt at working around the limitation I hit
> recently. My train of thought was that http:// is easier to scale than
> git://, if you say otherwise I can revert back to git:// clones.

Well...  It's not simple!  And also things change with time and server
resources and configuration.  So any answer I were to state today
would mutate into a wrong answer at some different point in time.

I understand your reasoning.  And if this were an infrastructure at,
say, Amazon AWS EC2, setup to be elastic and just scaled out then your
reasoning would be totally on target.  Increased load would scale out
to more parallel resources.  It is more typical to have a load
balancer in front of http/https protocol servers and so those can be
set up rather straight forward.  And also load balancers could be set
up in front of git:// protocol server too.

However the GNU Project is dedicated and directed to using only Free
Software.  And is hosted in this goal by the FSF.  Which means
everything is self-hosted.  Funded by the annual fund raising from
donors just like all of us who donate funding.  And resources are
limited.  Note that Github is not Free Software.  Therefore we cannot
endorse its use.  Same thing with Amazon AWS.  Although regardless of
this we know that many users do use them anyway.

Let's talk about the technical details of the differences between git
protocol servers and http/https protocol servers.

The abbreviated details of the git-daemon is that it is running as the
'nobody' user like this.

  git daemon --init-timeout=10 --timeout=28800 --max-connections=15 
--export-all --base-path=/srv/git --detach

In this configuration git-daemon acts as the supervisor process
managing its children.  The limit values are tweaked and learned from
having too many connections causing failures and tuned lower in order
to prevent this.

When connections exceed the max-connection limit then they will queue
to the kernel limit /proc/sys/net/core/somaxconn (defaults to 128) and
be serviced as able.  Any queued connections past the 128 default the
client will get a connection failure.  The behavior at that point is
client dependent.  It might retry.

The nginx configuration is this:

location /git/ {
autoindex on;
root /srv;
location ~ ^/git(/.*/(info/refs|git-upload-pack)$) {
gzip off;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
fastcgi_param SCRIPT_FILENAME 
/usr/local/sbin/git-http-backend;
fastcgi_param PATH_INFO $1;
fastcgi_param GIT_HTTP_EXPORT_ALL true;
fastcgi_param GIT_PROJECT_ROOT /srv/git;
client_max_body_size 0;
}
}

Looking at this now I see there is no rate limit being applied to this
section.  Therefore what I mentioned previously applies to the cgit
and gitweb sections which have been more problematic.  With no rate
limits all clients will be attempted.  Hmm...  I think that may have
been a mistake.  It is possible that 

Re: [Savannah-hackers-public] Emacs git repository clone limits

2020-05-27 Thread Bob Proulx
Hello Philippe,

Philippe Vaucher wrote:
> I'm the author of https://hub.docker.com/r/silex/emacs, dockerized Emacs
> images for a lot of versions.
> 
> You can see one of the builds here:
> https://gitlab.com/Silex777/docker-emacs/pipelines/148568952

Wow!  I see 40 different builds there.  Which is totally awesome btw.

> The build thus builds a lot of images at once, and thus I have many
> parallel "git clone https://git.sv.gnu.org/git/emacs.git;.

That can certainly be a problem.  Because Emacs is not small.  And in
conjunction with the rest of the Internet also cloning Emacs and
cloning other projects from the server the effect can brown out the
system.

> For a while this worked fine, but since 4-5 days I get a lot of http
> 502/504 and I found out that I can only do about 4 parralel git clones.
> 
> Is there a limit and/or maintainance going? Was I but in some sort of
> throttle-list?

We do have rate limits in place.  Because otherwise the general
background radiation activity of the Internet will break things.

However nothing has changed recently in regard to the rate limits for
a long time.  As I look at the logs the last rate limit change was Sat
Dec 7 06:31:50 2019 -0500 which seems long enough ago that it isn't
anything recent.  Meaning that this is probably simply just you
competing with other users on the Internet for resources.

Cloning with the https transport that uses git-http-backend for the
backend.  We are using Nginx rate limiting.  Which you can read about
how the algorithm works here.  It is basically a smoothing process.

  https://www.nginx.com/blog/rate-limiting-nginx/

How are you doing the clone?  Is it a fresh clone into an empty
directory?  Because that would be the worst case.  In that worst case
scenario it would need to transport 100% of the repository every time.
That's pretty heavy.  The Emacs git repository is about 359MB in
size.  Seeding that with the previous version and updating it would
make that transfer much more efficient.

Whenever I have set up continuous integration builds I always set up a
local mirror of all remote repositories.  I poll with a cronjob to
refresh those repositories.  Since the update is incremental it is
pretty efficient for keeping the local mirrors up to date.  Then all
of the continuous integration builds pull from the local mirror.

This has a pretty good result in that the LAN is very robust and only
shows infrastructure failures when there is something really
catastrophic happening on the local network.  Since 100% of everything
is local in that case.

Transient WAN glitches such as routing problems or server brownouts
happen periodically and are unavoidable but then will catch up the
local mirror image with the next cronjob refresh.  Can refresh on a
pretty quick cycle.  I would do every four hours for example.  Plus
those errors are much more understandable as a network issue separate
from the CI build errors.  It's a good way to protect the CI builds
and redunce noise from them to a minimum.

Periodically the Savannah vcs server is pummeled by network abuse.
People.  It's why we can't have nice things.  I have been seeing some
zero-dark-thirty times when things have been hammered into failure.
Which then will recover by the time I wake up and go peek at them.

Do the failures are are seeing have a periodic time of day cycle when
they are more likely to happen in the middle of the US nighttime?  If
so then that is probably related.

I have been needing to look at the logs and improve the fail2ban rules
to try to push back on the abuse a little more strongly.  I do this
periodically but the type of abuse keeps changing.  For example right
now I am seeing a new wp-content path (Wordpress attack).  However
note that there is no Wordpress running on these machines.  Anti-abuse
rule maintenance is always needed.

Bob



Re: [Savannah-hackers-public] (no subject)

2020-04-15 Thread Bob Proulx
Hello Héctor!

hector lacunza wrote:
> Hi, my name is Héctor. I am studing Network Administration Operative
> Systems (ASIR, in spanish) and i am interested in learn more about
> GNU/Linux. I think the best way yo learn is to do, so i would like
> to work with you.
> If i can be a contributor please let me know.

It's awesome that you would like to help the GNU Project!  You have
reached the folks who maintain Savannah, the software forge for people
committed to Free Software.  https://www.gnu.org/philosophy/free-sw.html
But we don't really know much about the individual projects.  It's all
very distributed.

There are many projects within the GNU Project.  Are there any that
you are more interested in than others?  The best thing is to find a
project that you have some interest and become active with that
project.  You might look at the list of high priority projects.  These
are very important projects that need more attention.

  https://www.fsf.org/campaigns/priority-projects/

Right now with everyone dealing with the COVID-19 stay at home
situation there is a lot of need for software to communicate.  You
might find something in this list that you are interested in.  These
are of particular interest right now.

  
https://www.fsf.org/blogs/community/better-than-zoom-try-these-free-software-tools-for-staying-in-touch

There is a list of all of the official GNU packages here.  But we also
encourage participation in non-GNU projects too.  But here is the
official GNU list.  If you are looking for something that interests
you then perhaps you will find something from here.  There are a lot
of projects!

  https://www.gnu.org/software/software.html

Also remember that good projects need good documentation.  In many
translated languages.  And with good web pages.  With nice logos and
artwork.  Often the best place to start is to start reading
documentation for a project that interests you and then upon finding
things that are not clear, mistaken, needs improvement, or whatever,
send suggestions for improving that documentation to the maintainers
of that project.  The best time to do this is when learning a new
project.  Because that is when omissions in the documentation are most
apparent.

Hope this helps!  And thanks for contributining to the community! :-)

Bob




Re: [Savannah-hackers-public] (no subject)

2020-03-19 Thread Bob Proulx
The Talk wrote:
> ViewVC 1.1.26

Did you want to make some comment to the Savannah Hackers?  If so then
the above was not sufficient to convey meaningful communication.

Bob



Re: [Savannah-hackers-public] Savannah Upgrades In Motion

2020-03-13 Thread Bob Proulx
Savannah Hackers,

> Let this be a general notice of very quickly moving system upgrades.
> I am in Boston at this time.  Ian and I have been working on various
> system upgrades.  With some effort, tenacity, persistence, and luck,
> we are hoping to upgrade the last of what was at one time the newer
> systems but are now older systems in need of upgrades.
> 
> Due to being here and trying to move quickly there won't be a lot of
> time available to give a lot of warning of changes.  Sorry.  But
> hopefully we will be able to get the upgrades done.  That's the goal.

And though we started strong at the beginning of the week and spun up
several new VMs with updated versions of the OS then things became
diverted.  Tuesday I was focused on a system rescue at a client site
of mine.  And of course the news of this week with the conference
going virtual and with cancelation of travel plans has just changed
the focus of everything for this week.  So even though I had high
hopes basically no upgrades have happened this week.  But the
foundation for doing them has been set and will continue next week.

Tomorrow is the virtual online LibrePlanet 2020.  I hope to see you
online!

Bob


signature.asc
Description: PGP signature


[Savannah-hackers-public] Savannah Upgrades In Motion

2020-03-08 Thread Bob Proulx
Savannah Hackers,

Let this be a general notice of very quickly moving system upgrades.
I am in Boston at this time.  Ian and I have been working on various
system upgrades.  With some effort, tenacity, persistence, and luck,
we are hoping to upgrade the last of what was at one time the newer
systems but are now older systems in need of upgrades.

Due to being here and trying to move quickly there won't be a lot of
time available to give a lot of warning of changes.  Sorry.  But
hopefully we will be able to get the upgrades done.  That's the goal.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Savannah Frontend Upgrade

2020-02-18 Thread Bob Proulx
Bob Proulx wrote:
> I have switched the DNS from the old frontend0 to the new frontend1.

Having sat for an appropriate quarentine period I have shutdown the
old frontend0 VM system.  The upgraded frontend1 is the live
production system for the PHP web UI frontend.

A continuing trickle of noise will be the EFF reminders that the
frontend0 Let's Encrypt certificates are expiring.  Yes they are as
the system is decommissioned now.  Once we get past the expiration
period then the EFF reminders will stop and we can simply ignore them
until then.

At some point I will ask the FSF admins to remove the 10GB VM system
image that still remains on disk.  But now that it is offline it is
just a very small bit of archive to maintain as a safety net.

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] emacs-devel list is unusually silent

2020-02-09 Thread Bob Proulx
Eli Zaretskii wrote:
> It is unusual for the list to have no messages in two days.  Could
> someone please take a look, to make sure there are no problems with
> email delivery?

That is odd.  Likely a problem.  I personally do not have the access
to look into it.  I am only responding here so you know someone read
your message.

Unfortunately this may need to wait until Monday when the FSF admins
can look into it.

Bob



[Savannah-hackers-public] Savannah Frontend Upgrade

2020-02-07 Thread Bob Proulx
Savannah Hackers,

After being stalled for a while due to Life, The Universe, and
Everything, work picked back up again on upgrading the OS hosting the
web UI frontend0.  To catch up on the happenings let me refresh the
events.

We asked the FSF admins to clone the VM image of frontend0 creating
frontend1 as a fork of it so that we could safely work on the upgrade
off to the side in parallel with but not affecting the running
production system frontend0.  Then we upgraded frontend1's Trisquel OS
from Belenos 7 to the latest release Trisquel Flidas 8.

That OS upgrade brings in many significant upgrades including an
upgrade from PHP 5 to PHP 7.  And also introduces systemd with another
collection of significant changes.

Ineiev did the majority of the real work in making the web UI upgrade
possible by working through the PHP code and making upgrades needed
for the PHP 7 changes.  That work made this upgrade possible!

I think the new system is ready to go.  It has actually been in this
state for some time but did not wan to make the switch until it would
happen at a time I would be available to monitor it and react to any
problem reports.  That time has now arrived.  I have switched the DNS
from the old frontend0 to the new frontend1.

If people would be so kind as to keep an eye on the web interface and
report any breakage that would be most appreciated.  Of course this
has no effect at all on the rest of the version control backend
services.  Except pessimist me knows that there will almost certainly
be something we missed that will be broken and need to be fixed in the
interaction between the systems.

A small block diagram type design change was made with this upgrade.
The web UI saves uploaded file attachments.  Previously this was
locally stored on the frontend0.  This has been changed to be shared
on NFS in the vcs partition where multiple frontend{0,1,n} systems can
share it.  Since there is no longer any saved local state local to the
web UI system it means we can switch between frontend systems as
needed without worry as to the local file state.  It would also mean
that we could scale-out with multiple load balancing frontend systems
if that were needed too.  In any case local saved attachments, the
only local saved state, is now on the shared NFS storage.

Now it is time to look at upgrades for the other systems.  Each as its
own unique challenges.  :-)

Bob


signature.asc
Description: PGP signature


Re: [Savannah-hackers-public] Problem with email delivery on fencepost?

2019-12-16 Thread Bob Proulx
Eli Zaretskii wrote:
> > Bob Proulx wrote:
> > Hmm...  debbugs is completely independent of eggs.  I don't see
> > anything wrong on debbugs.  It is receiving email and sending email.
> 
> If delivery of email on eggs stopped, then my email, sent through
> fencepost, would not get delivered to debbugs, right?

Hmm...  Initially I did not think so.  I almost posted saying that!
Because normally one would see this MX (mail exchange relay)
configuration and think it would go directly there.  (And I would have
been wrong!)

  $ host -t mx debbugs.gnu.org
  debbugs.gnu.org mail is handled by 10 debbugs.gnu.org.

But I decided to try a test of mail from fencepost to see how it went..

  Received: from fencepost.gnu.org ([2001:470:142:3::e]:34133)
by eggs.gnu.org with esmtp (Exim 4.71)
(envelope-from )
id 1igtjb-0003km-IX
for b...@proulx.com; Mon, 16 Dec 2019 12:00:39 -0500

That indicates that fencepost must be configured in the Exim
configuration to also relay outgoing mail through eggs.  I had not
known that.  So yes if eggs stalled in handling mail due to the disk
being full there then outgoing mail from fencepost would have also
been blocked behind it.

As long as I am here...  Mail from debbugs back to anyone @gnu.org
would definitely go through eggs too.  Same as for anyone else out in
the world.

  $ host -t mx gnu.org
  gnu.org mail is handled by 10 eggs.gnu.org.

> > Please send a message to help-debb...@gnu.org and ask about debbugs
> > being slow to respond to email.  One of Glenn, Michael, or Noam (or
> > potentially myself, but I don't know how the debbugs part operates)
> > will look at the problem there.  You need one of them to debug how
> > debbugs is working or not.
> 
> OK, will do next time something like this happens.

Now that we know in the above that outgoing mail from fencepost is
configured to relay through eggs and we know that eggs blocked for a
while due a spike in size of logfiles filling up the disk.  And we now
know that debbugs was also delayed in getting outgoing mail from
fencepost.  Therefore debbugs couldn't get the mail during the pause.
With all of the information in hand it all makes sense as to what
happened.

I like it when everything has an explanation that we can figure out. :-)

Bob



Re: [Savannah-hackers-public] Problem with email delivery on fencepost?

2019-12-15 Thread Bob Proulx
Eli Zaretskii wrote:
> Looks like some delay or another problem with emacs delivery: I was
> sent an email almost an hour ago, which didn't arrive.

It's the weekend so I would not expect a reply from sysadmin until at
least Monday.  But from some IRC chatter that I saw while lurking
earlier I think that the exim logs filled up the disk on eggs
preventing it from receiving email.  Around 2pm -0500 time it looked
like Ian and Ruben were discussing the problem and fixing it.  So I
think that was why the mailing lists stalled for a while today.  And
your mail here to this list came through so things must be okay again.

> And email I sent to debbugs wasn't received by debbugs, because my
> message was telling debbugs to close a bug report, and that bug
> report is still open (and my email is not on the bug page).

Hmm...  debbugs is completely independent of eggs.  I don't see
anything wrong on debbugs.  It is receiving email and sending email.
(Before anyone asks, the spamd is running there.)  My own email tests
through there is working okay.  From a very light surface look
everything looks okay.

For example if I send a help request to the control address then I get
a help response back okay.  It returns the expected help page within a
moment.  Pretty quick with the response too.

> Can someone look into this?

Please send a message to help-debb...@gnu.org and ask about debbugs
being slow to respond to email.  One of Glenn, Michael, or Noam (or
potentially myself, but I don't know how the debbugs part operates)
will look at the problem there.  You need one of them to debug how
debbugs is working or not.

Bob



Re: [Savannah-hackers-public] htaccess for website ?

2019-12-10 Thread Bob Proulx
Hello Patrik,

Patrik Dufresne wrote:
> After mush effort, I'm finally, admin of rdiff-backup project with Eric
> Zolf. I'm looking for various way to slowly redirect the traffic from
> savannah website to our new website. I'm wondering what are the tool
> available for me.
> 
> The easier way would be to setup a redirect for each pages. Is it possible
> for me to configure an htaccess file in my website project ?

As far as I know the answer is no.  There is only the ability to
upload html pages.  There is no ability to insert configuration using
a local .htacess file.

Note that Savannah is simply the version control host for the files.
All of the web page action is on the FSF web servers which checkout
the pages.  There is a separate WWW team.  At least two of that team
read this mailing list though and will hopefully comment if I say
anything incorrect. :-)

[[Aside: Generally use of .htaccess files is a really bad idea because
it really kills performance due to the need for Apache to dynamically
stat(2) up and down the directory tree for every page access looking
for local .htaccess files and then dynamically merge in the
configuration.  Also IMNHO the use of .htaccess locks the site into
using Apache and there are many reasons a site may want to use less
memory intensive and faster alternative servers.]]

Also there is the problem that Github runs non-free software.  The GNU
Project and the FSF cannot be endorsing non-free software.  Therefore
I think any http redirect from FSF resources to a non-free site is
just not possible.

Bob



[Savannah-hackers-public] Savannah Outage Event Today 2019-12-09

2019-12-09 Thread Bob Proulx
Savannah git, svn, hg, bzr, download, Outage Event Today 2019-12-09

We had a problem today that caused all of those services to fail for a
while this afternoon.  Things are back online again.  Sorry for the
inconvenience.  Everything is back working normally at this time.
Please make reports if you are seeing problems.

Here is the story of the events today:

The trigger was me upgrading nfs1 to the latest security upgrades and
rebooting it.  There were three packages with upgrades pending that
needed to be installed: systemd systemd-sysv libsystemd0 from Trisquel
8 versions 229-4ubuntu21.22 to 229-4ubuntu21.23.

The system had been running continuously for 90 days.  Which is a bit
of time.  Not a problem for a system. But long enough that some
forgotten change might have been made to affect things on a reboot.
If a system has been up for longer than even a week then as standard
operating practice I always reboot the system before doing any new
maintenance on it just as a paranoid practice.  Then I know if there
is a problem I know it was something previously existing and not
something I was just doing.  I rebooted nfs1.  All normal.

Then applied the upgrades listed above.  Then rebooted again.  On nfs1
all seemed perfectly normal.  But that is when vcs0 and download0
started reporting stale nfs handle on the mount point.  Nagios
detected this and sent out alerts.  Gack!  Jump and try to understand
the problem and fix it.

I poked and probed the patients.  Trying to mount the partition
manually failed with a "mount.nfs4: Connection timed out" error.  Yet
running tcpdump on the clients showed them communicating.  Regardless
of the timeout it did not appear to be a problem with them
communicating.  And in the tcpdump trace nfs1 kept returning "getattr
ERROR: Stale NFS file handle" errors.  Strange.  On the mount?

Without a good explanation I think nfs1 was behaving badly.  Even
though it had freshly been rebooted I decided to reboot it again.

And on this nfs1 reboot then on the clients I could mount the
partition.  Therefore this seems to indicate that nfs1 had for some
unknown reason latched into a bad state.

Rebooted vcs0 so that it would be a clean start from boot.  Ran
through the regression suite and all of the service tests passed.

During the problem event time download0 which has two mount points had
one stale and one okay.  Yet both are basically identical other than
different names.  After the reboot of nfs1 when things started working
then download0 could mount it on the stale partition and everything
was okay again there too.  And the mount point on frontend1 was okay
throughout too.  Odd that some client systems had problems and others
did not.

That leaves us with an unresolved and perhaps not possible to resolve
question of why nfs1 behaved badly.  Will need continued
investigation.  Sometimes things like this only become understood
after a longer research time.

At the start of things I asked Andrew to post a status update:

  https://hostux.social/@fsfstatus/

That's always a good URL to bookmark so as to be able to get an
out-of-band non-gnu-system update of infrastructure system status.
It's on a non-gnu system in case the gnu sites are offline, so that
there is a way to communicate in that event.

Bob



[Savannah-hackers-public] Please avoid removing ntp on nfs1

2019-12-09 Thread Bob Proulx
The time on nfs1 is drifing badly.  It is currently 1m8s fast.

  root@mgt0:~# dsh.sh date -R | sort
  + dsh -M -c -f hostlist date -R
  download0: Sat, 07 Dec 2019 15:22:01 -0500
  frontend-dev: Sat, 07 Dec 2019 15:22:00 -0500
  frontend0: Sat, 07 Dec 2019 15:22:01 -0500
  frontend1: Sat, 07 Dec 2019 15:22:01 -0500
  internal0: Sat, 07 Dec 2019 15:22:01 -0500
  mgt0: Sat, 07 Dec 2019 15:22:01 -0500
  nfs1: Sat, 07 Dec 2019 15:23:09 -0500
  vcs0: Sat, 07 Dec 2019 15:22:01 -0500
  vcs1: Sat, 07 Dec 2019 15:22:01 -0500

  root@mgt0:~# dsh.sh date +%s | sort
  + dsh -M -c -f hostlist date +%s
  download0: 1575750140
  frontend-dev: 1575750139
  frontend0: 1575750140
  frontend1: 1575750140
  internal0: 1575750140
  mgt0: 1575750140
  nfs1: 1575750208
  vcs0: 1575750140
  vcs1: 1575750140

In an NFS environment keeping system clock time in sync between client
and servers is a critical requirement.  Having them out of sync causes
many very strange problems that are hard to debug.  Been there.  Done
that.  System clock time being out of sync is one of the things I
routinely check now when I am looking at anything related to nfs.

I can see in the logs that although ntp has been installed a few times
that it always gets removed.  I am installing it again.  I'll mark it
'hold' so that it sticks around.

Thanks!
Bob



Re: [Savannah-hackers-public] [gnu.org #1456067] Re: Problems with email and debbugs?

2019-12-07 Thread Bob Proulx
Bob Proulx wrote:
> I started it back up again and immediately exim started to log
> normal activity in the log file.

I sent a test mailing through debbugs such that I would be able to
verify that basic email was working.  The test message was received
and delivered okay.

Bob



Re: [Savannah-hackers-public] [gnu.org #1456067] Re: Problems with email and debbugs?

2019-12-07 Thread Bob Proulx
From: "Andrew Engelbrecht via RT"
> I'll look into the spamassassain issue.

The Bayes issue is probably something that has been that way for a
long time.  Potentially it has always been that way.  The error
message I see in the logs is:

  Dec  7 21:38:57 debbugs spamd[13350]: bayes: cannot open bayes databases 
/var/lib/spamassassin/nobody/bayes_* R/O: tie failed: Permission denied

And the dates on the files are from 2013.

  -rw--- 1 debian-spamd debian-spamd 170110976 Jun  8  2013 bayes_seen

SpamAssassin itself will continue without Bayes in this case.  It will
use the other rules to evaluate the message.

So I think this is one of those things that has always been this way.

On the good side the /var/lib/spamassassin/* rules are current and
appear to be getting updated okay.

> It's possible that some of its internal files got corrupted from
> our recent RAID array failure event a couple weeks ago. We have
> backup files that we can selectively restore if needed.

I think this is the biggest urgent need on debbugs right now.  There
were a surprising number of random files dropped or corrupted across
the machines that needed some fixup and recovery from backup and
restore from pristine on the Savannah systems.  Glenn reported a
missing file that he fixed earlier.  I would not be surprised to find
that there were others that need rescuing.

Eli Zaretskii wrote:
> It seems something it still amiss.  Email to debbugs I sent more than
> 2 hours ago is still not acted upon by debbugs, and I received no
> responses from cont...@debbugs.gnu.org for directives I sent to that
> address.

I poked my nose in there and found that spamd was not running.  Exim
was complaining that it couldn't contact it.  I started it back up
again and immediately exim started to log normal activity in the log
file.

If email on debbugs was not down too long (past the expire time on 3rd
party servers which would then generate a bounce) then mail will have
queued on the 3rd party servers and retry for 3 to 5 days.  Now that
it seems to be receiving mail I expect that over the next hour or four
those 3rd party servers will retry sending the mail.  I am hoping that
the cofferdam will now be broken and we will see a bunch of backlogged
mail flowing through.

Bob



Re: [Savannah-hackers-public] Problems with email and debbugs?

2019-12-06 Thread Bob Proulx
Eli Zaretskii wrote:
> Looks like email delivery stopped at fencepost.  And debbugs takes a
> very long time to return results of a query.
> 
> Another DDoS attack on GNU servers?

I shouldn't be responding to this as I don't know first hand
information but I didn't see other responses so will contribute what I
know.

I see that Andrew restarted Apache on debbugs this morning and that
seems to have gotten things going.  The main problem may have been
Apache getting snarled up there.  Don't know.  Seems likely to be the
main problem today though.

I peeked in at the Apache logs on debbugs and while it does seem to
have some small number of bots crawling the site inappropriately it
did not seem to be overwhelmed by them.  I monkey tested the web
interface and though it wasn't really fast it also did respond in not
too long of a time.  I'll queue up a todo for me to see if I can block
some of the inappropriate bots on debbugs but that doesn't seem urgent
as it seems to be handling the load of them okay.

I didn't see any mail waiting in the mailq on debbugs.  The main
problem I see in the mail logs is a problem with the SpamAssassin
Bayes due to a permission error.  Therefore the Bayes subsystem is
probably not working for SpamAssassin on incoming mail there.

I do not know Exim very well but I didn't see any obvious errors
happening.  But me not seeing errors in exim is far from any statement
of happiness there!  When it comes to Exim and myself we are pretty
much an estranged couple.

I happen to have my fingers poked into these things and so here I am.
But for all things debbugs I recommend using the help-debb...@gnu.org
mailing list as a good place to reach Glenn and the other volunteers
looking after the debbugs system.

Just as background information there is a slow process of trying to
upgrade the debbugs system.  It has stalled a little bit of late but
will get back going again at some point.

Hope that helps,
Bob



[Savannah-hackers-public] Savannah DDoS Attack

2019-12-02 Thread Bob Proulx
Savannah Community,

Savannah systems are getting hit by a DDoS attack.  A botnet is
browning out the web UIs on three of the systems.  This has been going
on all weekend.  The botnet is hitting the web interface randomly
selecting every possible URL.  If you can imagine every version of
every project file in every project you will know what is happening.

The attack started late Friday.  It is at least 10k IP addresses
strong and probably a lot bigger.  It's somewhat hard to tell the
exact size.  I know that vcs0 was hit by 45k addresses in 24 hours on
Saturday but I do not know how many of those were the botnet and how
many were just nice people like you and I clicking on the web browser.
But that seems a likely upper end.

Unfortunately we weren't previously collecting trend data on that
particular statistic for vcs0 and so I don't know what is a normal
daily rate.  Not that high by a lot however.  But at least for the
future moving forward we will have this data.  Things are running
about 30 requests per second on just vcs0 at this moment.  5/s on vcs1
and 10/s on frontend0.  And sometimes it spikes significantly higher.

We are working as best we can to try to block the attack and keep the
system limping along.  But you know how these DDoS attacks go.  If
someone wants you offline then there is really no way to stop them.

In the meantime I suggest using ssh:// protocol member access for all
of the version control backends.  Because that is not http/https it is
faring better.  Checkouts and commits should still be working.  It's
really just the web UI that is problematic.  The 502 Bad Gateway for
the interfaces that use it is somewhat transient in that if one
retries then it will eventually succeed through the botnet.

Bob



Re: [Savannah-hackers-public] 502 Bad Gateway for project called tops

2019-12-02 Thread Bob Proulx
Dale Williamson wrote:
> It is possible to log in and update sources at "cvs -z3
> -d:ext:d...@cvs.sv.nongnu.org:/sources/tops" but from a web browser
> viewvc/tops has a bad gateway.

The problem is that savannah systems are getting hit by a botnet.  It
is browning out the web UIs on three of the systems.  This has been
going on all weekend.  The botnet is hitting the web interface
randomly selecting every possible URL.  If you can imagine every
version of every project file in every project you will know what is
happening.

The attack started late Friday.  It is at least 10k IP addresses
strong and probably a lot bigger.  It's somewhat hard to tell the
exact size.  I know that vcs0 was hit by 45k addresses in 24 hours on
Saturday but I do not know how many of those were the botnet and how
many were just nice people like you and I clicking on the web browser.
But that seems a likely upper end.

Unfortunately we weren't previously collecting trend data on that
particular statistic for vcs0 and so I don't know what is a normal
daily rate.  Not that high by a lot however.  But at least for the
future moving forward we will have this data.  Things are running
about 30 requests per second on just vcs0 at this moment.  5/s on vcs1
and 10/s on frontend0.  And sometimes it spikes significantly higher.

We are working as best we can to try to block the attack and keep the
system limping along.  But you know how these DDoS attacks go.  If
someone wants you offline then there is really no way to stop them.

In the meantime I suggest using ssh:// protocol member access for all
of the version control backends.  Because that is not http/https it is
faring better.  Checkouts and commits should still be working.  It's
really just the web UI that is problematic.

The 502 Bad Gateway is somewhat transient in that if one retries then
it will eventually succeed through the botnet.

> Screen shots in Chrome are shown below.

Thanks for the report but in the future please simply type it up.
Plain text reports are always better.  "502 Bad Gateway" is more than
sufficient.  Those two attachments were 485KB and 324KB in size for
the images making for an 810KB mail message!  That's really quite
large for a mail message.  But don't let this stop you from reporting
problems in the future.  Please we welcome problem reports!

Bob



Re: [Savannah-hackers-public] Accessibility to Savannah's Git repositories

2019-12-02 Thread Bob Proulx
Eli Zaretskii wrote:
> Something is wrong with access to Git repositories on Savannah via
> HTTPS: attempts to do so come back with error 502.
> 
> Could someone please look into fixing that?

The problem is that savannah systems are getting hit by a botnet.  It
is browning out the web UIs on three of the systems.  This has been
going on all weekend.  The botnet is hitting the web interface
randomly selecting every possible URL.  If you can imagine every
version of every project file in every project you will know what is
happening.

The attack started late Friday.  It is at least 10k IP addresses
strong and probably a lot bigger.  It's somewhat hard to tell the
exact size.  I know that vcs0 was hit by 45k addresses in 24 hours on
Saturday but I do not know how many of those were the botnet and how
many were just nice people like you and I clicking on the web browser.
But that seems a likely upper end.

Unfortunately we weren't previously collecting trend data on that
particular statistic for vcs0 and so I don't know what is a normal
daily rate.  Not that high by a lot however.  But at least for the
future moving forward we will have this data.  Things are running
about 30 requests per second on just vcs0 at this moment.  5/s on vcs1
and 10/s on frontend0.  And sometimes it spikes significantly higher.

We are working as best we can to try to block the attack and keep the
system limping along.  But you know how these DDoS attacks go.  If
someone wants you offline then there is really no way to stop them.

In the meantime I suggest using ssh:// protocol member access for all
of the version control backends.  Because that is not http/https it is
faring better.  Checkouts and commits should still be working.  It's
really just the web UI that is problematic.

Bob



Re: [Savannah-hackers-public] Error on guix

2019-09-11 Thread Bob Proulx
Hello Alberto,

Alberto Garcia wrote:
> #guix pull
> Updating channel 'guix' from Git repository at '
> https://git.savannah.gnu.org/git/guix.git'...
> guix pull: error: Git error: failed to connect to git.savannah.gnu.org:
> Network is unreachable

I assume that the "guix pull" command is like "git pull"?  I tested
this by git cloning the repository.  I was not able to recreate any
problems accessing it.

Could it have been a temporary failure of networking between you and
the repository?  Due to the error message "Network is unreachable" I
think that is the most likely thing that happened.

Bob



Re: [Savannah-hackers-public] Can't commit to the project called tops

2019-09-09 Thread Bob Proulx
Dale Williamson wrote:
>cvs [commit aborted]: cannot stat /var/lock/cvs/sources/tops: No such
> file or directory

Unfixed damage from this:

  
https://lists.gnu.org/archive/html/savannah-hackers-public/2019-09/msg00011.html

Sorry this wasn't noticed early.  Will fix it.  Thank you for
reporting it!

Bob



[Savannah-hackers-public] [task #15385] git.savannah.gnu.org does not find its repositories any more

2019-09-09 Thread Bob Proulx
Update of task #15385 (project administration):

 Assigned to:None => rwp
 Open/Closed:Open => Closed 

___

Follow-up Comment #2:

TL;DR: Bad Linux kernel upgrade.  Postmortem writeup here:

https://lists.gnu.org/archive/html/savannah-hackers-public/2019-09/msg00011.html


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.nongnu.org/




Re: [Savannah-hackers-public] HTTPS problems with elpa.gnu.org

2019-07-20 Thread Bob Proulx
Savannah Hackers,

Does anyone know who is working with elpa?  If elpa reports are going
to flow into this mailing list it would be nice to know who to
redirect them too.

Tim Cross wrote:
> Trying to install the delight package from elpa.gnu.org fails with a 'Bad
> Request" error when using https connection. This occurs with emacs -Q from
> a GNU Linux (Ubuntu 19.04). Using HTTPS to retrieve packages from the MELPA
> repository works fine. Changing to use HTTP works fine as well. This was
> working previously (last time approx 2 weeks ago).

I suggest filing a ticket with sysad...@gnu.org as I don't think any
of us Savannah Hackers know anything about elpa.

> To test, M-x package-refrresh-contents will give the error "Failed to
> download 'gnu'' archive. When tested previously on maxOS it appeard to be
> working. Testing again just now and it also fails on that platform.

Yep.  Sounds broken.

Good luck!
Bob




Re: [Savannah-hackers-public] ELPA is down 20190718

2019-07-18 Thread Bob Proulx
Hi Tom,

Tom Alexander wrote:
> Ah okay, sorry for the spam then and thanks for the URL!

No worries.  We are happy to chat here.  Just unable to help! :-)

If you find out who is involved in elpa or what mailing lists they use
please come back here and tell us the scoop on it.  I have no idea who
is working on that project.  If I did then I could redirect people
there when they show up here.

Bob



Re: [Savannah-hackers-public] ELPA is down 20190718

2019-07-18 Thread Bob Proulx
Tom Alexander wrote:
> https://downforeveryoneorjustme.com/elpa.gnu.org
> 
> Found a post on reddit from an emacs maintainer suggesting that we should
> email this list when ELPA is down.

Well...  I don't know why they would suggest that.  I don't think
anyone here has anything to do with elpa.gnu.org.  Nothing we can do
about it.  I suggest opening a ticket with sysad...@gnu.org for such
things.  But anyway...

> I tried to access
> https://lists.gnu.org/mailman/listinfo/savannah-hackers-public to find the
> web archive to see if its already been reported but that appears to also be
> down, so my apologies if this has already been reported.

Please bookmark this URL where out of band status is posted by the FSF
admins when things happen making access to all FSF & GNU resources
unavailable.

  https://quitter.im/fsfstatus

Bob



Re: [Savannah-hackers-public] Project tops showing old files

2019-06-22 Thread Bob Proulx
Hello Dale,

Dale Williamson wrote:
> I believe I have narrowed down what is probably the machine that made
> the most recent commit.  In our commit log it was on Thu Jun 13 15:04:52
> PDT 2019 for file viewvc/tops/tops/sys/math.v.

That's all good information but I am hoping that was from before your
report and before I updated the "tops" project and sync'd in those
commits.  When I wrote my previous message I had already determined
that the tops project had newer commits in the nfs1 version of the
repository.  I had therefore sync'd the CVS source repository forward
from there to the live production copy.

The live copy since my previous message should be completely correct
for the tops project.

> On your system I just viewed in a browser that file, and it shows:
>Log of /tops/tops/sys/math.v
>Revision 1.6.2 Thu Jun 13 22:07:23 2019 UTC

CVS (and RCS) revisions are always pairs of numbers.  So 1.6.2 is
perhaps 1.62?

> This time is consistent with the log time (PST) above.  In the browser I
> was able to view "Diff to previous 1.6.1" and it matches the changes
> made.

  https://cvs.savannah.gnu.org/viewvc/tops/tops/sys/math.v?view=log
  Revision 1.62 - (view) (download) (annotate) - [select for diffs] 
  Thu Jun 13 22:07:23 2019 UTC (9 days, 5 hours ago) by dwil

> But on your system viewvc/tops/tops/sys/?sortby=date shows for file
> math.v: Rev 1.54 Age 2 months.  In fact, many files show age 2 months.
> 
> That is at odds with what I just viewed in a browser as Rev 1.62 Thu Jun
> 13 22:07:23 2019 UTC.

I am hoping that we were just working in parallel and so our emails
passed each other in flight.

> While this discrepancy on your system exists, we won't make any updates
> or commits from our machines since their CVS/Entries will probably not
> square with yours and could cause a huge mess.

Thank you!  It would definitely cause a problem similar to the
two-brain problem if commits were made to both repositories
separately.  Because then the same revisions of a file could have two
different sets of changes.  However for your project "tops" I think
everything should be okay.

In particular I compared the CVSROOT/history file which contains every
action that has happened to the two repositories.  I could see that
the commits you had noticed that had gone missing were in the nfs1
system's version but not the vcs1 systems version.  vcs1 is the
current live production copy.  Therefore I sync'd the nfs1 version
over to vcs1 so all should be good for the "tops" project now.

> Wishing you best of luck and awaiting word of your success.

I have been analyzing the repositories.  It looks like I was quite
lucky in that there were no occurrences of commits happening to both
versions after the failed sync.  It looks like the web pages were
sync'd correctly.  It looks like there were 18 source projects with
commits to the nfs1 version that were failed to copy back.  Your
project "tops" was one of those 18.  I have sync'd tops.  So you
should be good to go.  That leaves only 17 projects for me to
carefully review manually and to sync forward.  And if I can do so
quickly I may avoid the messiest of the problems.

Bob



Re: [Savannah-hackers-public] Project tops showing old files

2019-06-21 Thread Bob Proulx
Hello Dale,

> > > Newest files for project tops are showing as 2 months old.
> >
> > You didn't say and so there is no way for us to know but for what
> > projects are you referring?  And is this source code or web pages?
>
> This is the tops project. 
>  
> As an example, I log in to do updates with: 
>cvs -z3 -d:ext:d...@cvs.sv.nongnu.org:/sources/tops update

Oh!  I am sorry but I did not recognize "tops" but saw "project tops"
or the tops of projects and I thought maybe that was "main trunk" or
something like that.  Sorry.  My bad.

Let me dig into the problem.

Bob



Re: [Savannah-hackers-public] Changes to the CVS repo don't propagate to the website

2019-06-21 Thread Bob Proulx
I think thinkgs are working as they were before again.

Bob Proulx wrote:
> > Is this related to migration to the new VM?
> 
> Almost certainly it must be related.  But when I test uploading to a
> test project I do see the web site updated with the changes.

Maybe yes and maybe no.  The stuck process timestamps from about the
time of the move.  But the problem was entirely on www.gnu.org and it
is a problem that has happened before.  So maybe no.

> The way this works is a little bit of a "Rube Goldberg" device.  CVS
> updates trigger the loginfo hook processing which is simply a curl hit
> of a URL on the web server machine.  A python script on the web server
> then goes and updates the web site copy of the project.

Actually the python script only touches a file in a directory
indicating that a project was updated.  Then a cron script uses flock
on /tmp/update-cvs.lock to semaphore running update-cvs.sh every two
minutes to do the update.  It's update-cvs.sh that was stuck.

> Can you point me to a specific repository that is seeing this
> behavior?

With help from bandali on IRC I started looking at this page which was
not updated.

  https://www.gnu.org/software/recent-releases-include.html

Yet the repository clearly had an update.

  
http://web.cvs.savannah.gnu.org/viewvc/www/www/software/recent-releases-include.html?r1=1.973=1.974

So then I started poking around on www.gnu.org to figure out where
things were stuck.  I found this process tree.

  wwwcvs   29893 29889  0 Jun20 ?00:00:00   /usr/bin/flock -w2 -x 
/tmp/update-cvs.lock /usr/local/bin/update-cvs.sh
  wwwcvs   29896 29893  0 Jun20 ?00:00:03 /bin/bash 
/usr/local/bin/update-cvs.sh
  wwwcvs   26857 29896  0 Jun20 ?00:00:00   /usr/bin/cvs -d 
:pserver:anon...@cvs.savannah.gnu.org:/webcvs/pdsplibrary checkout -P 
pdsplibrary

The last log line from it:

  Thu Jun 20 16:15:09 EDT 2019
  pdsplibrary: Beginning checkout (type non-gnu)
  cvs2 checkout: Updating pdsplibrary
  ...
  cvs2 checkout: Updating pdsplibrary/libpdssn/refman/html

It is Fri, 21 Jun 2019 18:21:55 -0600 now.  So that one checkout has
been running for 24+ hours?  That seems to have been the problem as to
why web pages are not updating.

And this problem has happened before.  There is another cronjob that
runs once a day and sends an email to sysadmin opening a ticket saying
the following when this happens.  I remember Lisa setting that up for
me on a previous occasion when this had happened before.

  /usr/local/bin/update-cvs.sh is stuck on gnu.org

This is all on www.gnu.org which was not part of the migration.  I
killed process 26857 above since that was stuck.  That allowed the
update-cvs.sh to move on.  Exiting in this case.  Then the */2 cron to
run it started again.  This run it walked through the queued work.  It
is still running.  But I think it will be able to do what it does now.
Looking at this page I see it is updated now.

  https://www.gnu.org/software/recent-releases-include.html

I am going to keep an eye on it until it is through running.  It
appears to be making progress through the queued checkout work.  I am
surprised by number of directories with work queued.  Literally
thousands because it was still over a thousand and reducing before I
thought to look to see how many.

Bob



Re: [Savannah-hackers-public] Project tops showing old files

2019-06-21 Thread Bob Proulx
Hello Dale,

Thank you for making this report.

Dale Williamson wrote:
> Newest files for project tops are showing as 2 months old.
> 
> Updates were made last week, so it appears the system has been restored
> from backup.
> 
> What is the status of files less than 2 months old?

Of course it sends chills down my spine because this should not be
happening.  As announced to the savannah-users mailing list we did
just migrate from one server to another one but for that process cvs
services were disabled, the data was copied to the new server, then
cvs services were enabled.  The theory would go that all files would
have been transferred successfully.  There shouldn't have been a time
when any commits could go in that were not copied.  And indeed for my
testing all appeared to have happened normally.  So maybe it is
something that is related but is really something different?

You didn't say and so there is no way for us to know but for what
projects are you referring?  And is this source code or web pages?

Bob



Re: [Savannah-hackers-public] New VM vcs1 to hold CVS data with working file ACLs

2019-06-19 Thread Bob Proulx
Savannah Hackers,

Bob Proulx wrote:
> I am going to disable cvs on vcs0, sync the data from old vcs over to
> new vcs1, switch the DNS names for all things cvs over to vcs1, and
> then hold my breath that everything is working moving forward.  It
> should be working okay at that point.  I'll ask for expanded testing
> then from the www team.  We have the rest of this week before the
> datacenter move and the deadline for this.

This has now been done.  The live server and backend storage for all
things cvs is now the vcs1 VM.

Bob



  1   2   3   4   5   6   >