[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Sunday, April 19, 2020 at 9:21:44 AM UTC-4, Jeffrey Walton wrote:
>
>
> On Friday, April 17, 2020 at 3:02:13 AM UTC-4, Jeffrey Walton wrote:
>>
>>
>> The Crypto++ website is down.
>>
>> It appears GoDaddy is suspending the VM due to high cpu usage. GoDaddy 
>> writes it is due to Apache.
>>
>> We are not sure what is going on. We have not made any configuration 
>> changes since last summer. The only changes are documentation updates last 
>> week, which is copied new HTML files to a docs/ directory.
>>
>> I suspect something is sideways due to the OOM killer on the VM. OOM 
>> probably killed MySQL service, which wreaked havoc with Apache.
>>
>> At the moment I cannot SSH into the machine.
>>
>
> The VM is available again.
>
> It looks like someone from China is hammering the web server with useless 
> requests, and it is causing a DoS:
>
> 59.64.129.175 - - [19/Apr/2020:08:53:53 -0400] "GET
> /w/index.php?title=Special:WhatLinksHere=50&
> printable=yes HTTP/1.1" 301 311
>
> We are trying to block the netblock, but the netfilter rule is not present 
> after rebooting the machine.
>
> iptables -A INPUT -s 59.64.128.0/19 -p TCP -j DROP
>
> We cannot ifdown/ifup because we lose access the the VM. We have to reboot 
> to machine to effect changes.
>
> We're now trying to figure out what is wrong with netfilter.
>

Also see 
https://groups.google.com/d/msg/cryptopp-users/aDE9ter5mv0/YEi4fKSKAgAJ. 

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/cryptopp-users/c70b83e1-1993-4b40-83f4-2bc3c933897a%40googlegroups.com.

Re: [cryptopp-users] Re: Website is down

[Scott Thorpe]

Might be silly but make sure the logs are on log rotate so the system
doesn’t run out of space.

On Sun, Apr 19, 2020 at 7:21 AM Jeffrey Walton  wrote:

>
> On Friday, April 17, 2020 at 3:02:13 AM UTC-4, Jeffrey Walton wrote:
>>
>>
>> The Crypto++ website is down.
>>
>> It appears GoDaddy is suspending the VM due to high cpu usage. GoDaddy
>> writes it is due to Apache.
>>
>> We are not sure what is going on. We have not made any configuration
>> changes since last summer. The only changes are documentation updates last
>> week, which is copied new HTML files to a docs/ directory.
>>
>> I suspect something is sideways due to the OOM killer on the VM. OOM
>> probably killed MySQL service, which wreaked havoc with Apache.
>>
>> At the moment I cannot SSH into the machine.
>>
>
> The VM is available again.
>
> It looks like someone from China is hammering the web server with useless
> requests, and it is causing a DoS:
>
> 59.64.129.175 - - [19/Apr/2020:08:53:53 -0400] "GET
> /w/index.php?title=Special:WhatLinksHere=50&
> printable=yes HTTP/1.1" 301 311
>
> We are trying to block the netblock, but the netfilter rule is not present
> after rebooting the machine.
>
> iptables -A INPUT -s 59.64.128.0/19 -p TCP -j DROP
>
> We cannot ifdown/ifup because we lose access the the VM. We have to reboot
> to machine to effect changes.
>
> We're now trying to figure out what is wrong with netfilter.
>
> Jeff
>
> --
> You received this message because you are subscribed to "Crypto++ Users".
> More information about Crypto++ and this group is available at
> http://www.cryptopp.com and
> http://groups.google.com/forum/#!forum/cryptopp-users.
> ---
> You received this message because you are subscribed to the Google Groups
> "Crypto++ Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cryptopp-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cryptopp-users/7e4e9f82-6f69-44a2-8d5c-91eac49d84d8%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/cryptopp-users/CACt5jOmByHXSHNh3t37oU8%3D%3D0rYgsLpzHQfYUtJjLU_cs-dZRg%40mail.gmail.com.

[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Friday, April 17, 2020 at 3:02:13 AM UTC-4, Jeffrey Walton wrote:
>
>
> The Crypto++ website is down.
>
> It appears GoDaddy is suspending the VM due to high cpu usage. GoDaddy 
> writes it is due to Apache.
>
> We are not sure what is going on. We have not made any configuration 
> changes since last summer. The only changes are documentation updates last 
> week, which is copied new HTML files to a docs/ directory.
>
> I suspect something is sideways due to the OOM killer on the VM. OOM 
> probably killed MySQL service, which wreaked havoc with Apache.
>
> At the moment I cannot SSH into the machine.
>

The VM is available again.

It looks like someone from China is hammering the web server with useless 
requests, and it is causing a DoS:

59.64.129.175 - - [19/Apr/2020:08:53:53 -0400] "GET
/w/index.php?title=Special:WhatLinksHere=50&
printable=yes HTTP/1.1" 301 311

We are trying to block the netblock, but the netfilter rule is not present 
after rebooting the machine.

iptables -A INPUT -s 59.64.128.0/19 -p TCP -j DROP

We cannot ifdown/ifup because we lose access the the VM. We have to reboot 
to machine to effect changes.

We're now trying to figure out what is wrong with netfilter.

Jeff

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/cryptopp-users/7e4e9f82-6f69-44a2-8d5c-91eac49d84d8%40googlegroups.com.

[cryptopp-users] Re: Website is down

[hccc]

Hi Jeff, 
The site has crashed again. Can you recheck it ?

Vào 23:09:56 UTC+7 Thứ Bảy, ngày 26 tháng 10 năm 2019, Jeffrey Walton đã 
viết:
>
> Hi Everyone,
>
> It looks like the VM is not accessible. I don't have access to the 
> management console so I can't can't troubleshoot it.
>
> We pinged Wei and asked him to look into it.
>
> Jeff
>

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/cryptopp-users/0ba8c41f-d205-4d87-963a-62b90bb6ecac%40googlegroups.com.

[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Wednesday, January 23, 2019 at 1:22:09 AM UTC-5, Jeffrey Walton wrote:
>
>
> A spammer got around our defenses and hit our wiki. It looks like they 
> created about 65 accounts (maybe more) and dropped 998 new spam pages 
> starting on 31 December 2018. New pages 999 and 1000 are ones I created on 
> 27 December 2018 for Release Signing.
>
> I put the wiki in Read-Only mode (
> https://www.mediawiki.org/wiki/Manual:Lock_the_database) and lost control 
> of things. The website and wiki are down, and I got booted off my SSH 
> connection. I can't reestablish the connection for some reason.
>
> The website and wiki are down, and I need some time to get things back up.
>

Two things here

First, the IP address happened to change while I was working on the 
machine. Wei changed DNS record and firt problem solved.

Second I had to cleanup the spam by hand. There were no extensions to do 
what we needed. There were 11,000+ spam accounts setup over the years. 
10,000+ happened this month. Each of the January 2019 spam account added 
30-100 wiki pages, so we were full of junk...

I could not clean the spam one-by-one. It was too big a job. I settled on 
deleting all users who user_id>5. Then I deleted all pages that no longer 
had an author. That cleared all of the spam and some real content. (The 
original five user_id are folks like Wei, me and Soren Dreijer. Account 6 
was spam from 2004).

Feel free to recreate missing pages or point out missing content.

Jeff

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Thursday, December 20, 2018 at 6:34:29 AM UTC-5, Jeffrey Walton wrote:
>
> Hi Everyone,
>
> The website and wiki is down at the moment. I accidentally blew it away 
> while cleaning some old cruft from root's home directory. I tried to delete 
> a local backup of the site called html. The problem was, it was a link to 
> /var/www/html and not a local backup.
>

The website is mostly up.

The html was easy to restore since we keep it in GitHub. The manual was 
available yesterday because we just need to 'make docs', scp to the web 
server, and unzip in the right location. The wiki was trickier, and we got 
it restored this afternoon from backup.

The wiki has current content because the database was not destroyed. 
However, the wiki software is version 1.25, so it is a little flakier than 
desired. I'm going to perform another backup now and then upgrade to 1.30.

Sorry about all the troubles I caused.

Jeff

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Thursday, December 20, 2018 at 10:42:51 PM UTC-5, pauljorgensen wrote:
>
> Hopefully you can get most of it back I guess...


Yeah, me too. There's a lot of info in the wiki and I would hate to lose it.

I have not started a restore yet. I want to get a good backup before moving 
forward.

Jeff

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cryptopp-users] Re: Website is down

[pauljorgensen]

Hopefully you can get most of it back I guess...

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Thursday, December 20, 2018 at 7:47:24 AM UTC-5, Jeffrey Walton wrote:
>
>
> On Thursday, December 20, 2018 at 6:34:29 AM UTC-5, Jeffrey Walton wrote:
>>
>> The website and wiki is down at the moment. I accidentally blew it away 
>> while cleaning some old cruft from root's home directory. I tried to delete 
>> a local backup of the site called html. The problem was, it was a link to 
>> /var/www/html and not a local backup.
>>
>> We have an offsite backup created with duplicity 0.7.18.2, which was 
>> released October 2018. I think the pain point (for me) will be doing the 
>> restore from Google Drive.
>>
>> I'll start the restore shortly.
>>
>
> I was looking at Google Drive cryptopp_com_backup, and it looks like 
> Duplicity stopped working after about two months. The last backup we have 
> is dated December 2017. I'm guessing we are going to lose some wiki 
> content. Derp...
>

Well, some good news... It looks like the database is still in tact. Now I 
need to get it wired up to a fresh Mediawiki install.

MariaDB [(none)]> SELECT table_schema "DB Name", ROUND(SUM(data_length + 
index_length) / 1024 / 1024, 1) "DB Size in MB" FROM 
information_schema.tables GROUP BY table_schema;
++---+
| DB Name| DB Size in MB |
++---+
| information_schema |   0.1 |
| mysql  |   0.6 |
| my_wiki| 153.0 |
| performance_schema |   0.0 |
++---+
4 rows in set (0.18 sec)

Jeff

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cryptopp-users] Re: Website is down

[Jeffrey Walton]

On Thursday, December 20, 2018 at 6:34:29 AM UTC-5, Jeffrey Walton wrote:
>
> The website and wiki is down at the moment. I accidentally blew it away 
> while cleaning some old cruft from root's home directory. I tried to delete 
> a local backup of the site called html. The problem was, it was a link to 
> /var/www/html and not a local backup.
>
> We have an offsite backup created with duplicity 0.7.18.2, which was 
> released October 2018. I think the pain point (for me) will be doing the 
> restore from Google Drive.
>
> I'll start the restore shortly.
>

I was looking at Google Drive cryptopp_com_backup, and it looks like 
Duplicity stopped working after about two months. The last backup we have 
is dated December 2017. I'm guessing we are going to lose some wiki content. 
Derp...

The really annoying thing about this is, I monitor the messages in dmesg 
and /var/log, and I don't recall seeing any messages about Duplicity 
failures. In fact this returns 0 hits:

[root@ftpit html]# grep -iIR duplicity /var/log
[root@ftpit html]#

But our cron job is still present:

# ls /etc/cron.daily/
yum-daily.cron  gdrive-backup  logrotate   mlocate

I can't help but wonder, who the fuck thought it was a good idea to _not_ 
log problems in well known locations, like dmesg or /var/log/. That fucking 
genius is probably the CTO of a fortune 500 company...

Jeff
 

-- 
You received this message because you are subscribed to "Crypto++ Users". More 
information about Crypto++ and this group is available at 
http://www.cryptopp.com and 
http://groups.google.com/forum/#!forum/cryptopp-users.
--- 
You received this message because you are subscribed to the Google Groups 
"Crypto++ Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to cryptopp-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.