Re: [Users] Is this mkswap message from OpenVZ?

[Jeffrey Walton]

On Wed, Jan 8, 2020 at 3:01 PM Paulo Coghi - Coghi IT
 wrote:
>
> By default, OpenVZ doesn't allow swap creation inside containers. And this is 
> a good thing.

I think that depends on the metric used.

A small swapfile would be most useful and economical for us.

I think the lack of a swapfile and OOM kills are the worst choice
because they create a DoS in practice. Users cannot access our wiki
because MySQL was killed.

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] Is this mkswap message from OpenVZ?

[Jeffrey Walton]

Hi Everyone,

I'm trying to track down the cause of the error message below. It is
encountered when we try to enable a swapfile for a low-end VM. The VPS
provider uses OpenVZ.

# ./vm-swapfile.sh
Swapfile already allocated
swapon: /swapfile: swapon failed: Operation not permitted
Failed to enable swapfile

The relevant command is:

if ! swapon /swapfile; then
echo "Failed to enable swapfile"
fi

Is this mkswap message from OpenVZ?

Thanks in advance.
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] OpenVZ,, CentOS SCL and failed cron jobs?

[Jeffrey Walton]

On Fri, Dec 21, 2018 at 5:03 AM Jeffrey Walton  wrote:
>
> On Fri, Dec 21, 2018 at 4:18 AM Konstantin Khorenko
>  wrote:
> >
> > On 12/20/2018 07:39 PM, Jeffrey Walton wrote:
> > >
> > > I'm performing a post-mortem on our [failed] disaster recovery procedures.
> > >
> > > We have an OpenVZ-based CentOS 7 VM. We use it for an open source
> > > project website and wiki. Our backup job in /etc/cron.daily has not
> > > been executing (nor has other cron jobs, like yum-daily.cron). We
> > > cannot find mention of the failures in dmesg or other logs in
> > > /var/log.
> > >
> > > It looks like things broke sometime around December 2017 based on the
> > > date of our last backup. (It is embarrassing, but like I said there
> > > were no logged failures so I did not know to investigate). I don't
> > > keep change control logs, but the best I can tell our last two major
> > > configuration changes were:
> > >
> > > * Migrate OpenVZ 7.1 -> 7.2, June 2016
> > > * Enable CentOS SCL, December 2017
> > ...
> > unfortunately i have not heard about issues related with OpenVZ + SCL,
> > seems you are challenged to investigate it.
> >
> > i'd start with checking if cron service is run at all,
>
> Thanks Konstantin.
>
> I tracked it down to a daily cron job. Backup ran for 7 seconds but
> did not log its error:
>
> Dec  19 08:04:43 ftpit run-parts(/etc/cron.daily)[14217]: starting
> gdrive-backup
> Dec  19 08:04:50 ftpit run-parts(/etc/cron.daily)[17162]: finished
> gdrive-backup
>
> Our site is static so a 7 second backup seems reasonable to me for an
> incremental. (https://www.cryptopp.com)
>
> In reality this is what was happening (from the command line):
>
> # duplicity --allow-source-mismatch ...
> sftp://:y...@zonk.example.com:22480/backup
> ... Failed: No module named paramiko
>
> There is a Paramiko in the original Python. However, I failed to
> install Paramiko for the SCL version of Python. And exercising
> duplicity from the command line failed to reveal the problem:
>
> # duplicity --version
> duplicity 0.7.18.2
>
> In the end it looks like an exercise in why airplanes crash...
>
>   1. CentOS 7 ships with antique software
>   - users have to do something special to get into a good state
>   - users must enable SCL
>   2. SCL is missing software
>   - users have to do something special to get into a good state
>   - Components like Duplicity have to be built from sources
>   3. Linux paths are still broken
>   - users have to do something special to get into a good state
>   - 20 years or so and counting
>   4. Cron misreports job results
>   - swallows exceptions and errors
>   5. User (me) configured machine incorrectly
>   - SCL configuration was wrong
>   6. User (me) monitored machine incorrectly
>   - Did not detect cron job failures
>
> I'd like to strangle the idiot who thought it was a good idea to allow
> Cron to swallow exceptions and allow things to silently fail. I bet
> that genius is a CTO of a Fortune 500 company.

Re:

> There is a Paramiko in the original Python. However, I failed to
> install Paramiko for the SCL version of Python.

Looking at reports like
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/959089 , the
problems are not a one-off problem for us. Its a chronic problem
across distros that has not been fixed.

Packages and software need to be in a good state. They have to "just
work" out of the box. When are distros going to learn that RTFM does
not work? If it was going to work it would have happened in the last
50 years or so.

The engineers responsible for this mess meet the definition of insane.
They keep doing the same thing over and over again expecting a
different outcome. It is completely irrational behavior.

(end gripe)

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] OpenVZ,, CentOS SCL and failed cron jobs?

[Jeffrey Walton]

On Fri, Dec 21, 2018 at 4:18 AM Konstantin Khorenko
 wrote:
>
> On 12/20/2018 07:39 PM, Jeffrey Walton wrote:
> >
> > I'm performing a post-mortem on our [failed] disaster recovery procedures.
> >
> > We have an OpenVZ-based CentOS 7 VM. We use it for an open source
> > project website and wiki. Our backup job in /etc/cron.daily has not
> > been executing (nor has other cron jobs, like yum-daily.cron). We
> > cannot find mention of the failures in dmesg or other logs in
> > /var/log.
> >
> > It looks like things broke sometime around December 2017 based on the
> > date of our last backup. (It is embarrassing, but like I said there
> > were no logged failures so I did not know to investigate). I don't
> > keep change control logs, but the best I can tell our last two major
> > configuration changes were:
> >
> > * Migrate OpenVZ 7.1 -> 7.2, June 2016
> > * Enable CentOS SCL, December 2017
> ...
> unfortunately i have not heard about issues related with OpenVZ + SCL,
> seems you are challenged to investigate it.
>
> i'd start with checking if cron service is run at all,

Thanks Konstantin.

I tracked it down to a daily cron job. Backup ran for 7 seconds but
did not log its error:

Dec  19 08:04:43 ftpit run-parts(/etc/cron.daily)[14217]: starting
gdrive-backup
Dec  19 08:04:50 ftpit run-parts(/etc/cron.daily)[17162]: finished
gdrive-backup

Our site is static so a 7 second backup seems reasonable to me for an
incremental. (https://www.cryptopp.com)

In reality this is what was happening (from the command line):

# duplicity --allow-source-mismatch ...
sftp://:y...@zonk.example.com:22480/backup
... Failed: No module named paramiko

There is a Paramiko in the original Python. However, I failed to
install Paramiko for the SCL version of Python. And exercising
duplicity from the command line failed to reveal the problem:

# duplicity --version
duplicity 0.7.18.2

In the end it looks like an exercise in why airplanes crash...

  1. CentOS 7 ships with antique software
  - users have to do something special to get into a good state
  - users must enable SCL
  2. SCL is missing software
  - users have to do something special to get into a good state
  - Components like Duplicity have to be built from sources
  3. Linux paths are still broken
  - users have to do something special to get into a good state
  - 20 years or so and counting
  4. Cron misreports job results
  - swallows exceptions and errors
  5. User (me) configured machine incorrectly
  - SCL configuration was wrong
  6. User (me) monitored machine incorrectly
  - Did not detect cron job failures

I'd like to strangle the idiot who thought it was a good idea to allow
Cron to swallow exceptions and allow things to silently fail. I bet
that genius is a CTO of a Fortune 500 company.

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] OpenVZ,, CentOS SCL and failed cron jobs?

[Jeffrey Walton]

Hi Everyone,

I'm performing a post-mortem on our [failed] disaster recovery procedures.

We have an OpenVZ-based CentOS 7 VM. We use it for an open source
project website and wiki. Our backup job in /etc/cron.daily has not
been executing (nor has other cron jobs, like yum-daily.cron). We
cannot find mention of the failures in dmesg or other logs in
/var/log.

It looks like things broke sometime around December 2017 based on the
date of our last backup. (It is embarrassing, but like I said there
were no logged failures so I did not know to investigate). I don't
keep change control logs, but the best I can tell our last two major
configuration changes were:

* Migrate OpenVZ 7.1 -> 7.2, June 2016
* Enable CentOS SCL, December 2017

The SCL is Software
Collections,https://wiki.centos.org/AdditionalResources/Repositories/SCL
 . We needed it because of the ancient versions of Apache, Python and
PHP provided with CentOS 7.

My question is, is there a bad interaction or adverse relationship
with OpenVZ, SCL repos and cron?

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] OT: Meltdown and Spectre cpu bugs

[Jeffrey Walton]

Hi Everyone,

It looks like PoCs are starting to be released for the CPU bugs. Or
there's a PoC in the wild for ARM processors.

Early reports:
* https://amp.reddit.com/r/sysadmin/comments/7nl8r0/intel_bug_incoming/
* https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/

Latest news:
* 
https://www.theverge.com/2018/1/4/16848976/how-to-protect-windows-pc-meltdown-security-flaw

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] X86_BUG_CPU_INSECURE

[Jeffrey Walton]

On Wed, Jan 3, 2018 at 8:41 AM, Scott Dowdle  wrote:
> Greetings,
> ...
> I've also read that while some other CPU arches have hardware specifically to 
> avoid the issue, others may also have a similar issue.  The fix is said to be 
> ported to aarch64 and at least one other arch that I don't recall.  Perhaps 
> they are just trying to be over protective?
>

That's kind of interesting.

Did they specifically mention ARM Trust Zones (or violating them)?

Jeff

___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] X86_BUG_CPU_INSECURE

[Jeffrey Walton]

On Wed, Jan 3, 2018 at 4:19 AM, Narcis Garcia  wrote:
> As I understand from dasunsrule32's post, affected CPUs show a flag
> X86_BUG_CPU_INSECURE (?!).
> Does this mean that Intel is distributing CPUs marking them as
> defective?! ...or is this flag from kernel detection?
>
> + Is somebody listing fixed CPU models?
>
> Note: I suppose neither OpenVZ 6 nor LXC are affected by this hardware bug.

As I understand things, it affects Intel processors. AMD processors
are safe, as are other architectures like Aarch64 and SPARC64.

Also see https://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/ :


... In an email to the Linux kernel mailing list over Christmas, AMD
said it is not affected. The wording of that message, though, rather
gives the game away as to what the underlying cockup is:

AMD processors are not subject to the types of attacks that the kernel
page table isolation feature protects against. The AMD
microarchitecture does not allow memory references, including
speculative references, that access higher privileged data when
running in a lesser privileged mode when that access would result in a
page fault.


Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] Local VM for testing production VM

[Jeffrey Walton]

Hi Everyone,

We are a free/open source project. We rent a OpenVZ-based CentOS 7 VM
for a website and wiki. We need to get off PHP 5.4 and MediaWiki 1.26.

I'm setting up a local VM to test the migration of the production VM.
I have a few questions.

1. Is there a way to clone a rented VM over SSH?
- an exact image would be great to use for testing

2. Is there a tool which duplicates a configuration?
- we would like to use the production config for the local config

3. What are the interesting items in a config for testing?
   - high level bullet points

For question (1), we don't get a lot of support from the provider. We
mostly have to rely on things we can do at the user side of things.

My apologies for (3). I know it is a crummy question. We have an idea
we want the same repos and packages, but we are not sure what else may
contribute to the config in a significant way.

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] CentOS 7 image, ModSecurity and Fail2Ban?

[Jeffrey Walton]

> In brief:
> * OpenVZ 6 (2.6.32-x kernels) does not allow to use ipset inside Containers
> (it's just not virtualized)
> * OpenVZ 7 (3.10.0-x kernels) does have ipset virtualized => it works inside
> Containers.
>
> If you try fail2ban in OpenVZ 7, please post here the results. :)
>
> Hope that helps.

Thanks Konstantin and Scott. I appreciate the responses, especially
when the questions are open-ended, non-specific, and mostly crummy.

Are there any recommendations for fail2ban-like functionality in the
down-level kernel? Perhaps another package I am not aware of?

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] CentOS 7 image, ModSecurity and Fail2Ban?

[Jeffrey Walton]

Hi Everyone,

I'd like to install fail2ban to stop the unfettered:

script '/var/www/html/wp-login.php' not found or unable to stat
...

I'm OK with a fail2ban config, but I don't know about the OpenVZ
container, CnetOS and other negative interatctions that may cause me
to lose access to the VM :(

It also looks like we are using the 2.6.32 kernel with OpenVZ build
113.17 (if I am parsing the OpenVZ info correctly):

$ uname -a
Linux cryptopp 2.6.32-042stab113.17 #1 SMP Wed Feb 10 18:31:00 MSK
2016 x86_64 x86_64 x86_64 GNU/Linux

Has anyone experienced any problems with OpenVZ, CentOS 7 and fail2ban?

My apologies for the broad question, and thanks in advance.

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] "Start the VPS" frequently required from web admin page since CentOS 7.2 migration

[Jeffrey Walton]

Hi Everyone,

We use a hosting provider to host a website and wiki. Last week we
performed the CentOS 7.1 -> 7.2 migration. We issued 'sudo yum update'
and everything migrated cleanly. The process seemed to go well.

Since the upgrade it appears the VPS service frequently stops or
crashes. We're not sure which at the moment because we don't have
access to Dmesg and Syslog (more correctly, they are 0-bytes). We just
noticed the trend, but it seems to be about a daily occurrence.

I've searched for the symptoms, but I'm not getting hits in the
context of an issue. Most results are related to creating a container
and starting the server.

My questions are:

 * Is there a well known issue that someone can point me to?
 * Is there an alternate set of log files available for troubleshooting?
 * How do I troubleshoot in the absence of logs?

My apologies for the lack of information. I know its not very helpful.

Thanks in advance.
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] Warnings treated as errors

[Jeffrey Walton]

> Could you please tell me what and in which version you use to build ploop, 
> vzctl, and vzquota? I am using GCC 5.3 and get a lot of warnings (Inline, …) 
> which are treated as errors according to -Werror in CFLAGS.
>
This does not answer your question on versioning, but it may help
resolve the problem.

These look like they are part of the OpenVZ API
(http://www.openvz.org/). Perhaps you should configure without OpenVZ
support; or compile and link against the OpenVZ API.

Run the following to see how to configure without OpenVZ support:

./configure --help | grep -i openvz

The package should give you the option.

Jeff

___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

Re: [Users] CentOS 7.2 Upgrade with OpenVZ container and 2.6 kernel

[Jeffrey Walton]

>  > $ uname -a
>  > Linux cryptopp 2.6.32-042stab112.15 #1 SMP Tue Oct 20 17:22:56 \
>  > MSK 2015 x86_64 x86_64 x86_64 GNU
>  > ...
>  >
>  > Is it safe to perform the upgrade from 7.1 to 7.2 with the 2.6 kernel?
>  > That is, is this a supported configuration that will update as
>  > expected? If not, then what are our options?
>
> systemd quit working when one of my OpenVZ machines upgraded from CentOS
> 7.1 to 7.2.  I tried re-installing, and it did the same then when
> upgrading to 7.2.  That machine is running kernel 2.6.32-042stab093.5
> It's now working fine with CentOS 6.
>
> I have CentOS 7.2 working fine on OpenVZ with kernels
> 2.6.32-042stab108.8 and 2.6.32-042stab111.12.
>
> I also have a machine with 2.6.32-042stab094.8, and I don't know if that
> works or not, but 2.6.32-042stab112.15 should be fine (at least until
> the next update).

Very good data points. Thank you very much.

Jeff
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users

[Users] CentOS 7.2 Upgrade with OpenVZ container and 2.6 kernel

[Jeffrey Walton]

Hello everyone,

We rent a CentOS 7.1 VM:

$ cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

CentOS 7.2 was released recently
(http://wiki.centos.org/Manuals/ReleaseNotes/CentOS7) and yum is
prompting us to upgrade. In addition, its not giving us 7.1 security
updates.

We want to upgrade, but we are not sure if our configuration is
tested. The configuration includes a 2.6 kernel (not a 3.x kernel),
and we believe is due to OpenVZ container
(http://unix.stackexchange.com/q/238317):

$ uname -a
Linux cryptopp 2.6.32-042stab112.15 #1 SMP Tue Oct 20 17:22:56 \
MSK 2015 x86_64 x86_64 x86_64 GNU

We can't determine if this a supported configuration, so we are not
sure if its expected to upgrade. The release notes don't discuss it
(http://wiki.centos.org/Manuals/ReleaseNotes/CentOS7), and we could
not find information on it searching the list archives
(http://www.google.com/search?q="CentOS+7.2"+site:lists.openvz.org).

Is it safe to perform the upgrade from 7.1 to 7.2 with the 2.6 kernel?
That is, is this a supported configuration that will update as
expected? If not, then what are our options?

Thanks in advance.
___
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users