Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-14 Thread Dale
Mark Knecht wrote:
>
> > I'm planning on my new rig having the Ryzen 5900X.  Is the 5950
> better?  While I've kinda picked that one, I'm open to ideas if it is
> faster and I can afford it.  As it is, I'm looking at between $300 and
> $350 for the 5900.  My last CPU cost a little over $100.
> >
>
> I'm not going to say one is better than the other. The 5950X has more
> cores, the 5900X runs at a higher speed. It depends on your workload
> which will be better for you. I do a lot of things based around
> machine learning where I felt I was better off having more cores -
> give 12 to the ML job, keep 4 for my personal use. It's worked out
> well. However you don't ever talk much about what you actually use
> your computers for other than having 250 disk drives and moving data
> around your network. Depending on how you are moving data you might be
> better off with 5900X going faster.
>
> You can use this site to get some comparative data:
>
> https://cpu.userbenchmark.com/Compare/AMD-Ryzen-9-5950X-vs-AMD-Ryzen-9-5900X/4086vs4087
>
> BTW - you probably know both of these CPUs have been superseded by the
> 7900X and 7950X. THere's also the 3D versions which have faster and
> larger cache.
>
> > While at it.  In the past, I always bought the mobo, CPU and memory
> from the same place.  Generally if one of those is bad, it's sometimes
> hard to know which one is bad.  Sometimes even the BIOS beep codes are
> no help because there may be none.  If the mobo doesn't boot up, worst
> case, send all three back to the same place.  Given how far things
> have come, do I need to worry about a bad one out of the box anymore? 
> I can save some money if I buy from different places.
>
> Cannot answer but you need a return policy from every vendor. If it
> doesn't boot and you cannot figure it out you send everything back to
> multiple vendors I guess.
>
> Until recently I built all my machines myself. My 5900X machine has
> water cooling and I had cash so I paid a local storefront here to
> build it. I bought right in the middle of the pandemic and the chip
> shortage cost me huge dollars. Most expensive machine I've ever owned.
> Probably could build it today for less than 50% of what I paid.


Now that you mention that, there isn't much difference in price between
the two, depending on who I buy from.  There are some good deals for
used but I'm kinda leery about used.  You never know if they overclocked
it and ran it to hot or something and it shorten its life.  One thing
tho, different CPU socket, different mobo is needed.  Gotta start my
research over again.  ;-)

Hardest thing is finding mobo with several PCIe slots.  They have cut
way back on those. 

Dale

:-)  :-) 



Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-14 Thread Michael
On Saturday, 14 October 2023 04:21:23 BST Dale wrote:
> Frank Steinmetzger wrote:
> > Am Thu, Oct 12, 2023 at 08:35:21PM -0500 schrieb Dale:
> >> Frank Steinmetzger wrote:
> >>> Am Thu, Oct 12, 2023 at 10:44:39PM +0100 schrieb Michael:
>  Why don't you test throughput without encryption to confirm your
>  assumption?>>> 
> >>> What does `cryptsetup benchmark` say? I used to use a Celeron G1840 in
> >>> my
> >>> NAS, which is Intel Haswell without AES_NI. It was able to do ~ 150 MB/s
> >>> raw encryption throughput when transferring to or from a LUKS’ed image
> >>> in a ramdisk, so almost 150 % of gigabit ethernet speed.
> >> 
> >> […]
> >> I've never used that benchmark.  Didn't know it exists.  This is the
> >> results.  Keep in mind, fireball is my main rig.  The FX-8350 thingy. 
> >> The NAS is currently the old 770T system.  Sometimes it is a old Dell
> >> Inspiron but not this time.  ;-)
> >> 
> >> root@fireball / # cryptsetup benchmark
> >> […]
> >> # Algorithm |   Key |  Encryption |  Decryption
> >> aes-cbc128b63.8 MiB/s51.4 MiB/s
> >> serpent-cbc128b90.9 MiB/s   307.6 MiB/s
> >> twofish-cbc128b   200.4 MiB/s   218.4 MiB/s
> >> aes-cbc256b54.6 MiB/s37.5 MiB/s
> >> serpent-cbc256b90.4 MiB/s   302.6 MiB/s
> >> twofish-cbc256b   198.2 MiB/s   216.7 MiB/s
> >> aes-xts256b68.0 MiB/s45.0 MiB/s
> >> serpent-xts256b   231.9 MiB/s   227.6 MiB/s
> >> twofish-xts256b   191.8 MiB/s   163.1 MiB/s
> >> aes-xts512b42.4 MiB/s18.9 MiB/s
> >> serpent-xts512b   100.9 MiB/s   124.6 MiB/s
> >> twofish-xts512b   154.8 MiB/s   173.3 MiB/s
> >> root@fireball / #
> > 
> > Phew, this looks vry slow. As you can clearly see, this is not enough
> > to even saturate Gbit ethernet. Unfortunately, I don’t have any benchmark
> > data left over from the mentioned celeron.
> > (Perhaps that’s why the industry chose to implement AES in hardware,
> > because it was the slowest of the bunch.)
> > 
> > It looks like there is no hardware acceleration involved. But according to
> > https://en.wikipedia.org/wiki/List_of_AMD_FX_processors#Piledriver-based
> > and https://www.cpu-world.com/CPUs/Bulldozer/AMD-FX-Series%20FX-8350.html
> > it has the extension. I’d say something is amiss in your kernel.

Yes, I also think AES_NI has not been enabled in Dale's kernel config.

I just ran 'cryptsetup benchmark' on an A10-7850K APU (Kaveri Steamroller core 
as opposed to the 2 year older FX-8350 Vishera Piledriver core) and aes-xts 
fares much better;

# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1  1028015 iterations per second for 256-bit key
PBKDF2-sha2561464491 iterations per second for 256-bit key
PBKDF2-sha5121123875 iterations per second for 256-bit key
PBKDF2-ripemd160  708497 iterations per second for 256-bit key
PBKDF2-whirlpool  389515 iterations per second for 256-bit key
argon2i   5 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-
bit key (requested 2000 ms time)
argon2id  5 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-
bit key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
aes-cbc128b   586.6 MiB/s  2169.8 MiB/s
serpent-cbc128b88.6 MiB/s   330.6 MiB/s
twofish-cbc128b   203.1 MiB/s   277.6 MiB/s
aes-cbc256b   443.0 MiB/s  1712.5 MiB/s
serpent-cbc256b89.7 MiB/s   329.8 MiB/s
twofish-cbc256b   204.0 MiB/s   277.3 MiB/s
aes-xts256b  1840.9 MiB/s  1857.6 MiB/s <==
serpent-xts256b   288.4 MiB/s   299.6 MiB/s
twofish-xts256b   240.6 MiB/s   252.9 MiB/s
aes-xts512b  1459.3 MiB/s  1474.2 MiB/s <==
serpent-xts512b   291.6 MiB/s   299.0 MiB/s
twofish-xts512b   242.8 MiB/s   252.7 MiB/s


Whether 256 or 512 bit aes-xts performance would fill up a 1Gbps pipe.  
Without AES_NI the performance on this CPU is ~10 times slower.  I expect the 
FX-8350 would produce comparable results once the kernel crypto options are 
sorted.


> >>> Ah right, you use NFS. If not, I’d have suggested not to use rsync over
> >>> ssh, because that would indeed introduce a lot of encryption overhead.
> >> 
> >> I thought nfs was the proper way.  I use ssh and I use rsync,
> >> separately.  Didn't know they can be used together tho. 
> > 
> > When you do `rsync -ai source host:/path/to/destination/`, you use ssh for
> > transport.
> 
> Well, I may be doing this all wrong.  First, I ssh into the NAS box.  I
> do the decrypt stuff and mount the LV in it's proper place.  Then I
> 

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-14 Thread Michael
On Friday, 13 October 2023 18:01:41 BST Dale wrote:
> Michael wrote:
> > On Friday, 13 October 2023 02:35:21 BST Dale wrote:
> >> root@fireball / # cryptsetup benchmark
> >> # Tests are approximate using memory only (no storage IO).
> >> PBKDF2-sha1   878204 iterations per second for 256-bit key
> >> PBKDF2-sha256 911805 iterations per second for 256-bit key
> >> PBKDF2-sha512 698119 iterations per second for 256-bit key
> >> PBKDF2-ripemd160  548418 iterations per second for 256-bit key
> >> PBKDF2-whirlpool  299251 iterations per second for 256-bit key
> >> argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> >> for 256-bit key (requested 2000 ms time)
> >> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> >> for 256-bit key (requested 2000 ms time)
> >> # Algorithm |   Key |  Encryption |  Decryption
> >> 
> >> aes-cbc128b63.8 MiB/s51.4 MiB/s
> >> 
> >> serpent-cbc128b90.9 MiB/s   307.6 MiB/s
> >> twofish-cbc128b   200.4 MiB/s   218.4 MiB/s
> >> 
> >> aes-cbc256b54.6 MiB/s37.5 MiB/s
> >> 
> >> serpent-cbc256b90.4 MiB/s   302.6 MiB/s
> >> twofish-cbc256b   198.2 MiB/s   216.7 MiB/s
> >> 
> >> aes-xts256b68.0 MiB/s45.0 MiB/s
> >> 
> >> serpent-xts256b   231.9 MiB/s   227.6 MiB/s
> >> twofish-xts256b   191.8 MiB/s   163.1 MiB/s
> >> 
> >> aes-xts512b42.4 MiB/s18.9 MiB/s
> >> 
> >> serpent-xts512b   100.9 MiB/s   124.6 MiB/s
> >> twofish-xts512b   154.8 MiB/s   173.3 MiB/s
> >> 
> >> root@fireball / #
> >> 
> >> 
> >> 
> >> root@nas:~# cryptsetup benchmark
> >> # Tests are approximate using memory only (no storage IO).
> >> PBKDF2-sha1   741567 iterations per second for 256-bit key
> >> PBKDF2-sha256 910222 iterations per second for 256-bit key
> >> PBKDF2-sha512 781353 iterations per second for 256-bit key
> >> PBKDF2-ripemd160  547845 iterations per second for 256-bit key
> >> PBKDF2-whirlpool  350929 iterations per second for 256-bit key
> >> argon2i   4 iterations, 571787 memory, 4 parallel threads (CPUs) for
> >> 256-bit key (requested 2000 ms time)
> >> argon2id  4 iterations, 524288 memory, 4 parallel threads (CPUs) for
> >> 256-bit key (requested 2000 ms time)
> >> # Algorithm |   Key |  Encryption |  Decryption
> >> 
> >> aes-cbc128b   130.6 MiB/s   128.0 MiB/s
> >> 
> >> serpent-cbc128b64.7 MiB/s   161.8 MiB/s
> >> twofish-cbc128b   175.4 MiB/s   218.8 MiB/s
> >> 
> >> aes-cbc256b   120.1 MiB/s   122.2 MiB/s
> >> 
> >> serpent-cbc256b84.5 MiB/s   210.8 MiB/s
> >> twofish-cbc256b   189.5 MiB/s   218.6 MiB/s
> >> 
> >> aes-xts256b   167.0 MiB/s   162.1 MiB/s
> >> 
> >> serpent-xts256b   173.9 MiB/s   204.5 MiB/s
> >> twofish-xts256b   204.4 MiB/s   213.2 MiB/s
> >> 
> >> aes-xts512b   127.9 MiB/s   122.9 MiB/s
> >> 
> >> serpent-xts512b   201.5 MiB/s   204.7 MiB/s
> >> twofish-xts512b   215.0 MiB/s   213.0 MiB/s
> >> 
> >> root@nas:~#
> >> 
> >> 
> >> 
> >> Is that about what you would expect?  Fireball is on a 970 mobo.  It's
> >> slightly newer.  I think the 770T is about 2 years older, maybe 3.
> > 
> > grep AES /usr/src/linux/.config
> > 
> > or,
> > 
> > zgrep AES /proc/config.gz
> > 
> > Or, grep your *current* kernel config wherever it is stored.
> 
> I got the idea but assuming you wanted that info from the NAS box, I had
> to dig a little.  It's Ubuntu.  It doesn't have kernel sources, no
> config.gz in /proc either.  I found this.  I assume it is accurate. 
> Hopefully. 
> 
> 
> root@nas:~# cat /boot/config-5.15.0-86-generic | grep -i aes
> CONFIG_SND_MAESTRO3=m
> CONFIG_SND_MAESTRO3_INPUT=y
> CONFIG_CRYPTO_AEGIS128_AESNI_SSE2=m
> CONFIG_CRYPTO_AES=y
> CONFIG_CRYPTO_AES_TI=m
> CONFIG_CRYPTO_AES_NI_INTEL=m
> CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
> CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
> CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64=m
> CONFIG_CRYPTO_SM4_AESNI_AVX2_X86_64=m
> CONFIG_CRYPTO_DEV_PADLOCK_AES=m
> CONFIG_CRYPTO_LIB_AES=y
> root@nas:~#
> 
> 
> I don't usually use modules.  So, this is not something I run into
> much.  I'm adding this info since I think it will help as well. 
> 
> 
> root@nas:~# lsmod | grep -i aes
> root@nas:~#

The module is not loaded, hence the pedestrian performance.

Check if the CPU has the AES instruction set - I expect it doesn't, or the 
module(s) would have been loaded:

grep -m1 -o aes /proc/cpuinfo

Ah! Just read your other message, the Phenom II 

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Dale
Frank Steinmetzger wrote:
> Am Thu, Oct 12, 2023 at 08:35:21PM -0500 schrieb Dale:
>
>> Frank Steinmetzger wrote:
>>> Am Thu, Oct 12, 2023 at 10:44:39PM +0100 schrieb Michael:
>>>
 Why don't you test throughput without encryption to confirm your 
 assumption?
>>> What does `cryptsetup benchmark` say? I used to use a Celeron G1840 in my 
>>> NAS, which is Intel Haswell without AES_NI. It was able to do ~ 150 MB/s 
>>> raw 
>>> encryption throughput when transferring to or from a LUKS’ed image in a 
>>> ramdisk, so almost 150 % of gigabit ethernet speed.
>> […]
>> I've never used that benchmark.  Didn't know it exists.  This is the
>> results.  Keep in mind, fireball is my main rig.  The FX-8350 thingy. 
>> The NAS is currently the old 770T system.  Sometimes it is a old Dell
>> Inspiron but not this time.  ;-)
>>
>> root@fireball / # cryptsetup benchmark
>> […]
>> # Algorithm |   Key |  Encryption |  Decryption
>>     aes-cbc    128b    63.8 MiB/s    51.4 MiB/s
>>     serpent-cbc    128b    90.9 MiB/s   307.6 MiB/s
>>     twofish-cbc    128b   200.4 MiB/s   218.4 MiB/s
>>     aes-cbc    256b    54.6 MiB/s    37.5 MiB/s
>>     serpent-cbc    256b    90.4 MiB/s   302.6 MiB/s
>>     twofish-cbc    256b   198.2 MiB/s   216.7 MiB/s
>>     aes-xts    256b    68.0 MiB/s    45.0 MiB/s
>>     serpent-xts    256b   231.9 MiB/s   227.6 MiB/s
>>     twofish-xts    256b   191.8 MiB/s   163.1 MiB/s
>>     aes-xts    512b    42.4 MiB/s    18.9 MiB/s
>>     serpent-xts    512b   100.9 MiB/s   124.6 MiB/s
>>     twofish-xts    512b   154.8 MiB/s   173.3 MiB/s
>> root@fireball / #
> Phew, this looks vry slow. As you can clearly see, this is not enough to 
> even saturate Gbit ethernet. Unfortunately, I don’t have any benchmark data 
> left over from the mentioned celeron.
> (Perhaps that’s why the industry chose to implement AES in hardware, because 
> it was the slowest of the bunch.)
>
> It looks like there is no hardware acceleration involved. But according to 
> https://en.wikipedia.org/wiki/List_of_AMD_FX_processors#Piledriver-based and 
> https://www.cpu-world.com/CPUs/Bulldozer/AMD-FX-Series%20FX-8350.html it has 
> the extension. I’d say something is amiss in your kernel.
>
> Heck, even my ultra-low-end eeepc with its no-AES Atom processor N450 from 
> 2009 is less than 50 % slower, and for aes-xts 512b it is actually faster! 
> And that was a snail even in its day. It is so low-end that its in-order 
> architecture is not vulnerable to spectre and meltdown. :D It just scrunched 
> several minutes on updating the GPG keyring of its arch linux installation.
>
> eeePC # LC_ALL=C cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   228348 iterations per second for 256-bit key
> PBKDF2-sha256 335222 iterations per second for 256-bit key
> PBKDF2-sha512 253034 iterations per second for 256-bit key
> PBKDF2-ripemd160  172690 iterations per second for 256-bit key
> PBKDF2-whirlpool   94705 iterations per second for 256-bit key
> argon2i   4 iterations, 71003 memory, 4 parallel threads (CPUs) for 
> 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 71506 memory, 4 parallel threads (CPUs) for 
> 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b31.0 MiB/s33.6 MiB/s
> serpent-cbc128b28.1 MiB/s62.9 MiB/s
> twofish-cbc128b28.6 MiB/s31.0 MiB/s
> aes-cbc256b24.0 MiB/s25.6 MiB/s
> serpent-cbc256b28.3 MiB/s62.7 MiB/s
> twofish-cbc256b28.6 MiB/s31.0 MiB/s
> aes-xts256b32.5 MiB/s33.4 MiB/s
> serpent-xts256b50.5 MiB/s60.5 MiB/s
> twofish-xts256b25.6 MiB/s30.7 MiB/s
> aes-xts512b25.0 MiB/s25.6 MiB/s
> serpent-xts512b60.2 MiB/s60.4 MiB/s
> twofish-xts512b30.2 MiB/s30.7 MiB/s
>
>> root@nas:~# cryptsetup benchmark
>> […]
>> # Algorithm |   Key |  Encryption |  Decryption
>>     aes-cbc    128b   130.6 MiB/s   128.0 MiB/s
>>     serpent-cbc    128b    64.7 MiB/s   161.8 MiB/s
>>     twofish-cbc    128b   175.4 MiB/s   218.8 MiB/s
>>     aes-cbc    256b   120.1 MiB/s   122.2 MiB/s
>>     serpent-cbc    256b    84.5 MiB/s   210.8 MiB/s
>>     twofish-cbc    256b   189.5 MiB/s   218.6 MiB/s
>>     aes-xts    256b   167.0 MiB/s   162.1 MiB/s
>>     serpent-xts    256b   173.9 MiB/s   204.5 MiB/s
>>     twofish-xts    256b   204.4 MiB/s   

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Frank Steinmetzger
Am Thu, Oct 12, 2023 at 08:35:21PM -0500 schrieb Dale:

> Frank Steinmetzger wrote:
> > Am Thu, Oct 12, 2023 at 10:44:39PM +0100 schrieb Michael:
> >
> >> Why don't you test throughput without encryption to confirm your 
> >> assumption?
> > What does `cryptsetup benchmark` say? I used to use a Celeron G1840 in my 
> > NAS, which is Intel Haswell without AES_NI. It was able to do ~ 150 MB/s 
> > raw 
> > encryption throughput when transferring to or from a LUKS’ed image in a 
> > ramdisk, so almost 150 % of gigabit ethernet speed.
> […]
> I've never used that benchmark.  Didn't know it exists.  This is the
> results.  Keep in mind, fireball is my main rig.  The FX-8350 thingy. 
> The NAS is currently the old 770T system.  Sometimes it is a old Dell
> Inspiron but not this time.  ;-)
> 
> root@fireball / # cryptsetup benchmark
> […]
> # Algorithm |   Key |  Encryption |  Decryption
>     aes-cbc    128b    63.8 MiB/s    51.4 MiB/s
>     serpent-cbc    128b    90.9 MiB/s   307.6 MiB/s
>     twofish-cbc    128b   200.4 MiB/s   218.4 MiB/s
>     aes-cbc    256b    54.6 MiB/s    37.5 MiB/s
>     serpent-cbc    256b    90.4 MiB/s   302.6 MiB/s
>     twofish-cbc    256b   198.2 MiB/s   216.7 MiB/s
>     aes-xts    256b    68.0 MiB/s    45.0 MiB/s
>     serpent-xts    256b   231.9 MiB/s   227.6 MiB/s
>     twofish-xts    256b   191.8 MiB/s   163.1 MiB/s
>     aes-xts    512b    42.4 MiB/s    18.9 MiB/s
>     serpent-xts    512b   100.9 MiB/s   124.6 MiB/s
>     twofish-xts    512b   154.8 MiB/s   173.3 MiB/s
> root@fireball / #

Phew, this looks vry slow. As you can clearly see, this is not enough to 
even saturate Gbit ethernet. Unfortunately, I don’t have any benchmark data 
left over from the mentioned celeron.
(Perhaps that’s why the industry chose to implement AES in hardware, because 
it was the slowest of the bunch.)

It looks like there is no hardware acceleration involved. But according to 
https://en.wikipedia.org/wiki/List_of_AMD_FX_processors#Piledriver-based and 
https://www.cpu-world.com/CPUs/Bulldozer/AMD-FX-Series%20FX-8350.html it has 
the extension. I’d say something is amiss in your kernel.

Heck, even my ultra-low-end eeepc with its no-AES Atom processor N450 from 
2009 is less than 50 % slower, and for aes-xts 512b it is actually faster! 
And that was a snail even in its day. It is so low-end that its in-order 
architecture is not vulnerable to spectre and meltdown. :D It just scrunched 
several minutes on updating the GPG keyring of its arch linux installation.

eeePC # LC_ALL=C cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1   228348 iterations per second for 256-bit key
PBKDF2-sha256 335222 iterations per second for 256-bit key
PBKDF2-sha512 253034 iterations per second for 256-bit key
PBKDF2-ripemd160  172690 iterations per second for 256-bit key
PBKDF2-whirlpool   94705 iterations per second for 256-bit key
argon2i   4 iterations, 71003 memory, 4 parallel threads (CPUs) for 256-bit 
key (requested 2000 ms time)
argon2id  4 iterations, 71506 memory, 4 parallel threads (CPUs) for 256-bit 
key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
aes-cbc128b31.0 MiB/s33.6 MiB/s
serpent-cbc128b28.1 MiB/s62.9 MiB/s
twofish-cbc128b28.6 MiB/s31.0 MiB/s
aes-cbc256b24.0 MiB/s25.6 MiB/s
serpent-cbc256b28.3 MiB/s62.7 MiB/s
twofish-cbc256b28.6 MiB/s31.0 MiB/s
aes-xts256b32.5 MiB/s33.4 MiB/s
serpent-xts256b50.5 MiB/s60.5 MiB/s
twofish-xts256b25.6 MiB/s30.7 MiB/s
aes-xts512b25.0 MiB/s25.6 MiB/s
serpent-xts512b60.2 MiB/s60.4 MiB/s
twofish-xts512b30.2 MiB/s30.7 MiB/s

> root@nas:~# cryptsetup benchmark
> […]
> # Algorithm |   Key |  Encryption |  Decryption
>     aes-cbc    128b   130.6 MiB/s   128.0 MiB/s
>     serpent-cbc    128b    64.7 MiB/s   161.8 MiB/s
>     twofish-cbc    128b   175.4 MiB/s   218.8 MiB/s
>     aes-cbc    256b   120.1 MiB/s   122.2 MiB/s
>     serpent-cbc    256b    84.5 MiB/s   210.8 MiB/s
>     twofish-cbc    256b   189.5 MiB/s   218.6 MiB/s
>     aes-xts    256b   167.0 MiB/s   162.1 MiB/s
>     serpent-xts    256b   173.9 MiB/s   204.5 MiB/s
>     twofish-xts    256b   204.4 MiB/s   213.2 MiB/s
>     aes-xts    512b   127.9 MiB/s   122.9 MiB/s
>     serpent-xts    512b   201.5 MiB/s   204.7 MiB/s

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Mark Knecht
> I'm planning on my new rig having the Ryzen 5900X.  Is the 5950 better?
While I've kinda picked that one, I'm open to ideas if it is faster and I
can afford it.  As it is, I'm looking at between $300 and $350 for the
5900.  My last CPU cost a little over $100.
>

I'm not going to say one is better than the other. The 5950X has more
cores, the 5900X runs at a higher speed. It depends on your workload which
will be better for you. I do a lot of things based around machine learning
where I felt I was better off having more cores - give 12 to the ML job,
keep 4 for my personal use. It's worked out well. However you don't ever
talk much about what you actually use your computers for other than having
250 disk drives and moving data around your network. Depending on how you
are moving data you might be better off with 5900X going faster.

You can use this site to get some comparative data:

https://cpu.userbenchmark.com/Compare/AMD-Ryzen-9-5950X-vs-AMD-Ryzen-9-5900X/4086vs4087

BTW - you probably know both of these CPUs have been superseded by the
7900X and 7950X. THere's also the 3D versions which have faster and larger
cache.

> While at it.  In the past, I always bought the mobo, CPU and memory from
the same place.  Generally if one of those is bad, it's sometimes hard to
know which one is bad.  Sometimes even the BIOS beep codes are no help
because there may be none.  If the mobo doesn't boot up, worst case, send
all three back to the same place.  Given how far things have come, do I
need to worry about a bad one out of the box anymore?  I can save some
money if I buy from different places.

Cannot answer but you need a return policy from every vendor. If it doesn't
boot and you cannot figure it out you send everything back to multiple
vendors I guess.

Until recently I built all my machines myself. My 5900X machine has water
cooling and I had cash so I paid a local storefront here to build it. I
bought right in the middle of the pandemic and the chip shortage cost me
huge dollars. Most expensive machine I've ever owned. Probably could build
it today for less than 50% of what I paid.


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Dale
Mark Knecht wrote:
>
>
> On Fri, Oct 13, 2023 at 10:02 AM Dale  > wrote:
> >
> > Michael wrote:
> >
> > On Friday, 13 October 2023 02:35:21 BST Dale wrote:
> >
> > root@fireball / # cryptsetup benchmark
> > # Tests are approximate using memory only (no storage IO).
> > PBKDF2-sha1       878204 iterations per second for 256-bit key
> > PBKDF2-sha256     911805 iterations per second for 256-bit key
> > PBKDF2-sha512     698119 iterations per second for 256-bit key
> > PBKDF2-ripemd160  548418 iterations per second for 256-bit key
> > PBKDF2-whirlpool  299251 iterations per second for 256-bit key
> > argon2i       4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> > for 256-bit key (requested 2000 ms time)
> > argon2id      4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> > for 256-bit key (requested 2000 ms time)
> > #     Algorithm |       Key |      Encryption |      Decryption
> >         aes-cbc        128b        63.8 MiB/s        51.4 MiB/s
> >     serpent-cbc        128b        90.9 MiB/s       307.6 MiB/s
> >     twofish-cbc        128b       200.4 MiB/s       218.4 MiB/s
> >         aes-cbc        256b        54.6 MiB/s        37.5 MiB/s
> >     serpent-cbc        256b        90.4 MiB/s       302.6 MiB/s
> >     twofish-cbc        256b       198.2 MiB/s       216.7 MiB/s
> >         aes-xts        256b        68.0 MiB/s        45.0 MiB/s
> >     serpent-xts        256b       231.9 MiB/s       227.6 MiB/s
> >     twofish-xts        256b       191.8 MiB/s       163.1 MiB/s
> >         aes-xts        512b        42.4 MiB/s        18.9 MiB/s
> >     serpent-xts        512b       100.9 MiB/s       124.6 MiB/s
> >     twofish-xts        512b       154.8 MiB/s       173.3 MiB/s
> > root@fireball / #
> >
> >
> >
> > root@nas:~# cryptsetup benchmark
> > # Tests are approximate using memory only (no storage IO).
> > PBKDF2-sha1       741567 iterations per second for 256-bit key
> > PBKDF2-sha256     910222 iterations per second for 256-bit key
> > PBKDF2-sha512     781353 iterations per second for 256-bit key
> > PBKDF2-ripemd160  547845 iterations per second for 256-bit key
> > PBKDF2-whirlpool  350929 iterations per second for 256-bit key
> > argon2i       4 iterations, 571787 memory, 4 parallel threads (CPUs) for
> > 256-bit key (requested 2000 ms time)
> > argon2id      4 iterations, 524288 memory, 4 parallel threads (CPUs) for
> > 256-bit key (requested 2000 ms time)
> > #     Algorithm |       Key |      Encryption |      Decryption
> >         aes-cbc        128b       130.6 MiB/s       128.0 MiB/s
> >     serpent-cbc        128b        64.7 MiB/s       161.8 MiB/s
> >     twofish-cbc        128b       175.4 MiB/s       218.8 MiB/s
> >         aes-cbc        256b       120.1 MiB/s       122.2 MiB/s
> >     serpent-cbc        256b        84.5 MiB/s       210.8 MiB/s
> >     twofish-cbc        256b       189.5 MiB/s       218.6 MiB/s
> >         aes-xts        256b       167.0 MiB/s       162.1 MiB/s
> >     serpent-xts        256b       173.9 MiB/s       204.5 MiB/s
> >     twofish-xts        256b       204.4 MiB/s       213.2 MiB/s
> >         aes-xts        512b       127.9 MiB/s       122.9 MiB/s
> >     serpent-xts        512b       201.5 MiB/s       204.7 MiB/s
> >     twofish-xts        512b       215.0 MiB/s       213.0 MiB/s
> > root@nas:~#
> >
> >
> >
> > Is that about what you would expect?  Fireball is on a 970 mobo.  It's
> > slightly newer.  I think the 770T is about 2 years older, maybe 3.
> >
>
> THis was just for kicks because I think somewhere, this thread or some
> other I think you mentioned a Ryzen 5900 and mine is a 5950, now about
> 18 months old:
>
> mark@science2:~$ cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1      2212185 iterations per second for 256-bit key
> PBKDF2-sha256    4161015 iterations per second for 256-bit key
> PBKDF2-sha512    1798586 iterations per second for 256-bit key
> PBKDF2-ripemd160  841553 iterations per second for 256-bit key
> PBKDF2-whirlpool  675628 iterations per second for 256-bit key
> argon2i      11 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000
> ms time)
> argon2id     11 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000
> ms time)
> #     Algorithm |       Key |      Encryption |      Decryption
>        aes-cbc        128b      1181.2 MiB/s      5132.1 MiB/s
>    serpent-cbc        128b       107.8 MiB/s       426.1 MiB/s
>    twofish-cbc        128b       221.1 MiB/s       418.1 MiB/s
>        aes-cbc        256b       890.1 MiB/s      4167.7 MiB/s
>    serpent-cbc        256b       116.0 MiB/s       428.3 MiB/s
>    twofish-cbc        256b       224.2 MiB/s       417.7 MiB/s
>        aes-xts        256b      4121.7 MiB/s      4115.7 MiB/s
>    serpent-xts        256b       385.9 MiB/s       401.6 MiB/s
>    twofish-xts        256b       394.5 MiB/s       405.0 MiB/s
> 

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Mark Knecht
On Fri, Oct 13, 2023 at 10:02 AM Dale  wrote:
>
> Michael wrote:
>
> On Friday, 13 October 2023 02:35:21 BST Dale wrote:
>
> root@fireball / # cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   878204 iterations per second for 256-bit key
> PBKDF2-sha256 911805 iterations per second for 256-bit key
> PBKDF2-sha512 698119 iterations per second for 256-bit key
> PBKDF2-ripemd160  548418 iterations per second for 256-bit key
> PBKDF2-whirlpool  299251 iterations per second for 256-bit key
> argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b63.8 MiB/s51.4 MiB/s
> serpent-cbc128b90.9 MiB/s   307.6 MiB/s
> twofish-cbc128b   200.4 MiB/s   218.4 MiB/s
> aes-cbc256b54.6 MiB/s37.5 MiB/s
> serpent-cbc256b90.4 MiB/s   302.6 MiB/s
> twofish-cbc256b   198.2 MiB/s   216.7 MiB/s
> aes-xts256b68.0 MiB/s45.0 MiB/s
> serpent-xts256b   231.9 MiB/s   227.6 MiB/s
> twofish-xts256b   191.8 MiB/s   163.1 MiB/s
> aes-xts512b42.4 MiB/s18.9 MiB/s
> serpent-xts512b   100.9 MiB/s   124.6 MiB/s
> twofish-xts512b   154.8 MiB/s   173.3 MiB/s
> root@fireball / #
>
>
>
> root@nas:~# cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   741567 iterations per second for 256-bit key
> PBKDF2-sha256 910222 iterations per second for 256-bit key
> PBKDF2-sha512 781353 iterations per second for 256-bit key
> PBKDF2-ripemd160  547845 iterations per second for 256-bit key
> PBKDF2-whirlpool  350929 iterations per second for 256-bit key
> argon2i   4 iterations, 571787 memory, 4 parallel threads (CPUs) for
> 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 524288 memory, 4 parallel threads (CPUs) for
> 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b   130.6 MiB/s   128.0 MiB/s
> serpent-cbc128b64.7 MiB/s   161.8 MiB/s
> twofish-cbc128b   175.4 MiB/s   218.8 MiB/s
> aes-cbc256b   120.1 MiB/s   122.2 MiB/s
> serpent-cbc256b84.5 MiB/s   210.8 MiB/s
> twofish-cbc256b   189.5 MiB/s   218.6 MiB/s
> aes-xts256b   167.0 MiB/s   162.1 MiB/s
> serpent-xts256b   173.9 MiB/s   204.5 MiB/s
> twofish-xts256b   204.4 MiB/s   213.2 MiB/s
> aes-xts512b   127.9 MiB/s   122.9 MiB/s
> serpent-xts512b   201.5 MiB/s   204.7 MiB/s
> twofish-xts512b   215.0 MiB/s   213.0 MiB/s
> root@nas:~#
>
>
>
> Is that about what you would expect?  Fireball is on a 970 mobo.  It's
> slightly newer.  I think the 770T is about 2 years older, maybe 3.
>

THis was just for kicks because I think somewhere, this thread or some
other I think you mentioned a Ryzen 5900 and mine is a 5950, now about
18 months old:

mark@science2:~$ cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1  2212185 iterations per second for 256-bit key
PBKDF2-sha2564161015 iterations per second for 256-bit key
PBKDF2-sha5121798586 iterations per second for 256-bit key
PBKDF2-ripemd160  841553 iterations per second for 256-bit key
PBKDF2-whirlpool  675628 iterations per second for 256-bit key
argon2i  11 iterations, 1048576 memory, 4 parallel threads (CPUs) for
256-bit key (requested 2000
ms time)
argon2id 11 iterations, 1048576 memory, 4 parallel threads (CPUs) for
256-bit key (requested 2000
ms time)
# Algorithm |   Key |  Encryption |  Decryption
   aes-cbc128b  1181.2 MiB/s  5132.1 MiB/s
   serpent-cbc128b   107.8 MiB/s   426.1 MiB/s
   twofish-cbc128b   221.1 MiB/s   418.1 MiB/s
   aes-cbc256b   890.1 MiB/s  4167.7 MiB/s
   serpent-cbc256b   116.0 MiB/s   428.3 MiB/s
   twofish-cbc256b   224.2 MiB/s   417.7 MiB/s
   aes-xts256b  4121.7 MiB/s  4115.7 MiB/s
   serpent-xts256b   385.9 MiB/s   401.6 MiB/s
   twofish-xts256b   394.5 MiB/s   405.0 MiB/s
   aes-xts512b  3480.2 MiB/s  3486.3 MiB/s
   serpent-xts512b   408.9 MiB/s   401.4 MiB/s
   twofish-xts512b   395.9 MiB/s   404.8 MiB/s
mark@science2:~$


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Dale
Michael wrote:
> On Friday, 13 October 2023 02:35:21 BST Dale wrote:
>
>> root@fireball / # cryptsetup benchmark
>> # Tests are approximate using memory only (no storage IO).
>> PBKDF2-sha1   878204 iterations per second for 256-bit key
>> PBKDF2-sha256 911805 iterations per second for 256-bit key
>> PBKDF2-sha512 698119 iterations per second for 256-bit key
>> PBKDF2-ripemd160  548418 iterations per second for 256-bit key
>> PBKDF2-whirlpool  299251 iterations per second for 256-bit key
>> argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
>> for 256-bit key (requested 2000 ms time)
>> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
>> for 256-bit key (requested 2000 ms time)
>> # Algorithm |   Key |  Encryption |  Decryption
>> aes-cbc128b63.8 MiB/s51.4 MiB/s
>> serpent-cbc128b90.9 MiB/s   307.6 MiB/s
>> twofish-cbc128b   200.4 MiB/s   218.4 MiB/s
>> aes-cbc256b54.6 MiB/s37.5 MiB/s
>> serpent-cbc256b90.4 MiB/s   302.6 MiB/s
>> twofish-cbc256b   198.2 MiB/s   216.7 MiB/s
>> aes-xts256b68.0 MiB/s45.0 MiB/s
>> serpent-xts256b   231.9 MiB/s   227.6 MiB/s
>> twofish-xts256b   191.8 MiB/s   163.1 MiB/s
>> aes-xts512b42.4 MiB/s18.9 MiB/s
>> serpent-xts512b   100.9 MiB/s   124.6 MiB/s
>> twofish-xts512b   154.8 MiB/s   173.3 MiB/s
>> root@fireball / #
>>
>>
>>
>> root@nas:~# cryptsetup benchmark
>> # Tests are approximate using memory only (no storage IO).
>> PBKDF2-sha1   741567 iterations per second for 256-bit key
>> PBKDF2-sha256 910222 iterations per second for 256-bit key
>> PBKDF2-sha512 781353 iterations per second for 256-bit key
>> PBKDF2-ripemd160  547845 iterations per second for 256-bit key
>> PBKDF2-whirlpool  350929 iterations per second for 256-bit key
>> argon2i   4 iterations, 571787 memory, 4 parallel threads (CPUs) for
>> 256-bit key (requested 2000 ms time)
>> argon2id  4 iterations, 524288 memory, 4 parallel threads (CPUs) for
>> 256-bit key (requested 2000 ms time)
>> # Algorithm |   Key |  Encryption |  Decryption
>> aes-cbc128b   130.6 MiB/s   128.0 MiB/s
>> serpent-cbc128b64.7 MiB/s   161.8 MiB/s
>> twofish-cbc128b   175.4 MiB/s   218.8 MiB/s
>> aes-cbc256b   120.1 MiB/s   122.2 MiB/s
>> serpent-cbc256b84.5 MiB/s   210.8 MiB/s
>> twofish-cbc256b   189.5 MiB/s   218.6 MiB/s
>> aes-xts256b   167.0 MiB/s   162.1 MiB/s
>> serpent-xts256b   173.9 MiB/s   204.5 MiB/s
>> twofish-xts256b   204.4 MiB/s   213.2 MiB/s
>> aes-xts512b   127.9 MiB/s   122.9 MiB/s
>> serpent-xts512b   201.5 MiB/s   204.7 MiB/s
>> twofish-xts512b   215.0 MiB/s   213.0 MiB/s
>> root@nas:~#
>>
>>
>>
>> Is that about what you would expect?  Fireball is on a 970 mobo.  It's
>> slightly newer.  I think the 770T is about 2 years older, maybe 3. 
> grep AES /usr/src/linux/.config
>
> or,
>
> zgrep AES /proc/config.gz
>
> Or, grep your *current* kernel config wherever it is stored.


I got the idea but assuming you wanted that info from the NAS box, I had
to dig a little.  It's Ubuntu.  It doesn't have kernel sources, no
config.gz in /proc either.  I found this.  I assume it is accurate. 
Hopefully. 


root@nas:~# cat /boot/config-5.15.0-86-generic | grep -i aes
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_CRYPTO_AEGIS128_AESNI_SSE2=m
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_TI=m
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_SM4_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_LIB_AES=y
root@nas:~#


I don't usually use modules.  So, this is not something I run into
much.  I'm adding this info since I think it will help as well. 


root@nas:~# lsmod | grep -i aes
root@nas:~#


I see the main aes option is built in so it shouldn't be listed above if
I recall correctly.  The other two options are modules but not loaded. 
That said, I don't know if they are needed either.  On my main rig, I
have AES_TI built in.  Anyway, I thought I would include that in case it
helps. 

I was thinking about later on upgrading the CPU to a 6 core version.  I
may research and see if it includes the aes instruction set.  It may
help.  It may not.  Right now, I don't know if the 770T is even going to
be a NAS box and need encryption. 

It could be that given that mobo and CPUs age, it's doing the best it
can.  After all, the Dell box was also fairly 

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-13 Thread Michael
On Friday, 13 October 2023 02:35:21 BST Dale wrote:

> root@fireball / # cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   878204 iterations per second for 256-bit key
> PBKDF2-sha256 911805 iterations per second for 256-bit key
> PBKDF2-sha512 698119 iterations per second for 256-bit key
> PBKDF2-ripemd160  548418 iterations per second for 256-bit key
> PBKDF2-whirlpool  299251 iterations per second for 256-bit key
> argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b63.8 MiB/s51.4 MiB/s
> serpent-cbc128b90.9 MiB/s   307.6 MiB/s
> twofish-cbc128b   200.4 MiB/s   218.4 MiB/s
> aes-cbc256b54.6 MiB/s37.5 MiB/s
> serpent-cbc256b90.4 MiB/s   302.6 MiB/s
> twofish-cbc256b   198.2 MiB/s   216.7 MiB/s
> aes-xts256b68.0 MiB/s45.0 MiB/s
> serpent-xts256b   231.9 MiB/s   227.6 MiB/s
> twofish-xts256b   191.8 MiB/s   163.1 MiB/s
> aes-xts512b42.4 MiB/s18.9 MiB/s
> serpent-xts512b   100.9 MiB/s   124.6 MiB/s
> twofish-xts512b   154.8 MiB/s   173.3 MiB/s
> root@fireball / #
> 
> 
> 
> root@nas:~# cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   741567 iterations per second for 256-bit key
> PBKDF2-sha256 910222 iterations per second for 256-bit key
> PBKDF2-sha512 781353 iterations per second for 256-bit key
> PBKDF2-ripemd160  547845 iterations per second for 256-bit key
> PBKDF2-whirlpool  350929 iterations per second for 256-bit key
> argon2i   4 iterations, 571787 memory, 4 parallel threads (CPUs) for
> 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 524288 memory, 4 parallel threads (CPUs) for
> 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b   130.6 MiB/s   128.0 MiB/s
> serpent-cbc128b64.7 MiB/s   161.8 MiB/s
> twofish-cbc128b   175.4 MiB/s   218.8 MiB/s
> aes-cbc256b   120.1 MiB/s   122.2 MiB/s
> serpent-cbc256b84.5 MiB/s   210.8 MiB/s
> twofish-cbc256b   189.5 MiB/s   218.6 MiB/s
> aes-xts256b   167.0 MiB/s   162.1 MiB/s
> serpent-xts256b   173.9 MiB/s   204.5 MiB/s
> twofish-xts256b   204.4 MiB/s   213.2 MiB/s
> aes-xts512b   127.9 MiB/s   122.9 MiB/s
> serpent-xts512b   201.5 MiB/s   204.7 MiB/s
> twofish-xts512b   215.0 MiB/s   213.0 MiB/s
> root@nas:~#
> 
> 
> 
> Is that about what you would expect?  Fireball is on a 970 mobo.  It's
> slightly newer.  I think the 770T is about 2 years older, maybe 3. 

grep AES /usr/src/linux/.config

or,

zgrep AES /proc/config.gz

Or, grep your *current* kernel config wherever it is stored.

signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Dale
Frank Steinmetzger wrote:
> Am Thu, Oct 12, 2023 at 10:44:39PM +0100 schrieb Michael:
>
>> Why don't you test throughput without encryption to confirm your assumption?
> What does `cryptsetup benchmark` say? I used to use a Celeron G1840 in my 
> NAS, which is Intel Haswell without AES_NI. It was able to do ~ 150 MB/s raw 
> encryption throughput when transferring to or from a LUKS’ed image in a 
> ramdisk, so almost 150 % of gigabit ethernet speed.

When I first set up the old 770T system, I did that.  It was faster with
no encryption on the 770T end but I did have encryption on my main rig's
end.  The difference was a pretty good bit.  Pretty much all my stuff is
encrypted.  Anyway, I was still using the old mount options and it was
still faster. 

I've never used that benchmark.  Didn't know it exists.  This is the
results.  Keep in mind, fireball is my main rig.  The FX-8350 thingy. 
The NAS is currently the old 770T system.  Sometimes it is a old Dell
Inspiron but not this time.  ;-)



root@fireball / # cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1   878204 iterations per second for 256-bit key
PBKDF2-sha256 911805 iterations per second for 256-bit key
PBKDF2-sha512 698119 iterations per second for 256-bit key
PBKDF2-ripemd160  548418 iterations per second for 256-bit key
PBKDF2-whirlpool  299251 iterations per second for 256-bit key
argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
for 256-bit key (requested 2000 ms time)
argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
for 256-bit key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
    aes-cbc    128b    63.8 MiB/s    51.4 MiB/s
    serpent-cbc    128b    90.9 MiB/s   307.6 MiB/s
    twofish-cbc    128b   200.4 MiB/s   218.4 MiB/s
    aes-cbc    256b    54.6 MiB/s    37.5 MiB/s
    serpent-cbc    256b    90.4 MiB/s   302.6 MiB/s
    twofish-cbc    256b   198.2 MiB/s   216.7 MiB/s
    aes-xts    256b    68.0 MiB/s    45.0 MiB/s
    serpent-xts    256b   231.9 MiB/s   227.6 MiB/s
    twofish-xts    256b   191.8 MiB/s   163.1 MiB/s
    aes-xts    512b    42.4 MiB/s    18.9 MiB/s
    serpent-xts    512b   100.9 MiB/s   124.6 MiB/s
    twofish-xts    512b   154.8 MiB/s   173.3 MiB/s
root@fireball / #



root@nas:~# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1   741567 iterations per second for 256-bit key
PBKDF2-sha256 910222 iterations per second for 256-bit key
PBKDF2-sha512 781353 iterations per second for 256-bit key
PBKDF2-ripemd160  547845 iterations per second for 256-bit key
PBKDF2-whirlpool  350929 iterations per second for 256-bit key
argon2i   4 iterations, 571787 memory, 4 parallel threads (CPUs) for
256-bit key (requested 2000 ms time)
argon2id  4 iterations, 524288 memory, 4 parallel threads (CPUs) for
256-bit key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
    aes-cbc    128b   130.6 MiB/s   128.0 MiB/s
    serpent-cbc    128b    64.7 MiB/s   161.8 MiB/s
    twofish-cbc    128b   175.4 MiB/s   218.8 MiB/s
    aes-cbc    256b   120.1 MiB/s   122.2 MiB/s
    serpent-cbc    256b    84.5 MiB/s   210.8 MiB/s
    twofish-cbc    256b   189.5 MiB/s   218.6 MiB/s
    aes-xts    256b   167.0 MiB/s   162.1 MiB/s
    serpent-xts    256b   173.9 MiB/s   204.5 MiB/s
    twofish-xts    256b   204.4 MiB/s   213.2 MiB/s
    aes-xts    512b   127.9 MiB/s   122.9 MiB/s
    serpent-xts    512b   201.5 MiB/s   204.7 MiB/s
    twofish-xts    512b   215.0 MiB/s   213.0 MiB/s
root@nas:~#



Is that about what you would expect?  Fireball is on a 970 mobo.  It's
slightly newer.  I think the 770T is about 2 years older, maybe 3. 


 If you're copying over the network, that will be the limiting factor.
>>> Someone posted some extra options to mount with and add to exports
>>> file.
> Ah right, you use NFS. If not, I’d have suggested not to use rsync over ssh, 
> because that would indeed introduce a lot of encryption overhead.
>

I thought nfs was the proper way.  I use ssh and I use rsync,
separately.  Didn't know they can be used together tho. 


>>> I still think encryption is slowing it down some.  As you say tho,
>>> ethernet isn't helping which is why I may look into other options later,
>>> faster ethernet or fiber if I can find something cheap enough. 
>> There are a lot of hypotheses in your statements, but not much testing to 
>> prove or disprove any of them.
>>
>> Why don't you try to isolate the cause by testing one system element at a 
>> time 
>> and see what results you get.
>> […]
>> 

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Frank Steinmetzger
Am Thu, Oct 12, 2023 at 10:44:39PM +0100 schrieb Michael:

> >  It only does this when I'm copying files over.  Right now I'm copying
> >  about 26TBs of data over ethernet and it is taking a while.  Once I
> >  stop it or it finishes the copy, the CPU goes to about nothing,
> >  unless I'm doing something else.  So it has something to do with the
> >  copy process.
> > >>> 
> > >>> Or the network. What are you using to copy? If you use rsync, you can
> > >>> make use the the --bwlimit option to reduce the speed and network
> > >>> load.
> > >> 
> > >> Reduce?  I wouldn't complain if it went faster.  I think it is about as
> > >> fast as it is going to get tho.
> > > 
> > > And that may be contributing to the CPU usage. Slowing down the flow may
> > > make the comouter more usable, and you're never going to copy 26TB
> > > quickly, especially over ethernet.
> > > 
> > >> While I'm not sure what is keeping me from copying as fast as the drives
> > >> themselves can go, I suspect it is the encryption.
> 
> Why don't you test throughput without encryption to confirm your assumption?

What does `cryptsetup benchmark` say? I used to use a Celeron G1840 in my 
NAS, which is Intel Haswell without AES_NI. It was able to do ~ 150 MB/s raw 
encryption throughput when transferring to or from a LUKS’ed image in a 
ramdisk, so almost 150 % of gigabit ethernet speed.

> > > If you're copying over the network, that will be the limiting factor.
> > 
> > Someone posted some extra options to mount with and add to exports
> > file.

Ah right, you use NFS. If not, I’d have suggested not to use rsync over ssh, 
because that would indeed introduce a lot of encryption overhead.

> > I still think encryption is slowing it down some.  As you say tho,
> > ethernet isn't helping which is why I may look into other options later,
> > faster ethernet or fiber if I can find something cheap enough. 
> 
> There are a lot of hypotheses in your statements, but not much testing to 
> prove or disprove any of them.
> 
> Why don't you try to isolate the cause by testing one system element at a 
> time 
> and see what results you get.
> […]
> Unless you're running Pentium 4 or some other old CPU, it is almost certain 
> your CPU is capable of using AES-NI to offload to hardware some/all of the 
> encryption/decryption load - as long as you have the crypto module built in 
> your kernel.

The FX-8350 may be old, but it actually does have AES instructions.

Here is my Haswell i5 (only two years younger than the FX) with AES_NI:

~ LC_ALL=C cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1  1323959 iterations per second for 256-bit key
PBKDF2-sha2561724631 iterations per second for 256-bit key
PBKDF2-sha5121137284 iterations per second for 256-bit key
PBKDF2-ripemd160  706587 iterations per second for 256-bit key
PBKDF2-whirlpool  510007 iterations per second for 256-bit key
argon2i   7 iterations, 1048576 memory, 4 parallel threads (CPUs) for 
256-bit key (requested 2000 ms time)
argon2id  7 iterations, 1048576 memory, 4 parallel threads (CPUs) for 
256-bit key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
aes-cbc128b   679.8 MiB/s  2787.0 MiB/s
serpent-cbc128b91.4 MiB/s   582.1 MiB/s
twofish-cbc128b   194.9 MiB/s   368.3 MiB/s
aes-cbc256b   502.3 MiB/s  2155.4 MiB/s
serpent-cbc256b90.3 MiB/s   582.5 MiB/s
twofish-cbc256b   194.0 MiB/s   368.6 MiB/s
aes-xts256b  2470.8 MiB/s  2478.7 MiB/s
serpent-xts256b   537.4 MiB/s   526.1 MiB/s
twofish-xts256b   347.3 MiB/s   347.3 MiB/s
aes-xts512b  1932.6 MiB/s  1958.0 MiB/s
serpent-xts512b   532.9 MiB/s   522.9 MiB/s
twofish-xts512b   348.4 MiB/s   348.9 MiB/s

The 6 Watts processor in my Surface Go yields:
aes-xts512b  1122,2 MiB/s  1123,7 MiB/s

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

The severity of the itch is inversely proportional to the reach.


signature.asc
Description: PGP signature


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Michael
On Thursday, 12 October 2023 21:50:28 BST Dale wrote:
> Neil Bothwick wrote:
> > On Thu, 12 Oct 2023 07:36:17 -0500, Dale wrote:
> >> Neil Bothwick wrote:
> >>> On Wed, 11 Oct 2023 17:52:58 -0500, Dale wrote:
>  It only does this when I'm copying files over.  Right now I'm copying
>  about 26TBs of data over ethernet and it is taking a while.  Once I
>  stop it or it finishes the copy, the CPU goes to about nothing,
>  unless I'm doing something else.  So it has something to do with the
>  copy process.
> >>> 
> >>> Or the network. What are you using to copy? If you use rsync, you can
> >>> make use the the --bwlimit option to reduce the speed and network
> >>> load.
> >> 
> >> Reduce?  I wouldn't complain if it went faster.  I think it is about as
> >> fast as it is going to get tho.
> > 
> > And that may be contributing to the CPU usage. Slowing down the flow may
> > make the comouter more usable, and you're never going to copy 26TB
> > quickly, especially over ethernet.
> > 
> >> While I'm not sure what is keeping me from copying as fast as the drives
> >> themselves can go, I suspect it is the encryption.

Why don't you test throughput without encryption to confirm your assumption?


> > If you're copying over the network, that will be the limiting factor.
> 
> Someone posted some extra options to mount with and add to exports
> file.  Those added options almost doubled the speed.  I watch gkrellm
> and I think it is going about as fast as it can.  My problem is, some
> software uses one unit to measure things while another uses something
> else.  It makes it hard to figure out what is doing what.  Still, using
> gkrellm which is something I'm used to watching when it comes to drive
> read/write data, I think it is as good as it is going to get.  Not that
> I'm not open to trying other options that might speed things up.  I
> still think encryption is slowing it down some.  As you say tho,
> ethernet isn't helping which is why I may look into other options later,
> faster ethernet or fiber if I can find something cheap enough. 

There are a lot of hypotheses in your statements, but not much testing to 
prove or disprove any of them.

Why don't you try to isolate the cause by testing one system element at a time 
and see what results you get.

Copy a large file from tmpfs to tmpfs to see how fast it can transfer across 
your LAN - or use iperf3 as already recommended.  Use a file size large enough 
to saturate your network and give you a real life max throughput.

Repeat, but this time copy the large file over to disk.

Repeat, but this time try different filesystems, disks, volumes, strides/
stripes, add encryption, compression, whatnot.

You may spend an hour or two, but you'd soon isolate the major contributing 
factor(s) causing the observed slowdown.

Unless you're running Pentium 4 or some other old CPU, it is almost certain 
your CPU is capable of using AES-NI to offload to hardware some/all of the 
encryption/decryption load - as long as you have the crypto module built in 
your kernel.

PS. Keep notes and flush caches between tests to avoid drawing conclusions on 
spurious results.

signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Dale
Neil Bothwick wrote:
> On Thu, 12 Oct 2023 07:36:17 -0500, Dale wrote:
>
>> Neil Bothwick wrote:
>>> On Wed, 11 Oct 2023 17:52:58 -0500, Dale wrote:
>>>  
 It only does this when I'm copying files over.  Right now I'm copying
 about 26TBs of data over ethernet and it is taking a while.  Once I
 stop it or it finishes the copy, the CPU goes to about nothing,
 unless I'm doing something else.  So it has something to do with the
 copy process.  
>>> Or the network. What are you using to copy? If you use rsync, you can
>>> make use the the --bwlimit option to reduce the speed and network
>>> load.
>>>
>>>  
>>
>> Reduce?  I wouldn't complain if it went faster.  I think it is about as
>> fast as it is going to get tho.
> And that may be contributing to the CPU usage. Slowing down the flow may
> make the comouter more usable, and you're never going to copy 26TB
> quickly, especially over ethernet.
>
>> While I'm not sure what is keeping me from copying as fast as the drives
>> themselves can go, I suspect it is the encryption.
> If you're copying over the network, that will be the limiting factor.
>
>


Someone posted some extra options to mount with and add to exports
file.  Those added options almost doubled the speed.  I watch gkrellm
and I think it is going about as fast as it can.  My problem is, some
software uses one unit to measure things while another uses something
else.  It makes it hard to figure out what is doing what.  Still, using
gkrellm which is something I'm used to watching when it comes to drive
read/write data, I think it is as good as it is going to get.  Not that
I'm not open to trying other options that might speed things up.  I
still think encryption is slowing it down some.  As you say tho,
ethernet isn't helping which is why I may look into other options later,
faster ethernet or fiber if I can find something cheap enough. 

My new CPU cooler is on it's way.  I'm picking up the smaller stuff at
times.  I couldn't pass up a good deal on that CPU cooler.  The big
spend is CPU, mobo and memory.  Oh, that MASSIVE case too.  O_O 

Dale

:-)  :-) 



Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Neil Bothwick
On Thu, 12 Oct 2023 07:36:17 -0500, Dale wrote:

> Neil Bothwick wrote:
> > On Wed, 11 Oct 2023 17:52:58 -0500, Dale wrote:
> >  
> >> It only does this when I'm copying files over.  Right now I'm copying
> >> about 26TBs of data over ethernet and it is taking a while.  Once I
> >> stop it or it finishes the copy, the CPU goes to about nothing,
> >> unless I'm doing something else.  So it has something to do with the
> >> copy process.  
> > Or the network. What are you using to copy? If you use rsync, you can
> > make use the the --bwlimit option to reduce the speed and network
> > load.
> >
> >  
> 
> 
> Reduce?  I wouldn't complain if it went faster.  I think it is about as
> fast as it is going to get tho.

And that may be contributing to the CPU usage. Slowing down the flow may
make the comouter more usable, and you're never going to copy 26TB
quickly, especially over ethernet.

> While I'm not sure what is keeping me from copying as fast as the drives
> themselves can go, I suspect it is the encryption.

If you're copying over the network, that will be the limiting factor.


-- 
Neil Bothwick

Nixon's Principal: If 2 wrongs don't make a right, try 3.


pgpv3iDV0bfu7.pgp
Description: OpenPGP digital signature


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Dale
Neil Bothwick wrote:
> On Wed, 11 Oct 2023 17:52:58 -0500, Dale wrote:
>
>> It only does this when I'm copying files over.  Right now I'm copying
>> about 26TBs of data over ethernet and it is taking a while.  Once I stop
>> it or it finishes the copy, the CPU goes to about nothing, unless I'm
>> doing something else.  So it has something to do with the copy process.
> Or the network. What are you using to copy? If you use rsync, you can
> make use the the --bwlimit option to reduce the speed and network load.
>
>


Reduce?  I wouldn't complain if it went faster.  I think it is about as
fast as it is going to get tho.  I may one day try to get a fiber card
to go between my computer and some sort of NAS box or something.  I'm
slowly building up my number of rigs so I can separate some things
around.  I got my old Gigabyte 770T running but no case yet.  I use it
for my backups instead of the old Dell Inspiron.  Actually, I found a
good deal on a CPU cooler last night for the new build and bought it. 
About half price or so.  Reviews claim it is a really nice cooler. 
Fairly good size with two fans. 

While I'm not sure what is keeping me from copying as fast as the drives
themselves can go, I suspect it is the encryption.  I think for these
old CPUs, it uses more CPU time.  Htop with the meters shows the CPU
doing something but not exactly what.  I'm pretty sure it is either
encryption, LVM or both. 

Now it's time for my weekly pokings at the Doctor and my weekly grocery
shopping.  ;-)

Dale

:-)  :-) 



Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-12 Thread Neil Bothwick
On Wed, 11 Oct 2023 17:52:58 -0500, Dale wrote:

> It only does this when I'm copying files over.  Right now I'm copying
> about 26TBs of data over ethernet and it is taking a while.  Once I stop
> it or it finishes the copy, the CPU goes to about nothing, unless I'm
> doing something else.  So it has something to do with the copy process.

Or the network. What are you using to copy? If you use rsync, you can
make use the the --bwlimit option to reduce the speed and network load.


-- 
Neil Bothwick

Is that "woof" feed me; "woof" walk me; "woof" there's a burglar? What??


pgpYMkd7E9Si0.pgp
Description: OpenPGP digital signature


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-11 Thread Dale
Jude DaShiell wrote:
> Linux is being targeted by ransomware and other forms of malware so it may
> be worthwhile to run ferensics on your backup and find what ferensics have
> to tell you.  After check and see if any of what you found out were false
> positives.
>
>
> -- Jude  "There are four boxes to be used in
> defense of liberty: soap, ballot, jury, and ammo. Please use in that
> order." Ed Howdershelt 1940.
>
>


It only does this when I'm copying files over.  Right now I'm copying
about 26TBs of data over ethernet and it is taking a while.  Once I stop
it or it finishes the copy, the CPU goes to about nothing, unless I'm
doing something else.  So it has something to do with the copy process. 
Since the LVs are encrypted, I think a good bit of it is related to
that.  I read somewhere that new CPUs have a instruction set that makes
encryption really efficient.  Mine doesn't have that, that I'm aware of
anyway.  It's yet another reason for me building a newer rig.  One thing
I've noticed, as my drive space increases, this seems to use more CPU. 

One thing I hope to do, have a system that does nothing but torrent
stuff.  Right now, I have a lot of stuff running on this rig.  For its
age, it's a good size load at times.  I'd like to put some of that load
on another machine. 

I'm looking at that AMD Ryzen 9 5900X which I've been told will run
circles around this old FX-8350.  I also need more memory too.  I
sometimes have to close Firefox to free up some memory especially during
some compiles.  I'm working on it.  Takes time tho. 

Dale

:-)  :-) 



Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-11 Thread Jude DaShiell
Linux is being targeted by ransomware and other forms of malware so it may
be worthwhile to run ferensics on your backup and find what ferensics have
to tell you.  After check and see if any of what you found out were false
positives.


-- Jude  "There are four boxes to be used in
defense of liberty: soap, ballot, jury, and ammo. Please use in that
order." Ed Howdershelt 1940.

On Wed, 11 Oct 2023, Dale wrote:

> Frank Steinmetzger wrote:
> > Am Sun, Oct 08, 2023 at 07:44:06PM -0500 schrieb Dale:
> >
> >> Just as a update.  The file system I was trying to do a file system
> >> check on was my large one, about 40TBs worth.  While running the file
> >> system check, it started using HUGE amounts of memory.  It used almost
> >> all my 32GBs and most of swap as well.  It couldn't finish due to not
> >> enough memory, it literally crashed itself.  So, I don't know if this is
> >> because of some huge problem or what but if this is expected behavior,
> >> don't try to do a file system check on devices that large unless you
> >> have a LOT of memory. 
> > Or use a different filesystem. O:-)
>
> I'm using ext4 which is said to be one of the most reliable and widely
> used file systems.  I do wonder tho, am I creating file systems that may
> be to large or that it just has trouble with???  I doubt that but I'm up
> to about 40TBs now.  I just can't figure out a way to split that data
> up, yet.
>
>
> >> I ended up recreating the LVM devices from scratch and redoing the
> >> encryption as well.  I have backups tho.  This all started when using
> >> pvmove to replace a hard drive with a larger drive.  I guess pvmove
> >> isn't always safe.
> > I think that may be a far-fetched conclusion. If it weren’t safe, it
> > wouldn’t be in the software – or at least not advertised as safe.
> >
>
> Well, something went sideways.  Honestly, I think it might not be pvmove
> but something happened with the file system itself. After all, LVM
> wasn't complaining at all and everything showed the move completed with
> no errors.  I guess it is possible pvmove had a problem but given it was
> the file system that complained so loudly, I'm leaning to it having a
> issue. 
>
>
> >> P. S.  I currently have my backup system on my old Gigabyte 770T mobo
> >> and friends.  It is still a bit slower than copying when no encryption
> >> is used so I guess encryption does slow things down a bit.  That said,
> >> the CPU does hang around 50% most of the time.  htop doesn't show what
> >> is using that so it must be IO or encryption.
> > You can add more widgets (“meters”) to htop, one of them shows disk
> > throughput. But there is none for I/O wait. One tool that does show that is
> > glances. And also dstat which I mentioned a few days ago. Not only can dstat
> > tell you the total percentage, but also which process is the most expensive
> > one.
> >
> > I set up bash aliases for different use cases of dstat:
> > alias ,d='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? 
> > /dev/mmcblk? 2>/dev/null | tr "\n" ,) --net --mem --swap'
> > alias ,dd='dstat --time --cpu --disk --disk-util -D $(ls /dev/sd? 
> > /dev/nvme?n? /dev/mmcblk? 2>/dev/null | tr "\n" ,) --mem-adv'
> > alias ,dm='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? 
> > /dev/mmcblk? 2>/dev/null | tr "\n" ,) --net --mem-adv --swap'
> > alias ,dt='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? 
> > /dev/mmcblk? 2>/dev/null | tr "\n" ,) --net --mem --swap --top-cpu 
> > --top-bio --top-io --top-mem'
> >
> > Because I attach external storage once in a while, I use a dynamic list of
> > devices to watch that is passed to the -D argument. If I don’t use -D, dstat
> > will only show a total for all drives.
> >
> > The first is a simple overview (d = dstat).
> >
> > The second is the same but only for disk statistics (dd = dstat disks). I
> > use it mostly on my NAS (five SATA drives in total, which creates a very
> > wide table).
> >
> > The third shows more memory details like dirty cache (dm = dstat memory),
> > which is interesting when copying large files.
> >
> > And the last one shows the top “pigs”, i.e. expensive processes in terms of
> > CPU, IO and memory (dt = dstat top).
> >
> >> Or something kernel
> >> related that htop doesn't show.  No idea. 
> > Perhaps my tool tips give you ideas. :)
> >
> > -- Grüße | Greetings | Salut | Qapla’ Please do not share anything
> > from, with or about me on any social network. What is the difference
> > between two flutes? – A semitone.
>
>
> Dang, I have a lot of drives here to add to all that.  Bad thing is,
> every time I reboot, all but two I think tend to move around, even tho I
> haven't moved anything.  This is why I use either labels or UUIDs by the
> way.  Once ages ago, I saw a way to make commands/scripts see all drives
> on a system with some sort of inclusive trick.  I think it used brackets
> but not sure.  I can't find that trick anymore.  I should have saved
> that thing. 
>
> I used some command, 

Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-11 Thread Dale
Frank Steinmetzger wrote:
> Am Sun, Oct 08, 2023 at 07:44:06PM -0500 schrieb Dale:
>
>> Just as a update.  The file system I was trying to do a file system
>> check on was my large one, about 40TBs worth.  While running the file
>> system check, it started using HUGE amounts of memory.  It used almost
>> all my 32GBs and most of swap as well.  It couldn't finish due to not
>> enough memory, it literally crashed itself.  So, I don't know if this is
>> because of some huge problem or what but if this is expected behavior,
>> don't try to do a file system check on devices that large unless you
>> have a LOT of memory. 
> Or use a different filesystem. O:-)

I'm using ext4 which is said to be one of the most reliable and widely
used file systems.  I do wonder tho, am I creating file systems that may
be to large or that it just has trouble with???  I doubt that but I'm up
to about 40TBs now.  I just can't figure out a way to split that data
up, yet.


>> I ended up recreating the LVM devices from scratch and redoing the
>> encryption as well.  I have backups tho.  This all started when using
>> pvmove to replace a hard drive with a larger drive.  I guess pvmove
>> isn't always safe.
> I think that may be a far-fetched conclusion. If it weren’t safe, it 
> wouldn’t be in the software – or at least not advertised as safe.
>

Well, something went sideways.  Honestly, I think it might not be pvmove
but something happened with the file system itself. After all, LVM
wasn't complaining at all and everything showed the move completed with
no errors.  I guess it is possible pvmove had a problem but given it was
the file system that complained so loudly, I'm leaning to it having a
issue. 


>> P. S.  I currently have my backup system on my old Gigabyte 770T mobo
>> and friends.  It is still a bit slower than copying when no encryption
>> is used so I guess encryption does slow things down a bit.  That said,
>> the CPU does hang around 50% most of the time.  htop doesn't show what
>> is using that so it must be IO or encryption.
> You can add more widgets (“meters”) to htop, one of them shows disk 
> throughput. But there is none for I/O wait. One tool that does show that is 
> glances. And also dstat which I mentioned a few days ago. Not only can dstat 
> tell you the total percentage, but also which process is the most expensive 
> one.
>
> I set up bash aliases for different use cases of dstat:
> alias ,d='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? 
> /dev/mmcblk? 2>/dev/null | tr "\n" ,) --net --mem --swap'
> alias ,dd='dstat --time --cpu --disk --disk-util -D $(ls /dev/sd? 
> /dev/nvme?n? /dev/mmcblk? 2>/dev/null | tr "\n" ,) --mem-adv'
> alias ,dm='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? 
> /dev/mmcblk? 2>/dev/null | tr "\n" ,) --net --mem-adv --swap'
> alias ,dt='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? 
> /dev/mmcblk? 2>/dev/null | tr "\n" ,) --net --mem --swap --top-cpu --top-bio 
> --top-io --top-mem'
>
> Because I attach external storage once in a while, I use a dynamic list of 
> devices to watch that is passed to the -D argument. If I don’t use -D, dstat 
> will only show a total for all drives.
>
> The first is a simple overview (d = dstat).
>
> The second is the same but only for disk statistics (dd = dstat disks). I 
> use it mostly on my NAS (five SATA drives in total, which creates a very 
> wide table).
>
> The third shows more memory details like dirty cache (dm = dstat memory), 
> which is interesting when copying large files.
>
> And the last one shows the top “pigs”, i.e. expensive processes in terms of 
> CPU, IO and memory (dt = dstat top).
>
>> Or something kernel
>> related that htop doesn't show.  No idea. 
> Perhaps my tool tips give you ideas. :)
>
> -- Grüße | Greetings | Salut | Qapla’ Please do not share anything
> from, with or about me on any social network. What is the difference
> between two flutes? – A semitone.


Dang, I have a lot of drives here to add to all that.  Bad thing is,
every time I reboot, all but two I think tend to move around, even tho I
haven't moved anything.  This is why I use either labels or UUIDs by the
way.  Once ages ago, I saw a way to make commands/scripts see all drives
on a system with some sort of inclusive trick.  I think it used brackets
but not sure.  I can't find that trick anymore.  I should have saved
that thing. 

I used some command, can't recall which it was, and I think it is the
kernel itself using so much CPU time.  Given when it does it, I think it
is either processing the encryption or working to send the data to the
disks, or both.  I'd suspect both but I dunno. 

Anyway, I'm restoring from a fresh LVM rebuild now.  No way to test
anything to see what the problem was now. 

Dale

:-)  :-) 


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-09 Thread Frank Steinmetzger
Am Sun, Oct 08, 2023 at 07:44:06PM -0500 schrieb Dale:

> Just as a update.  The file system I was trying to do a file system
> check on was my large one, about 40TBs worth.  While running the file
> system check, it started using HUGE amounts of memory.  It used almost
> all my 32GBs and most of swap as well.  It couldn't finish due to not
> enough memory, it literally crashed itself.  So, I don't know if this is
> because of some huge problem or what but if this is expected behavior,
> don't try to do a file system check on devices that large unless you
> have a LOT of memory. 

Or use a different filesystem. O:-)

> I ended up recreating the LVM devices from scratch and redoing the
> encryption as well.  I have backups tho.  This all started when using
> pvmove to replace a hard drive with a larger drive.  I guess pvmove
> isn't always safe.

I think that may be a far-fetched conclusion. If it weren’t safe, it 
wouldn’t be in the software – or at least not advertised as safe.

> P. S.  I currently have my backup system on my old Gigabyte 770T mobo
> and friends.  It is still a bit slower than copying when no encryption
> is used so I guess encryption does slow things down a bit.  That said,
> the CPU does hang around 50% most of the time.  htop doesn't show what
> is using that so it must be IO or encryption.

You can add more widgets (“meters”) to htop, one of them shows disk 
throughput. But there is none for I/O wait. One tool that does show that is 
glances. And also dstat which I mentioned a few days ago. Not only can dstat 
tell you the total percentage, but also which process is the most expensive 
one.

I set up bash aliases for different use cases of dstat:
alias ,d='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? /dev/mmcblk? 
2>/dev/null | tr "\n" ,) --net --mem --swap'
alias ,dd='dstat --time --cpu --disk --disk-util -D $(ls /dev/sd? /dev/nvme?n? 
/dev/mmcblk? 2>/dev/null | tr "\n" ,) --mem-adv'
alias ,dm='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? /dev/mmcblk? 
2>/dev/null | tr "\n" ,) --net --mem-adv --swap'
alias ,dt='dstat --time --cpu --disk -D $(ls /dev/sd? /dev/nvme?n? /dev/mmcblk? 
2>/dev/null | tr "\n" ,) --net --mem --swap --top-cpu --top-bio --top-io 
--top-mem'

Because I attach external storage once in a while, I use a dynamic list of 
devices to watch that is passed to the -D argument. If I don’t use -D, dstat 
will only show a total for all drives.

The first is a simple overview (d = dstat).

The second is the same but only for disk statistics (dd = dstat disks). I 
use it mostly on my NAS (five SATA drives in total, which creates a very 
wide table).

The third shows more memory details like dirty cache (dm = dstat memory), 
which is interesting when copying large files.

And the last one shows the top “pigs”, i.e. expensive processes in terms of 
CPU, IO and memory (dt = dstat top).

> Or something kernel
> related that htop doesn't show.  No idea. 

Perhaps my tool tips give you ideas. :)

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

What is the difference between two flutes? – A semitone.


signature.asc
Description: PGP signature


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-08 Thread Dale
Dale wrote:
> Mark Knecht wrote:
>>
>>
>> On Sun, Oct 8, 2023 at 10:56 AM Dale > > wrote:
>> >
>> > Howdy,
>> >
>> > I use Konsole a lot, that thing within KDE that acts like a console.
>> > Anyway, I'm running a offline file system check on a rather large file
>> > system.  For some reason, Konsole decided to crash.  I can see the file
>> > system is still running with top, ps etc but I can't see anything to
>> > know what it is doing.  Is there a way to get that back?  Should I kill
>> > it and restart now that Konsole is running again?  I'd think a regular
>> > term signal would give it a safe stopping place but still kinda chicken
>> > to do it.  Then again, what if it stops and needs my input or worse
>> yet,
>> > it displays a error that I can't see but I need to know and see?
>> >
>> > Any thoughts?  Is there a way to get it back?  Kill it and restart?  Do
>> > nothing and hope for the best?
>> >
>> > Thanks.
>> >
>> > Dale
>> >
>> > :-)  :-)
>>
>> I would suggest you learn screen - a very simple app that allows you to
>> start an app and then disconnect from it. You can then log out, close
>> your terminal, or in your case if konsole really crashed, you just open
>> a new konsole and reconnect. 
>>
>> The screen process keeps all the terminal output so you can review
>> it while the process is running or after it has finished.
>>
>> I do not know how to reliably get access to your process if it's 
>> really still running. Someone else here can probably give you
>> better instructions on that.
>>
>> HTH,
>> Mark
>>
>>
>>
>
>
> I was hoping I would catch a real quick response, even tho that wasn't
> very likely.  After about 45 minutes or so, I did a pkill on it.  I
> seem to recall it is about the same as a ctrl c which is a polite
> 'stop what you doing' when safely possible, in most cases anyway.  I
> then started a new screen process and restarted the file system
> check.  It's still working on it on the other desktop.  So, even tho I
> hadn't read your reply yet, I still did what you advised.  It's
> running in a screen process now.  I can reattach if Konsole dies
> again.  Good advice tho.  Should have did that before.  ;-)
>
> I don't know what happened to Konsole tho.  It's crashed once before a
> month or so ago and then again a bit ago.  Before that, I can't recall
> it ever crashing on me before.  It appears someone is adding a feature
> that includes the occasional crash as a added bonus.  ROFL 
>
> I'm glad I made new backups.  Before Konsole crashed, it was spitting
> out a LOT of stuff that I'm not sure is good.  It even mentioned
> possible lost data.  I got a new 18TB hard drive and was in the
> process of moving data to it and resizing the file system when this
> all started.  I can't mount right now so no idea if it is still there
> or not.  Now let us pray. 
>
> Dale
>
> :-)  :-) 


Just as a update.  The file system I was trying to do a file system
check on was my large one, about 40TBs worth.  While running the file
system check, it started using HUGE amounts of memory.  It used almost
all my 32GBs and most of swap as well.  It couldn't finish due to not
enough memory, it literally crashed itself.  So, I don't know if this is
because of some huge problem or what but if this is expected behavior,
don't try to do a file system check on devices that large unless you
have a LOT of memory. 

I ended up recreating the LVM devices from scratch and redoing the
encryption as well.  I have backups tho.  This all started when using
pvmove to replace a hard drive with a larger drive.  I guess pvmove
isn't always safe.  It took going on two days to move. 

Oh, Mark had good advice too.  Do important stuff in screen just in case
something crashes, like Konsole.  :/ 

Thanks.  Hope someone learns from my boo boo. 

Dale

:-) :-) 

P. S.  I currently have my backup system on my old Gigabyte 770T mobo
and friends.  It is still a bit slower than copying when no encryption
is used so I guess encryption does slow things down a bit.  That said,
the CPU does hang around 50% most of the time.  htop doesn't show what
is using that so it must be IO or encryption.  Or something kernel
related that htop doesn't show.  No idea. 


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-08 Thread Dale
Mark Knecht wrote:
>
>
> On Sun, Oct 8, 2023 at 10:56 AM Dale  > wrote:
> >
> > Howdy,
> >
> > I use Konsole a lot, that thing within KDE that acts like a console.
> > Anyway, I'm running a offline file system check on a rather large file
> > system.  For some reason, Konsole decided to crash.  I can see the file
> > system is still running with top, ps etc but I can't see anything to
> > know what it is doing.  Is there a way to get that back?  Should I kill
> > it and restart now that Konsole is running again?  I'd think a regular
> > term signal would give it a safe stopping place but still kinda chicken
> > to do it.  Then again, what if it stops and needs my input or worse yet,
> > it displays a error that I can't see but I need to know and see?
> >
> > Any thoughts?  Is there a way to get it back?  Kill it and restart?  Do
> > nothing and hope for the best?
> >
> > Thanks.
> >
> > Dale
> >
> > :-)  :-)
>
> I would suggest you learn screen - a very simple app that allows you to
> start an app and then disconnect from it. You can then log out, close
> your terminal, or in your case if konsole really crashed, you just open
> a new konsole and reconnect. 
>
> The screen process keeps all the terminal output so you can review
> it while the process is running or after it has finished.
>
> I do not know how to reliably get access to your process if it's 
> really still running. Someone else here can probably give you
> better instructions on that.
>
> HTH,
> Mark
>
>
>


I was hoping I would catch a real quick response, even tho that wasn't
very likely.  After about 45 minutes or so, I did a pkill on it.  I seem
to recall it is about the same as a ctrl c which is a polite 'stop what
you doing' when safely possible, in most cases anyway.  I then started a
new screen process and restarted the file system check.  It's still
working on it on the other desktop.  So, even tho I hadn't read your
reply yet, I still did what you advised.  It's running in a screen
process now.  I can reattach if Konsole dies again.  Good advice tho. 
Should have did that before.  ;-)

I don't know what happened to Konsole tho.  It's crashed once before a
month or so ago and then again a bit ago.  Before that, I can't recall
it ever crashing on me before.  It appears someone is adding a feature
that includes the occasional crash as a added bonus.  ROFL 

I'm glad I made new backups.  Before Konsole crashed, it was spitting
out a LOT of stuff that I'm not sure is good.  It even mentioned
possible lost data.  I got a new 18TB hard drive and was in the process
of moving data to it and resizing the file system when this all
started.  I can't mount right now so no idea if it is still there or
not.  Now let us pray. 

Dale

:-)  :-) 


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-08 Thread Mark Knecht
On Sun, Oct 8, 2023 at 10:56 AM Dale  wrote:
>
> Howdy,
>
> I use Konsole a lot, that thing within KDE that acts like a console.
> Anyway, I'm running a offline file system check on a rather large file
> system.  For some reason, Konsole decided to crash.  I can see the file
> system is still running with top, ps etc but I can't see anything to
> know what it is doing.  Is there a way to get that back?  Should I kill
> it and restart now that Konsole is running again?  I'd think a regular
> term signal would give it a safe stopping place but still kinda chicken
> to do it.  Then again, what if it stops and needs my input or worse yet,
> it displays a error that I can't see but I need to know and see?
>
> Any thoughts?  Is there a way to get it back?  Kill it and restart?  Do
> nothing and hope for the best?
>
> Thanks.
>
> Dale
>
> :-)  :-)

I would suggest you learn screen - a very simple app that allows you to
start an app and then disconnect from it. You can then log out, close
your terminal, or in your case if konsole really crashed, you just open
a new konsole and reconnect.

The screen process keeps all the terminal output so you can review
it while the process is running or after it has finished.

I do not know how to reliably get access to your process if it's
really still running. Someone else here can probably give you
better instructions on that.

HTH,
Mark


Re: [gentoo-user] Getting output of a program running in background after a crash

2023-10-08 Thread Jude DaShiell
If I understand your question, this may help.  Understand prog is the
program that errors out in this example:
prog 2>&1 | tee prog.err
Look for all output including errors in the file prog.err which tee will
have created for you and before opening prog.err try wc-l prog.err and
grep -i error prog.err to do initial inspection.  If the wc command
returns 0 then no need to do the grep search since the file is empty.


-- Jude  "There are four boxes to be used in
defense of liberty: soap, ballot, jury, and ammo. Please use in that
order." Ed Howdershelt 1940.

On Sun, 8 Oct 2023, Dale wrote:

> Howdy,
>
> I use Konsole a lot, that thing within KDE that acts like a console. 
> Anyway, I'm running a offline file system check on a rather large file
> system.  For some reason, Konsole decided to crash.  I can see the file
> system is still running with top, ps etc but I can't see anything to
> know what it is doing.  Is there a way to get that back?  Should I kill
> it and restart now that Konsole is running again?  I'd think a regular
> term signal would give it a safe stopping place but still kinda chicken
> to do it.  Then again, what if it stops and needs my input or worse yet,
> it displays a error that I can't see but I need to know and see? 
>
> Any thoughts?  Is there a way to get it back?  Kill it and restart?  Do
> nothing and hope for the best? 
>
> Thanks.
>
> Dale
>
> :-)  :-) 
>
>



[gentoo-user] Getting output of a program running in background after a crash

2023-10-08 Thread Dale
Howdy,

I use Konsole a lot, that thing within KDE that acts like a console. 
Anyway, I'm running a offline file system check on a rather large file
system.  For some reason, Konsole decided to crash.  I can see the file
system is still running with top, ps etc but I can't see anything to
know what it is doing.  Is there a way to get that back?  Should I kill
it and restart now that Konsole is running again?  I'd think a regular
term signal would give it a safe stopping place but still kinda chicken
to do it.  Then again, what if it stops and needs my input or worse yet,
it displays a error that I can't see but I need to know and see? 

Any thoughts?  Is there a way to get it back?  Kill it and restart?  Do
nothing and hope for the best? 

Thanks.

Dale

:-)  :-)