Re: [Koha-devel] Memory footprint of Koha processes

2023-10-24 Thread Philippe Blouin via Koha-devel 
Thanks everyone for the replies!  I will give a look at everything, 
because I'm in dire need of solutions.


Just a cue on the tests I did yesterday:

#!/usr/bin/perl

use Modern::Perl;
use C4::Context;
use Koha::BackgroundJobs;

while (1){
    sleep 10;
}

I use htop to see the memory usage.  The RES is the value I care about.  
And the total memory usage of the machine tells me it is indeed additive 
(unlike mariadb or apache that show me the same value for each thread)


Now, the code above uses 50K on _some_ instances, and 150M on others.  I 
still haven't figured out the why for the discrepency, and I've tried 
about 20 instances, our our debian servers, our clients, and my 
coworkers ubuntu's.  Something in the config?  The 50K happens more 
often in 23.05, but when I restart them, they go to 150M and stay 
there.  I'll be examining the koha-conf.xml today...



As for plack, I have a very bare understanding on how it works. I 
suppose it loads all the code, so I suppose the 200M is justified?  Of 
course, making it slower makes it useless, so I won't go that route.



Thanks again, all!  I'll be checking the BZ you suggested.


Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
philippe.blo...@inlibro.com

inLibro | pour esprit libre | www.inLibro.com 
On 2023-10-24 03:25, Thomas Klausner via Koha-devel wrote:

Hi!

On Mon, Oct 23, 2023 at 03:57:16PM -0400, Philippe Blouin via Koha-devel wrote:
  

background_jobs_worker.pl for instance uses 170M per instance, and plack
uses 200M without anything being loaded.

How are you calculating the memory usage? I remember from mod_perl times
that if you preload as much code as possible before forking of
plack/redis workers much of the memory will be shared between processes.

So maybe the actual memory usage is much lower than what you expect it
to be. And if it's not, then we should investigate how to pull as much
`use` statements into pieces of code that are loaded before a `fork`
happens.

Greetings,
domm

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Memory footprint of Koha processes

2023-10-24 Thread Thomas Klausner via Koha-devel 
Hi!

On Mon, Oct 23, 2023 at 03:57:16PM -0400, Philippe Blouin via Koha-devel wrote:
 
> background_jobs_worker.pl for instance uses 170M per instance, and plack
> uses 200M without anything being loaded.

How are you calculating the memory usage? I remember from mod_perl times 
that if you preload as much code as possible before forking of 
plack/redis workers much of the memory will be shared between processes.

So maybe the actual memory usage is much lower than what you expect it 
to be. And if it's not, then we should investigate how to pull as much 
`use` statements into pieces of code that are loaded before a `fork` 
happens.

Greetings,
domm


-- 
Thomas Klausner   domm  https://domm.plix.at
Just another ( Perl | Postgres | Bicycle | Food | Photo | Vinyl ) Hacker

Moneglia, Italy, via Puck, 2 day exposure
[ 2023-10-14 10:00 > https://domm.plix.at/potd/2023-10-14.html ]

Long and good read: http://bit-player.org/2023/ai-and-the-end-of-programming
[ 2023-09-14 12:01 > https://domm.plix.at/microblog.html ]
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Memory footprint of Koha processes

2023-10-24 Thread Jonathan Druart via Koha-devel 
Some ideas:
1. You could hack the worker to process all queues, then you could have
only one worker (?)
2. Stop the worker, modify Koha::BackgroundJob->enqueue to call ->process
(not with $self, you will need to refetch the job)
3. You can remove the "use" statements from plack.psgi to reduce plack's
memory footprint (but will be slower then)
4. Investigate why the worker is now taking 150M while it was 100 after
28413
5. Care about my bugs when I am working on memory footprint, so that I
don't have the feeling to do something nobody cares about... 28410 ;)

Le lun. 23 oct. 2023 à 21:57, Philippe Blouin via Koha-devel <
koha-devel@lists.koha-community.org> a écrit :

> Howdy!
>
> I'm trying to run a server with MANY koha instances (for educational
> purposes), but although the usage is very small, the resources required
> just make everything explode.
>
> background_jobs_worker.pl for instance uses 170M per instance, and plack
> uses 200M without anything being loaded.
>
> Any suggestion how this could be hacked (locally) to allow for 200 koha
> instances on a machine without requiring 100G of memory ?  The object model
> seems at fault here, but I see no way to go around "use
> Koha::BackgroundJobs;" at the top of the script.
>
> Thanks!
> --
> Philippe Blouin,
> Directeur de la technologie
>
> Tél.  : (833) 465-4276, poste 230
> philippe.blo...@inlibro.com
> inLibro | pour esprit libre | www.inLibro.com
> ___
> Koha-devel mailing list
> Koha-devel@lists.koha-community.org
> https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
> website : https://www.koha-community.org/
> git : https://git.koha-community.org/
> bugs : https://bugs.koha-community.org/
>
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Memory footprint of Koha processes

2023-10-23 Thread David Cook via Koha-devel 
Hi Philippe,

 

I run quite a few Koha instances, so I feel your pain.

 

A couple months ago I opened 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34477 but I haven’t 
dedicated any time to work on it yet (and I have no plans to work on it anytime 
soon).

 

The ORM eats up a lot of memory, and if I recall correctly C4::Context also 
requires a fair bit of memory. I don’t think you’ll really be able to hack this 
one per se as there are quite a few changes that are needed. That said… I 
suppose you could always create an alternative background jobs worker and use 
that instead. If I recall correctly, I don’t think you use the RabbitMQ, so you 
could exclude lots of code. 

 

In theory Koha::Database->dbh() should provide a database handle with a smaller 
memory footprint, but I haven’t tested it. 

 

But yeah… I think you’d basically need to do a rewrite. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
Philippe Blouin via Koha-devel
Sent: Tuesday, 24 October 2023 6:57 AM
To: koha-devel@lists.koha-community.org
Subject: [Koha-devel] Memory footprint of Koha processes

 

Howdy!

I'm trying to run a server with MANY koha instances (for educational purposes), 
but although the usage is very small, the resources required just make 
everything explode.

background_jobs_worker.pl for instance uses 170M per instance, and plack uses 
200M without anything being loaded.

Any suggestion how this could be hacked (locally) to allow for 200 koha 
instances on a machine without requiring 100G of memory ?  The object model 
seems at fault here, but I see no way to go around   
"use Koha::BackgroundJobs;" at the top of the script.

Thanks!

-- 

Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
 <mailto:philippe.blo...@inlibro.com> philippe.blo...@inlibro.com 

inLibro | pour esprit libre |  <http://www.inLibro.com> www.inLibro.com 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Memory footprint of Koha processes

2023-10-23 Thread Philippe Blouin via Koha-devel 

PS this is 22.05.06

Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
philippe.blo...@inlibro.com

inLibro | pour esprit libre | www.inLibro.com 
On 2023-10-23 15:57, Philippe Blouin wrote:


Howdy!

I'm trying to run a server with MANY koha instances (for educational 
purposes), but although the usage is very small, the resources 
required just make everything explode.


background_jobs_worker.pl for instance uses 170M per instance, and 
plack uses 200M without anything being loaded.


Any suggestion how this could be hacked (locally) to allow for 200 
koha instances on a machine without requiring 100G of memory ?  The 
object model seems at fault here, but I see no way to go around "use 
Koha::BackgroundJobs;" at the top of the script.


Thanks!

--
Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
philippe.blo...@inlibro.com

inLibro | pour esprit libre | www.inLibro.com ___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Memory footprint of Koha processes

2023-10-23 Thread Philippe Blouin via Koha-devel 

Howdy!

I'm trying to run a server with MANY koha instances (for educational 
purposes), but although the usage is very small, the resources required 
just make everything explode.


background_jobs_worker.pl for instance uses 170M per instance, and plack 
uses 200M without anything being loaded.


Any suggestion how this could be hacked (locally) to allow for 200 koha 
instances on a machine without requiring 100G of memory ?  The object 
model seems at fault here, but I see no way to go around "use 
Koha::BackgroundJobs;" at the top of the script.


Thanks!

--
Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
philippe.blo...@inlibro.com

inLibro | pour esprit libre | www.inLibro.com ___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/