Re: [Koha-devel] Optimizing Starman startup

[dcook]

Hi Ere,

I think you're right about the refs. While they get resolved by the OpenAPI 
plugin, you probably have to resolve them before trying to dynamically inject 
the routes from plugins. 

Jan Thorsen (the author of Mojolicious::Plugin::OpenAPI and JSON::Validator) 
thinks that the ref resolution is actually what's taking so long. I looked it 
up and I think we have over 400 different references in the main OpenAPI spec 
alone. I haven't profiled it but something to think about.

At some point, I'm going to have a play with newer versions of the modules. I'm 
gong to look at Ubuntu 20.04 and newer Debian versions to see what I can get 
away with in terms of newness. Needs more investigation, but I am really hoping 
that this is an issue that can be solved by just upgrading the OS. 

I find Jan's code to be unnecessarily opaque (could use more descriptive 
comments and function naming) but... I'll investigate. Probably not right away 
as I have a bunch of other priorities that I have to address but... this is on 
my mind.

Starman startup time is probably the thing about Koha annoying me the most 
right now and probably the most practical thing I can improve at the moment...

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Ere Maijala  
Sent: Wednesday, 21 April 2021 6:31 PM
To: dc...@prosentient.com.au; koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup

Hi David,

I wish I'd remember all the details, but my memory fails me. I think not using 
JSON had something to do with how the refs are resolved. That may or may not 
have been the reason, but if everything works with JSON module, I can't think 
of a reason not to use it.

Thanks for taking a look!

--Ere

dc...@prosentient.com.au kirjoitti 21.4.2021 klo 3.28:
> Hi Ere,
> 
> Thanks for your reply. 24700 looks much better. I'll look at backporting it 
> locally.
> 
> Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at 
> https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm
>  and it says "Do not use this module directly. Use 
> Mojolicious::Plugin::OpenAPI instead." I notice that you're using the 
> "bundle" method. Do we really need that there? Why don't we just load the 
> JSON using the JSON module, merge with the plugin spec files, and then pass 
> it to the OpenAPI plugin? Shouldn't the plugin take care of the $ref 
> replacement?
> 
> Hmm... I didn't realize until now that the OpenAPI plugin was doing a 
> validate behind the scenes. That's tricky.
> 
> At a glance, we might be able to pre-load the app into the Starman 
> master process pre-fork. There are warnings about doing that with open 
> database connections, so we'd need to review plack.psgi, but a quick 
> glance suggests it might be OK. (Alternatively, I have wondered about 
> running the REST API as a separate process apart from Starman using 
> hypnotoad. According to 
> https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, 
> Mojo::Server::Prefork preloads the application in the manager/master 
> process, and Hypnotoad is based off that, so that would help.)
> 
> It does seem like changes to the OpenAPI plugin would be needed for caching.
> 
> I'm going to try backporting your change and try pre-loading and see how far 
> that gets me.
> 
> David Cook
> Software Engineer
> Prosentient Systems
> Suite 7.03
> 6a Glen St
> Milsons Point NSW 2061
> Australia
> 
> Office: 02 9212 0899
> Online: 02 8005 0595
> 
> -Original Message-
> From: Koha-devel  On 
> Behalf Of Ere Maijala
> Sent: Tuesday, 20 April 2021 4:48 PM
> To: koha-devel@lists.koha-community.org
> Subject: Re: [Koha-devel] Optimizing Starman startup
> 
> Hi,
> 
> I did some work on improving it here:
> 
> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
> 
> That shaved a good bit of time from it, but it's still a heavy 
> operation, and it would make sense to
> 
> 1.) avoid doing it too often
> 
> 2.) cache the results and avoid doing it if results are cached
> 
> If you could address the first one, that'd go a long way. I'm afraid the 
> second one would require changes to the OpenAPI plugin to support caching.
> 
> --Ere
> 
> dc...@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
>> Hi all,
>>
>> Do you despair when you see the following periodically in “top” when 
>> a starman worker is recreated ?
>>
>> PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
>> COMMAND
>>
>> 9529 my-koha  20   0  460108 197212  17172 R 100.0  0.4   0:03.41
>> /usr/share/koha/api/v1/app.pl
>>
>> Or the following in top when you install koha-common package or 
>> restart the koha-common service?
>>
>> 11101 1-koha  20   0  447232 193320  16076 R   10.6  0.4   0:09.09
>> /usr/share/koha/api/v1/app.pl
>>
>> 11168 1-koha  20   0  447240 193264  16056 R   10.6 

Re: [Koha-devel] Optimizing Starman startup

[dcook]

It's too bad that we're not using Unix sockets out of the box (like we do with 
Starman) as that would make Bug 26625 less pressing but maybe I'll look more at 
that...

The only downside with starting Koha as a Mojolicious application at the moment 
seems to be the experimentalness of it. I'm wondering how we can start a 
paradigm shift towards using it for production for everyone globally...

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
Julian Maurice
Sent: Wednesday, 21 April 2021 7:00 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup

Le 21/04/2021 à 09:26, dc...@prosentient.com.au a écrit :
> I recall someone (Julian or Kyle?) did some work to make Koha into a 
> Mojolicious application

Yes, since 20.11 Koha can be started as a Mojolicious application.

`hypnotoad bin/intranet`

But you'll probably need
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26625 in order to 
configure the listen port/address.

--
Julian Maurice
BibLibre
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/


___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Optimizing Starman startup

[Julian Maurice]

Le 21/04/2021 à 09:26, dc...@prosentient.com.au a écrit :
I recall someone (Julian or Kyle?) did some work to make Koha into a 
Mojolicious application


Yes, since 20.11 Koha can be started as a Mojolicious application.

`hypnotoad bin/intranet`

But you'll probably need 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26625 in order 
to configure the listen port/address.


--
Julian Maurice
BibLibre
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Optimizing Starman startup

[Ere Maijala]

Awesome, good to hear it's been useful. :) Even better if you can 
improve it further.


--Ere

dc...@prosentient.com.au kirjoitti 21.4.2021 klo 3.46:

Thanks for that optimization. My restart time has gone from 2 minutes 45 
seconds to 1 minute 30 seconds. Much better.

Now I'm going to try the preload and see if I can get it down to 45 seconds or 
better...

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
dc...@prosentient.com.au
Sent: Wednesday, 21 April 2021 10:28 AM
To: 'Ere Maijala' ; koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup

Hi Ere,

Thanks for your reply. 24700 looks much better. I'll look at backporting it 
locally.

Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at 
https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm
 and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." 
I notice that you're using the "bundle" method. Do we really need that there? Why don't 
we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to 
the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?

Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate 
behind the scenes. That's tricky.

At a glance, we might be able to pre-load the app into the Starman master 
process pre-fork. There are warnings about doing that with open database 
connections, so we'd need to review plack.psgi, but a quick glance suggests it 
might be OK. (Alternatively, I have wondered about running the REST API as a 
separate process apart from Starman using hypnotoad. According to 
https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork 
preloads the application in the manager/master process, and Hypnotoad is based 
off that, so that would help.)

It does seem like changes to the OpenAPI plugin would be needed for caching.

I'm going to try backporting your change and try pre-loading and see how far 
that gets me.

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of Ere 
Maijala
Sent: Tuesday, 20 April 2021 4:48 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup

Hi,

I did some work on improving it here:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700

That shaved a good bit of time from it, but it's still a heavy operation, and 
it would make sense to

1.) avoid doing it too often

2.) cache the results and avoid doing it if results are cached

If you could address the first one, that'd go a long way. I'm afraid the second 
one would require changes to the OpenAPI plugin to support caching.

--Ere

dc...@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:

Hi all,

Do you despair when you see the following periodically in “top” when a
starman worker is recreated ?

PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
COMMAND

9529 my-koha  20   0  460108 197212  17172 R 100.0  0.4   0:03.41
/usr/share/koha/api/v1/app.pl

Or the following in top when you install koha-common package or
restart the koha-common service?

11101 1-koha  20   0  447232 193320  16076 R   10.6  0.4   0:09.09
/usr/share/koha/api/v1/app.pl

11168 1-koha  20   0  447240 193264  16056 R   10.6  0.4   0:08.72
/usr/share/koha/api/v1/app.pl

11306 2-koha  20   0  447220 193148  16000 R   10.6  0.4   0:08.07
/usr/share/koha/api/v1/app.pl

11543 2-koha  20   0  447232 193036  15828 R   10.6  0.4   0:07.07
/usr/share/koha/api/v1/app.pl

11784 3-koha  20   0  441536 189664  16172 R   10.6  0.4   0:06.04
/usr/share/koha/api/v1/app.pl

11830 3-koha  20   0  439548 187212  15748 R   10.6  0.4   0:05.82
/usr/share/koha/api/v1/app.pl

11831 4-koha  20   0  438620 186344  15748 R   10.6  0.4   0:05.81
/usr/share/koha/api/v1/app.pl

11853 4-koha  20   0  437680 185672  16000 R   10.6  0.4   0:05.79
/usr/share/koha/api/v1/app.pl

Well, I still have a lot of investigation left to do, but I notice 1
place that a lot of time taken is here (per worker):

  my $validator = JSON::Validator::OpenAPI::Mojolicious->new;

  $validator->load_and_validate_schema(

  $self->home->rel_file("api/v1/swagger/swagger.json"),

  {

allow_invalid_ref  => 1,

  }

);

  push @{$self->routes->namespaces}, 'Koha::Plugin';

  my $spec = $validator->schema->data;

  $self->plugin(

  'Koha::REST::Plugin::PluginRoutes' => {

  spec  => $spec,

  validator => $validator

  }

  );

  $self->plugin(

  OpenAPI => {

  spec  => $spec,

  

Re: [Koha-devel] Optimizing Starman startup

[Ere Maijala]

Hi David,

I wish I'd remember all the details, but my memory fails me. I think not 
using JSON had something to do with how the refs are resolved. That may 
or may not have been the reason, but if everything works with JSON 
module, I can't think of a reason not to use it.


Thanks for taking a look!

--Ere

dc...@prosentient.com.au kirjoitti 21.4.2021 klo 3.28:

Hi Ere,

Thanks for your reply. 24700 looks much better. I'll look at backporting it 
locally.

Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at 
https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm
 and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead." 
I notice that you're using the "bundle" method. Do we really need that there? Why don't 
we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to 
the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?

Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate 
behind the scenes. That's tricky.

At a glance, we might be able to pre-load the app into the Starman master 
process pre-fork. There are warnings about doing that with open database 
connections, so we'd need to review plack.psgi, but a quick glance suggests it 
might be OK. (Alternatively, I have wondered about running the REST API as a 
separate process apart from Starman using hypnotoad. According to 
https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork 
preloads the application in the manager/master process, and Hypnotoad is based 
off that, so that would help.)

It does seem like changes to the OpenAPI plugin would be needed for caching.

I'm going to try backporting your change and try pre-loading and see how far 
that gets me.

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of Ere 
Maijala
Sent: Tuesday, 20 April 2021 4:48 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup

Hi,

I did some work on improving it here:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700

That shaved a good bit of time from it, but it's still a heavy operation, and 
it would make sense to

1.) avoid doing it too often

2.) cache the results and avoid doing it if results are cached

If you could address the first one, that'd go a long way. I'm afraid the second 
one would require changes to the OpenAPI plugin to support caching.

--Ere

dc...@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:

Hi all,

Do you despair when you see the following periodically in “top” when a
starman worker is recreated ?

PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+
COMMAND

9529 my-koha  20   0  460108 197212  17172 R 100.0  0.4   0:03.41
/usr/share/koha/api/v1/app.pl

Or the following in top when you install koha-common package or
restart the koha-common service?

11101 1-koha  20   0  447232 193320  16076 R   10.6  0.4   0:09.09
/usr/share/koha/api/v1/app.pl

11168 1-koha  20   0  447240 193264  16056 R   10.6  0.4   0:08.72
/usr/share/koha/api/v1/app.pl

11306 2-koha  20   0  447220 193148  16000 R   10.6  0.4   0:08.07
/usr/share/koha/api/v1/app.pl

11543 2-koha  20   0  447232 193036  15828 R   10.6  0.4   0:07.07
/usr/share/koha/api/v1/app.pl

11784 3-koha  20   0  441536 189664  16172 R   10.6  0.4   0:06.04
/usr/share/koha/api/v1/app.pl

11830 3-koha  20   0  439548 187212  15748 R   10.6  0.4   0:05.82
/usr/share/koha/api/v1/app.pl

11831 4-koha  20   0  438620 186344  15748 R   10.6  0.4   0:05.81
/usr/share/koha/api/v1/app.pl

11853 4-koha  20   0  437680 185672  16000 R   10.6  0.4   0:05.79
/usr/share/koha/api/v1/app.pl

Well, I still have a lot of investigation left to do, but I notice 1
place that a lot of time taken is here (per worker):

  my $validator = JSON::Validator::OpenAPI::Mojolicious->new;

  $validator->load_and_validate_schema(

  $self->home->rel_file("api/v1/swagger/swagger.json"),

  {

allow_invalid_ref  => 1,

  }

);

  push @{$self->routes->namespaces}, 'Koha::Plugin';

  my $spec = $validator->schema->data;

  $self->plugin(

  'Koha::REST::Plugin::PluginRoutes' => {

  spec  => $spec,

  validator => $validator

  }

  );

  $self->plugin(

  OpenAPI => {

  spec  => $spec,

  route =>
$self->routes->under('/api/v1')->to('Auth#under'),

  allow_invalid_ref =>

1,# required by our spec because $ref directly
under

  # Paths-, Parameters-, Definitions- & Info-object

  # is not allowed by the OpenAPI specification.

  }

);

Anyone have ideas for improving 

Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

[dcook]

I like to think that we’re two peas in a pod. (I don’t know why I use so many 
English idioms on the listserv. I swear that I don’t actually use them in my 
every day life.)

 

I’m not sure how easy it is to split a package up with DEB packages but it’s so 
easy with RPM packages. I feel like there’s no harm in trying. 

 

I haven’t been motivated to code outside of work hours for a while now, but 
maybe I would if I had that koha-libs package. I like the idea of spending free 
time working on little services that can live in their own Docker containers. 

 

(Actually, I already have something like that for Shibboleth. You can’t install 
it on Ubuntu 18.04 due to broken dependencies, so what I do is have a Docker 
container running Debian and do everything with Shibboleth in there, and I just 
proxy from the Docker host that runs Koha to the Docker container to do the 
auth. I then pass back a one-shot auth token to authenticate back into Koha. 
Works a treat. If I had access to the Koha core libs, I’d be able to do it even 
more seamlessly…)

 

David Cook

Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Renvoize, Martin  
Sent: Wednesday, 21 April 2021 4:49 PM
To: David Cook 
Cc: Mason James ; Koha Devel 
; Tomas Cohen Arazi 
Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

 

This is another area where I think our thoughts align David.  I've also wanted 
to split his into even smaller units for a while..  a clear example is our move 
to background jobs. I see those as prime targets for horizontal scaling.. but 
as you've highlighted, at the moment to get the library of core code you may 
want to write a job you need to install the whole app.. koha-libs would 
certainly help with that

 

On Tue, 20 Apr 2021, 2:12 am , mailto:dc...@prosentient.com.au> > wrote:

Hi Mason,

[FYI this started as a response and turned into a 
stream-of-consciousness.][Also really keen to hear from Tomas and Agustin about 
their work with Docker/Kubernetes/Helm/etc.]

The idea with "koha-core" was to separate out the third-party systems away from 
the core application. In this case, "koha-libs" would be about separating the 
core libraries away from the core application. 

Consider that the "koha-core" package contains the full Koha application 
(including code, templates, images, other assets, etc) and its application 
dependencies (e.g. at, cron, daemon, starman, sudo, unzip, xmlstarlet, yaz, 
mysql-client). My latest koha-core package alone was 38MB. A "koha-libs" 
package would probably be measured in terms of KB.

I'm thinking about the absolute minimum required to run things like the SIP 
server or some other service/microservice (e.g. z3950-responder, REST service, 
etc). The koha-libs package would need to contain the Koha .pm files and 
koha-libs would depend on koha-perldeps. 

In theory, over time, some of the libraries and dependencies would probably 
move out of koha-libs and koha-perldeps to live in the individual services, but 
I don't have a full vision for that yet.  

You could think about a koha-libs package as a Perl API rather than a HTTP API, 
which could be used by Koha's existing background services, and it could be 
used by new separate services.

I have a vision for creating a service that depends on koha-libs, points at a 
koha-conf.xml file, and then implements its own functionality separate from the 
main Koha codebase. 

It could allow for a lot of power and flexibility. 

--

That said, maybe I'm thinking too small and too Debian-focused. Maybe a CPAN 
distribution of Koha/ modules makes more sense (or both koha-libs and a CPAN 
distribution). Apparently we could use the cpanfile to specify the dependencies 
for the CPAN distribution too. Then you could install it on any suitable 
OS/container. In a container environment, you could always install the Koha/ 
modules and dependencies on a volume, and then share that volume with your 
containers. That would have some pros and cons. 

I'm mostly just brainstorming. Thinking about how we can re-organize things to 
take advantage of modern technology. But then to what end?

One advantage of Docker is to run different versions of the same software on 
the same virtual machine...

While I was thinking about 1+ containers per Koha library, maybe we do just go 
with a 20.05 Koha container, a 20.11 Koha container, a 21.05 Koha container. I 
suppose each of those versions could have multiple containers though. For 
example, a SIP server container, a Z39.50 responder container, a Zebra indexer 
container, a Zebra server container, etc. Technically speaking, if we planned 
the volumes right, the Koha "etc" config and "share" and "var" volumes could 
all be shared across those containers. Of course, that means that all the 
containers have to be running on the same host. At the scale of most Koha 
libraries, that would 

Re: [Koha-devel] Optimizing Starman startup

[dcook]

He did answer back fairly quickly but not a very helpful way. Maybe I just 
wasn’t asking the right questions.

 

Agreed. I wish that we kept up better with Mojolicious. 

 

I was actually wondering why we don’t use Hypnotoad for the REST API. Is it 
just because of the extra process management? I recall someone (Julian or 
Kyle?) did some work to make Koha into a Mojolicious application and I have 
some pending work that uses Mojolicious more, so it would be nice to further 
embrace it and to use Hypnotoad. 

 

David Cook

Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Renvoize, Martin  
Sent: Wednesday, 21 April 2021 4:40 PM
To: David Cook 
Cc: Koha Devel 
Subject: Re: [Koha-devel] Optimizing Starman startup

 

I've always found the OpenAPI Dev finally and responsive. 

 

However, it's worth remembering were really not using the module as it's 
intended. Mojolicious D the ecosystem of modules around it are generally fast 
moving and we don't keep up at all.. we also deliver into the guts frequently 
and do things the authors never expected and finally we not using any of the 
recommended deployment options. Hypnotoad is by far the most optimal way of 
running mojo and this module.

 

We have so many layers of obfuscation in our code it's hard to comprehend.. 
various layers virtualizing other layers etc.

 

On Wed, 21 Apr 2021, 6:23 am , mailto:dc...@prosentient.com.au> > wrote:

It looks like newer version of the plugin might have cached spec/schema 
functionality, but I'm not finding the plugin author very helpful so far.  My 
guess is that maybe... maybe... Mojolicious::Plugin::OpenAPI 3.41+ with 
JSON::Validator 4.10+ might perform better. But it's just a guess at this point.

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of 
dc...@prosentient.com.au  
Sent: Wednesday, 21 April 2021 1:21 PM
To: 'Ere Maijala' mailto:ere.maij...@helsinki.fi> >; 
koha-devel@lists.koha-community.org 
 
Subject: Re: [Koha-devel] Optimizing Starman startup

Looking at newer versions of Mojolicious::Plugin::OpenAPI and the coding 
choices are... interesting. 

I opened a ticket against JSON::Validator 
(https://github.com/jhthorsen/json-validator/issues/246), but maybe I should 
open it against Mojolicious::Plugin::OpenAPI. We'll see how responsive the 
author is on Github.

Of course, we're using such ancient versions of the Mojolicious modules that 
even convincing the author to cache validation or skip validation won't really 
help us in the short- to medium- term. 

At the moment, I think that I might just have to resign myself to Koha being 
slow to start up, as we're beholden to this third-party module.

That said, we could fork Mojolicious::Plugin::OpenAPI. 

I suppose I could just do that locally too. H. 

David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of 
dc...@prosentient.com.au  
Sent: Wednesday, 21 April 2021 12:12 PM
To: 'Ere Maijala' mailto:ere.maij...@helsinki.fi> >; 
koha-devel@lists.koha-community.org 
 
Subject: Re: [Koha-devel] Optimizing Starman startup

That was an interesting experience.

When you run koha-plack --start, it spins through a long list of instances 
quickly, because it returns immediately after starting the Starman master, 
which doesn't really do much, but fork off worker processes, which then consume 
a huge amount of resources concurrently. 

However, when you preload the app using --preload-app, the resource consumption 
is in the Starman master, and the command doesn't return until after the 
Starman master has loaded the app. So you synchronously move through the list 1 
by 1.

This has pros and cons.

Pro: You can start 60 Koha instances with very little resource usage, because 
you're only starting 1 Koha at a time. Yay!
Con: "service koha-common restart" will stop all Plack instances and then start 
them all. This is suboptimal, as it took me about 3.5 minutes to start 60 Koha 
instances. That's a lot of downtime for the last Koha instance in that list. 

Of course, we could change "koha-common" init service to actually restart 
instead of stop/start on a "restart" action. 

But the other gotcha is Debian package installs/upgrades (hello continuous 
deployment). According to 
https://manpages.debian.org/testing/debhelper/dh_installinit.1.en.html, the 
init script is "stopped" in the prerm and 

Re: [Koha-devel] Packaging idea (for decoupling systems, Docker, etc)

[Renvoize, Martin]

This is another area where I think our thoughts align David.  I've also
wanted to split his into even smaller units for a while..  a clear example
is our move to background jobs. I see those as prime targets for horizontal
scaling.. but as you've highlighted, at the moment to get the library of
core code you may want to write a job you need to install the whole app..
koha-libs would certainly help with that

On Tue, 20 Apr 2021, 2:12 am ,  wrote:

> Hi Mason,
>
> [FYI this started as a response and turned into a
> stream-of-consciousness.][Also really keen to hear from Tomas and Agustin
> about their work with Docker/Kubernetes/Helm/etc.]
>
> The idea with "koha-core" was to separate out the third-party systems away
> from the core application. In this case, "koha-libs" would be about
> separating the core libraries away from the core application.
>
> Consider that the "koha-core" package contains the full Koha application
> (including code, templates, images, other assets, etc) and its application
> dependencies (e.g. at, cron, daemon, starman, sudo, unzip, xmlstarlet, yaz,
> mysql-client). My latest koha-core package alone was 38MB. A "koha-libs"
> package would probably be measured in terms of KB.
>
> I'm thinking about the absolute minimum required to run things like the
> SIP server or some other service/microservice (e.g. z3950-responder, REST
> service, etc). The koha-libs package would need to contain the Koha .pm
> files and koha-libs would depend on koha-perldeps.
>
> In theory, over time, some of the libraries and dependencies would
> probably move out of koha-libs and koha-perldeps to live in the individual
> services, but I don't have a full vision for that yet.
>
> You could think about a koha-libs package as a Perl API rather than a HTTP
> API, which could be used by Koha's existing background services, and it
> could be used by new separate services.
>
> I have a vision for creating a service that depends on koha-libs, points
> at a koha-conf.xml file, and then implements its own functionality separate
> from the main Koha codebase.
>
> It could allow for a lot of power and flexibility.
>
> --
>
> That said, maybe I'm thinking too small and too Debian-focused. Maybe a
> CPAN distribution of Koha/ modules makes more sense (or both koha-libs and
> a CPAN distribution). Apparently we could use the cpanfile to specify the
> dependencies for the CPAN distribution too. Then you could install it on
> any suitable OS/container. In a container environment, you could always
> install the Koha/ modules and dependencies on a volume, and then share that
> volume with your containers. That would have some pros and cons.
>
> I'm mostly just brainstorming. Thinking about how we can re-organize
> things to take advantage of modern technology. But then to what end?
>
> One advantage of Docker is to run different versions of the same software
> on the same virtual machine...
>
> While I was thinking about 1+ containers per Koha library, maybe we do
> just go with a 20.05 Koha container, a 20.11 Koha container, a 21.05 Koha
> container. I suppose each of those versions could have multiple containers
> though. For example, a SIP server container, a Z39.50 responder container,
> a Zebra indexer container, a Zebra server container, etc. Technically
> speaking, if we planned the volumes right, the Koha "etc" config and
> "share" and "var" volumes could all be shared across those containers. Of
> course, that means that all the containers have to be running on the same
> host. At the scale of most Koha libraries, that would probably be fine. I
> suppose the main change here would be to the debian scripts and koha-common
> init script...
>
> --
>
> Overall, I suppose though most vendors would win from improved
> multi-tenancy. Having 100 Starman instances for 50 Koha libraries where
> only 3 libraries might have requests at a time is massive overhead.
>
> I imagine improved multi-tenancy coupled with multiple versions per host
> via Docker... that would be a nicer setup.
>
> Preloading Koha::REST::V1 and getting its work done in the master starman
> process rather than the child processes would be better too...
>
> Ok, that's enough ranting for me for now heh. Too many ideas and never
> enough time.
>
> David Cook
> Software Engineer
> Prosentient Systems
> Suite 7.03
> 6a Glen St
> Milsons Point NSW 2061
> Australia
>
> Office: 02 9212 0899
> Online: 02 8005 0595
>
> -Original Message-
> From: Koha-devel  On Behalf
> Of Mason James
> Sent: Friday, 16 April 2021 3:50 PM
> To: koha-devel@lists.koha-community.org
> Subject: Re: [Koha-devel] Packaging idea (for decoupling systems, Docker,
> etc)
>
> hi David
>
> i think we may already have this functionality with the 'koha-core'
> package?
>
>
> On 15/04/21 3:32 pm, dc...@prosentient.com.au wrote:
> >
> > Hey all,
> >
> > What do people think about creating a “koha-libs” package which just
> contains Koha’s libraries (ie C4/, Koha/, etc)?
> >
> > I’m not as