Hi,
just want to share our experience with 17.05 to give you some hints where to go.
We have been using it in production since it was released and for us it has
been stable enough.
We are a large public library, 400k users, 19branches, some 70+ self checkout
machines using SIP, and a quite high item turnover.
For sure, mysql _needs_ quite a lot of ram, we specifically allocated 16G to
the innondb buffer pool (innodb-buffer-pool-size=16G).
This is needed since Koha rarely has been optimized for DB queries.
Second, plack, apache and sip should have a fair amount of workers. In our
example:
SIP_WORKERS: 125
APACHE_MINSERVERS: 20
APACHE_TIMEOUT: 1000
PLACK_WORKERS: "30"
PLACK_MAX_REQUESTS: "200"
Now, on the few troubles we had so far:
- when employees ran some troublesome SQL reports which consumed high amounts
of memory it could overload, by starting to swap memory to disk and stall the
system
since there is no limits on what to run from the Reports. We resolved this by
limiting the swap space (tmp) on the mysql server (tmpfs: /tmp:rw,size=655360k)
Troublesome queries that exceed swap size would now die. Not the perfect
solution, but a simple hack.
- patron lookups in intra was sometimes overloading system if using mysql
specific identifiers, e.g. percentage, '%a'. We resolved this by filtering with
javascript.
Again not perfect, but a simple hack.
We also, even prior to 17.05, had complaints on slow performance and started
investigating. The amount of unneccessary DB calls turned out to be absurd, so
a perl expert
helped us in resolving this my a memoized cached we named purefunctions. It's
been discussed in
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17941 which
is a bug fixing some performance issues in CanbookBeRenewed. In short
PureFunctions is a cache map to be used on the "immutable" functions called
often, usually Getting
info on items, patrons, etc. It is a short-lived cache, that is, it is function
scope, so it will not live after function is returned. As an example, we ran a
call on /svc/checkouts on
a patron with 5 checkouts with some items that had many reserves and saw that
about 15k db calls were saved and about 20sec.
We have been running this stable for more than 6 months in production, so we
are sure it is stable. Perhaps this should be added to the community? Our patch
is here:
https://gitlab.deichman.no/digibib/Koha/commit/919c65f1b521d051bc0dc137efaef0b62fd8d28b
As you can se, we have added it to any Get function that can be deemed safe,
and it has improved our Koha performance immensely.
To be sure, this is not fixing poorly optimized code, only bypassing it.
Regards
Benjamin Rokseth
Oslo Public Library
________________________________________
Fra: Koha <[email protected]> på vegne av
[email protected] <[email protected]>
Sendt: 12. oktober 2017 16:34
Til: [email protected]
Emne: Koha Digest, Vol 144, Issue 18
Send Koha mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.katipo.co.nz/mailman/listinfo/koha
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Koha digest..."
Today's Topics:
1. Re: Major, multiple problems with our Koha 17.05 system.
(Tajoli Zeno)
2. Elasticsearch on BWS Sandboxes (BOUIS Sonia)
3. Re: Major, multiple problems with our Koha 17.05 system.
(Tomas Cohen Arazi)
4. Re: Major, multiple problems with our Koha 17.05 system.
(Jonathan Druart)
----------------------------------------------------------------------
Message: 1
Date: Thu, 12 Oct 2017 14:29:53 +0200
From: Tajoli Zeno <[email protected]>
To: Raymund Delahunty <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: [Koha] Major, multiple problems with our Koha 17.05
system.
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Raymund,
Il 12/10/2017 13:29, Raymund Delahunty ha scritto:
> VERY long sorry, but our Koha system is virtually unusable and we need help!
generaly speaking, you can try to tune your MySQL setup, I find useful
this tool: https://github.com/major/MySQLTuner-perl
Attention that could you need more RAM on your server !!
About SIP, do a search on bugzilla:
https://bugs.koha-community.org/bugzilla3/
and see the open problems
Bye
Zeno Tajoli
--
Zeno Tajoli
/SVILUPPO PRODOTTI CINECA/ - Automazione Biblioteche
Email: [email protected] Fax: 051/6132198
*CINECA* Consorzio Interuniversitario - Sede operativa di Segrate (MI)
------------------------------
Message: 2
Date: Thu, 12 Oct 2017 12:29:56 +0000
From: BOUIS Sonia <[email protected]>
To: "[email protected]" <[email protected]>
Subject: [Koha] Elasticsearch on BWS Sandboxes
Message-ID:
<[email protected]>
Content-Type: text/plain; charset="utf-8"
What a good news !
Thank you Nick !
Sonia
------------------------------
Message: 6
Date: Thu, 12 Oct 2017 11:26:29 +0000
From: Nick Clemens <[email protected]>
To: Koha Devel <[email protected]>, Koha
<[email protected]>, ByWater Partners
<[email protected]>
Subject: [Koha] Elasticsearch on BWS Sandboxes
Message-ID:
<caa_ex3nao1ktbrn4w00nxkauslbgn4segqvaywpuq75narr...@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Hi All,
Just an update to ES testing, we have now made Elasticsearch available on
the BWS sandboxes:
http://sandbox.bywatersolutions.com/cgi-bin/sandbox-dashboard.pl
You will need to set the 'SearchEngine' system preference once the sandbox
is setup, but records should be indexed and ready to go.
If you have any questions or problems please let me know via email or in
IRC.
-Nick (kidclamp)
------------------------------
Message: 7
Date: Thu, 12 Oct 2017 11:29:17 +0000
From: Raymund Delahunty <[email protected]>
To: "[email protected]" <[email protected]>
Subject: [Koha] Major, multiple problems with our Koha 17.05 system.
Message-ID:
<he1pr0601mb2156a4dcc397d0a6feb95796dc...@he1pr0601mb2156.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
VERY long sorry, but our Koha system is virtually unusable and we need help!
We are experiencing dozens of outages daily (both OPAC and intranet, more often
only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl 5.020002, mysql
14.14, apache 2.4.10). Searches can take 15+ seconds. While university staff
provide first tier support, we don't have server access, and the second tier
and hosting is via an external company. Of course we are working with that
company to have them identify the cause, but I have decided to ask the
community of any user on 17.05 has the problems we have.
Our system is quite extensively customised, with major work done to the OPAC at
https://libsearch.arts.ac.uk<https://libsearch.arts.ac.uk/>. We have some
auto-renewal functionality we funded working, while we work on more
improvements (we are NOT using the template toolkit, relying on workarounds to
offer reasonably useful auto-renewal features).
Our problems started in the summer vacation soon after the upgrade, where Koha
returns functionality became unstable. It was taking 8 seconds to return one
item. Our support company advised that it was the AddReturn (and, in
particular, MarkIssueReturned) which was the pinch point. Seems some code to
resolve the auto_increment bug works slower in 17.05 and our support company
was correcting that with a view (I understand) to adding it to Master.
Things improved a little, but with the start of term almost 2 weeks ago we are
encountering so many problems...
"We were alerted to issues at 4.22pm. and found that the plack server was no
longer responding. We restarted plack and then apache and the service restored.
We are trying to get to the bottom of what caused this issue. It appears to be
completely separate from the high CPU problem that we are also seeing."
I understand more resources have been allocated to mySQL and additional plack
helpers have been allocated (whatever that means).
We are heavy users of SIP (80% of transactions are via self issue). But there
are so many problems with the returns unit that they are almost unusable. Front
line staff are fed up! The sorters and the kiosks configured for returns (as
well as issue) fail repeatedly dozens of times daily (ERR_SIP_COM_RECV).
Reconnection does happen- takes about 40seconds or so, leading to queues. And
the items have NOT come off the account and flow into the exceptions bin. At
the same time the intranet report 500 server errors. I am pretty sure most of
our problems are related to returns code, but could well be wrong/
oversimplifying things.
Our support company advised:
"We have added extra logging and can see that some checkin requests via the
SIP2 protocol are causing the sipserver process to abort. The book is returned
within Koha but as the process returns no information to the SIP2 client
therefore the unit responds with an error. After waiting for the timeout period
it then will reconnect to the server. We are adding further debug to the logs
to work out why the sipserver process is aborting. It does seem to be a special
subset of the checkin requests. I'll update you when we have more news."
And more problems relating to total outages (with proxy errors 502 relating to
invalid responses from an upstream server):
"There appears to be a number of issues you are experiencing. Friday you had
high CPU usage which is different to today's issues which we think are down to
the apache webserver not restarting correctly. I will double check first thing
tomorrow that the overnight apache restart has occurred properly. We are
hoping the database connections change may address the high CPU issue. We will
monitor closely for the errors we have seen in the apache error logs tomorrow."
We are a university with 20,000 students across 6 colleges, and right now
inductions and training in OPAC use is going on with new students. These
continual problems are deeply embarrassing, and faith in the Koha system is
dropping rapidly. Has any other Koha 17.05 user (especially one heavily reliant
on SIP) experienced anything like this? Has anyone out there any suggestions?
Ray Delahunty
University of the Arts London
This email and any attachments are intended solely for the addressee and may
contain confidential information. If you are not the intended recipient of this
email and/or its attachments you must not take any action based upon them and
you must not copy or show them to anyone. Please send the email back to us and
immediately and permanently delete it and its attachments. Where this email is
unrelated to the business of University of the Arts London or of any of its
group companies the opinions expressed in it are the opinions of the sender and
do not necessarily constitute those of University of the Arts London (or the
relevant group company). Where the sender's signature indicates that the email
is sent on behalf of UAL Short Courses Limited the following also applies: UAL
Short Courses Limited is a company registered in England and Wales under
company number 02361261. Registered Office: University of the Arts London, 272
High Holborn, London WC1V 7EY
------------------------------
Message: 8
Date: Thu, 12 Oct 2017 12:25:13 +0000
From: Marcel de Rooy <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: [Koha] Major, multiple problems with our Koha 17.05
system.
Message-ID:
<vi1pr05mb3487f8e520f21ea675ca2957ce...@vi1pr05mb3487.eurprd05.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
Very sorry to hear about your problems, Raymond.
Cant say very much about it, since we do not yet use 17.05.
But you should be searching on Bugzilla for the reports that got recently
backported to 17.05.x.
Might just include some that you need. Upgrading to the latest 17.05 would then
perhaps address some issues.
Just a side note: Very long delays in the past have been the result of date
calculations with very far future dates like 9999 in expiry dates etc. Setting
them back temporarily has been helpful in the past. (No guarantees; may be
other factors involved.)
Since your system is extensively customized, it could be helpful (for you) to
know if your system without custom changes in a test environment would still
experience the same problems..
And what about: Does your server have enough CPU and memory, etc ?
Just a few remarks.
Marcel
-----Oorspronkelijk bericht-----
Van: Koha [mailto:[email protected]] Namens Raymund Delahunty
Verzonden: donderdag 12 oktober 2017 13:29
Aan: [email protected]
Onderwerp: [Koha] Major, multiple problems with our Koha 17.05 system.
VERY long sorry, but our Koha system is virtually unusable and we need help!
We are experiencing dozens of outages daily (both OPAC and intranet, more often
only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl 5.020002, mysql
14.14, apache 2.4.10). Searches can take 15+ seconds. While university staff
provide first tier support, we don't have server access, and the second tier
and hosting is via an external company. Of course we are working with that
company to have them identify the cause, but I have decided to ask the
community of any user on 17.05 has the problems we have.
Our system is quite extensively customised, with major work done to the OPAC at
https://libsearch.arts.ac.uk<https://libsearch.arts.ac.uk/>. We have some
auto-renewal functionality we funded working, while we work on more
improvements (we are NOT using the template toolkit, relying on workarounds to
offer reasonably useful auto-renewal features).
Our problems started in the summer vacation soon after the upgrade, where Koha
returns functionality became unstable. It was taking 8 seconds to return one
item. Our support company advised that it was the AddReturn (and, in
particular, MarkIssueReturned) which was the pinch point. Seems some code to
resolve the auto_increment bug works slower in 17.05 and our support company
was correcting that with a view (I understand) to adding it to Master.
Things improved a little, but with the start of term almost 2 weeks ago we are
encountering so many problems...
"We were alerted to issues at 4.22pm. and found that the plack server was no
longer responding. We restarted plack and then apache and the service restored.
We are trying to get to the bottom of what caused this issue. It appears to be
completely separate from the high CPU problem that we are also seeing."
I understand more resources have been allocated to mySQL and additional plack
helpers have been allocated (whatever that means).
We are heavy users of SIP (80% of transactions are via self issue). But there
are so many problems with the returns unit that they are almost unusable. Front
line staff are fed up! The sorters and the kiosks configured for returns (as
well as issue) fail repeatedly dozens of times daily (ERR_SIP_COM_RECV).
Reconnection does happen- takes about 40seconds or so, leading to queues. And
the items have NOT come off the account and flow into the exceptions bin. At
the same time the intranet report 500 server errors. I am pretty sure most of
our problems are related to returns code, but could well be wrong/
oversimplifying things.
Our support company advised:
"We have added extra logging and can see that some checkin requests via the
SIP2 protocol are causing the sipserver process to abort. The book is returned
within Koha but as the process returns no information to the SIP2 client
therefore the unit responds with an error. After waiting for the timeout period
it then will reconnect to the server. We are adding further debug to the logs
to work out why the sipserver process is aborting. It does seem to be a special
subset of the checkin requests. I'll update you when we have more news."
And more problems relating to total outages (with proxy errors 502 relating to
invalid responses from an upstream server):
"There appears to be a number of issues you are experiencing. Friday you had
high CPU usage which is different to today's issues which we think are down to
the apache webserver not restarting correctly. I will double check first thing
tomorrow that the overnight apache restart has occurred properly. We are
hoping the database connections change may address the high CPU issue. We will
monitor closely for the errors we have seen in the apache error logs tomorrow."
We are a university with 20,000 students across 6 colleges, and right now
inductions and training in OPAC use is going on with new students. These
continual problems are deeply embarrassing, and faith in the Koha system is
dropping rapidly. Has any other Koha 17.05 user (especially one heavily reliant
on SIP) experienced anything like this? Has anyone out there any suggestions?
Ray Delahunty
University of the Arts London
This email and any attachments are intended solely for the addressee and may
contain confidential information. If you are not the intended recipient of this
email and/or its attachments you must not take any action based upon them and
you must not copy or show them to anyone. Please send the email back to us and
immediately and permanently delete it and its attachments. Where this email is
unrelated to the business of University of the Arts London or of any of its
group companies the opinions expressed in it are the opinions of the sender and
do not necessarily constitute those of University of the Arts London (or the
relevant group company). Where the sender's signature indicates that the email
is sent on behalf of UAL Short Courses Limited the following also applies: UAL
Short Courses Limited is a company registered in England and Wales under
company number 02361261. Registered Office: University of the Arts London, 272
High Holborn, London WC1V 7EY _______________________________________________
Koha mailing list http://koha-community.org [email protected]
https://lists.katipo.co.nz/mailman/listinfo/koha
------------------------------
Subject: Digest Footer
_______________________________________________
Koha mailing list
[email protected]
https://lists.katipo.co.nz/mailman/listinfo/koha
------------------------------
End of Koha Digest, Vol 144, Issue 17
*************************************
------------------------------
Message: 3
Date: Thu, 12 Oct 2017 14:16:22 +0000
From: Tomas Cohen Arazi <[email protected]>
To: koha <[email protected]>
Subject: Re: [Koha] Major, multiple problems with our Koha 17.05
system.
Message-ID:
<CABZfb=xgdsbvrifwyjv5767r+tyrwsd7j-tga989jxo9bov...@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
El jue., 12 oct. 2017 a las 8:29, Raymund Delahunty (<[email protected]>)
escribió:
> VERY long sorry, but our Koha system is virtually unusable and we need
> help!
>
> We are experiencing dozens of outages daily (both OPAC and intranet, more
> often only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl
> 5.020002, mysql 14.14, apache 2.4.10). Searches can take 15+ seconds. While
> university staff provide first tier support, we don't have server access,
> and the second tier and hosting is via an external company. Of course we
> are working with that company to have them identify the cause, but I have
> decided to ask the community of any user on 17.05 has the problems we have.
It seems you haven't enabled Plack on your setup.
Those numbers are really wrong.
--
Tomás Cohen Arazi
Theke Solutions (https://theke.io <http://theke.io/>)
✆ +54 9351 3513384
GPG: B2F3C15F
------------------------------
Message: 4
Date: Thu, 12 Oct 2017 14:34:13 +0000
From: Jonathan Druart <[email protected]>
To: Raymund Delahunty <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: [Koha] Major, multiple problems with our Koha 17.05
system.
Message-ID:
<CAJzKNY4-1u6LfY_=n+H0DOs-4LN=2q+ewauiods++cgbykv...@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
I am a bit desperate because I already told several times on this list that
some versions of Koha must not be used.
See https://wiki.koha-community.org/wiki/DBMS_auto_increment_fix
and you will see that 17.05.01 is a buggy version
My recommendation is to upgrade ASAP to the latest 17.05.x
Maybe it will not change anything, but you should not use this version
anyway.
Regards,
Jonathan
On Thu, 12 Oct 2017 at 08:29 Raymund Delahunty <[email protected]>
wrote:
> VERY long sorry, but our Koha system is virtually unusable and we need
> help!
>
> We are experiencing dozens of outages daily (both OPAC and intranet, more
> often only OPAC) with our Koha 17.05.01.000 (Debian 3.2.89-2, perl
> 5.020002, mysql 14.14, apache 2.4.10). Searches can take 15+ seconds. While
> university staff provide first tier support, we don't have server access,
> and the second tier and hosting is via an external company. Of course we
> are working with that company to have them identify the cause, but I have
> decided to ask the community of any user on 17.05 has the problems we have.
>
> Our system is quite extensively customised, with major work done to the
> OPAC at https://libsearch.arts.ac.uk<https://libsearch.arts.ac.uk/>. We
> have some auto-renewal functionality we funded working, while we work on
> more improvements (we are NOT using the template toolkit, relying on
> workarounds to offer reasonably useful auto-renewal features).
>
> Our problems started in the summer vacation soon after the upgrade, where
> Koha returns functionality became unstable. It was taking 8 seconds to
> return one item. Our support company advised that it was the AddReturn
> (and, in particular, MarkIssueReturned) which was the pinch point. Seems
> some code to resolve the auto_increment bug works slower in 17.05 and our
> support company was correcting that with a view (I understand) to adding it
> to Master.
>
> Things improved a little, but with the start of term almost 2 weeks ago we
> are encountering so many problems...
>
> "We were alerted to issues at 4.22pm. and found that the plack server was
> no longer responding. We restarted plack and then apache and the service
> restored. We are trying to get to the bottom of what caused this issue. It
> appears to be completely separate from the high CPU problem that we are
> also seeing."
>
> I understand more resources have been allocated to mySQL and additional
> plack helpers have been allocated (whatever that means).
>
> We are heavy users of SIP (80% of transactions are via self issue). But
> there are so many problems with the returns unit that they are almost
> unusable. Front line staff are fed up! The sorters and the kiosks
> configured for returns (as well as issue) fail repeatedly dozens of times
> daily (ERR_SIP_COM_RECV). Reconnection does happen- takes about 40seconds
> or so, leading to queues. And the items have NOT come off the account and
> flow into the exceptions bin. At the same time the intranet report 500
> server errors. I am pretty sure most of our problems are related to returns
> code, but could well be wrong/ oversimplifying things.
>
> Our support company advised:
> "We have added extra logging and can see that some checkin requests via
> the SIP2 protocol are causing the sipserver process to abort. The book is
> returned within Koha but as the process returns no information to the SIP2
> client therefore the unit responds with an error. After waiting for the
> timeout period it then will reconnect to the server. We are adding further
> debug to the logs to work out why the sipserver process is aborting. It
> does seem to be a special subset of the checkin requests. I'll update you
> when we have more news."
>
> And more problems relating to total outages (with proxy errors 502
> relating to invalid responses from an upstream server):
>
> "There appears to be a number of issues you are experiencing. Friday you
> had high CPU usage which is different to today's issues which we think are
> down to the apache webserver not restarting correctly. I will double check
> first thing tomorrow that the overnight apache restart has occurred
> properly. We are hoping the database connections change may address the
> high CPU issue. We will monitor closely for the errors we have seen in the
> apache error logs tomorrow."
>
> We are a university with 20,000 students across 6 colleges, and right now
> inductions and training in OPAC use is going on with new students. These
> continual problems are deeply embarrassing, and faith in the Koha system is
> dropping rapidly. Has any other Koha 17.05 user (especially one heavily
> reliant on SIP) experienced anything like this? Has anyone out there any
> suggestions?
>
>
> Ray Delahunty
> University of the Arts London
>
>
> This email and any attachments are intended solely for the addressee and
> may contain confidential information. If you are not the intended recipient
> of this email and/or its attachments you must not take any action based
> upon them and you must not copy or show them to anyone. Please send the
> email back to us and immediately and permanently delete it and its
> attachments. Where this email is unrelated to the business of University of
> the Arts London or of any of its group companies the opinions expressed in
> it are the opinions of the sender and do not necessarily constitute those
> of University of the Arts London (or the relevant group company). Where the
> sender's signature indicates that the email is sent on behalf of UAL Short
> Courses Limited the following also applies: UAL Short Courses Limited is a
> company registered in England and Wales under company number 02361261.
> Registered Office: University of the Arts London, 272 High Holborn,
> London WC1V 7EY
> <https://maps.google.com/?q=272+High+Holborn,+London+WC1V+7EY&entry=gmail&source=g>
> _______________________________________________
> Koha mailing list http://koha-community.org
> [email protected]
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
------------------------------
Subject: Digest Footer
_______________________________________________
Koha mailing list
[email protected]
https://lists.katipo.co.nz/mailman/listinfo/koha
------------------------------
End of Koha Digest, Vol 144, Issue 18
*************************************
_______________________________________________
Koha mailing list http://koha-community.org
[email protected]
https://lists.katipo.co.nz/mailman/listinfo/koha
