[Koha-devel] Bug 33092 - Add ability to search within a list

[David Cook via Koha-devel]

Hi all,

 

Has anyone looked at this one?

 

My first thought is that the list ID would need to be added to a repeatable
subfield (e.g. 999$l) and indexed. That way it would be easy to allow any
search engine query and then just wrap it like "() and (list_id=X)".

 

But. it would mean that we'd have to re-index records for list operations,
and I don't love that. 

 

Does anyone have any other ideas? Or happy to support this one? 

 

It would be a fairly significant change to lists, so I think it would need
to be well thought out.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Google OpenID Connect

[David Cook via Koha-devel]

The Google OpenID Connect specific functionality only targeted the OPAC. 
Probably because it’s the most common use case. (I wrote my own OpenID Connect 
integration for Koha about 10 years ago, and it only targeted the OPAC too.)

 

However, Koha has a different more generic Oauth2/OpenID Connect feature that 
targets both the OPAC and the staff interface. It should be usable with Google 
as well, if you want to target it.

 

I think at some point the Google OpenID Connect specific code will be removed 
from Koha.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
long_sam.tw via Koha-devel
Sent: Thursday, 18 April 2024 7:22 AM
To: koha-devel 
Subject: [Koha-devel] Google OpenID Connect

 

Hi, all

 

Google OpenID Connect is used in koha. Why does opac have this identity 
authentication login, but staff does not have this function? What is the main 
reason?

 

With respect, long_sam

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] SelfCheckoutByLogin

[David Cook via Koha-devel]

Hi long_sam,

 

Thanks for your reply. You’re right; SAML does have single log out 
functionality.  However, from my understanding it’s a “front-channel logout” 
mechanism that requires the redirecting of the user in the browser.

 

With OpenID Connect, the “back-channel logout” is a request sent from the 
server to the IdP without needing to redirect the user in the browser. As far 
as I know, SAML doesn’t have “back-channel logout” functionality. 

 

That being said, I suppose there’s no reason you couldn’t do a front-channel 
logout. In fact, that might be preferable, because then the user would know 
that they’ve been logged out of the Identity Provider (or that there was some 
failure to logout, so they could take manual steps to end their SSO session).

 

--

 

I like the idea of being able to use credentials from a centralized Identity 
Provider to log into the self-checkout, but personally I wouldn’t want to use 
SSO on a shared system that could potentially leave my SSO session open beyond 
my interactions with the self-checkout. And I’m a tech-savvy user. I fear for 
users who are not tech-savvy. 

 

Another technical detail. At the moment, Koha is set up to allow SAML or OpenID 
Connect for creating authenticated user sessions. In the case of the 
self-checkout, the actual user session belongs to a self-checkout user. When 
you put in your cardnumber or username/password, you’re not actually creating a 
new user session. You’re really just authorizing the self-checkout user to work 
with your account. So the existing code for the SAML/OpenID Connect can’t be 
easily re-used I think. We’d need totally different implementations just for 
the self-checkout. (I suppose one option would be to allow SSO login, confirm 
the authenticated session, and then immediately do a SLO (single log out) from 
the IdP to help reduce the time the SSO session exists.)

 

--

 

So overall… there are ways of doing it. A few of us discussed it at 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30444 but ultimately 
no one decided to pursue it. 

 

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30979 provided a 
method for allowing OPAC users to checkout to themselves, so you’d login as the 
OPAC user (using SSO) and then checkout to yourself. I think the idea here is 
that you do it on a private device though. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: long_sam.tw  
Sent: Monday, 15 April 2024 10:45 PM
To: koha-devel@lists.koha-community.org; David Cook 
Subject: Re: [Koha-devel] SelfCheckoutByLogin

 

Hi, David 

 

I found the AWS SAML SSO logout example.

 

SAML sign-out flow - Amazon Cognito 

 

 

 

 

 

SAML SSO logout IDP, security issues.

 

 

When logging out of a SAML SSO IDP (Identity Provider), there are several 
security considerations to keep in mind:

1. Single Logout (SLO) Support: Ensure that your IDP supports Single Logout 
functionality, which logs the user out from all related Service Providers (SPs) 
when they log out from one, maintaining session consistency and security.

2. Logout Request Validation: When the IDP receives a logout request from an 
SP, it must validate the request to prevent malicious requests or Cross-Site 
Request Forgery (CSRF) attacks. Validation can be achieved through digital 
signatures or other secure mechanisms.

3. Security of Callback URLs: Ensure that the callback URLs used during logout 
are secure, avoiding the use of vulnerable or unauthorized URLs.

4. Session Management: Ensure that the IDP correctly terminates relevant 
sessions and clears user authentication information and session data upon 
logout to prevent session hijacking or replay attacks.

5. Security Event Monitoring: Establish monitoring mechanisms for logout 
operations and related session management events to promptly detect abnormal 
behavior or security incidents and take necessary response measures.

6. Security Auditing and Logging: Conduct thorough auditing and logging of 
logout operations and related security events to facilitate audit 
investigations or security incident tracing when needed.

7. Integration with Other Security Mechanisms: Integrate the logout 
functionality of SAML SSO with other security mechanisms such as Multi-Factor 
Authentication (MFA), Access Control Lists (ACLs), etc., to enhance the overall 
security of the system.

8. Regular Security Assessments: Conduct regular security assessments and 
vulnerability scans of the SAML SSO logout process, and promptly address any 
identified security issues to ensure the security and stability of the system.

In summary, logging out of a SAML SSO IDP requires attention to ensuring Single 
Logout support, secure logout request validation, security of callback 

Re: [Koha-devel] SelfCheckoutByLogin

[David Cook via Koha-devel]

Part of the reason is that it’s considerably more complicated and error-prone. 

 

If you log in using Google OpenID Connect, the self-checkout browser will 
retain your Google user session beyond your Koha self-checkout user session. 
Also, when Koha goes back to Google to authenticate someone else, it will 
auto-detect that you’re still logged in, and use your account instead. 

 

In theory, we could do a back channel logout against Google (or whatever other 
OpenID Connect identity provider), but if that failed to run for whatever 
reason you’re risking someone else at a public terminal accessing your personal 
Google account.

 

SAML doesn’t even have options for back channel logout, which makes it not an 
option at all. 

 

If someone can think of a really good way of making this work, I’d be happy to 
discuss it further, but I can’t think of a safe way to do this on a public 
terminal at the moment.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
Katrin Fischer via Koha-devel
Sent: Monday, 15 April 2024 6:29 AM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] SelfCheckoutByLogin

 

Hi,

I think there is probably no specific reason, it's just not been developed yet.

As a next step you could search Bugzilla 
(https://bugs.koha-community.org/bugzilla3/) for any related bugs. If there is 
no existing report yet, you could file a new enhancement request.

Hope this helps,

Katrin

On 12.04.24 23:49, long_sam.tw via Koha-devel wrote:

Hi, all

 

Koha SelfCheckoutByLogin 

 

https://koha-community.org/manual/latest/en/html/circulationpreferences.html#selfcheckoutbylogin

 

 

I found that only local account authentication and cardnumber are supported, 
but other authentication methods are not supported,

such as google openid Oauth2, are not supported.

 

Can anyone explain the reason?

 

With respect, long_sam

 

 





___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL

[David Cook via Koha-devel]

I’m just finishing up the next version of the plugin which finds problem bib 
records, problem item records, and lets you fix 1 bib record and its X problem 
item records at a time.

 

I’ll be putting it up online sometime in the next couple weeks. (I’m away a lot 
these couple of weeks, so tough to give a firm time.)

 

But I’m about to use it to find problem item records in one of my systems…

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Tomas Cohen Arazi  
Sent: Sunday, 14 April 2024 2:49 AM
To: Magnus Enger 
Cc: David Cook ; koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL

 

Yeah, dump it somewhere and we send pull requests to add missing bits

 

El El vie, 12 abr 2024 a la(s) 9:50 a. m., Magnus Enger via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > escribió:

Den 12.04.2024 08:24, skrev David Cook:
> Not yet. At some point I need to get better at sharing Koha plugins more 
> widely.

Just dump it to Github or similar, and set the version to 0.0.1. ;-) 
Release early, release often.

Best regards,
Magnus
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] [Koha] Error 500 when searching in Koha 23.11.0

[David Cook via Koha-devel]

Hi Michael,

I don't think your database was properly upgraded, and doing these manual 
corrections isn't really going to be enough unless you go line by line through 
the upgrade scripts and figure out all the things that were missed.

If you're able to, I would go back and try the upgrade again. If you can't... 
then I would carefully look through every database revision. 

You might also want to consider looking at ./misc/maintenance/audit_database.pl 
as that could save you time. (Just don't blindly run the suggestions made by 
the script. They're just hints.)

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
Michael Kuhn via Koha-devel
Sent: Saturday, 13 April 2024 9:28 AM
To: k...@lists.katipo.co.nz; Koha-devel 
Subject: Re: [Koha-devel] [Koha] Error 500 when searching in Koha 23.11.0

Hi

Just for the record: I updated from Koha 21.11.10 to 23.11.04.

When searching the catalogue I got a result list, but when clicking a single 
hit I got an error 500. In file "plack-error.log" I found this:

DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st 
execute failed: Unknown column 'issue.renewals_count' in 'field list' at 
/usr/share/koha/lib/Koha/Objects.pm line 317

In Koha 21.11 this column was called "issues.renewals" and in Koha 23.11 it 
should be called "issues.renewals_count" - but for whatever reason the upgrade 
did not rename it properly. I have done that myself:

ALTER TABLE issues RENAME COLUMN renewals TO renewals_count;

Now single hits show up.

PS1: The error message says the column is "issue.renewals_count" but actually 
it is "issues.renewals_count".

PS2: https://schema.koha-community.org/23_11/tables/issues.html says the type 
of this column is tinyint(3) while it actually is tinyint(4).

Best wishes: Michael
--
Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin 
Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 
· E m...@adminkuhn.ch · W www.adminkuhn.ch

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL

[David Cook via Koha-devel]

Not yet. At some point I need to get better at sharing Koha plugins more 
widely. 

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
Magnus Enger via Koha-devel
Sent: Friday, 12 April 2024 4:09 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL



Den 12.04.2024 04:13, skrev David Cook via Koha-devel:
> I have a RepairRecord plugin, so I might do a version in that first, 
> and if that goes well I could look at upstreaming a patch…

Intriguing! :-) Is it available somewhere?

Best regards,
Magnus
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL

[David Cook via Koha-devel]

Yeah, I was thinking it would be good to add to the SQL Report Library. I've 
just been flat out today... (but wanted to make sure I shared it with you folk 
at least)

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
Magnus Enger via Koha-devel
Sent: Friday, 12 April 2024 4:06 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL

Hi!

Den 12.04.2024 03:36, skrev David Cook via Koha-devel:
> Hi all,
> 
> I just wanted to share a (MariaDB) SQL report that I wrote for finding 
> bib records with invalid XML characters:
> 
> select biblionumber from biblio_metadata where metadata REGEXP 
> '[^\\x{0009}\\x{000A}\\x{000D}\\x{0020}-\\x{D7FF}\\x{E000}-\\x{FFFD}\\
> x{1}-\\x{10}]+';
> 
> Newer versions of Koha strip invalid character from the XML so that 
> you can fix your records. I figure this report is very valuable when 
> coupled with that functionality. In fact, I just advised a library 
> today to use them together to fix up some bad data in their catalogue.
> 
> --
> 
> On a related note, I’ve noticed that you can have a record with good 
> bib XML but invalid item XML, and you won’t notice until your record 
> fails to be indexed. So I’m planning on writing a report for that too.
> 
> I’m thinking it might be good to add these reports to core Koha, so 
> that people can find and fix their own metadata problems. What do people 
> think?

Sounds like an excellent idea! Sounds kind of similar to "MARC bibliographic 
framework test" at /cgi-bin/koha/admin/checkmarc.pl

The report could also be added to
https://wiki.koha-community.org/wiki/SQL_Reports_Library for older Kohas and to 
be immediately useful.

Best regards,
Magnus
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Bug 36586 - Self-checkouts will get CSRF errors if left inactive for 8 hours

[David Cook via Koha-devel]

Hi all,

 

We recently noticed that self-checkouts will generate CSRF errors when left
unattended for over 8 hours (the lifetime of the CSRF token).

 

Our solution is to use a timer to refresh the page every 8 hours or so, but
open to other ideas. 

 

This will potentially affect Vue.js driven pages that don't reload the whole
page. Although maybe Jonathan has already come up with a solution for that.
I'm not sure. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Finding invalid XML characters in Koha data via SQL

[David Cook via Koha-devel]

Alas, I couldn't think of a really clever way of doing the items table, so I
think it'll need a Perl-based solution.

 

I have a RepairRecord plugin, so I might do a version in that first, and if
that goes well I could look at upstreaming a patch.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of
David Cook via Koha-devel
Sent: Friday, 12 April 2024 11:36 AM
To: 'Koha-devel' 
Subject: [Koha-devel] Finding invalid XML characters in Koha data via SQL

 

Hi all,

 

I just wanted to share a (MariaDB) SQL report that I wrote for finding bib
records with invalid XML characters:

select biblionumber from biblio_metadata where metadata REGEXP
'[^\\x{0009}\\x{000A}\\x{000D}\\x{0020}-\\x{D7FF}\\x{E000}-\\x{FFFD}\\x{1000
0}-\\x{10}]+';

 

Newer versions of Koha strip invalid character from the XML so that you can
fix your records. I figure this report is very valuable when coupled with
that functionality. In fact, I just advised a library today to use them
together to fix up some bad data in their catalogue.

 

--

 

On a related note, I've noticed that you can have a record with good bib XML
but invalid item XML, and you won't notice until your record fails to be
indexed. So I'm planning on writing a report for that too. 

 

I'm thinking it might be good to add these reports to core Koha, so that
people can find and fix their own metadata problems. What do people think?

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Finding invalid XML characters in Koha data via SQL

[David Cook via Koha-devel]

Hi all,

 

I just wanted to share a (MariaDB) SQL report that I wrote for finding bib
records with invalid XML characters:

select biblionumber from biblio_metadata where metadata REGEXP
'[^\\x{0009}\\x{000A}\\x{000D}\\x{0020}-\\x{D7FF}\\x{E000}-\\x{FFFD}\\x{1000
0}-\\x{10}]+';

 

Newer versions of Koha strip invalid character from the XML so that you can
fix your records. I figure this report is very valuable when coupled with
that functionality. In fact, I just advised a library today to use them
together to fix up some bad data in their catalogue.

 

--

 

On a related note, I've noticed that you can have a record with good bib XML
but invalid item XML, and you won't notice until your record fails to be
indexed. So I'm planning on writing a report for that too. 

 

I'm thinking it might be good to add these reports to core Koha, so that
people can find and fix their own metadata problems. What do people think?

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Update to Koha 23.11.04 fails

[David Cook via Koha-devel]

Hi Michael,

I've noticed that a bunch of the database changes for the additional_contents 
table can half-complete and not fatally fail in very ungraceful ways. 

In this case, I'd say an earlier update probably failed to rename "idnew" to 
"id".

Hopefully this was a test upgrade, because otherwise you're going to have pain 
trying to fix it...

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
Michael Kuhn via Koha-devel
Sent: Friday, 12 April 2024 7:14 AM
To: Koha-devel 
Subject: [Koha-devel] Update to Koha 23.11.04 fails

Hi

Today I tried to update a Koha 21.11.10 database to Koha 23.11.04 but the 
update process failed at the following point:

Upgrade to 23.06.00.068  [22:55:59]: Bug 23798 - Convert OpacMaintenanceNotice 
system preference to additional contents ERROR - {UNKNOWN}: DBI Exception: 
DBD::mysql::db selectrow_array failed: 
Unknown column 'id' in 'field list' at
/usr/share/koha/lib/C4/Installer.pm line 741

I couldn't find anything regarding this error in Bugzilla, but maybe it is 
related to
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23798#c12
which shows a similar error regarding an "Unknown column 'title'".

How should I proceed?

Best wishes: Michael
--
Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin 
Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 
· E m...@adminkuhn.ch · W www.adminkuhn.ch 
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] XZ Utils backdoor for Linux

[David Cook via Koha-devel]

Hi all,

 

This isn't related to Koha per se, but I thought I'd share it for anyone who
hasn't seen it: https://en.wikipedia.org/wiki/XZ_Utils_backdoor

 

All the more reason to have things like Kohacon so that we actually know
each other.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Elasticsearch sans Zebra

[David Cook via Koha-devel]

Can you be more specific? If there’s a problem, a ticket should be raised.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
Philippe Blouin via Koha-devel
Sent: Wednesday, 27 March 2024 11:30 PM
To: Fridolin SOMERS ; 
koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Elasticsearch sans Zebra

 

Well, not really.  Hammat asked about limitations on Feb 28 on this very list, 
because attributes are not well handled.


 <https://inlibro.com/> 

Philippe Blouin 
Directeur de la technologie 

T833-INLIBRO (465-4276), poste 230
C   <mailto:philippe.blo...@inlibro.com> philippe.blo...@inlibro.com 

 <https://inLibro.com> www.inLibro.com 

On 2024-03-27 06:21, Fridolin SOMERS via Koha-devel wrote:

Ah nope, z39.50 can be handled by misc/z3950_responder.pl 
https://wiki.koha-community.org/wiki/Setting_up_the_Z39.50_and_SRU_Server 

Le 25/03/2024 à 13:59, Philippe Blouin a écrit : 



Hi! 

We have a scenario (albeit an exception) where both are used, due to the 
limitation of ES's z3950. 

So we fill zebraqueue uselessly in 199 Koha, but really really need it in 1. 

Logo inLibro  <https://inLibro.com> <https://inLibro.com> Philippe Blouin 
Directeur de la technologie 

T 833-INLIBRO (465-4276)   , poste 230 
C philippe.blo...@inlibro.com <mailto:philippe.blo...@inlibro.com>  

www.inLibro.com <http://www.inLibro.com>   <https://inLibro.com> 
<https://inLibro.com> 

On 2024-03-25 08:54, Fridolin SOMERS via Koha-devel wrote: 



Hi, 

We have patch since a few years for that because filling zebraqueue is useless 
and can't be removed with cleanup_database.pl : 
https://git.biblibre.com/biblibre/kohac/commit/50ae1a44d200c6fa46940107d15bef6d66e55383
 

It would be nice to have a more official way ;) 

Best regards 

Le 25/03/2024 à 12:29, David Schmidt via Koha-devel a écrit : 



Hi David 

the majority of our Koha installations use Elastiscearch and we have a process 
that deactivates zebra but we would love to see the change you proposed. 

I vaguely remember a bug where not having zebra running caused a problem (with 
background jobs i think) even if ES is in charge of indexing. 

cheers 
david (HKS3/www.koha-support.eu) 

On Mon, 25 Mar 2024, at 2:22 AM, David Cook via Koha-devel wrote: 




Hi all, 


I have started using Elasticsearch more with Koha, and we’re at a point of 
wanting to use Elasticsearch without Zebra for Koha. 

I came across “Bug 21820 - Zebraqueue should not be added to when only 
Elasticsearch is used” and it seems like other people are interested in this 
too. 


I imagine it’s just a case of getting a patch written? Overall, it’s an idea 
that people support? 


David Cook 

Senior Software Engineer 

Prosentient Systems 

Suite 7.03 

6a Glen St 

Milsons Point NSW 2061 

Australia 


Office: 02 9212 0899 

Online: 02 8005 0595 


___ 
Koha-devel mailing list 
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org>   
<mailto:Koha-devel@lists.koha-community.org> 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel  
<https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel> 
<https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel>
website : https://www.koha-community.org/  <https://www.koha-community.org/> 
<https://www.koha-community.org/> 
git : https://git.koha-community.org/  <https://git.koha-community.org/> 
<https://git.koha-community.org/> 
bugs : https://bugs.koha-community.org/  <https://bugs.koha-community.org/> 
<https://bugs.koha-community.org/> 



___ 
Koha-devel mailing list 
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org>  
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel 
website : https://www.koha-community.org/ 
git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/ 

 

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Elasticsearch indexing does not handle exceptions well

[David Cook via Koha-devel]

Hi all,

 

I'm re-indexing Elasticsearch and I notice occasionally Elasticsearch will
throw an exception when committing (e.g. Elasticsearch exception thrown:
Request), and Koha just ignores it.

 

That means that whole commit gets missed, so a default of 5000 records don't
get indexed. 

 

You can probably find those inconsistencies using
/usr/share/koha/bin/maintenance/compare_es_to_db.pl but that doesn't provide
you a method for indexing those missed records either.

 

Am I missing something obvious here? 

 

I'll need to look at increasing the verbosity to get more of the error
message, and I'll play around with some settings to make it so that the
Elasticsearch server doesn't thrown an error, but this seems weird to me. 

 

Perhaps we should have an automatic re-try, or at least a record of what
records failed to be committed, so that they can be manually retried via a
different mechanism?

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Elasticsearch sans Zebra

[David Cook via Koha-devel]

Hi all,

 

I have started using Elasticsearch more with Koha, and we're at a point of
wanting to use Elasticsearch without Zebra for Koha. 

I came across "Bug 21820 - Zebraqueue should not be added to when only
Elasticsearch is used" and it seems like other people are interested in this
too.

 

I imagine it's just a case of getting a patch written? Overall, it's an idea
that people support?

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Random XSLT knowledge

[David Cook via Koha-devel]

Hi Marcel,

 

Sure!

 

Before Koha started using "http://exslt.org/strings; to encode URIs, I used
to use register_function to add my own function to do that.

 

1.  vi C4/XSLT.pm
2.  Add a line near the top like:

a.
XML::LibXSLT->register_function("http://.prosentient.com.au/xsltperl;,
"uri-escape",\::Prosentient::Biblio::Urls::xslt_uri_escape);

3.  Write a function like this:

sub xslt_uri_escape {

my ($uri) = @_;

#$uri should always be a XML::LibXML::Nodelist, even if that Nodelist
just contains 1 URI

if (ref $uri eq "XML::LibXML::NodeList"){

$uri->foreach(

sub {

my ( $node ) = @_;

#If the node has child (text) nodes

if ($node->hasChildNodes()){

my @childNodes = $node->childNodes();

if (@childNodes){

foreach my $childNode (@childNodes){

if (my $textdata = $childNode->data){

#Trim whitespace

$textdata =~ s/^\s+|\s+$//g;

my $encoded_url =
URI::Encode::uri_encode($textdata, { encode_reserved => 0 });

if ($encoded_url){

#Replace the existing URI data with the
encoded URI data

$childNode->setData($encoded_url);

}

}

}

}

}

}

);

}

return $uri;

}

 

4.  Add the namespace at the top of your XSLT:

a.  xmlns:pro="http://.prosentient.com.au/xsltperl;

5.  Call the function on some XML:

a.  

 

--

 

More recently, I've done more complicated things like looking up item urls
for that biblio record, deduplicating them against the 856$u, and adding
them to the online access in the search results. I continue to take into
account OpacHiddenItems and its associated preferences. Since it's operating
at the Perl level, I'm able to check the C4::Context->userenv for the patron
details as well.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Marcel de Rooy  
Sent: Friday, 8 March 2024 6:13 PM
To: 'Koha Devel' ; David Cook

Subject: Re: [Koha-devel] Random XSLT knowledge

 

Hi David,

Thanks for sharing.

Would you have an example of how you use register_function with Koha
functions to share?

 

Marcel

 

  _  

Van: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > namens David Cook via
Koha-devel mailto:koha-devel@lists.koha-community.org> >
Verzonden: vrijdag 8 maart 2024 03:02
Aan: 'Koha Devel' mailto:koha-devel@lists.koha-community.org> >
Onderwerp: [Koha-devel] Random XSLT knowledge 

 

Hi all,

 

I've been working on performance issues, and in the process I got looking at
XSLTs.

 

I just wanted to share that it's possible to pass strings to the XSLT's
transform() method:

 

-return $engine->transform($xmlrecord, $xslfilename ); #file or URL

+return $engine->transform({

+xml => $xmlrecord,

+file => $xslfilename,

+parameters => {

+test => "'$test_str'",

+},

+}); #file or URL

 

It's somewhat limited in that you can only pass strings and I think there's
a small limit on the number of parameters you can pass (not sure if it's 32
or 255), but I thought it was interesting. It would allow you to pass some
data that you have at hand on a per-XML record basis without having to
mangle the XML record (like we do with items and system preferences).

 

In the end, I didn't end up using it though. Instead, I use
XML::LibXSLT->register_function() to provide access to Koha functions from
the XSLT, and in this case that meets my needs.

 

Anyway, back to it..

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] SMTP servers requiring XOAUTH2

[David Cook via Koha-devel]

Hey folks,

 

Is anyone using Koha with SMTP servers requiring XOAUTH2?

 

It doesn't look like the Perl libraries used by Koha are anywhere near ready
for it, but it looks like Postfix has an experimental plugin available:
https://github.com/tarickb/sasl-xoauth2 or
https://packages.debian.org/experimental/sasl-xoauth2

 

I think both Gmail and Microsoft require it these days.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Random XSLT knowledge

[David Cook via Koha-devel]

Hi all,

 

I've been working on performance issues, and in the process I got looking at
XSLTs.

 

I just wanted to share that it's possible to pass strings to the XSLT's
transform() method:

 

-return $engine->transform($xmlrecord, $xslfilename ); #file or URL

+return $engine->transform({

+xml => $xmlrecord,

+file => $xslfilename,

+parameters => {

+test => "'$test_str'",

+},

+}); #file or URL

 

It's somewhat limited in that you can only pass strings and I think there's
a small limit on the number of parameters you can pass (not sure if it's 32
or 255), but I thought it was interesting. It would allow you to pass some
data that you have at hand on a per-XML record basis without having to
mangle the XML record (like we do with items and system preferences).

 

In the end, I didn't end up using it though. Instead, I use
XML::LibXSLT->register_function() to provide access to Koha functions from
the XSLT, and in this case that meets my needs.

 

Anyway, back to it..

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] PatternFly for design?

[David Cook via Koha-devel]

Hi all,

 

Has anyone used PatternFly for designing? https://www.patternfly.org/

 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Future of mod_security

[David Cook via Koha-devel]

Hi all,

 

mod_security has had a troubled existence the past few years, but it looks
like the company behind it has transferred it over to OWASP:
https://owasp.org/blog/2024/01/09/ModSecurity.html 

 

Hopefully this means mod_security starts getting updated more. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Override sysprefs in Apache config

[David Cook via Koha-devel]

Martin, you legend. That’s what I was looking for and missing I think. Thanks 
for making me feel less crazy : ). 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Renvoize, Martin  
Sent: Tuesday, 27 February 2024 9:51 AM
To: David Cook 
Cc: Magnus Enger ; Koha Devel 

Subject: Re: [Koha-devel] Override sysprefs in Apache config

 

That might be what the _NAMES override is for.. I believe you pass it a list of 
the Prefs your overriding and it'll then display a message in the sysprefs if 
the staff client stating their likely override at the virtual host level.

 

On Sun, 18 Feb 2024, 10:21 pm David Cook via Koha-devel, 
mailto:koha-devel@lists.koha-community.org> > wrote:

Hi Magnus,

Yeah, I was thinking it wouldn't make sense to affect the staff interface UI, 
since you might be overriding 5 different OPAC virtual hosts. But I had a vague 
memory that there was some sort of signifier that the syspref was being 
overridden. Maybe a false memory in the end. 

In the end, I did use X-Koha-SetEnv to set a syspref for one of two OPAC 
virtual hosts, so that worked well enough. 

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of Magnus 
Enger via Koha-devel
Sent: Friday, 16 February 2024 6:26 PM
To: koha-devel@lists.koha-community.org 
<mailto:koha-devel@lists.koha-community.org> 
Subject: Re: [Koha-devel] Override sysprefs in Apache config

Hi!

Den 16.02.2024 06:56, skrev David Cook via Koha-devel:
> I think it might be partially broken.
> 
> As per https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16520
> <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16520> I 
> added the following to the OPAC and the staff interface:
> 
> RequestHeader add X-Koha-SetEnv "OVERRIDE_SYSPREF_LibraryName Potato\, 
> Potato"
> 
> It did indeed load into the OPAC.
> 
> In the past, I think it used to fill in and lock the syspref in the 
> staff interface as well…  but maybe not anymore. Maybe because it 
> would just affect the staff interface, and yet sometimes maybe you 
> only want to affect the staff interface… hmm…

We have used this a fair bit to set up alternative/infividual OPACs for 
individual libraries, but I have never seen a setting like that fill in or 
otherwise affect the actual syspref. And how would that work when you have 5 
different VirtualHost configs all setting their own 
OVERRIDE_SYSPREF_LibraryName?

I can confirm that settings like these work for us:

RequestHeader add X-Koha-SetEnv "OVERRIDE_SYSPREF_LibraryName Grundskolan"
RequestHeader add X-Koha-SetEnv "OPAC_SEARCH_LIMIT branch:multibranchlimit-23"
RequestHeader add X-Koha-SetEnv "OPAC_LIMIT_OVERRIDE 1"

Best regards,
Magnus
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] DataTables 2.0.0

[David Cook via Koha-devel]

Hi all,

 

I do not intend to draw your attention to this in any meaningful way. I just
wanted to point out that DataTables 2.0.0 was released on February 15th 2024
(https://cdn.datatables.net/2.0.0/). Just squirrel away that information in
the back of your brain for later.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Override sysprefs in Apache config

[David Cook via Koha-devel]

Hi Magnus,

Yeah, I was thinking it wouldn't make sense to affect the staff interface UI, 
since you might be overriding 5 different OPAC virtual hosts. But I had a vague 
memory that there was some sort of signifier that the syspref was being 
overridden. Maybe a false memory in the end. 

In the end, I did use X-Koha-SetEnv to set a syspref for one of two OPAC 
virtual hosts, so that worked well enough. 

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Koha-devel  On Behalf Of 
Magnus Enger via Koha-devel
Sent: Friday, 16 February 2024 6:26 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Override sysprefs in Apache config

Hi!

Den 16.02.2024 06:56, skrev David Cook via Koha-devel:
> I think it might be partially broken.
> 
> As per https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16520
> <https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16520> I 
> added the following to the OPAC and the staff interface:
> 
> RequestHeader add X-Koha-SetEnv "OVERRIDE_SYSPREF_LibraryName Potato\, 
> Potato"
> 
> It did indeed load into the OPAC.
> 
> In the past, I think it used to fill in and lock the syspref in the 
> staff interface as well…  but maybe not anymore. Maybe because it 
> would just affect the staff interface, and yet sometimes maybe you 
> only want to affect the staff interface… hmm…

We have used this a fair bit to set up alternative/infividual OPACs for 
individual libraries, but I have never seen a setting like that fill in or 
otherwise affect the actual syspref. And how would that work when you have 5 
different VirtualHost configs all setting their own 
OVERRIDE_SYSPREF_LibraryName?

I can confirm that settings like these work for us:

RequestHeader add X-Koha-SetEnv "OVERRIDE_SYSPREF_LibraryName Grundskolan"
RequestHeader add X-Koha-SetEnv "OPAC_SEARCH_LIMIT branch:multibranchlimit-23"
RequestHeader add X-Koha-SetEnv "OPAC_LIMIT_OVERRIDE 1"

Best regards,
Magnus
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/ 
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Override sysprefs in Apache config

[David Cook via Koha-devel]

I think it might be partially broken.

 

As per https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16520 I
added the following to the OPAC and the staff interface:

 

RequestHeader add X-Koha-SetEnv "OVERRIDE_SYSPREF_LibraryName Potato\,
Potato"

 

It did indeed load into the OPAC. 

 

In the past, I think it used to fill in and lock the syspref in the staff
interface as well.  but maybe not anymore. Maybe because it would just
affect the staff interface, and yet sometimes maybe you only want to affect
the staff interface. hmm.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: David Cook  
Sent: Friday, 16 February 2024 4:52 PM
To: 'koha-devel' 
Subject: Override sysprefs in Apache config

 

Hi all,

 

I've read this wiki page and tried this many times, and never ever gotten it
to work. Is there something missing from these instructions?

 

https://wiki.koha-community.org/wiki/Override_sysprefs_in_Apache_config

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Override sysprefs in Apache config

[David Cook via Koha-devel]

Hi all,

 

I've read this wiki page and tried this many times, and never ever gotten it
to work. Is there something missing from these instructions?

 

https://wiki.koha-community.org/wiki/Override_sysprefs_in_Apache_config

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Session corruption in koha-testing-docker master

[David Cook via Koha-devel]

Hi all,

 

The last two days I've noticed that my session is getting corrupted while
I'm working on master in koha-testing-docker. I'm not touching the session
myself, but seemingly randomly the session gets all kinds of DIBC data
stuffed into it, and Koha seems to be parse it and Koha throws a fatal
error, which can only be resolved by deleting the problematic session from
the table.

 

Just curious if anyone else is seeing this. I really don't think it's
related to anything I'm working on, but I also can't reliably reproduce it. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Growing size of koha/koha-testing:master

[David Cook via Koha-devel]

That's interesting. I think that adds up to about 2.5GB. I wonder where the 
other 2GB comes from. 

I have been wondering if we should be doing some of these steps in a "builder" 
image and copying across files in case some of the bloat is in the layering. 

Something to play with perhaps. It's not a top priority at the moment, but I 
thought I'd raise it.

A few large koha-testing-docker images start adding up in terms of SDD space. I 
upgraded workstations not that long ago, and now I'm thinking maybe I need to 
do it again just for the storage...

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-Original Message-
From: Victor Grousset/tuxayo  
Sent: Wednesday, 7 February 2024 1:33 PM
To: David Cook 
Cc: koha-devel 
Subject: Re: [Koha-devel] Growing size of koha/koha-testing:master

On 24-02-04 23:30, David Cook via Koha-devel wrote:
> Do we know what’s contributing to this?

After running `ncdu /` from inside KTD Debian 10 (the smallest image it
seems)

- 661 M /kohadevbox/Cypress
- 347 M /kohadevbox/node_modules
- 95 M /kohadevbox/misc4dev
- 224 M /usr/lib/locale (does locales are needed for Koha? because all are 
generated)
- 410 M /usr/lib/x86_64-linux-gnu , various lib, maybe libLLVM and guile are 
not needed
- 93 M /usr/lib/gcc
- 270 M /usr/share/locale & /usr/share/doc not sure that prunable depending on 
how packages are made in Debian. Maybe part can be pruned via a dirty rm -rf 
like it seems to be done for the apt repo metadata in dockerfiles. Seen that a 
few times.
- 400 M /usr/local/share/.cache/yarn likely everything is needed to be able to 
rebuild CSS & vue without having to redownload. Which is done on start so not 
only needed when working with UI. Ok maybe some stuff is not needed? Same 
questions with /kohadevbox/node_modules

npm(node) and yarn have redundancy in their cache >_< They share all of their 
top 10 packages (some with same size, other with quite different) ^^"
I guess one is used for SCSS compilation and another one for vue and cypress. 
But they still have in common the packages of the 3, hmm


Cheers,

--
Victor Grousset/tuxayo

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Growing size of koha/koha-testing:master

[David Cook via Koha-devel]

Hi all,

 

I've noticed recently that koha/koha-testing:master has been steadily
growing. At this point, it's about 4.37GB in size. For 21.11 I think it used
to be about 2GB and 22.11 was 3.5GB. 

 

Do we know what's contributing to this? 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] January point releases

[David Cook via Koha-devel]

Hi all,

 

I think the January point releases have been done. Just curious when we'll
see those flow through to the main repository?

 

Thanks,

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Dockerfile for koha/elasticsearch-icu:7.x image

[David Cook via Koha-devel]

Hi Tomas,

 

Thanks for that. I had looked at that URL but I couldn’t find a Dockerfile 
there. Just LICENSE and README.md files. But I see now if I had selected a 
particular branch I would’ve been able to find the Dockerfile.

 

Thanks again,

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Tomas Cohen Arazi  
Sent: Monday, 29 January 2024 10:16 PM
To: David Cook 
Cc: koha-devel 
Subject: Re: [Koha-devel] Dockerfile for koha/elasticsearch-icu:7.x image

 

You can find the Dokerfiles here

 

https://gitlab.com/koha-community/docker/koha-elasticsearch-icu

 

and yes, they were just installing the ICU plugin

 

El lun, 29 ene 2024 1:47, David Cook via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > escribió:

Hi all,

 

Is there a published version of the Dockerfile for the 
koha/elasticsearch-icu:7.x image?

 

Is it just running “RUN /usr/share/elasticsearch/bin/plugin install --batch 
analysis-icu” or does it make other changes to the standard image as well?

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Dockerfile for koha/elasticsearch-icu:7.x image

[David Cook via Koha-devel]

Hi all,

 

Is there a published version of the Dockerfile for the
koha/elasticsearch-icu:7.x image?

 

Is it just running "RUN /usr/share/elasticsearch/bin/plugin install --batch
analysis-icu" or does it make other changes to the standard image as well?

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] 22.11.12 having package dependency problems?

[David Cook via Koha-devel]

I help out a few libraries running 22.11 straight from the community, and
two libraries lately have had Koha go down during automatic upgrades to
22.11.12-1. 

 

koha-common mostly installs but fails to complete due to unmet dependencies:

 

apt install koha-common -s

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

koha-common is already the newest version (22.11.12-1).

You might want to run 'apt --fix-broken install' to correct these.

The following packages have unmet dependencies:

koha-common : Depends: koha-l10n but it is not going to be installed

E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or
specify a solution).

 

You can remedy this by running apt-get install koha-l10n, but I'm curious
why this is happening. I imagine it must be happening to more libraries
around the world as well.

 

Maybe when the koha-common upgrade ran the koha-l10n package wasn't
available? 

 

Has anyone else bumped into this?

 

I didn't realize that we were backporting koha-l10n to other releases than
23.11, so it hasn't been on my radar until now. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] "Online access" vs "Online resources"

[David Cook via Koha-devel]

Hi all,

 

Detail pages say "Online resources" but search results say "Online access".
I don't know if I've never noticed before or what but I had a librarian
raise it today.

 

I think it makes sense to harmonise so that they're both the same, but how
do we choose which one? 

 

I've raised a ticket for this:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35410

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Using Elasticsearch in koha-testing-docker

[David Cook via Koha-devel]

Thanks, Jonathan. 

 

I’ve done some playing around and I think I’m learning a lot about Koha’s 
integration.

 

I’m a bit surprised that there isn’t an “any”, “all”, or “anywhere” index. I 
notice “ElasticsearchMARCFormat” can make the full MARC record searchable by 
storing the MARC record as a JSON object, but it seems like that’s not 
recommended. I don’t know why we wouldn’t just index the full record like we do 
with Zebra. It would make for a bigger data store, but it could be done fairly 
simply. 

 

Who all is actively contributing to the Elasticsearch work? Might be useful to 
get in on the conversation…

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Jonathan Druart  
Sent: Tuesday, 7 November 2023 8:09 PM
To: David Cook 
Cc: koha-devel 
Subject: Re: [Koha-devel] Using Elasticsearch in koha-testing-docker

 

About ktd you need to include one of the ES/OS compose files (docker-compose.es 
<http://docker-compose.es> * docker-compose.os*), there are bin/ktd switches.

You also need KOHA_ELASTICSEARCH set to "yes" in the .env file.

 

Le mar. 7 nov. 2023 à 03:55, David Cook via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > a écrit :

Hi all,

 

I’m trying to use Elasticsearch in koha-testing-docker, and I would appreciate 
some assistance.

 

Here’s the steps I’m taking:

1.  Launch koha-testing-docker
2.  Update Koha’s sysprefs

a.  Change SearchEngine from “Zebra” to “Elasticsearch” at 
http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search 
<http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search=SearchEngine>
 =SearchEngine

3.  (Re)build Elasticsearch indexes:

a.  koha-elasticsearch --rebuild kohadev

4.  Start Elasticsearch indexer:

a.  koha-es-indexer --start kohadev

 

Is there anything else to do? In a prod install, do I just need to update the 
“elasticsearch” config in koha-conf.xml? 

 

Do I leave all the Zebra things running or can I turn those off? I recall 
reading somewhere that zebra_queue might fill up if I turn it off?

 

Thanks all!

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Using Elasticsearch in koha-testing-docker

[David Cook via Koha-devel]

Hi all,

 

I'm trying to use Elasticsearch in koha-testing-docker, and I would
appreciate some assistance.

 

Here's the steps I'm taking:

1.  Launch koha-testing-docker
2.  Update Koha's sysprefs

a.  Change SearchEngine from "Zebra" to "Elasticsearch" at
http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search
 =SearchEngine

3.  (Re)build Elasticsearch indexes:

a.  koha-elasticsearch --rebuild kohadev

4.  Start Elasticsearch indexer:

a.  koha-es-indexer --start kohadev

 

Is there anything else to do? In a prod install, do I just need to update
the "elasticsearch" config in koha-conf.xml? 

 

Do I leave all the Zebra things running or can I turn those off? I recall
reading somewhere that zebra_queue might fill up if I turn it off?

 

Thanks all!

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Last month before the release

[David Cook via Koha-devel]

Also, regarding 
https://jenkins.koha-community.org/job/Koha_Master/lastCompletedBuild/testReport/
 :

 

I’ve got a fix for “t_db_dependent_Koha_Patrons_Import_t.No warning raised by 
import_patrons” at 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35264

 

It looks like Katrin fixed 
“xt_author_valid_templates_t./kohadevbox/koha/koha-tmpl/intranet-tmpl/prog/en/modules/circ/circulation_batch_checkouts.tt”

 

And I can’t reproduce the problem on master for 
“t_db_dependent_selenium_patrons_search_t.Search patrons”, so I’m guessing 
someone has already fixed that one?

 

Hopefully the next run will be fail free :D

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: David Cook  
Sent: Monday, 6 November 2023 10:32 AM
To: 'Jonathan Druart' 
Cc: 'Tomas Cohen Arazi' ; 'koha-devel' 

Subject: RE: [Koha-devel] Last month before the release

 

Thanks, Jonathan. Yeah, I think having a live chat sounds like a plan. 
Communication is good.

 

Timezones, especially mine, make it tough, but being able to point at 
documentation for policies and procedures is helpful. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Jonathan Druart mailto:jonathan.dru...@bugs.koha-community.org> > 
Sent: Friday, 3 November 2023 7:10 PM
To: David Cook mailto:dc...@prosentient.com.au> >
Cc: Tomas Cohen Arazi mailto:tomasco...@gmail.com> >; 
koha-devel mailto:koha-devel@lists.koha-community.org> >
Subject: Re: [Koha-devel] Last month before the release

 

David, if you don't know Jenkins and you are part of the QA team, I guess you 
are not alone to be lost with our CI structure and it means we have a big 
problem. Either motivation, or tools or communication.
I am not answering you now but maybe we need to document a bit better how it 
works for the next release, responsibilize people (asking them to fix what they 
broke instead of fixing for them) and finally maybe have a live chat beginning 
of the cycle with RMaints and QA team to (re) explain how it works.

Le vendredi 3 novembre 2023, David Cook via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > a écrit :
> It’s not clear to me though how you look at Jenkins runs. Is it going to 
> https://dashboard.koha-community.org/ and then clicking on one of the badges 
> to go to Jenkins like https://jenkins.koha-community.org/job/Koha_Master_U22/ 
> and then clicking on “Latest Test Result”?
>
>  
>
> David Cook
>
> Senior Software Engineer
>
> Prosentient Systems
>
> Suite 7.03
>
> 6a Glen St
>
> Milsons Point NSW 2061
>
> Australia
>
>  
>
> Office: 02 9212 0899
>
> Online: 02 8005 0595
>
>  
>
> From: Tomas Cohen Arazi mailto:tomasco...@gmail.com> >
> Sent: Friday, 3 November 2023 11:25 AM
> To: David Cook mailto:dc...@prosentient.com.au> >
> Cc: koha-devel  <mailto:koha-devel@lists.koha-community.org> >
> Subject: Re: [Koha-devel] Last month before the release
>
>  
>
> The suggested sequence is:
>
>  
>
> 1. You notice your stuff for pushed, yay!
>
> 2. You look at Jenkins runs for the relevant branches (e.g. Koha_Master_* 
> tasks)
>
> 3. If some test gets broken that day, you try too see if it was you. Beware 
> sometimes it is not obvious.
>
>  
>
> Today there was a failure in the holds API tests, because of a bug that 
> didn't touch the API, but made the code stricter (?) so the loose tests we 
> had written had to be tweaked so they didn't trigger a silly error.
>
>  
>
> So the devs did right, didn't introduce a bug, but made the tests fail. But 
> it is all of us, specially those who got patches pushed, who should be 
> looking after those failures. Team work
>
>  
>
> Thanks!
>
>  
>
> El jue, 2 nov 2023 21:15, David Cook  <mailto:dc...@prosentient.com.au> > escribió:
>
> What’s the best way to do this?
>
>  
>
> I think Jenkins might send emails, but I think my mail server blocked it ages 
> ago as spam.
>
>  
>
> David Cook
>
> Senior Software Engineer
>
> Prosentient Systems
>
> Suite 7.03
>
> 6a Glen St
>
> Milsons Point NSW 2061
>
> Australia
>
>  
>
> Office: 02 9212 0899
>
> Online: 02 8005 0595
>
>  
>
> From: Koha-devel  <mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of Tomas 
> Cohen Arazi via Koha-devel
> Sent: Friday, 3 November 2023 12:31 AM
> To: koha-devel  <mailto:koha-devel@lists.koha-community.org> >
> Subject: [Koha-devel] Last month before the release
>
>  
>
&g

Re: [Koha-devel] Last month before the release

[David Cook via Koha-devel]

Thanks, Jonathan. Yeah, I think having a live chat sounds like a plan. 
Communication is good.

 

Timezones, especially mine, make it tough, but being able to point at 
documentation for policies and procedures is helpful. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Jonathan Druart  
Sent: Friday, 3 November 2023 7:10 PM
To: David Cook 
Cc: Tomas Cohen Arazi ; koha-devel 

Subject: Re: [Koha-devel] Last month before the release

 

David, if you don't know Jenkins and you are part of the QA team, I guess you 
are not alone to be lost with our CI structure and it means we have a big 
problem. Either motivation, or tools or communication.
I am not answering you now but maybe we need to document a bit better how it 
works for the next release, responsibilize people (asking them to fix what they 
broke instead of fixing for them) and finally maybe have a live chat beginning 
of the cycle with RMaints and QA team to (re) explain how it works.

Le vendredi 3 novembre 2023, David Cook via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > a écrit :
> It’s not clear to me though how you look at Jenkins runs. Is it going to 
> https://dashboard.koha-community.org/ and then clicking on one of the badges 
> to go to Jenkins like https://jenkins.koha-community.org/job/Koha_Master_U22/ 
> and then clicking on “Latest Test Result”?
>
>  
>
> David Cook
>
> Senior Software Engineer
>
> Prosentient Systems
>
> Suite 7.03
>
> 6a Glen St
>
> Milsons Point NSW 2061
>
> Australia
>
>  
>
> Office: 02 9212 0899
>
> Online: 02 8005 0595
>
>  
>
> From: Tomas Cohen Arazi mailto:tomasco...@gmail.com> >
> Sent: Friday, 3 November 2023 11:25 AM
> To: David Cook mailto:dc...@prosentient.com.au> >
> Cc: koha-devel  <mailto:koha-devel@lists.koha-community.org> >
> Subject: Re: [Koha-devel] Last month before the release
>
>  
>
> The suggested sequence is:
>
>  
>
> 1. You notice your stuff for pushed, yay!
>
> 2. You look at Jenkins runs for the relevant branches (e.g. Koha_Master_* 
> tasks)
>
> 3. If some test gets broken that day, you try too see if it was you. Beware 
> sometimes it is not obvious.
>
>  
>
> Today there was a failure in the holds API tests, because of a bug that 
> didn't touch the API, but made the code stricter (?) so the loose tests we 
> had written had to be tweaked so they didn't trigger a silly error.
>
>  
>
> So the devs did right, didn't introduce a bug, but made the tests fail. But 
> it is all of us, specially those who got patches pushed, who should be 
> looking after those failures. Team work
>
>  
>
> Thanks!
>
>  
>
> El jue, 2 nov 2023 21:15, David Cook  <mailto:dc...@prosentient.com.au> > escribió:
>
> What’s the best way to do this?
>
>  
>
> I think Jenkins might send emails, but I think my mail server blocked it ages 
> ago as spam.
>
>  
>
> David Cook
>
> Senior Software Engineer
>
> Prosentient Systems
>
> Suite 7.03
>
> 6a Glen St
>
> Milsons Point NSW 2061
>
> Australia
>
>  
>
> Office: 02 9212 0899
>
> Online: 02 8005 0595
>
>  
>
> From: Koha-devel  <mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of Tomas 
> Cohen Arazi via Koha-devel
> Sent: Friday, 3 November 2023 12:31 AM
> To: koha-devel  <mailto:koha-devel@lists.koha-community.org> >
> Subject: [Koha-devel] Last month before the release
>
>  
>
> Hi all.
>
>  
>
> We are almost done with the release, a couple medium/big things to push 
> today/tomorrow and then bug fixing.
>
>  
>
> I wanted to ask y'all to please view the Jenkins tasks status anytime after 
> your stuff has been pushed. I know I said this many times, but some devs 
> clearly break some tests that are trivially fixable and the team needs you to 
> take care of your breakages.
>
>  
>
> I'm doing my best to track things and fix, but you should all be worried 
> about that.
>
>  
>
> --
>
> Tomás Cohen Arazi
>
> Theke Solutions (https://theke.io)
> ✆ +54 9351 3513384
> GPG: B2F3C15F 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Last month before the release

[David Cook via Koha-devel]

It’s not clear to me though how you look at Jenkins runs. Is it going to 
https://dashboard.koha-community.org/ and then clicking on one of the badges to 
go to Jenkins like https://jenkins.koha-community.org/job/Koha_Master_U22/ and 
then clicking on “Latest Test Result”? 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Tomas Cohen Arazi  
Sent: Friday, 3 November 2023 11:25 AM
To: David Cook 
Cc: koha-devel 
Subject: Re: [Koha-devel] Last month before the release

 

The suggested sequence is:

 

1. You notice your stuff for pushed, yay!

2. You look at Jenkins runs for the relevant branches (e.g. Koha_Master_* tasks)

3. If some test gets broken that day, you try too see if it was you. Beware 
sometimes it is not obvious.

 

Today there was a failure in the holds API tests, because of a bug that didn't 
touch the API, but made the code stricter (?) so the loose tests we had written 
had to be tweaked so they didn't trigger a silly error.

 

So the devs did right, didn't introduce a bug, but made the tests fail. But it 
is all of us, specially those who got patches pushed, who should be looking 
after those failures. Team work

 

Thanks!

 

El jue, 2 nov 2023 21:15, David Cook mailto:dc...@prosentient.com.au> > escribió:

What’s the best way to do this?

 

I think Jenkins might send emails, but I think my mail server blocked it ages 
ago as spam. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of Tomas Cohen 
Arazi via Koha-devel
Sent: Friday, 3 November 2023 12:31 AM
To: koha-devel mailto:koha-devel@lists.koha-community.org> >
Subject: [Koha-devel] Last month before the release

 

Hi all.

 

We are almost done with the release, a couple medium/big things to push 
today/tomorrow and then bug fixing.

 

I wanted to ask y'all to please view the Jenkins tasks status anytime after 
your stuff has been pushed. I know I said this many times, but some devs 
clearly break some tests that are trivially fixable and the team needs you to 
take care of your breakages.

 

I'm doing my best to track things and fix, but you should all be worried about 
that.


 

-- 

Tomás Cohen Arazi

Theke Solutions (https://theke.io)
✆ +54 9351 3513384
GPG: B2F3C15F

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Last month before the release

[David Cook via Koha-devel]

What’s the best way to do this?

 

I think Jenkins might send emails, but I think my mail server blocked it ages 
ago as spam. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
Tomas Cohen Arazi via Koha-devel
Sent: Friday, 3 November 2023 12:31 AM
To: koha-devel 
Subject: [Koha-devel] Last month before the release

 

Hi all.

 

We are almost done with the release, a couple medium/big things to push 
today/tomorrow and then bug fixing.

 

I wanted to ask y'all to please view the Jenkins tasks status anytime after 
your stuff has been pushed. I know I said this many times, but some devs 
clearly break some tests that are trivially fixable and the team needs you to 
take care of your breakages.

 

I'm doing my best to track things and fix, but you should all be worried about 
that.


 

-- 

Tomás Cohen Arazi

Theke Solutions (https://theke.io)
✆ +54 9351 3513384
GPG: B2F3C15F

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Memory footprint of Koha processes

[David Cook via Koha-devel]

Hi Philippe,

 

I run quite a few Koha instances, so I feel your pain.

 

A couple months ago I opened 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34477 but I haven’t 
dedicated any time to work on it yet (and I have no plans to work on it anytime 
soon).

 

The ORM eats up a lot of memory, and if I recall correctly C4::Context also 
requires a fair bit of memory. I don’t think you’ll really be able to hack this 
one per se as there are quite a few changes that are needed. That said… I 
suppose you could always create an alternative background jobs worker and use 
that instead. If I recall correctly, I don’t think you use the RabbitMQ, so you 
could exclude lots of code. 

 

In theory Koha::Database->dbh() should provide a database handle with a smaller 
memory footprint, but I haven’t tested it. 

 

But yeah… I think you’d basically need to do a rewrite. 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
Philippe Blouin via Koha-devel
Sent: Tuesday, 24 October 2023 6:57 AM
To: koha-devel@lists.koha-community.org
Subject: [Koha-devel] Memory footprint of Koha processes

 

Howdy!

I'm trying to run a server with MANY koha instances (for educational purposes), 
but although the usage is very small, the resources required just make 
everything explode.

background_jobs_worker.pl for instance uses 170M per instance, and plack uses 
200M without anything being loaded.

Any suggestion how this could be hacked (locally) to allow for 200 koha 
instances on a machine without requiring 100G of memory ?  The object model 
seems at fault here, but I see no way to go around   
"use Koha::BackgroundJobs;" at the top of the script.

Thanks!

-- 

Philippe Blouin,
Directeur de la technologie

Tél.  : (833) 465-4276, poste 230
  philippe.blo...@inlibro.com 

inLibro | pour esprit libre |   www.inLibro.com 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Anyone else having issues attaching patches to Bugzilla?

[David Cook via Koha-devel]

Turns out it was this problem: 
https://gitlab.com/koha-community/git-bz/-/issues/8#note_1180787597

 

I’d put the wrong bug number in the title line of the commit and that was 
causing the problem. Yikes.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of 
David Cook via Koha-devel
Sent: Monday, 23 October 2023 1:47 PM
To: 'David Nind' 
Cc: 'Koha Devel' 
Subject: Re: [Koha-devel] Anyone else having issues attaching patches to 
Bugzilla?

 

Thanks for confirming, David!

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: David Nind mailto:da...@davidnind.com> > 
Sent: Monday, October 23, 2023 1:43 PM
To: David Cook mailto:dc...@prosentient.com.au> >
Cc: Koha Devel mailto:koha-devel@lists.koha-community.org> >
Subject: Re: [Koha-devel] Anyone else having issues attaching patches to 
Bugzilla?

 

Hi David.

 

It's working for me, just signed of a patch - git bz attach -e X HEAD

 

So it must be you! 8-)

 

David Nind

New Zealand

 

On Mon, 23 Oct 2023 at 15:25, David Cook via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > wrote:

Hi all,

 

Trying to attach a patch to Bugzilla this afternoon and I get the following 
error:

 

Cleaning up back to original state on error

Traceback (most recent call last):

  File "/usr/bin/git-bz", line 2840, in 

do_attach(*args)

  File "/usr/bin/git-bz", line 2168, in do_attach

add_url(bug, commits)

  File "/usr/bin/git-bz", line 1719, in add_url

add_url_to_head_commit(commit, bug)

  File "/usr/bin/git-bz", line 1682, in add_url_to_head_commit

git.commit(file="-", amend=True, _input=input)

  File "/usr/bin/git-bz", line 197, in f

o = git_run(command, *args, **kwargs)

  File "/usr/bin/git-bz", line 167, in git_run

output, error = process.communicate(input)

  File "/usr/lib/python3.9/subprocess.py", line 1134, in communicate

stdout, stderr = self._communicate(input, endtime, timeout)

  File "/usr/lib/python3.9/subprocess.py", line 1961, in _communicate

input_view = memoryview(self._input)

TypeError: memoryview: a bytes-like object is required, not 'str'

 

I figure it’s probably a “me problem” but just want to check…

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] I'm not getting your emails on bugs...

[David Cook via Koha-devel]

Hi all,

 

It looks like my "bugmail has been disabled" for the last 10 days on
Bugzilla, so if I haven't responded to a comment on Bugzilla, it's not
because I'm ignoring you. It's because I'm not getting the emails. (If an
administrator sees this, please re-enable my bugmail. Thanks!)

 

I've been trying to weed through the contents from
koha-b...@lists.koha-community.org
  which I still receive but
there's so many emails I'm likely to miss the vast majority. 

 

So if you need anything from me urgently, shoot me an email directly? 

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Anyone else having issues attaching patches to Bugzilla?

[David Cook via Koha-devel]

Thanks for confirming, David!

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: David Nind  
Sent: Monday, October 23, 2023 1:43 PM
To: David Cook 
Cc: Koha Devel 
Subject: Re: [Koha-devel] Anyone else having issues attaching patches to 
Bugzilla?

 

Hi David.

 

It's working for me, just signed of a patch - git bz attach -e X HEAD

 

So it must be you! 8-)

 

David Nind

New Zealand

 

On Mon, 23 Oct 2023 at 15:25, David Cook via Koha-devel 
mailto:koha-devel@lists.koha-community.org> > wrote:

Hi all,

 

Trying to attach a patch to Bugzilla this afternoon and I get the following 
error:

 

Cleaning up back to original state on error

Traceback (most recent call last):

  File "/usr/bin/git-bz", line 2840, in 

do_attach(*args)

  File "/usr/bin/git-bz", line 2168, in do_attach

add_url(bug, commits)

  File "/usr/bin/git-bz", line 1719, in add_url

add_url_to_head_commit(commit, bug)

  File "/usr/bin/git-bz", line 1682, in add_url_to_head_commit

git.commit(file="-", amend=True, _input=input)

  File "/usr/bin/git-bz", line 197, in f

o = git_run(command, *args, **kwargs)

  File "/usr/bin/git-bz", line 167, in git_run

output, error = process.communicate(input)

  File "/usr/lib/python3.9/subprocess.py", line 1134, in communicate

stdout, stderr = self._communicate(input, endtime, timeout)

  File "/usr/lib/python3.9/subprocess.py", line 1961, in _communicate

input_view = memoryview(self._input)

TypeError: memoryview: a bytes-like object is required, not 'str'

 

I figure it’s probably a “me problem” but just want to check…

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org 
<mailto:Koha-devel@lists.koha-community.org> 
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Keycloak as Identity Provider using Koha's users

[David Cook via Koha-devel]

Sounds good. Thanks, Paul!

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Paul Derscheid  
Sent: Friday, October 20, 2023 8:07 PM
To: David Cook ; 'Koha Devel' 

Cc: 'Renvoize, Martin' 
Subject: Re: Keycloak as Identity Provider using Koha's users

 

Hi David,

thanks for putting so much work into this. I will try to use it to for a third 
party integration a little down the road.
I'll share my experience with you if you're interested (or write an issue if 
something is broken :D).

Thanks again

Paul

On 10/19/23 03:01, David Cook wrote:

Hi all,

 

I’ve posted the Keycloak extension for using Koha’s user database as Keycloak’s 
user datastore: https://gitlab.com/minusdavid/keycloak-user-storage-koha

 

This effectively allows you to use Koha (version 22.11.03 and upwards) as an 
Identity Provider – mediated via Keycloak (version 20 and up). 

 

So you can set up Koha to use Keycloak via SAML or OIDC and then have other 
third-party services like EBSCO EDS using the Keycloak, and thus you have SSO 
across the board while the users are fully managed through Koha, which is 
pretty cool.

 

At some point, I might look into how I can host the built JAR on Gitlab as 
well, but I’ve provided the command to build the JAR using Docker, so it is 
very easy to do even if you’re not a Java expert. 

 

Let me know if you need any help trying it out.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

-- 
LMSCloud GmbH
Paul Derscheid - Software Engineer
Bismarckstr. 3 -  D-72764 Reutlingen
m +49 174 2436308
e paul.dersch...@lmscloud.de  
w www.lmscloud.de  
___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

Re: [Koha-devel] Background jobs quirks

[David Cook via Koha-devel]

@lists.koha-community.org> >
Cc: 'Jonathan Druart' mailto:jonathan.dru...@bugs.koha-community.org> >; 'Tomas Cohen Arazi' 
mailto:tomasco...@gmail.com> >
Subject: RE: [Koha-devel] Background jobs quirks

 

Success!

 

I created the persistent database connection between the background job worker 
and the database, and then I restarted the database.

 

When I tried to enqueue a background job, it’s now stuck in a “New” status with 
a Progress of “null/1”, and /var/log/koha/kohadev/worker-error.log says the 
following:

20231019 05:02:14 kohadev-koha-worker-long_tasks: client (pid 23247) killed by 
signal 13, respawning

 

Now Starman seems to have managed to re-establish a connection without any 
errors. It’s not clear why that might be. I suppose Starman/Plack might be 
suppressing SIGPIPE errors. We might have some subtle issues under the hood 
without even realizing it. 

 

(That suddenly reminds me of that MYSQL_OPT_RECONNECT issue from August. MySQL 
8.0.34/8.1.0 deprecate automatic reconnection. I would hope that DBD::MySQL 
would take care of that, but as we might recall that module isn’t really 
maintained…)

 

I am testing on 22.11 so maybe it’s a bit different in master… but it’s 
interesting.  

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: David Cook mailto:dc...@prosentient.com.au> > 
Sent: Thursday, October 19, 2023 3:55 PM
To: 'David Cook' mailto:dc...@prosentient.com.au> >; 
'Koha Devel' mailto:koha-devel@lists.koha-community.org> >
Subject: RE: [Koha-devel] Background jobs quirks

 

On a standard Debian package install of Koha, it looks like there are 3 
persistent connections to the database: 2x Starman workers and 1x Zebra Indexer.

 

Then if you use the background jobs, the background job worker for that queue 
will create a persistent connection as well. 

 

I had to work out the MySQL/MariaDB ID through trial and error since there 
wasn’t enough information in the processlist in the database, but I got the ID, 
so I’ll check the logs in the morning to see if that’s the problem… 

 

--

 

I’m losing some confidence in this theory as I’m looking at the 
koha-testing-docker database instance and I’m seeing similar timeouts, but it 
hasn’t triggered the “killed by signal 13, respawning”.

 

2023-10-19  3:06:09 44 [Warning] Aborted connection 44 to db: 'koha_kohadev' 
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication 
packets)

2023-10-19  3:06:09 32 [Warning] Aborted connection 32 to db: 'koha_kohadev' 
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication 
packets)

 

--

 

I’m thinking the best thing to do is just ignore SIGPIPE and let our regular 
error handling take care of it, as we should then get a more nuanced error 
message somewhere…

 

Looking at https://metacpan.org/dist/Net-Stomp/source/lib/Net/Stomp.pm and it 
ignores SIGPIPE when it’s sending frames… so it’s probably not the $conn->ack() 
doing it.

 

I do wonder if it’s “my $job = Koha::BackgroundJobs->find($args->{job_id});”

 

I suppose we’ll see in the morning…

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of David Cook 
via Koha-devel
Sent: Thursday, October 19, 2023 2:41 PM
To: 'Koha Devel' mailto:koha-devel@lists.koha-community.org> >
Subject: [Koha-devel] Background jobs quirks

 

Hi all,

 

Have you seen something like the following in your worker-error.log?

 

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888) killed 
by signal 13, respawning

20231019 07:21:26 instance-koha-worker-long_tasks: client (pid 4082852) killed 
by signal 13, respawning

 

I found some discussion in a thread 
https://www.mail-archive.com/koha@lists.katipo.co.nz/msg30046.html

 

Signal 13 would be a SIGPIPE I believe. At first, I assumed it was the 
connection between the background worker and RabbitMQ, but based off the 
RabbitMQ logs[1] that seemed unlikely.

 

MariaDB is another possibility. Looking at its logs [2] and I see some timed 
out connections. 

 

I might monitor the connections between the background workers and the database 
tonight and see if it’s related. It might also explain why it happens on my 
MariaDB system but not my MySQL system…

 

Anyway, just putting out some feelers…

 

[1]

2023-10-18 07:44:06.492400+11:00 [info] <0.30487.131> accepting STOMP 
connection <0.30487.131> (127.0.0.1:59120 <http://127.0.0.1:59120>  -> 
127.0.0.1:61613 <http://127.0.0.1:61613> )

2023-10-18 07:44:06.500534+11:00 [info] <0.30487.131> closing STOMP connection 
<0.30487.131> (127.0.0.1:59120 <http://127.0.0.1:59120>

Re: [Koha-devel] Background jobs quirks

[David Cook via Koha-devel]

it looks like there are 3
persistent connections to the database: 2x Starman workers and 1x Zebra
Indexer.

 

Then if you use the background jobs, the background job worker for that
queue will create a persistent connection as well. 

 

I had to work out the MySQL/MariaDB ID through trial and error since there
wasn't enough information in the processlist in the database, but I got the
ID, so I'll check the logs in the morning to see if that's the problem. 

 

--

 

I'm losing some confidence in this theory as I'm looking at the
koha-testing-docker database instance and I'm seeing similar timeouts, but
it hasn't triggered the "killed by signal 13, respawning".

 

2023-10-19  3:06:09 44 [Warning] Aborted connection 44 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

2023-10-19  3:06:09 32 [Warning] Aborted connection 32 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

 

--

 

I'm thinking the best thing to do is just ignore SIGPIPE and let our regular
error handling take care of it, as we should then get a more nuanced error
message somewhere.

 

Looking at https://metacpan.org/dist/Net-Stomp/source/lib/Net/Stomp.pm and
it ignores SIGPIPE when it's sending frames. so it's probably not the
$conn->ack() doing it.

 

I do wonder if it's "my $job = Koha::BackgroundJobs->find($args->{job_id});"

 

I suppose we'll see in the morning.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of David
Cook via Koha-devel
Sent: Thursday, October 19, 2023 2:41 PM
To: 'Koha Devel' mailto:koha-devel@lists.koha-community.org> >
Subject: [Koha-devel] Background jobs quirks

 

Hi all,

 

Have you seen something like the following in your worker-error.log?

 

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888)
killed by signal 13, respawning

20231019 07:21:26 instance-koha-worker-long_tasks: client (pid 4082852)
killed by signal 13, respawning

 

I found some discussion in a thread
https://www.mail-archive.com/koha@lists.katipo.co.nz/msg30046.html

 

Signal 13 would be a SIGPIPE I believe. At first, I assumed it was the
connection between the background worker and RabbitMQ, but based off the
RabbitMQ logs[1] that seemed unlikely.

 

MariaDB is another possibility. Looking at its logs [2] and I see some timed
out connections. 

 

I might monitor the connections between the background workers and the
database tonight and see if it's related. It might also explain why it
happens on my MariaDB system but not my MySQL system.

 

Anyway, just putting out some feelers.

 

[1]

2023-10-18 07:44:06.492400+11:00 [info] <0.30487.131> accepting STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.500534+11:00 [info] <0.30487.131> closing STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.510426+11:00 [info] <0.20712.130> closing STOMP
connection <0.20712.130> (127.0.0.1:57508 -> 127.0.0.1:61613)

2023-10-18 07:44:08.703380+11:00 [info] <0.30518.131> accepting STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

 

2023-10-19 07:21:26.254065+11:00 [info] <0.27344.133> accepting STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.259129+11:00 [info] <0.27344.133> closing STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.269652+11:00 [info] <0.30518.131> closing STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

2023-10-19 07:21:26.660604+11:00 [info] <0.27372.133> accepting STOMP
connection <0.27372.133> (127.0.0.1:46178 -> 127.0.0.1:61613)

 

[2]

Oct 18 02:08:28 awesome-host mariadbd[959]: 2023-10-18  2:08:28 6308
[Warning] Aborted connection 6308 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:15:58 awesome-host mariadbd[959]: 2023-10-18  2:15:58 6304
[Warning] Aborted connection 6304 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:16:06 awesome-host mariadbd[959]: 2023-10-18  2:16:06 6303
[Warning] Aborted connection 6303 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 21:10:08 awesome-host mariadbd[959]: 2023-10-18 21:10:08 6380
[Warning] Aborted connection 6380 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 19 02:21:17 awesome-host mariadbd[959]: 2023-10-19  2:21:17 6431
[Warning] Abor

Re: [Koha-devel] Background jobs quirks

[David Cook via Koha-devel]

Success!

 

I created the persistent database connection between the background job
worker and the database, and then I restarted the database.

 

When I tried to enqueue a background job, it's now stuck in a "New" status
with a Progress of "null/1", and /var/log/koha/kohadev/worker-error.log says
the following:

20231019 05:02:14 kohadev-koha-worker-long_tasks: client (pid 23247) killed
by signal 13, respawning

 

Now Starman seems to have managed to re-establish a connection without any
errors. It's not clear why that might be. I suppose Starman/Plack might be
suppressing SIGPIPE errors. We might have some subtle issues under the hood
without even realizing it. 

 

(That suddenly reminds me of that MYSQL_OPT_RECONNECT issue from August.
MySQL 8.0.34/8.1.0 deprecate automatic reconnection. I would hope that
DBD::MySQL would take care of that, but as we might recall that module isn't
really maintained.)

 

I am testing on 22.11 so maybe it's a bit different in master. but it's
interesting.  

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: David Cook  
Sent: Thursday, October 19, 2023 3:55 PM
To: 'David Cook' ; 'Koha Devel'

Subject: RE: [Koha-devel] Background jobs quirks

 

On a standard Debian package install of Koha, it looks like there are 3
persistent connections to the database: 2x Starman workers and 1x Zebra
Indexer.

 

Then if you use the background jobs, the background job worker for that
queue will create a persistent connection as well. 

 

I had to work out the MySQL/MariaDB ID through trial and error since there
wasn't enough information in the processlist in the database, but I got the
ID, so I'll check the logs in the morning to see if that's the problem. 

 

--

 

I'm losing some confidence in this theory as I'm looking at the
koha-testing-docker database instance and I'm seeing similar timeouts, but
it hasn't triggered the "killed by signal 13, respawning".

 

2023-10-19  3:06:09 44 [Warning] Aborted connection 44 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

2023-10-19  3:06:09 32 [Warning] Aborted connection 32 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

 

--

 

I'm thinking the best thing to do is just ignore SIGPIPE and let our regular
error handling take care of it, as we should then get a more nuanced error
message somewhere.

 

Looking at https://metacpan.org/dist/Net-Stomp/source/lib/Net/Stomp.pm and
it ignores SIGPIPE when it's sending frames. so it's probably not the
$conn->ack() doing it.

 

I do wonder if it's "my $job = Koha::BackgroundJobs->find($args->{job_id});"

 

I suppose we'll see in the morning.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel mailto:koha-devel-boun...@lists.koha-community.org> > On Behalf Of David
Cook via Koha-devel
Sent: Thursday, October 19, 2023 2:41 PM
To: 'Koha Devel' mailto:koha-devel@lists.koha-community.org> >
Subject: [Koha-devel] Background jobs quirks

 

Hi all,

 

Have you seen something like the following in your worker-error.log?

 

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888)
killed by signal 13, respawning

20231019 07:21:26 instance-koha-worker-long_tasks: client (pid 4082852)
killed by signal 13, respawning

 

I found some discussion in a thread
https://www.mail-archive.com/koha@lists.katipo.co.nz/msg30046.html

 

Signal 13 would be a SIGPIPE I believe. At first, I assumed it was the
connection between the background worker and RabbitMQ, but based off the
RabbitMQ logs[1] that seemed unlikely.

 

MariaDB is another possibility. Looking at its logs [2] and I see some timed
out connections. 

 

I might monitor the connections between the background workers and the
database tonight and see if it's related. It might also explain why it
happens on my MariaDB system but not my MySQL system.

 

Anyway, just putting out some feelers.

 

[1]

2023-10-18 07:44:06.492400+11:00 [info] <0.30487.131> accepting STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.500534+11:00 [info] <0.30487.131> closing STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.510426+11:00 [info] <0.20712.130> closing STOMP
connection <0.20712.130> (127.0.0.1:57508 -> 127.0.0.1:61613)

2023-10-18 07:44:08.703380+11:00 [info] <0.30518.131> accepting STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

 

2023-10-19 07:21:26.254065+11:00 [info] <0.27344.133> accepting STOMP
connection <0.27344.133> (127.0.0

Re: [Koha-devel] Background jobs quirks

[David Cook via Koha-devel]

On a standard Debian package install of Koha, it looks like there are 3
persistent connections to the database: 2x Starman workers and 1x Zebra
Indexer.

 

Then if you use the background jobs, the background job worker for that
queue will create a persistent connection as well. 

 

I had to work out the MySQL/MariaDB ID through trial and error since there
wasn't enough information in the processlist in the database, but I got the
ID, so I'll check the logs in the morning to see if that's the problem. 

 

--

 

I'm losing some confidence in this theory as I'm looking at the
koha-testing-docker database instance and I'm seeing similar timeouts, but
it hasn't triggered the "killed by signal 13, respawning".

 

2023-10-19  3:06:09 44 [Warning] Aborted connection 44 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

2023-10-19  3:06:09 32 [Warning] Aborted connection 32 to db: 'koha_kohadev'
user: 'koha_kohadev' host: '172.21.0.4' (Got an error reading communication
packets)

 

--

 

I'm thinking the best thing to do is just ignore SIGPIPE and let our regular
error handling take care of it, as we should then get a more nuanced error
message somewhere.

 

Looking at https://metacpan.org/dist/Net-Stomp/source/lib/Net/Stomp.pm and
it ignores SIGPIPE when it's sending frames. so it's probably not the
$conn->ack() doing it.

 

I do wonder if it's "my $job = Koha::BackgroundJobs->find($args->{job_id});"

 

I suppose we'll see in the morning.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

From: Koha-devel  On Behalf Of
David Cook via Koha-devel
Sent: Thursday, October 19, 2023 2:41 PM
To: 'Koha Devel' 
Subject: [Koha-devel] Background jobs quirks

 

Hi all,

 

Have you seen something like the following in your worker-error.log?

 

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888)
killed by signal 13, respawning

20231019 07:21:26 instance-koha-worker-long_tasks: client (pid 4082852)
killed by signal 13, respawning

 

I found some discussion in a thread
https://www.mail-archive.com/koha@lists.katipo.co.nz/msg30046.html

 

Signal 13 would be a SIGPIPE I believe. At first, I assumed it was the
connection between the background worker and RabbitMQ, but based off the
RabbitMQ logs[1] that seemed unlikely.

 

MariaDB is another possibility. Looking at its logs [2] and I see some timed
out connections. 

 

I might monitor the connections between the background workers and the
database tonight and see if it's related. It might also explain why it
happens on my MariaDB system but not my MySQL system.

 

Anyway, just putting out some feelers.

 

[1]

2023-10-18 07:44:06.492400+11:00 [info] <0.30487.131> accepting STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.500534+11:00 [info] <0.30487.131> closing STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.510426+11:00 [info] <0.20712.130> closing STOMP
connection <0.20712.130> (127.0.0.1:57508 -> 127.0.0.1:61613)

2023-10-18 07:44:08.703380+11:00 [info] <0.30518.131> accepting STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

 

2023-10-19 07:21:26.254065+11:00 [info] <0.27344.133> accepting STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.259129+11:00 [info] <0.27344.133> closing STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.269652+11:00 [info] <0.30518.131> closing STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

2023-10-19 07:21:26.660604+11:00 [info] <0.27372.133> accepting STOMP
connection <0.27372.133> (127.0.0.1:46178 -> 127.0.0.1:61613)

 

[2]

Oct 18 02:08:28 awesome-host mariadbd[959]: 2023-10-18  2:08:28 6308
[Warning] Aborted connection 6308 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:15:58 awesome-host mariadbd[959]: 2023-10-18  2:15:58 6304
[Warning] Aborted connection 6304 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:16:06 awesome-host mariadbd[959]: 2023-10-18  2:16:06 6303
[Warning] Aborted connection 6303 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 21:10:08 awesome-host mariadbd[959]: 2023-10-18 21:10:08 6380
[Warning] Aborted connection 6380 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 19 02:21:17 awesome-host mariadbd[959]: 2023-10-19  2:21:17 6431
[Warning] Aborted connection 6431 to db: 'koha_instance' user:
'koha_i

[Koha-devel] Background jobs quirks

[David Cook via Koha-devel]

Hi all,

 

Have you seen something like the following in your worker-error.log?

 

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888)
killed by signal 13, respawning

20231019 07:21:26 instance-koha-worker-long_tasks: client (pid 4082852)
killed by signal 13, respawning

 

I found some discussion in a thread
https://www.mail-archive.com/koha@lists.katipo.co.nz/msg30046.html

 

Signal 13 would be a SIGPIPE I believe. At first, I assumed it was the
connection between the background worker and RabbitMQ, but based off the
RabbitMQ logs[1] that seemed unlikely.

 

MariaDB is another possibility. Looking at its logs [2] and I see some timed
out connections. 

 

I might monitor the connections between the background workers and the
database tonight and see if it's related. It might also explain why it
happens on my MariaDB system but not my MySQL system.

 

Anyway, just putting out some feelers.

 

[1]

2023-10-18 07:44:06.492400+11:00 [info] <0.30487.131> accepting STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.500534+11:00 [info] <0.30487.131> closing STOMP
connection <0.30487.131> (127.0.0.1:59120 -> 127.0.0.1:61613)

2023-10-18 07:44:06.510426+11:00 [info] <0.20712.130> closing STOMP
connection <0.20712.130> (127.0.0.1:57508 -> 127.0.0.1:61613)

2023-10-18 07:44:08.703380+11:00 [info] <0.30518.131> accepting STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

 

2023-10-19 07:21:26.254065+11:00 [info] <0.27344.133> accepting STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.259129+11:00 [info] <0.27344.133> closing STOMP
connection <0.27344.133> (127.0.0.1:54116 -> 127.0.0.1:61613)

2023-10-19 07:21:26.269652+11:00 [info] <0.30518.131> closing STOMP
connection <0.30518.131> (127.0.0.1:59882 -> 127.0.0.1:61613)

2023-10-19 07:21:26.660604+11:00 [info] <0.27372.133> accepting STOMP
connection <0.27372.133> (127.0.0.1:46178 -> 127.0.0.1:61613)

 

[2]

Oct 18 02:08:28 awesome-host mariadbd[959]: 2023-10-18  2:08:28 6308
[Warning] Aborted connection 6308 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:15:58 awesome-host mariadbd[959]: 2023-10-18  2:15:58 6304
[Warning] Aborted connection 6304 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 02:16:06 awesome-host mariadbd[959]: 2023-10-18  2:16:06 6303
[Warning] Aborted connection 6303 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 18 21:10:08 awesome-host mariadbd[959]: 2023-10-18 21:10:08 6380
[Warning] Aborted connection 6380 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 19 02:21:17 awesome-host mariadbd[959]: 2023-10-19  2:21:17 6431
[Warning] Aborted connection 6431 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

Oct 19 02:24:02 awesome-host mariadbd[959]: 2023-10-19  2:24:02 6432
[Warning] Aborted connection 6432 to db: 'koha_instance' user:
'koha_instance' host: 'localhost' (Got timeout reading communication
packets)

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] MIssing emails for CC list?

[David Cook via Koha-devel]

Hi all,

 

I'm used to getting emails for issues I'm CCed to, but I don't think I've
gotten any for the last 1 week. 

 

Is this happening to anyone else or is it just me?

 

I'm still getting emails addressed to koha-b...@lists.koha-community.org
  but not to me specifically. 

 

I'll check the mail server next I guess but figured I'd ask the question.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/

[Koha-devel] Keycloak as Identity Provider using Koha's users

[David Cook via Koha-devel]

Hi all,

 

I've posted the Keycloak extension for using Koha's user database as
Keycloak's user datastore:
https://gitlab.com/minusdavid/keycloak-user-storage-koha

 

This effectively allows you to use Koha (version 22.11.03 and upwards) as an
Identity Provider - mediated via Keycloak (version 20 and up). 

 

So you can set up Koha to use Keycloak via SAML or OIDC and then have other
third-party services like EBSCO EDS using the Keycloak, and thus you have
SSO across the board while the users are fully managed through Koha, which
is pretty cool.

 

At some point, I might look into how I can host the built JAR on Gitlab as
well, but I've provided the command to build the JAR using Docker, so it is
very easy to do even if you're not a Java expert. 

 

Let me know if you need any help trying it out.

 

David Cook

Senior Software Engineer

Prosentient Systems

Suite 7.03

6a Glen St

Milsons Point NSW 2061

Australia

 

Office: 02 9212 0899

Online: 02 8005 0595

 

___
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/