Re: [Evergreen-general] osrf_http_translator - EG 3.11.1 - Cache error and failed transaction begin
Thank you Galen, you helped me solve it... and I'm embarrassed to say it
was just a bad/old setting for the OSRFTranslatorCacheServer in
eg_vhost.conf. I missed that setting in my checklist.
Switching from --servers localhost to --servers 127.0.0.1 also fixed my
issue with the memcdump command error.
Josh
On Tue, Oct 24, 2023 at 2:01 PM Galen Charlton wrote:
> Hi,
>
> One other thing that occurred to me: "localhost" resolves to both IPv4 and
> IPv6 loopback addresses by default, so it may be a good idea to ensure that
> IPv6 is out of the picture by forcing use of 127.0.0.1 for memcdump,
> OSRFTranslatorCachedServer, and the cache configuration in opensrf.xml.
>
> I did get the same connection refused error when attempting "memcdump
> --servers localhost" on a fresh Bookworm system whose only difference from
> stock was having memcached and libmemcached-tools installed. On that box,
> "memcdump --servers 127.0.0.1" worked as expected.
>
> Regards,
>
> Galen
>
> On Tue, Oct 24, 2023 at 2:38 PM Galen Charlton wrote:
>
>> Hi,
>>
>> On Tue, Oct 24, 2023 at 10:55 AM Josh Stompro via Evergreen-general <
>> evergreen-general@list.evergreen-ils.org> wrote:
>>
>>> I have a test system(Copy of production) with EG 3.11.1 running on
>>> Debian 11.8 with memcached 1.6.9, that doesn't show this issue. So I'm
>>> leaning towards there being some bug in Memcached in Debian 12.
>>>
>>
>> I tend to agree, though at the moment I suspect libmemcached more than
>> memcached itself. From what I've gleaned just now, libmemcached was
>> unchanged with the 1.0.18 release from February 2014 until it was picked up
>> by a new maintainer in 2020. Bookworm happens to be the first Debian
>> release to package the newer version, and Ubuntu was only a few months
>> earlier to include it.
>>
>>
>>> Has anyone run into this before? Any other suggestions for things to
>>> look into? Maybe memcached with verbose logging turned on?
>>>
>>
>> Agreed, bumping up memcached's verbosity may help.
>>
>> Also, please try this: create a file named
>> "0.10253501323116211698110395845" with the contents
>> {"ip":"10.0.100.6","jid":"opensrf@private.localhost/open-ils.pcrud_drone_virt-egstaff1_1698110384.796304_2053","service":"open-ils.pcrud"}
>> and use memccp to load it into the cache. If a subsequent memcdump fails,
>> that would suggest a (weird) data-dependent bug in libmemcached or memached
>> itself.
>>
>> Regards,
>>
>> Galen
>> --
>> Galen Charlton
>> Implementation and IT Manager
>> Equinox Open Library Initiative
>> g...@equinoxoli.org
>> https://www.equinoxOLI.org
>> phone: 877-OPEN-ILS (673-6457)
>> direct: 770-709-5581
>>
>>
>
>
> --
> Galen Charlton
> Implementation and IT Manager
> Equinox Open Library Initiative
> g...@equinoxoli.org
> https://www.equinoxOLI.org
> phone: 877-OPEN-ILS (673-6457)
> direct: 770-709-5581
>
>
___
Evergreen-general mailing list
Evergreen-general@list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
Re: [Evergreen-general] osrf_http_translator - EG 3.11.1 - Cache error and failed transaction begin
Hi,
One other thing that occurred to me: "localhost" resolves to both IPv4 and
IPv6 loopback addresses by default, so it may be a good idea to ensure that
IPv6 is out of the picture by forcing use of 127.0.0.1 for memcdump,
OSRFTranslatorCachedServer, and the cache configuration in opensrf.xml.
I did get the same connection refused error when attempting "memcdump
--servers localhost" on a fresh Bookworm system whose only difference from
stock was having memcached and libmemcached-tools installed. On that box,
"memcdump --servers 127.0.0.1" worked as expected.
Regards,
Galen
On Tue, Oct 24, 2023 at 2:38 PM Galen Charlton wrote:
> Hi,
>
> On Tue, Oct 24, 2023 at 10:55 AM Josh Stompro via Evergreen-general <
> evergreen-general@list.evergreen-ils.org> wrote:
>
>> I have a test system(Copy of production) with EG 3.11.1 running on Debian
>> 11.8 with memcached 1.6.9, that doesn't show this issue. So I'm leaning
>> towards there being some bug in Memcached in Debian 12.
>>
>
> I tend to agree, though at the moment I suspect libmemcached more than
> memcached itself. From what I've gleaned just now, libmemcached was
> unchanged with the 1.0.18 release from February 2014 until it was picked up
> by a new maintainer in 2020. Bookworm happens to be the first Debian
> release to package the newer version, and Ubuntu was only a few months
> earlier to include it.
>
>
>> Has anyone run into this before? Any other suggestions for things to
>> look into? Maybe memcached with verbose logging turned on?
>>
>
> Agreed, bumping up memcached's verbosity may help.
>
> Also, please try this: create a file named
> "0.10253501323116211698110395845" with the contents
> {"ip":"10.0.100.6","jid":"opensrf@private.localhost/open-ils.pcrud_drone_virt-egstaff1_1698110384.796304_2053","service":"open-ils.pcrud"}
> and use memccp to load it into the cache. If a subsequent memcdump fails,
> that would suggest a (weird) data-dependent bug in libmemcached or memached
> itself.
>
> Regards,
>
> Galen
> --
> Galen Charlton
> Implementation and IT Manager
> Equinox Open Library Initiative
> g...@equinoxoli.org
> https://www.equinoxOLI.org
> phone: 877-OPEN-ILS (673-6457)
> direct: 770-709-5581
>
>
--
Galen Charlton
Implementation and IT Manager
Equinox Open Library Initiative
g...@equinoxoli.org
https://www.equinoxOLI.org
phone: 877-OPEN-ILS (673-6457)
direct: 770-709-5581
___
Evergreen-general mailing list
Evergreen-general@list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
Re: [Evergreen-general] osrf_http_translator - EG 3.11.1 - Cache error and failed transaction begin
Hi,
On Tue, Oct 24, 2023 at 10:55 AM Josh Stompro via Evergreen-general <
evergreen-general@list.evergreen-ils.org> wrote:
> I have a test system(Copy of production) with EG 3.11.1 running on Debian
> 11.8 with memcached 1.6.9, that doesn't show this issue. So I'm leaning
> towards there being some bug in Memcached in Debian 12.
>
I tend to agree, though at the moment I suspect libmemcached more than
memcached itself. From what I've gleaned just now, libmemcached was
unchanged with the 1.0.18 release from February 2014 until it was picked up
by a new maintainer in 2020. Bookworm happens to be the first Debian
release to package the newer version, and Ubuntu was only a few months
earlier to include it.
> Has anyone run into this before? Any other suggestions for things to look
> into? Maybe memcached with verbose logging turned on?
>
Agreed, bumping up memcached's verbosity may help.
Also, please try this: create a file named
"0.10253501323116211698110395845" with the contents
{"ip":"10.0.100.6","jid":"opensrf@private.localhost/open-ils.pcrud_drone_virt-egstaff1_1698110384.796304_2053","service":"open-ils.pcrud"}
and use memccp to load it into the cache. If a subsequent memcdump fails,
that would suggest a (weird) data-dependent bug in libmemcached or memached
itself.
Regards,
Galen
--
Galen Charlton
Implementation and IT Manager
Equinox Open Library Initiative
g...@equinoxoli.org
https://www.equinoxOLI.org
phone: 877-OPEN-ILS (673-6457)
direct: 770-709-5581
___
Evergreen-general mailing list
Evergreen-general@list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
[Evergreen-general] osrf_http_translator - EG 3.11.1 - Cache error and failed transaction begin
Hello, I noticed that with our EG 3.11.1 system it isn't possible to create
a new part in the AngularJS catalog/Dojo interface,
/eg/conify/global/biblio/monograph_part?r=188963 for example.
When the save button is pressed to create a new part, there is a CONNECT
call and then a call to open-ils.pcrud.transaction.begin. The transaction
begin call fails with a http 400 Bad Request page error.
I enabled debug logging to see this in the gateway logs.
osrf_http_translator 2023-10-23 20:19:58 [ERR
:2026:osrf_cache.c:87:169811038420260] Failed to cache key:value
[0.10253501323116211698110395845]:[{"ip":"10.0.100.6","jid":"opensrf@private.localhost/open-ils.pcrud_drone_virt-egstaff1_1698110384.796304_2053","service":"open-ils.pcrud"}]
- SYSTEM ERROR
So I think I understand that my IP is trying to be cached, so that
subsequent requests will go to the same backend, but that is failing.
And then later on when the transaction begin call wants to grab the cached
ip, that also fails.
osrf_http_translator 2023-10-23 20:19:58
[DEBG:2026:osrf_cache.c:105:169811038420261] Failed to get key
[0.10253501323116211698110395845] - SERVER HAS FAILED AND IS DISABLED UNTIL
TIMED RETRY
This is on a Debian 12.2 (Bookworm) system, with memcached 1.6.18 running
locally on the same machine. My memcached is vanilla, only change is
allowing it to use more memory for the cache.
memcached seems to be working in general, most functionality is fine. But
memcdump --server=localhost does stop listing early with an error on the
Debian 12 systems... with the error
"Failed to dump keys:(0x7ffec29c9770) CONNECTION FAILURE(Connection
refused), host: localhost:11211 -> ./src/libmemcached/io.cc:145"
I have a test system(Copy of production) with EG 3.11.1 running on Debian
11.8 with memcached 1.6.9, that doesn't show this issue. So I'm leaning
towards there being some bug in Memcached in Debian 12.
Has anyone run into this before? Any other suggestions for things to look
into? Maybe memcached with verbose logging turned on?
I put fuller logs at
https://gitlab.com/LARL/evergreen-larl/-/snippets/3614421
Thanks
Josh
[image: Company logo]
*Josh Stompro*
IT Director
stomp...@gsuite.larl.org | 218-233-3757 ext. 139 | 218-790-2110
*Lake Agassiz Regional Library *
118 5th ST S
Moorhead MN 56560
www.larl.org
*Our mission is to enrich lives and strengthen communities.*
___
Evergreen-general mailing list
Evergreen-general@list.evergreen-ils.org
http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-general
