Bug#1020989: trafficserver: failed to load plugin
Hello, Thanks for the detailled report. The only solution I got for now is to update /etc/trafficserver/record.config to disable dynamic reload for plugins: CONFIG proxy.config.plugin.dynamic_reload_mode INT 0 This way, dynamic plugin will be disable, as well as the copy and rename operations in the /run directory. Please note however that any update in remap.config will require a full service restart. I'm preparing a new package release to disable the dynamic reload by default. Best, Jean Baptiste On 12/29/23 4:21 PM, Yavor Doganov wrote: On Mon, Oct 24, 2022 at 10:39:45AM +0200, Jean Baptiste Favre wrote: Please provide some more details on you issue: * which plugin do you use? Is there any specific configuration? I am not YunQiang Su but I'm pretty sure we're talking about the same thing. Any remap plugin will fail to load; for example, add the following to /etc/trafficserver/remap.config (example from the manual): map http://a.com http://b.com @plugin=regex_remap.so @pparam=maps.reg With contents of /etc/trafficserver/maps.reg (again from the manual): ^/(ogre.*)/more http://www.ogre.com/$h/$0/$1 Global plugins are not affected by this issue. * do you have some trafficserver's logs to provide? Here you are (/var/log/trafficserver/diags.log): [Dec 28 20:00:59.369] traffic_server ERROR: plugin 'regex_remap.so' failed to load: /run/trafficserver/adb67a84-e739-4e8f-bc9d-c4797cec16e4/usr/lib/trafficserver/modules/regex_remap.so: failed to map segment from shared object [Dec 28 20:00:59.369] traffic_server ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 199: /run/trafficserver/adb67a84-e739-4e8f-bc9d-c4797cec16e4/usr/lib/trafficserver/modules/regex_remap.so: failed to map segment from shared objectfailed to remove runtime copy: Success [Dec 28 20:00:59.369] traffic_server WARNING: something failed during BuildTable() -- check your remap plugins! [Dec 28 20:00:59.369] traffic_server FATAL: remap.config failed to load [Dec 28 19:46:23.246] traffic_server ERROR: plugin 'cachekey.so' failed to load: /run/trafficserver/c4edf320-f56b-4193-8554-7db2dccfa0a7/usr/lib/trafficserver/modules/cachekey.so: failed to map segment from shared object [Dec 28 19:46:23.246] traffic_server ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 197: /run/trafficserver/c4edf320-f56b-4193-8554-7db2dccfa0a7/usr/lib/trafficserver/modules/cachekey.so: failed to map segment from shared objectfailed to remove runtime copy: Success [Dec 28 19:46:23.246] traffic_server WARNING: something failed during BuildTable() -- check your remap plugins! [Dec 28 19:46:23.246] traffic_server FATAL: remap.config failed to load There are errors from dlopen. As the OP said, the culprit is that it copies the plugins from /usr/lib to /run and tries to dlopen them there. This is doomed to fail because, exactly as the OP said, /run is mounted with the "noexec" option. Upstream logic behind this is to dynamically reload plugins that are changed while the server is running. I guess it makes sense with custom plugins and some other scenarios but there's no way for this to work on Debian systems. The attached dirty hack removes this code but also leads to test failures of two test programs where this dynamic reloading functionality is tested. It is not a proper fix but at least makes remap plugins load again. Like the OP, I also believe this bug deserves RC severity. Additionally (and this is an entirely separate issue, completely unrelated with the problem described above), some global plugins fail to load as well, for various reasons: [Dec 28 19:34:21.066] traffic_server NOTE: loading plugin '/usr/lib/trafficserve r/modules/prefetch.so' [Dec 28 19:34:21.066] traffic_server ERROR: unable to find TSPluginInit function in '/usr/lib/trafficserver/modules/prefetch.so': /usr/lib/trafficserver/modules/prefetch.so: undefined symbol: TSPluginInit [Dec 28 19:34:21.066] traffic_server FATAL: unable to find TSPluginInit function in '/usr/lib/trafficserver/modules/prefetch.so': /usr/lib/trafficserver/modules/prefetch.so: undefined symbol: TSPluginInit There is indeed no such function, and according to the plugin API there should be. [Dec 29 14:31:12.743] traffic_server ERROR: [geoip_acl] No Geo library available! [Dec 29 14:31:12.743] traffic_server ERROR: [ReverseProxy] failed to add remap rule at /etc/trafficserver/remap.config line 206: failed to initialize plugin geoip_acl.so: Unknown plugin errorfailed to remove runtime copy: Success [Dec 29 14:31:12.744] traffic_server WARNING: something failed during BuildTable() -- check your remap plugins! [Dec 29 14:31:12.744] traffic_server FATAL: remap.config failed to load I have tested only those that are pcre-based.
Bug#1020989: trafficserver: failed to load plugin
On Mon, Oct 24, 2022 at 10:39:45AM +0200, Jean Baptiste Favre wrote:
> Please provide some more details on you issue:
> * which plugin do you use? Is there any specific configuration?
I am not YunQiang Su but I'm pretty sure we're talking about the same
thing. Any remap plugin will fail to load; for example, add the
following to /etc/trafficserver/remap.config (example from the manual):
map http://a.com http://b.com @plugin=regex_remap.so @pparam=maps.reg
With contents of /etc/trafficserver/maps.reg (again from the manual):
^/(ogre.*)/more http://www.ogre.com/$h/$0/$1
Global plugins are not affected by this issue.
> * do you have some trafficserver's logs to provide?
Here you are (/var/log/trafficserver/diags.log):
[Dec 28 20:00:59.369] traffic_server ERROR: plugin 'regex_remap.so' failed to
load:
/run/trafficserver/adb67a84-e739-4e8f-bc9d-c4797cec16e4/usr/lib/trafficserver/modules/regex_remap.so:
failed to map segment from shared object
[Dec 28 20:00:59.369] traffic_server ERROR: [ReverseProxy] failed to add remap
rule at /etc/trafficserver/remap.config line 199:
/run/trafficserver/adb67a84-e739-4e8f-bc9d-c4797cec16e4/usr/lib/trafficserver/modules/regex_remap.so:
failed to map segment from shared objectfailed to remove runtime copy: Success
[Dec 28 20:00:59.369] traffic_server WARNING: something failed during
BuildTable() -- check your remap plugins!
[Dec 28 20:00:59.369] traffic_server FATAL: remap.config failed to load
[Dec 28 19:46:23.246] traffic_server ERROR: plugin 'cachekey.so' failed to
load:
/run/trafficserver/c4edf320-f56b-4193-8554-7db2dccfa0a7/usr/lib/trafficserver/modules/cachekey.so:
failed to map segment from shared object
[Dec 28 19:46:23.246] traffic_server ERROR: [ReverseProxy] failed to add remap
rule at /etc/trafficserver/remap.config line 197:
/run/trafficserver/c4edf320-f56b-4193-8554-7db2dccfa0a7/usr/lib/trafficserver/modules/cachekey.so:
failed to map segment from shared objectfailed to remove runtime copy: Success
[Dec 28 19:46:23.246] traffic_server WARNING: something failed during
BuildTable() -- check your remap plugins!
[Dec 28 19:46:23.246] traffic_server FATAL: remap.config failed to load
There are errors from dlopen. As the OP said, the culprit is that it
copies the plugins from /usr/lib to /run and tries to dlopen them
there. This is doomed to fail because, exactly as the OP said, /run
is mounted with the "noexec" option. Upstream logic behind this is to
dynamically reload plugins that are changed while the server is
running. I guess it makes sense with custom plugins and some other
scenarios but there's no way for this to work on Debian systems.
The attached dirty hack removes this code but also leads to test
failures of two test programs where this dynamic reloading
functionality is tested. It is not a proper fix but at least makes
remap plugins load again.
Like the OP, I also believe this bug deserves RC severity.
Additionally (and this is an entirely separate issue, completely
unrelated with the problem described above), some global plugins fail
to load as well, for various reasons:
[Dec 28 19:34:21.066] traffic_server NOTE: loading plugin '/usr/lib/trafficserve
r/modules/prefetch.so'
[Dec 28 19:34:21.066] traffic_server ERROR: unable to find TSPluginInit
function in '/usr/lib/trafficserver/modules/prefetch.so':
/usr/lib/trafficserver/modules/prefetch.so: undefined symbol: TSPluginInit
[Dec 28 19:34:21.066] traffic_server FATAL: unable to find TSPluginInit
function in '/usr/lib/trafficserver/modules/prefetch.so':
/usr/lib/trafficserver/modules/prefetch.so: undefined symbol: TSPluginInit
There is indeed no such function, and according to the plugin API
there should be.
[Dec 29 14:31:12.743] traffic_server ERROR: [geoip_acl] No Geo library
available!
[Dec 29 14:31:12.743] traffic_server ERROR: [ReverseProxy] failed to add remap
rule at /etc/trafficserver/remap.config line 206: failed to initialize plugin
geoip_acl.so: Unknown plugin errorfailed to remove runtime copy: Success
[Dec 29 14:31:12.744] traffic_server WARNING: something failed during
BuildTable() -- check your remap plugins!
[Dec 29 14:31:12.744] traffic_server FATAL: remap.config failed to load
I have tested only those that are pcre-based.
--- trafficserver-9.2.3+ds.orig/proxy/http/remap/PluginDso.cc
+++ trafficserver-9.2.3+ds/proxy/http/remap/PluginDso.cc
@@ -72,7 +72,7 @@
result = false;
} else {
PluginDebug(_tag, "plugin '%s' effective path: %s", _configPath.c_str(), _effectivePath.c_str());
-
+#if 0
/* Copy the installed plugin DSO to a runtime directory if dynamic reload enabled */
std::error_code ec;
if (isDynamicReloadEnabled() && !copy(_effectivePath, _runtimePath, ec)) {
@@ -88,9 +88,9 @@
fs::file_status fs = fs::status(_effectivePath, ec);
_mtime = fs::modification_time(fs);
PluginDebug(_tag, "plugin '%s' modification time %ld", _configPath.c_str(), _mtime);
-
+#endif
/* Now attempt
Bug#1020989: trafficserver: failed to load plugin
Waiting for your answer, I lower the severity of this bug to avoid it being removed from archive. Without any further news, this bug might be closed by the end of the month Best, Jean Baptiste On 10/24/22 10:39, Jean Baptiste Favre wrote: Hello, Made some tests today: loading plugin works well for me, with a default Debian install. Please provide some more details on you issue: * which plugin do you use? Is there any specific configuration? * do you have some trafficserver's logs to provide? A possible workaround would be to tune the RUNDI variable in /etc/default/trafficserver. Please try this and update this bug. Best, Jean Baptiste On 10/21/22 16:51, Jean Baptiste Favre wrote: Hello, Thanks for your report. After a quick look, I couldn't find any easy solution to this issue. Will have a deeper as soon as possible. Best, Jean Baptiste On 9/30/22 09:05, YunQiang Su wrote: Package: trafficserver Version: 9.1.3+ds-2 Severity: grave Since ATS 9, it tries to load plugins to /run/trafficserver, and then load them, and it asks for the plugins to have execution permission. While the /run filesystem on Debian is mounted with noexec option.
Bug#1020989: trafficserver: failed to load plugin
Hello, Made some tests today: loading plugin works well for me, with a default Debian install. Please provide some more details on you issue: * which plugin do you use? Is there any specific configuration? * do you have some trafficserver's logs to provide? A possible workaround would be to tune the RUNDI variable in /etc/default/trafficserver. Please try this and update this bug. Best, Jean Baptiste On 10/21/22 16:51, Jean Baptiste Favre wrote: Hello, Thanks for your report. After a quick look, I couldn't find any easy solution to this issue. Will have a deeper as soon as possible. Best, Jean Baptiste On 9/30/22 09:05, YunQiang Su wrote: Package: trafficserver Version: 9.1.3+ds-2 Severity: grave Since ATS 9, it tries to load plugins to /run/trafficserver, and then load them, and it asks for the plugins to have execution permission. While the /run filesystem on Debian is mounted with noexec option. OpenPGP_signature Description: OpenPGP digital signature
Bug#1020989: trafficserver: failed to load plugin
Hello, Thanks for your report. After a quick look, I couldn't find any easy solution to this issue. Will have a deeper as soon as possible. Best, Jean Baptiste On 9/30/22 09:05, YunQiang Su wrote: Package: trafficserver Version: 9.1.3+ds-2 Severity: grave Since ATS 9, it tries to load plugins to /run/trafficserver, and then load them, and it asks for the plugins to have execution permission. While the /run filesystem on Debian is mounted with noexec option. OpenPGP_signature Description: OpenPGP digital signature
Bug#1020989: trafficserver: failed to load plugin
Package: trafficserver Version: 9.1.3+ds-2 Severity: grave Since ATS 9, it tries to load plugins to /run/trafficserver, and then load them, and it asks for the plugins to have execution permission. While the /run filesystem on Debian is mounted with noexec option. -- YunQiang Su
