Re: [naviserver-devel] refproxy module/ns_connchan: no driver for protocol 'https' found

[Gustaf Neumann (sslmail)]

> On 30.04.2024, at 20:53, Georg Lehner  wrote:
> 
> Hello,
> 
> I am trying to set up the revproxy module on a current naviserver git 
> checkout.
> 
> Upon connection to the frontend I get:
> 
> Request Error
> Error during opening connection to backend http://localhost:65193/ failed.
> Error message: no driver for protocol 'http' found
> 
> NaviServer/5.0.0a
> 

Probably, you have no nssock module installed. ns_sock uses the network drivers 
installed, and in case, your configuration network does not include it, it is 
not available. If you want to use a driver module, but you do not want to 
listen on a port, configure the port with the value 0

> The same message I get when starting 'nsd -c' and running either of
> 
> ns_connchan open http://example.com  -> no driver 
> for protocol 'http' found
> 
> ns_connchan open https://example.com  -> no driver 
> for protocol 'https' found
> 
Same as above. When “nsd -c” is called, no network drivers are loaded

-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] reverseproxymode with multiple X-Forwarded-For values

[Gustaf Neumann (sslmail)]

ded 
137.208.116.31}



> On 29.04.2024, at 10:47, David Osborne  wrote:
> 
> Hi Gustaf,
> 
> From your description it sounds like we could certainly work round our issue 
> using the ReverseProxyTrustedServers config.
> Thank you very much for your time on this.
> 
> On Fri, 26 Apr 2024 at 14:09, Gustaf Neumann (sslmail)  <mailto:neum...@wu.ac.at>> wrote:
>> Hi David,
>> 
>> I have now implemented the following (but not yet committed,
>> since i was side-tracked by some tcl9 issues and i am running out of
>> time.
>> 
>> From my understanding, this should address your problems now, 
>> and when “proxy 2” is removed.
>> 
>> An easy extension of this would be to let the site-admin configure
>> an alternative header field (like x-real-ip), which could bypass
>> the search through the list of candidate addresses.
>> 
>> all the best
>> -gn
>> 
> 
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] reverseproxymode with multiple X-Forwarded-For values

[Gustaf Neumann (sslmail)]

Hi David,

I have now implemented the following (but not yet committed,
since i was side-tracked by some tcl9 issues and i am running out of
time.

From my understanding, this should address your problems now, 
and when “proxy 2” is removed.

An easy extension of this would be to let the site-admin configure
an alternative header field (like x-real-ip), which could bypass
the search through the list of candidate addresses.

all the best
-gn

=

Traditionally, NaviServer has implemented x-forwarded-for handling in
the simplest way, by treating the first value in the x-forwarded-for
header as the client (see also [1] for a brief description). This
works well for the most part in many production environments.

There are two potential problems with this:

a) If the client is connecting from a private network through
   potentially multiple proxy servers, the private address
   (e.g. 192.168.1.100) may end up as the peer (and also e.g. in the
   access.log), which is not useful for any kind of analysis.

b) If the client itself sends a (faked) "X-Forwarded-For" header, it
   can privide whatever it wants as header value.

One measure is for a reverse proxy to only accept X-Forwarded-For
headers from trusted sources, or to always treat its peer as a client
(providing the peer address in the header field).

This can lead to some loss of information if there are multiple proxy
servers in the chain, and sometimes, it is not possible for an
application to influence the proxy server configuration.

The new version now has two modes, the first being an extension of the
classic mode, and the second being better for untrusted header
content.

The default mode processes the address list from left to right,
skipping private addresses.  If the received "x-forwarded-for header
contains, for example, "192.168.1.100 1.1.1.1 2.2.2.2", then "1.1.1.1"
will be used as the client IP address.

Alternatively, when the NaviServer site admin configures trusted
reverse proxy servers in the configuration file (either by IP address
or in CDIR notation)

ReverseProxyTrustedServers {192.168.0.0/16 137.208.89.213}

then the "x-forwarded-for" header will only be accepted from trusted
servers (otherwise the header will be ignored). In addition, the
address list is processed from right to left, skipping known trusted
reverse proxies. This reduces the problem of spoofed addresses. In the
address list "142.251.208.110 1.1.1.1 192.168.2.10", the trusted
reverse proxy "192.168.2.10" is skipped and "1.1.1.1" is used (instead
of the spoofed Google IP).

Note: Only when NaviServer receives multiple entries in the
"x-forwarded-for" header is there a difference between these two
modes.

[1] https://en.wikipedia.org/wiki/X-Forwarded-For



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] reverseproxymode with multiple X-Forwarded-For values

[Gustaf Neumann (sslmail)]

> On 23.04.2024, at 18:07, David Osborne  wrote:
> 
> But the Client can initiate requests which have X-Forwarded-For Headers 
> already present, then we run into difficulties
> 
> Client: IP 1.1.1.1 : sends X-Forwarded-For: 1.2.3.4
> |
> Proxy 1: sends X-Forwarded-For: 1.2.3.4,1.1.1.1
> Proxy 2: sends X-Forwarded-For: 1.2.3.4,1.1.1.1,2.2.2.2
> Naviserver: peeraddr -source forwarded = 1.2.3.4

This is the case where proxy 1 should not accept the x-forwarded-for header 
from an untrusted upstream server. In case, proxy1 is a nginx server, it should 
use  
  proxy_set_header X-Forwarded-For $remote_addr;
for untrusted upstream requests. Therefore, proxy 2 will receive 
x-forwarded-for with 1.1.1.1, 2.2.2.2 etc. and everything is fine.

> 
> We have got around this by using proxy2 to clobber the X-Forwared-For header 
> using the Nginx real_ip module which has the logic that it will use the 
> rightmost IP if from a trusted peer.
> 
> Client: IP 1.1.1.1 : sends X-Forwarded-For: 1.2.3.4
> |
> Proxy 1: sends X-Forwarded-For: 1.2.3.4,1.1.1.1
> Proxy 2: sends X-Forwarded-For: 1.1.1.1 (as calculated by real_ip  
> logic - we trust 
> proxy1, so use the most recent IP - the rightmost)
> |
> Naviserver: peeraddr -source forwarded = 1.1.1.1

by this “trick” one can skip in your configuration the “untrusted”  
X-Forwarded-For: 1.2.3.4

However, without the massaging in proxy2, you would end up with 

X-Forwarded-For: 1.2.3.4,1.1.1.1,2.2.2.2

where the rightmost address is as well incorrect. So, even by the search from 
the right, one should skip the known proxy servers from the list (here 
2.2.2.2).  Also, one should aim for a configuration that works also, when one 
more different proxy is added. 

> However, we are looking into dropping proxy2. Unfortunately proxy1 does not 
> have the flexibility to manipulate headers in such a way

Does your proxy1 set something like the x-real-ip header?

-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] reverseproxymode with multiple X-Forwarded-For values

[Gustaf Neumann (sslmail)]

> On 22.04.2024, at 11:01, David Osborne  wrote:
> 
> In reverseproxymode, when there is a list of IPs in X-Forwarded-For header, 
> it's always the leftmost IP which is chosen by NaviServer for accesslogs (and 
> ns_conn peeraddr):
> 
> X-Forwarded-For 1.1.1.1,2.2.2.2
> ns_conn peeraddr -source forwarded = 1.1.1.1
> 
> Is there any mechanism by which we can resolve to the rightmost IP for the 
> access logs instead?


In case, there is only one forwarded-for entry received by NaviServer, 
everything is fine in all situations. In case x-forwarded-for has multiple 
values and gets its content from trusted and well-configured proxies, the 
leftmost value is the original value of the client
 X-Forwarded-For: client, proxy1, proxy2
The rightmost value is from is the last proxy. 

From a security point of view, if there is no trust in the proxies, it is 
certainly easy for an attacker to inject a wrong IP-address of the client to 
the forwarded-for header. Therefore, various reverse proxy server have 
mechanisms to define trusted upstream proxies, from which these values are 
accepted. So, probably the right approach is to accept in nginx configuration 
only  incoming x-forwarded-for headers from trusted sources (or drop all, if 
there are no trusted upstream servers). nginx-rule for replacing incoming 
x-forwarded-for headers with the remote address:
  proxy_set_header X-Forwarded-For $remote_addr;
Note that it is possible that a request can contain multiple x-forwarded-for 
headers, which should be logically concatenated. NaviServer just takes the 
first one (corresponding the leftmost value) and shows therefore a consistent 
behaviour.

However, the implementation in NaviServer is certainly not perfect. When 
handling of multiple proxy servers is required (larger network of proxies) and 
e.g. a client and a proxy are from a private network (having non-routable 
addresses), and these requests are routed over proxies in public networks, the 
internal addresses would not be informative for most purposes (geo-location, 
etc.) and should not be used. Instead, the first routable, valid address from 
the left should be taken.

As another improvement, NaviServer should support the standardised “Forwarded” 
header, but its content can't be trusted as well. So, i am not sure, this would 
help.

Why are you looking into the issue? Would the filtering of private addresses 
help you?

-gn

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] FreeBSD 14 - TCL 9.0 - NaviServer GIT Latest - error: incompatible function pointer - log.c

[Gustaf Neumann (sslmail)]

Dear David,

I installed freebsd14 and could reproduce the issue. The issue is that 
Tcl_PanicProc is defined in Tcl9 with the attribute declaration TCL_NORETURN1, 
but it seems, this attribute definition has to be provided as well for the  
prototype. It would be certainly better, if this attribute definition would be 
part of the type declaration of Tcl_PanicProc in Tcl9. Not very long ago, the 
same code worked fine, so something has changed in Tcl9.

The patch below addresses this issue, but this is not nice. Maybe, the tcl-core 
people have a suggestion, since this will effect as well other applications 
requiring the use of Tcl_SetPanicProc().

Please update your checkout of NaviServer from git, since i have fixed one more 
small issues unrelated to this.

all the best
-g


--- a/nsd/log.c
+++ b/nsd/log.c
@@ -96,7 +96,11 @@ static void LogEntriesFree(void *arg);
  */
 
 static Ns_TlsCleanup FreeCache;
-static Tcl_PanicProc Panic;
+static
+#ifndef NS_TCL_PRE9
+ TCL_NORETURN1
+#endif
+ Tcl_PanicProc Panic;
 
 static Ns_LogFilter LogToFile;
 static Ns_LogFilter LogToTcl;




> On 21.04.2024, at 16:18, Not Spam  wrote:
> 
> Good afternoon,
> I'm currently rebuilding my machine and I am having some issues with 
> NaviServer (4.99-main #defd765) compiling on FreeBSD 14, with TCL 9.0b2 or 
> TCL 9.0b1
>  
> TCL 9.0b1
> ===
> cc  -O2 -Wall -fPIC  -pipe -finput-charset=UTF-8  -DNDEBUG -DSYSTEM_MALLOC 
> -std=c99 -I../include -I"/usr/local/include"   -DHAVE_CONFIG_H   -c -o log.o 
> log.c
> log.c:262:22: error: incompatible function pointer types passing 
> 'Tcl_PanicProc' (aka 'void (const char *, ...)') to parameter of type 'void 
> (*)(const char *, ...) __attribute__((noreturn))' 
> [-Wincompatible-function-pointer-types]
> Tcl_SetPanicProc(Panic);
>  ^
> /usr/local/include/tcl.h:2366:37: note: passing argument to parameter 
> 'panicProc' here
> TCL_NORETURN1 Tcl_PanicProc *panicProc);
>   
>^
>  
> TCL 9.0b2
> ===
> cc  -O2 -Wall -fPIC  -pipe -finput-charset=UTF-8  -DNDEBUG -DSYSTEM_MALLOC 
> -std=c99 -I../include -I"/usr/local/include"   -DHAVE_CONFIG_H   -c -o log.o 
> log.c
> log.c:262:22: error: incompatible function pointer types passing 
> 'Tcl_PanicProc' (aka 'void (const char *, ...)') to parameter of type 'void 
> (*)(const char *, ...) __attribute__((noreturn))' 
> [-Wincompatible-function-pointer-types]
> Tcl_SetPanicProc(Panic);
>  ^
> /usr/local/include/tcl.h:2366:37: note: passing argument to parameter 
> 'panicProc' here
> TCL_NORETURN1 Tcl_PanicProc *panicProc);
>   
>^
> 1 error generated.
>  
> I'm using the latest GIT repo version.
> Any advice would be nice, I wish to use the ZIPFS that TCL9 brings.
>  
> Regards,
> David F
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Error on startup

[Gustaf Neumann (sslmail)]

The error indicates inconsistent usage of memory allocation/deallocation 
functions. Typical reasons are inconsistent usage of library functions (e.g. 
different tcl versions during compilation and runtime of nsd or some of the c 
modules) or inconsistent usage of memory allocator options (e.g. malloc 
libraries in use).

One can obtain details what happened via debugging options during compilation.

-gn

> On 18.03.2024, at 18:39, Maksym Zinchenko  wrote:
> 
> Hello, what can cause this errors, how can I get more information about, 
> debug:
> 
> Fatal: alloc: invalid block: 0x7fd5285e5670: 0 0
> Fatal: received fatal signal 11
> 
> Thank you
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] (no subject)

[Gustaf Neumann]

Dear Maksym,

in order to help you, I tried to mimic your configuration:

- nginx running in a docker container (listening on port 1000)
- nsd s1.com running in a docker container (listening on port 8081)
- nsd s2.com running on the docker host (listening on port 8082, no 
container involved)

- s1 and s2 have info.tcl installed

On the docker host, i made the following tests
1) request to s1.com via containerized nginx
2) request to s2.com via containerized nginx
3) direct request to s2.com

All requests are made via curl.
To me, everything looks ok. I still do not understand, what you see as a 
problem.


When you make a request to nginx with a host name neither s1.com or s2.com,
the first entry is chosen, and the request goes to s1.com. This is nginx 
behavior.


all the best

-g


 case 1 
% curl -H "Host: s1.com" http://localhost:1000/info.tcl Host: s1.com X-Real-IP: 192.168.65.1 
X-Forwarded-For: 192.168.65.1 X-Forwarded-Proto: http X-Name: s1.com 
Connection: close User-Agent: curl/8.6.0 Accept: */* ns_conn host: 
s1.com  case 2  % curl -H "Host: s2.com" http://localhost:1000/info.tcl Host: s2.com X-Real-IP: 192.168.65.1 
X-Forwarded-For: 192.168.65.1 X-Forwarded-Proto: http X-Name: s2.com 
Connection: close User-Agent: curl/8.6.0 Accept: */* ns_conn host: 
s2.com  case 3  % curl -H "Host: s2.com" http://localhost:8082/info.tcl Host: s2.com User-Agent: curl/8.6.0 
Accept: */* ns_conn host: s2.com info.tcl: 
### ns_return 
200 text/plain [subst [ns_trim -delimiter | { |[join [lmap {key value} 
[ns_set array [ns_conn headers]] {set _ "$key: $value\n"}] ""]

|
|ns_conn host: [ns_conn host]
}]]
###


nginx configuration:
###
server {
listen 80;
server_name s1.com;

location / {

proxy_passhttp://nsd:8081/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Name s1.com;
}
}

server {
listen 80;
server_name s2.com;

location / {

proxy_passhttp://host.docker.internal:8082/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Name s2.com;
}
}
###

docker-compose.yaml
###
version: '3'

services:
  nginx:
image: nginx:alpine
ports:
  - 1000:80
volumes:
  - /opt/docker/nginx/conf.d:/etc/nginx/conf.d
depends_on:
  - nsd

  nsd:
image: gustafn/naviserver:latest-alpine
container_name: nsd
restart: unless-stopped
command: /usr/local/ns/bin/nsd -f -t /usr/local/ns/conf/nsd-config.tcl -u 
nsadmin -g nsadmin
ports:
  - 8081:8081
environment:
  - TZ=${TZ:-Europe/Vienna}
  - nsd_httpport=8081
  - nsd_httpsport=
###

On 05.03.24 20:25, Maksym Zinchenko wrote:
Hello, no right now i have single nsd with many subdomains clients, 
without docker.


For example: sub1.domain.com <http://sub1.domain.com>, sub2.domain.com 
<http://sub2.domain.com> etc. What i want is to move each one of this 
subdomains to a separeted docker container. But I need to move them 
one by one, because I still testing my application on docker, so I 
want to move for example sub1 today test it, next time sub2 test it 
and so on.


While im working on one subdomain i still want to serve other subdomains.

So my idea was to move host's nsd to other ports 8080 and 8443 and 
install nginx reverse proxy, so i can redirect some subdomains to 
host's nsd and some to dockerized nsd.


Thanj you

On Tue, 5 Mar 2024, 18:08 Gustaf Neumann (sslmail),  
wrote:


Hi Maxsym,

if i understand correctly, you are sending from the docker host
requests to a single dockerized nginx instance, that forwards
these requests to a single dockerized nsd backend instance. The
nginx instance distinguishes the incoming requests to the same
port based on the host header field. And the same should happen as
well on the backend (nsd).

Is this a correct understanding of your setup?

This is somewhat unusual, since typically one uses nginx to
forward requests to multiple backend instances. So i am not sure,
why you are doing this. I am not an nginx expert, but my suspicion
is that one does not need 2 nginx “serv

Re: [naviserver-devel] (no subject)

[Gustaf Neumann (sslmail)]

Hi Maxsym,

if i understand correctly, you are sending from the docker host requests to a 
single dockerized nginx instance, that forwards these requests to a single 
dockerized nsd backend instance. The nginx instance distinguishes the incoming 
requests to the same port based on the host header field. And the same should 
happen as well on the backend (nsd).

Is this a correct understanding of your setup?

This is somewhat unusual, since typically one uses nginx to forward requests to 
multiple backend instances. So i am not sure, why you are doing this. I am not 
an nginx expert, but my suspicion is that one does not need 2 nginx “server” 
definitions for this.

The problem that you are seeing is that on the backend, you see always the same 
host header field, which is used for virtual hosting. Correct?

Maybe install for your backend servers the following script e.g. under the name 
“info.tcl” and show the result of “curl -H ….” requests, and what you are 
expecting.

all the best
-g

ns_return 200 text/plain [subst [ns_trim -delimiter | {
|[join [lmap {key value} [ns_set array [ns_conn headers]] {set _ "$key: 
$value\n"}] ""]
|   
  
|ns_conn host: [ns_conn host]   
  
|ns_conn peer: [ns_conn peeraddr]
|ns_conn peer -source configured: [ns_conn peeraddr -source configured]
|ns_conn peer -source direct: [ns_conn peeraddr -source direct]   
|ns_conn peer -source forwarded: [ns_conn peeraddr -source forwarded] 
}]]



> On 04.03.2024, at 11:45, Maksym Zinchenko  wrote:
> 
> I have a Naviserver with a couple of virtual web servers listening on a 
> single ip:port. I want to move all those virtual servers to Docker containers 
> gradually. So I'm trying to install Nginx Reverse Proxy on my Docker 
> container and configure it to redirect requests to my host Naviserver.
> 
> I've changed my Naviserver config to listen to 8080 and 8443 ports on docker 
> host gateway IP:
> 
>> /opt/ns/bin/nsd -w -u nsadmin -t /opt/ns/conf/dz_nsd.tcl -b 172.17.0.1:8080 
>> ,172.17.0.1:8443 
> In my Nginx config I have 2 listeners one for dev and dummy1 subdomains:
> 
>>  server {
>> listen 80;
>> server_name dev.daidze.org ;
>> 
>> location / {
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header Host $http_host;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_pass http://172.17.0.1:8080/;
>> }
>> }
>> 
>> server {
>> listen 443 ssl;
>> server_name dev.daidze.org ;
>> ssl_certificate /opt/ns/modules/nsssl/fullchain.pem;
>> ssl_certificate_key  /opt/ns/modules/nsssl/privkey.pem;
>> ssl_prefer_server_ciphers on;
>> 
>> location / {
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header Host $http_host;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_pass https://172.17.0.1:8443/;
>> }
>> }
>> 
>> server {
>> listen 80;
>> server_name dummy1.daidze.org ;
>> 
>> location / {
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header Host $http_host;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_pass http://172.17.0.1:8080/;
>> }
>> }
>> 
>> server {
>> listen 443 ssl;
>> server_name dummy1.daidze.org ;
>> ssl_certificate /opt/ns/modules/nsssl/fullchain.pem;
>> ssl_certificate_key  /opt/ns/modules/nsssl/privkey.pem;
>> ssl_prefer_server_ciphers on;
>> 
>> location / {
>> proxy_set_header X-Real-IP $remote_addr;
>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
>> proxy_set_header Host $http_host;
>> proxy_set_header X-Forwarded-Proto $scheme;
>> proxy_pass https://172.17.0.1:8443/;
>> }
>> }
> 
> I'm using proxy_set_header Host $http_host; to redirect the headers real Host 
> parameter to my host Naviserver installation, but I'm getting response only 
> from the dev Virtual server. It doesn't matter what url I request 
> dev.daidze.org  or dummy1.daidze.org 
> . Here an example of request to 
> https://dummy1.daidze.org/ :
> 
>> dev server
>> X-Real-IP: 172.64.238.37 
>> X-Forwarded-For: 165.90.99.154, 172.64.238.37 
>> Host: dummy1.daidze.org  
>> X-Forwarded-Proto: https 
>> Connection: close 
>> accept-encoding: gzip, br 
>> CF-RAY: 85f13d442cd66671-MAD 
>> CF-Visitor: {"scheme":"https"} 
>> user-agent: Mozilla/5.0 

[naviserver-devel] NaviServer 4.99.30 available

[Gustaf Neumann]

Dear all,

I am glad to announce that the release of NaviServer 4.99.30 is 
available at SourceForge [1].  This release contains a small improvement 
to reduce the log file verbosity on busy sites (many thanks to Brian for 
the suggestion) and contains minor fixes for certain configurations, 
improved timing information under error conditions, and improved support 
for cluster configurations for OpenACS.


As discussed before, the repository of NaviServer was moved from 
Bitbucket to GitHub [2]. The plan is to empty the Bitbucket repository 
in the near future. Please adjust your install/update scripts to point 
to the new link. The installation scripts from [3] have been adjusted to 
GitHub as well.


All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.30/
[2] https://github.com/orgs/naviserver-project/repositories
[3] https://github.com/gustafn/install-ns

===
NaviServer 4.99.30, released 2024-01-26
===

 35 files changed, 700 insertions(+), 333 deletions(-)

New Features:
-

- Global Parameter "sockacceptlog" (Commit a8b5f33c):

  Introduced a new global parameter "sockacceptlog" to define a
  threshold for logging, when to inform the user about a high number
  of incoming connections in a single accept callback.

Bug Fixes:
--

- Compilation Issues (Commit 0b72baa7): Fixed compilation issues
  when compiled without OpenSSL or zlib support.

- Meaningful End-Times on Connection Errors (Commit 929d55be):
  Ensured meaningful end-times are provided during connection
  errors in ns_http.

- Rework of "rejectalreadyclosedconn" (Commit f0963834): Reworked
  the "rejectalreadyclosedconn" for safer operations and logging.

- Shutdown Callbacks and Spoolers (Commit 0f0ea439): Change
  shutdown order to let shutdown callbacks use spoolers. This is
  beneficial for cluster mode in OpenACS, where other cluster
  nodes should be be informed about the shutdown of a node (Used
  by OpenACS).

Changes in Sample Configuration Files:
--

- Oracle Startup Configurable (Commit 1ee7fef): Made startup
  configuration for Oracle databases adjustable in
  openacs-config.tcl.

- Use Uppercase Memory Units (Commit 2f96dc62): Standardized
  memory units to uppercase in openacs-config.tcl.

- Removed Custom Setting for "connsperthread" (Commit 823b68b2):
  Removed the custom setting for "connsperthread" in
  openacs-config.tcl to avoid misconceptions.

- Backported nsd-config.tcl and openacs-config.tcl from main
  branch to improve maintainability

Misc Improvements:
--

- Library Initialization After Command Line Processing (Commit
  fc3ee12b): Moved library initialization to occur after command
  line processing to avoid potential unexpected output.

- Timeout Information in Warning Messages (Commit 12fe7805):
  Enhanced warning messages to include timeout value information.

- Improved Robustness of Regression Rest (Commit bcc309716)

- Compatibility Improvements:
  * Improved Forward Compatibility for Tcl9 (Commit d9e15fb6)
  * Improved Cross Compilation for certain ARM Processors in .m4
setup (Commit e1071fcf)
  * Improved Include Path for macOS (Commit 120dcafce): Updated
the include path for cppcheck under newer macOS versions.
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Compile error for nsdbpg

[Gustaf Neumann]

Dear Wolfgang,

You have to compile the newest version of nsdbpg either against

- the newest version of NaviServer from the branch "main", or against
- the newest version of NaviServer from the branch "release/4.99"

General rule: Take either the releases with matching version number from 
sourceforge, or take the newest versions of NaviServer and the modules 
from github (typically unreleased versions).


The following combinations are tested on the latest code

all the best

-g

Tested environments for NaviServer 4.99 (branch release/4.99)

  - os: ubuntu-latest
    compiler: gcc-10
    tcltag: core-8-6-13
    nsf_version: 2.3.0
    tdom_version: 0.9.1

  - os: ubuntu-latest
    compiler: gcc-11
    tcltag: core-8-7-a5
    ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd
    nsf_version: 2.4.0
    tdom_version: 0.9.3

Tested environments for NaviServer 5 (branch main)

  - os: ubuntu-latest
    compiler: gcc-10
    tcltag: core-8-6-13
    nsf_version: 2.3.0
    tdom_version: 0.9.1

  - os: ubuntu-latest
    compiler: gcc-12
    tcltag: core-8-7-a5
    ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd
    nsf_version: 2.4.0
    tdom_version: 0.9.3

  - os: ubuntu-latest
    compiler: gcc-12
    tcltag: main
    ns_modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd nsloopctl
    nsf_version: HEAD
    tdom_version: 0.9.3


On 22.11.23 08:18, Wolfgang Winkler via naviserver-devel wrote:


Hi all!

When compiling the current version of nsdbg from

https://github.com/naviserver-project/nsdbpg/tree/main

I get the error:

tclcmds.c:1635:37:error: expected ‘)’ before ‘PRITcl_Size’
sprintf(segment_pos, "%d, %"PRITcl_Size", '%s')",
^~~~
)
I locally changed the line 1635 to the old version, which works.

-    sprintf(segment_pos, "%d, %" PRITcl_Size ", '%s')",
-    segment, segment_len, out_buf);
+sprintf(segment_pos, "%d, %d, '%s')", segment, segment_len, out_buf);

Regards,

wiwo

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf Neumann]

Hi Brian,


There is no plan to change the policy of NaviServer on sourceforge. We 
will continue to use sourceforge for NaviServer releases in the form of 
tar files, along side with tar files for the matching modules. Github is 
for developer, code-inspection, PRs, etc.



all the best

-g


On 20.11.23 16:21, Brian Fenton wrote:

Dear Gustaf

thanks for this good work. Github does appear to be superior to 
Bitbucket on many fronts.


Will the tar.gz files for each release continue to be published to 
https://sourceforge.net/projects/naviserver/files/naviserver/ ?


best wishes
Brian
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf Neumann]

Dear all,

This is a follow-up to my own mail. The problem due to the changes of 
the billing rules on Bitbucket (see below) are somewhat sorted out, I 
have again access to the repository, but many of the former contributors 
are deleted from the "naviserver" group to get it functioning. There was 
also a change with "managed accounts", which made the problem for me 
even more complicated. In total, I count 31 emails with Atlassian 
support to sort these things out. Anyhow, these changes are signs that 
Atlassian does not have a high priority on open source projects with 
free accounts.


We have discussed this with Zoran, and we came to the conclusion that it 
is the best to move the NaviServer repositories to GitHub


https://github.com/orgs/naviserver-project/repositories

We have now on GitHub an organization "naviserver-project" which 
contains now the 54 sub-repositories. This is essentially the same 
structure we had before. We also have a certain hope, that the move to 
GitHub will rather improve the visibility of NaviServer. All the newest 
commits were done on these new repositories. The plan is to move 
completely to GitHub and to delete the repositories on Bitbucket to 
avoid confusion.


Please adjust your install/update scripts to point to the new link.

All the best

-g

On 02.11.23 15:30, I wrote:
So far, these changes are only available on sourceforge, since i have 
lost write access to the repository at bitbucket. The people on 
Atlassian seem to have changed some account types, and - on to of this 
- they annouced via the Blog post on September 27, 2023, that the 
billing model changed (where they also refer to the 
"unified-user-management"). It took me a while to figure out, what 
happened. The blog post states:


/From October 31st, 2023, Bitbucket Cloud will begin counting all
workspace members as a billable user. 

Free plans: If you're on a free plan and your billable user count
is higher than 5 as per the new definition of billable user, all
repositories in your workspace will become read-only until you
remove some users or upgrade your workspace to a paid plan./

It seems that the users of the "naviserver" group are now counted as 
"billable users", and it contains 19 users. According to support, we 
have to reduce this number to 5, otherwise nobody will be able to 
commit anything.


Due to the ability with PRs, i think the reduction will be possible 
without too much loss in functionality. If nobody objects, i will go 
back in history and reduce the number of commit-member based on the 
most recent direct commits. I hope, that non of the "old members" will 
be offended by this. One other option would be to upgrade to a paid 
plain - but i am not sure, who is gonna pay for this.


___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Gustaf Neumann]

On 08.11.23 15:42, Brian Fenton wrote:
Also regarding configuring a second driver thread - just to be clear 
are you referring to this mail you sent back in 2016 
https://sourceforge.net/p/naviserver/mailman/message/35502664/ i.e. 
enable reuseport and set driverthreads = 2? It's unclear to me if this 
has benefit when running Naviserver in a Docker container - definitely 
beyond my level of comprehension. See 
https://github.com/moby/moby/issues/7536#issuecomment-1039443599 for 
discussion of SO_REUSEPORT and Docker.


SO_REUSEPORT allows multiple sockets on the same host to bind to the 
same port at the same time, therefore, multiple driver threads can 
listen at the same time at the same port. I do not see, why this should 
not be possible in a container, ... but i am not a container expert. You 
can try a configuration with two driver threads, if it does not reduce 
the number of entries, then undo it.



On a configuration with e.g. two spooler threads, it should look like 
the following, distributing the requests more or less evenly



   thread nsssl:1 module nsssl received 890K spooled 422 partial 437K 
errors 0
   thread nsssl:0 module nsssl received 935K spooled 27K partial 491K 
errors 0



If the load distribution does not work on your docker machine (only one 
driver thread is doing all the work), then a configuration with two 
driver threads is indeed useless. Since the driver threads will work on 
different cores, this improving concurrency and scalability. Giving 
cores something to do is always preferable. But also, when one is 
defining two driver threads for the same port, don't expect that the 
messages will go away completely.



3. "exiting: exceeded max connections per thread" - actually we used 
to have "connsperthread" set to 1, but I reduced to 1000 when we 
moved to Linux, as I noticed that was the value in latest OpenACS 
config, even though the default is still 1 
https://bitbucket.org/naviserver/naviserver/src/main/openacs-config.tcl 
Is 1000 the recommended value for OpenACS?


The value of 1000 is probably a "traditional" value, which is forgiving 
for sloppy developers not cleaning up their garbage after a request. I 
have commented out the line in the repository such that users don't draw 
wrong conclusions.



-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] test ns_config-7.4.1, Stack around the variable 'filter' was corrupted

[Gustaf Neumann]

On 07.11.23 22:56, Andrew Piskorski wrote:

On the current NaviServer head, that problem has gone away!  The
ns_config-7.4.1 test now runs fine for me on Windows.


You see: time heals all wounds! -g
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] A selection of messages appearing in our error logs

[Gustaf Neumann]

Hi Brian,


In general, we try to follow with the logging severities the guideline 
[1]. "Notice" is informational, "warning" is something 
unexpected/unwanted (hitting some limits, got some data that looks like 
an attack, ...), "error" is something which should be looked at by an 
engineer.



    [1] https://openacs.org/xowiki/logging-conventions


below, I comment inline.



On 07.11.23 18:28, Brian Fenton wrote:

Hi

with the goal of improving signal-to-noise in our error logs, and also 
to increase my understanding of Naviserver internals, I have a few 
questions about various different messages we are seeing regularly in 
our logs.


 1. [06/Nov/2023:14:58:08][77320.7f7b9a7fd640][-driver:nsssl:0-]
Notice: ... sockAccept accepted 2 connections
this is by far the most common entry in our logs - we see a LOT of
these. Are they merely informative or indicative of some tuning
action we could take? I don't see any way to disable them. I would
love to remove these.



This indicates some stress situation of the server, where multiple 
requests come in at the same time. The value of 2 is not of a concern, 
if you see values like 10 or 20, you should consider a change in your 
configuration (e.g., configuring a second driver thread). It would be 
possible to define a configuration value for the network driver to set 
the threshold when the notice is written to the log file.




1.

2.
[06/Nov/2023:14:57:28][77320.7f7b63fff640][-conn:openacs:default:6:6292-]
Notice: ns_cache create entry collision cache util_memoize key
'__util_memoize_installed_p', no timeout
We also see a quite a few of these, always for
'__util_memoize_installed_p', and always with 'no timeout'. From
looking at cache.c I see  that this means that Naviserver failed
to create the cache entry. I checked the 'util_memoize' cache
utilization in case the cache was full, but it's quite low. What
other cause could there be for this?

This means that concurrent operations are executed to obtain the same 
cache value, and that the second request (here 
conn:openacs:default:6:6292) has to wait until the locking process has 
finished. The problem can become serious, when the locking process 
"hangs", i.e. taking a very long time to finish. This means, that more 
and more of caching requests for the same entry will pile up, 
potentially until a restart, since no timeout was specified. The warning 
was introduced to give the developer a hint, why suddenly the server is 
hanging (before the message was introduced, the cause was very hard to 
determine).



This message tells as well something else:
* It looks as if the installation is based on a very old version of 
openacs-core (including in particular acs-tcl) installed. At least since 
9 years, the code has changed to perform this test just at startup. 
Upgrade to recent versions will improve the performance and increase 
security.



* The message tells me as well that the application has a stress on the 
util-memoize cache. Old versions of OpenACS used this "util-memoize" 
cache as a kitchen-sink cache for everything. I've seen instances having 
300K + entries in this cache. The size itself is not the problem, but 
there were/are many operations in OpenACS that obtain all keys from the 
cache. It means that a lock is created, a list with 300K+ entries is 
created and handed to the interpreter before the cache is released. In 
the meantime, *all* caching requests for this cache have to be halted, 
... causing in high load situations degraded performance and limited 
concurrency. I would not be surprised, if you see high lock times on 
this cache.



There are caches aside of the util-memoize cache, where cache collisions 
might happen. See e.g. [2] for more background. Cache partitioning is in 
this situation the instrument for scaling.



   [2] https://openacs.org/xowiki/cache-size-tuning



1.
 2. These 5 appear usually as a group:
[07/Nov/2023:13:46:07][38.7f352cff9640][-conn:openacs:default:3:4088-]
Notice: exiting: exceeded max connections per thread
[07/Nov/2023:13:46:07][38.7f3525b84640][-driver:nsssl:0-] Notice:
NsEnsureRunningConnectionThreads wantCreate 1 waiting 1 idle 0
current 2
[07/Nov/2023:13:46:07][38.7f352d7fa640][-conn:openacs:default:5:0-]
Notice: start update interpreter openacs to epoch 1, concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-]
Notice: update interpreter openacs to epoch 1 done, trace none,
time 0.373006 secs concurrent 1
[07/Nov/2023:13:46:08][38.7f352d7fa640][-conn:openacs:default:5:0-]
Notice: thread initialized (0.388878 secs)
We see these pretty regularly, over 50 per day on  a  local
development server.  "maxconnections" is set to the default value,
which is 100, I believe.  "maxthreads" and "minthreads" are also
set to default values. What could  be causing these so regularly
on a system with 1 logged-in 

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf Neumann]

Hi Brian,


The proper solution is not to skip the error message, but to skip the 
full connection output operations in error situations. Since this 
involves many commands (from ns_return*, ns_write, ns_cookie* ...) this 
is a larger change. I will look into this in the next days. Should be 
doable with moderate effort.



all the best

-g

PS: bitbucket recovery still takes time, i am now at >15 email messages 
with support from support of atlassian and/or bitbucket.



On 06.11.23 15:11, Brian Fenton wrote:

Hi Gustaf

my apologies, I hadn't realised that silencing the log would lead to 
different behaviour. If this is a bigger job than expected, please 
feel free to revert back to previous version. It's a nice-to-have 
feature for us.


The attached script reproduces the issue when the parameter is set to 
false.


thanks
Brian


*From:* Gustaf Neumann 
*Sent:* Monday 6 November 2023 1:59 pm
*To:* naviserver-devel@lists.sourceforge.net 


*Subject:* Re: [naviserver-devel] NaviServer 4.99.29 available

Hi Brian,


as stated several times, the right action is to fix your script (as 
you did) rather than "silencing" NaviServer. I am not surprised, that 
attempts to write on detached connections can lead to error conditions 
on several occasions (generating errors avoids this).



But since we offer this silencing parameter, i do agree, the crashing 
is harsh. If you could send a short script triggering the problem it 
would help to work on such cases.



all the best

-g


On 06.11.23 14:25, Brian Fenton wrote:

Hi Gustaf

I just built and ran some tests on the "rejectalreadyclosedconn" 
parameter to see how it handles code that triggers the "connection 
socket is detached" error.


 If I set "rejectalreadyclosedconn"to false, and browse to a page 
that triggers the "connection socket is detached" error, Naviserver 
crashes with the following error message:


[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 
1030431 stream 00 chan (nil) fd -1
[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
Fatal: received fatal signal 11


If I then fix the code that was triggering the "connection socket is 
detached" error, by adding the missing "return" after the offending 
"ad_returnredirect", everything works fine.


Let me know if you need more info to help reproduce this.
thanks,
Brian

----
*From:* Gustaf Neumann  <mailto:neum...@wu.ac.at>
*Sent:* Thursday 2 November 2023 2:30 pm
*To:* Navidevel  
<mailto:naviserver-devel@lists.sourceforge.net>

*Subject:* [naviserver-devel] NaviServer 4.99.29 available
Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a potentiall serious memory leak 
when working with PostgreSQL and large text contents. Furthermore, 
the release contains a small enhancement as requested by Brian not 
very long ago on this list.


See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have 
lost write access to the repository at bitbucket. The people on 
Atlassian seem to have changed some account types, and - on to of 
this - they annouced via the Blog post on September 27, 2023, that 
the billing model changed (where they also refer to the 
"unified-user-management"). It took me a while to figure out, what 
happened. The blog post states:


/From October 31st, 2023, Bitbucket Cloud will begin counting all
workspace members as a billable user. 

Free plans: If you're on a free plan and your billable user count
is higher than 5 as per the new definition of billable user, all
repositories in your workspace will become read-only until you
remove some users or upgrade your workspace to a paid plan./

It seems that the users of the "naviserver" group are now counted as 
"billable users", and it contains 19 users. According to support, we 
have to reduce this number to 5, otherwise nobody will be able to 
commit anything.


Due to the ability with PRs, i think the reduction will be possible 
without too much loss in functionality. If nobody objects, i will go 
back in history and reduce the number of commit-member based on the 
most recent direct commits. I hope, that non of the "old members" 
will be offended by this. One other option would be to upgrade to a 
paid plain - but i am not sure, who is gonna pay for this.


All the best!

-gustaf neumann

[1] 
https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/ 
<https://sourceforge.net/projects/n

Re: [naviserver-devel] NaviServer 4.99.29 available

[Gustaf Neumann]

Hi Brian,


as stated several times, the right action is to fix your script (as you 
did) rather than "silencing" NaviServer. I am not surprised, that 
attempts to write on detached connections can lead to error conditions 
on several occasions (generating errors avoids this).



But since we offer this silencing parameter, i do agree, the crashing is 
harsh. If you could send a short script triggering the problem it would 
help to work on such cases.



all the best

-g


On 06.11.23 14:25, Brian Fenton wrote:

Hi Gustaf

I just built and ran some tests on the "rejectalreadyclosedconn" 
parameter to see how it handles code that triggers the "connection 
socket is detached" error.


 If I set "rejectalreadyclosedconn"to false, and browse to a page that 
triggers the "connection socket is detached" error, Naviserver crashes 
with the following error message:


[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
Warning: NsWriterQueue: called without sockPtr size 414 bufs 1 flags 
1030431 stream 00 chan (nil) fd -1
[06/Nov/2023:13:13:21][39.7f3489fb9640][-conn:openacs:default:1:30-] 
Fatal: received fatal signal 11


If I then fix the code that was triggering the "connection socket is 
detached" error, by adding the missing "return" after the offending 
"ad_returnredirect", everything works fine.


Let me know if you need more info to help reproduce this.
thanks,
Brian

--------
*From:* Gustaf Neumann 
*Sent:* Thursday 2 November 2023 2:30 pm
*To:* Navidevel 
*Subject:* [naviserver-devel] NaviServer 4.99.29 available
Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a potentiall serious memory leak when 
working with PostgreSQL and large text contents. Furthermore, the 
release contains a small enhancement as requested by Brian not very 
long ago on this list.


See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have 
lost write access to the repository at bitbucket. The people on 
Atlassian seem to have changed some account types, and - on to of this 
- they annouced via the Blog post on September 27, 2023, that the 
billing model changed (where they also refer to the 
"unified-user-management"). It took me a while to figure out, what 
happened. The blog post states:


/From October 31st, 2023, Bitbucket Cloud will begin counting all
workspace members as a billable user. 

Free plans: If you're on a free plan and your billable user count
is higher than 5 as per the new definition of billable user, all
repositories in your workspace will become read-only until you
remove some users or upgrade your workspace to a paid plan./

It seems that the users of the "naviserver" group are now counted as 
"billable users", and it contains 19 users. According to support, we 
have to reduce this number to 5, otherwise nobody will be able to 
commit anything.


Due to the ability with PRs, i think the reduction will be possible 
without too much loss in functionality. If nobody objects, i will go 
back in history and reduce the number of commit-member based on the 
most recent direct commits. I hope, that non of the "old members" will 
be offended by this. One other option would be to upgrade to a paid 
plain - but i am not sure, who is gonna pay for this.


All the best!

-gustaf neumann

[1] 
https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/ 
<https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/>
[2] https://bitbucket.org/blog/billing-model-change 
<https://bitbucket.org/blog/billing-model-change>


===
NaviServer 4.99.29, released 2023-11-01
===

  37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-
  - Eased configuration of simple setups
* don't require to specify a "defaultserver" when a single
  server is in use.

* reduce warnings for per-server network drivers. This is a not
  recommended but possible configuration, global network drivers
  should be used.

  - The configuration option "rejectalreadyclosedconn", which warns
about attempts to send data to the web client at times, when the
connection is not available anymore, is now applied on closed and
detached connections. Before it was only applied on closed
connections, causing potentially many warnings for legacy
applications.

Bug Fixes:
--

   - Fixed a potential memory leak introduced two releases ago (in
 4.99.27).

   - Fixed a potential compilation problem with glibc 2.38 or newer
 (released 31 Jul 2023)

   - Fixed r

[naviserver-devel] NaviServer 4.99.29 available

[Gustaf Neumann]

Dear all,

I am glad to announce that the release of NaviServer 4.99.29 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a potentiall serious memory leak when 
working with PostgreSQL and large text contents. Furthermore, the 
release contains a small enhancement as requested by Brian not very long 
ago on this list.


See below for a summary of the changes.

So far, these changes are only available on sourceforge, since i have 
lost write access to the repository at bitbucket. The people on 
Atlassian seem to have changed some account types, and - on to of this - 
they annouced via the Blog post on September 27, 2023, that the billing 
model changed (where they also refer to the "unified-user-management"). 
It took me a while to figure out, what happened. The blog post states:


   /From October 31st, 2023, Bitbucket Cloud will begin counting all
   workspace members as a billable user. 

   Free plans: If you're on a free plan and your billable user count is
   higher than 5 as per the new definition of billable user, all
   repositories in your workspace will become read-only until you
   remove some users or upgrade your workspace to a paid plan./

It seems that the users of the "naviserver" group are now counted as 
"billable users", and it contains 19 users. According to support, we 
have to reduce this number to 5, otherwise nobody will be able to commit 
anything.


Due to the ability with PRs, i think the reduction will be possible 
without too much loss in functionality. If nobody objects, i will go 
back in history and reduce the number of commit-member based on the most 
recent direct commits. I hope, that non of the "old members" will be 
offended by this. One other option would be to upgrade to a paid plain - 
but i am not sure, who is gonna pay for this.


All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.29/
[2] https://bitbucket.org/blog/billing-model-change

===
NaviServer 4.99.29, released 2023-11-01
===

 37 files changed, 261 insertions(+), 132 deletions(-)

New Features:
-
 - Eased configuration of simple setups
   * don't require to specify a "defaultserver" when a single
 server is in use.

   * reduce warnings for per-server network drivers. This is a not
 recommended but possible configuration, global network drivers
 should be used.

 - The configuration option "rejectalreadyclosedconn", which warns
   about attempts to send data to the web client at times, when the
   connection is not available anymore, is now applied on closed and
   detached connections. Before it was only applied on closed
   connections, causing potentially many warnings for legacy
   applications.

Bug Fixes:
--

  - Fixed a potential memory leak introduced two releases ago (in
4.99.27).

  - Fixed a potential compilation problem with glibc 2.38 or newer
(released 31 Jul 2023)

  - Fixed reloading of certificates for mass virtual hosting

Code Maintenance:
-

  - fixed typos
  - fixed enum/int conversion flagged by gcc13


Modules:

The following list contains the most important changes:

 - module nsdbpg:
   fixed memory leak (see above).
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Gustaf Neumann]

On 29.09.23 12:56, Brian Fenton wrote:

Hi

this discussion here 
https://openacs.org/forums/message-view?message_id=5659253 suggests 
that the "rejectalreadyclosedconn" parameter will prevent the 
"connection socket is detached" message from showing in the logs. 
However, this doesn't appear to work (tried in 4.99.28). Looking at 
"NsConnRequire" in conn.c, it looks like 
"reject_already_closed_connection" is only used in the "connection 
already closed" case. 
https://bitbucket.org/naviserver/naviserver/src/b21b6d75a8f2ff41cd00dd40cb4c143d9a8a55ce/nsd/conn.c#lines-2748


It would be good to be able to turn off the "connection socket is 
detached" (understanding of course that we should in the meantime 
resolve the source of those messages). Have I misunderstood something 
or is this not possible?


thanks
Brian


___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] "connection already closed" vs "connection socket is detached" and the "rejectalreadyclosedconn" parameter

[Gustaf Neumann]

Hi Brian,

The parameter "rejectalreadyclosedconn" does what it is supposed to do 
(controls error messages, when someone tries to write to a connection, 
which was already closed). The parameter was introduced at a time, 
before the distinction between closed and detached connections was 
introduced in the code. So, I read your request to extend the meaning of 
the parameter to cover detached connections as well.


> Have I misunderstood something or is this not possible?

There is no hard reason rendering this as impossible. One could either 
introduce (a) another parameter "rejectalreadydetachedconn" or (b) make 
"rejectalreadyclosedconn" slightly misleading from the name to cover as 
well the detached cases.


Since the parameter is just for legacy applications, which did not care 
about the semantics, when someone tries to write to a connection which 
is not available anymore (leading to potentially complex debugging 
questions), the updated code [1] follows the (b) approach. I see the 
potential misuse of the parameter to encourage lazy coding practices, 
but i do understand as well code-maintainers who see large amounts of 
errors in their log files on busy sites, not having the resources right 
now to address these...


-gn

[1] 
https://bitbucket.org/naviserver/naviserver/commits/725cf9475fe78f495918ed4549bf434cf2daaf96






___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NX and Naviserver init

[Gustaf Neumann]

Dear Maksym,

On 15.09.23 01:09, Maksym Zinchenko wrote:
Hello, I've been struggling with a problem for a few days now. First I 
thought it's something with my understanding of NX, but I think its 
has to do something with Naviserver.


actually, the problem is not with NaviServer, but it is rooted in the 
XOTcl/NX serializer. When a NaviServer blueprint is created, NaviServer 
performs "::Serializer all" to serialize the current workspace. 
Actually, it does not serialize the full workspace, since the workspace 
contains as well the NX internal class definitions. Therefore, the 
serializer has "ignore patterns", including everything from, e.g., the 
"::ns"f namespace [1].


When objects are created with "new", these are named by default 
"::nsf::__#1", "::nsf::__#2". These names match the ignore pattern, and 
therefore, these names are excluded from the blueprint.


If you create the objects in your code with the "-childof" flag (see 
below), you can place these outside the "::nsf" namespace, and 
everything should be fine


Hope this helps

-g


[1] 
https://github.com/gustafn/nsf/blob/master/library/serialize/serializer.tcl#L868



nx::Class create B {
:method init {} {
ns_log notice "Im class [self]"
}
}

nx::Class create C {
:method init {} {
ns_log notice "Im class [self]"
}
}

nx::Class create A {
:property -accessor public inBObj:object
:property -accessor public inCObj:object

:method init {} {
set :inBObj [B new -childof [::nsf::current object]]
set :inCObj [C new -childof [::nsf::current object]]
}

:public method myObjs {} {
ns_log notice  "My B obj is: [${:inBObj} info name]"
ns_log notice  "My C obj is: [${:inCObj} info name]"
}
}


___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NaviServer 4.99.28 available

[Gustaf Neumann]

Dear all,

Part of the announcement of NaviServer 4.99.28 was the update of the 
nsdbbdb module (Berkley DB driver via nsdb). I did a few tests about its 
performance, that might interest a few here.


The test of this module configured with lmdb is quite impressive (see 
below). For comparison, there are also accesses to nsv, ns_cache and 
similar included. All timings are from my notebook (Apple Silicon M1). 
This is just a micro-benchmark and measures buffered access through the 
NaviServer DB driver infrastructure. More detailed information about 
LMDB in comparison to Berkley DB concerning large Applications and more 
micro-benchmarks is in [1]


all the best

-gb

183 ns nsv_set foo x 1; time {nsv_get foo x} 10 213 ns time 
{ns_cache_eval ns:memoize 1 {set x 1}} 10 253 ns time {array set x 
{a 1 b 2 c 3}} 10 308 ns ns_urlspace set -key foo1 /*.adp A; time 
{ns_urlspace get -key foo1 /static/test.adp} 10 >>> 511 ns set ::db 
[ns_db gethandle lmdb]; time {ns_db 0or1row $::db "GET key1"} 10 
3633 ns time {parameter::get -package_id [ad_conn subsite_id] -parameter 
DefaultMaster -default "x"} 10 36893 ns time {xo::dc get_value 
dbqd..qn {select title from acs_objects where object_id=221}} 10 
42523 ns time {::mongo::collection::query $::mongoColl {name string 
Gustaf}} 10

Times are in nano seconds, the "slowest" is here the access to MongoDB (via nsf 
package) and PostgreSQL.
The PostgreSQL access takes 36 micro seconds, this means, one can run ~27K of 
those in a second.
But with LMDB it is possible to gets nearly 2 Mio DB accesses per second.

[1]http://www.lmdb.tech/media/20130329-devox-MDB.pdf
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] NaviServer 4.99.28 available

[Gustaf Neumann]

Dear all,

I am glad to announce that the release of NaviServer 4.99.28 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users.

See below for a summary of the changes.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.28/

===
NaviServer 4.99.28, released 2023-09-06
===

 49 files changed, 851 insertions(+), 328 deletions(-)

New Features:
-
  - None

Bug Fixes:
--

  - Fixed incorrect list quoting for "filter" in urlspace
introspection. This lead to display problems, e.g. in the nsstats
module.

  - Fixed naming Inconsistency: Unified the term "fastpath" across
configuration, API-interface, and documentation.

  - Fixed potential crash in Ns_SetIUpdateSz(): Fixed a bug that could
lead to a crash when updating the content-type header field with a
lowercase field name via the old C API.

Many thanks to Brian Fenton for reporting the problem and his help
for debugging it.

Code Maintenance:
-

  - New API Function Ns_PlainUrlPath: Checks if a given URL path is
"plain," meaning it doesn't contain a query or fragment. Useful
for data validation in configuration files and Tcl API calls.

  - Improved Validation when registering request handler. Previously,
it was possible to register request handlers for paths containing
query variables and fragments, which were never matched.

  - Added Missing "-server" Argument: The "-server" argument was added
to ns_unregister_op and ns_unregister_url2file, allowing for more
centralized management (e.g. from the nsstats module).

  - Added a more Portable Way of Printing NULL Pointer: Improved
portability for printing NULL pointers across different compilers,
which could lead to false positives in the regression test.

  - Extended and Simplified Test Cases: Several commits were made to
improve and extend test cases across different modules and
scenarios.

  - Improved and corrected documentation, including styling changes,
bug fixes, and providing more detailed information.


Modules:

The following list contains the most important changes:

 - Extending nsstats:

   * Introduced templating to improve style-ability
   * Provided a more ,odern GUI based on the CSS classes from w3schools
   * Added management of request handlers
   * Added management of url2file mappings

 - Rework of module nsdbbdb:
* properly initializing mutex variables (many thanks to John Buckman)
* support for LMDB (Lightning Memory-Mapped Database)
* Improved configurability for Makefile
* Modernization of C code (better following style guide)
* Added debug support via Debug(bdb)

 Build nsdbbdb with Berkeley DB 6.2.32 (results in nsdbbdb.so)
make DBINCLUDE=/opt/local/include/db62  DBLIB=/opt/local/lib/db62/

 Build nsdbbdb with LMDB (Lightning Memory-Mapped Database) (results in 
nsdblmdb.so)
make LMDB=1 DBINCLUDE=/opt/local/include DBLIB=/opt/local/lib/

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf Neumann]

On 05.09.23 14:40, John at Decent wrote:
Gustav, I think my confusion is as to how this works. As it's not 
documented, one needs to figure it out from the source.


the man page 
https://naviserver.sourceforge.io/n/nsproxy/files/ns_proxy.html is not a 
perfect documentation, but this is what we have now, ... and this is 
certainly more than reading the source. Improvements are always welcome!


I can't really tell what each ns_proxy::exec does if the executed 
process doesn't exit.


a worker process is potentially started by "ns_proxy get ..." but is 
influenced by the definition of the proxy pool (e.g. "maxworkers", ...).



Does it reuse it for the next request? It seems so, but I'm not clear.


by "request" you probably mean the next invocation of "nsproxy eval". It 
is reused, but as before, the reuse is controlled by the pool definition 
(e.g. "idletimeout", "maxruns", ...)


If it does reuse the still-running process, how does it tell it’s the 
same executable?


the name of the executable (per default 
/usr/local/ns/bin/nsproxy-helper) is defined via config file or via  
"nsproxy configure ... -exec XXX". The running instances are managed via 
the handles. If you want to have different executables, define different 
nsproxy pools.


-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf Neumann]

On 05.09.23 11:39, Gustaf Neumann wrote:

On 05.09.23 09:17, John at Decent wrote:


However, won’t that cause the external process to get reloaded every 
time I call proxy::exec ?

see

https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl#L74-L83

in the link i have sent in my last mail


Maybe there is some confusion with the term "reloaded"
Check also the "-maxuns" parameter...
Or do you want to have - technically speaking - one fork per request? 

-g
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf Neumann]

On 05.09.23 09:17, John at Decent wrote:

Thanks Gustaf.

I knew about ns_proxy, but as it is documented as being a proxy to 
external Tcl processes, I didn’t think to use it as a general process 
pool.


The part I was missing was
> proxy::exec

However, won’t that cause the external process to get reloaded every 
time I call proxy::exec ?

see

https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl#L74-L83

in the link i have sent in my last mail

-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] “worker process pool” feature?

[Gustaf Neumann]

On 05.09.23 08:10, John at Decent wrote:
I think the best way is to have a naviserver module that implements a 
“worker pool of processes”.  The module would launch X numbers of this 
external process (say, undroidwish), controlled via STDOUT/STDIN, and 
dispatch requests free members of the pool, returning the result to 
the caller.  This would be an efficient way to have slow-loading Unix 
processes brought into Naviserver efficiently.


The NaviServer module nsproxy does exactly this, i.e. running processes 
in the background talking to NaviServer via pipes, providing queues, 
cancellation, . The processes, which are executed via the nsproxy 
module, are the "nsproxy-helpers", which are executing the incoming Tcl 
commands and return the results back via pipe, such that for a caller, 
it looks like a local "exec". The connection to the workers is 
established via handles.


See [1] for an example, how nsproxy is used for implemented "exec" in 
NaviServer, see [2] for the nsproxy API and configuration options.


-g

[1] 
https://github.com/openacs/openacs-core/blob/oacs-5-10/packages/acs-tcl/tcl/proxy-procs.tcl

[2] https://naviserver.sourceforge.io/n/nsproxy/files/ns_proxy.html

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf Neumann]

> FYI, also the "ns_set put" crashes the server e.g. if I run this in 
/ds/shell:



it is clear. The problem were not the "ns_set update" or the "ns_set 
put" commands itself, but the fact, that the key "content-type" was in 
lower-case and the normalization of the output headers was causing the 
crash. Some ages ago, it was possible to set two different header values 
with upper and lower case keys, leading to some unspecified behaviors in 
browsers.




On 18.08.23 15:55, Brian Fenton wrote:

Hi again Gustaf

Great news, this seems to have worked according to all my tests. I am 
assuming that I built this correctly with the "version_ns=GIT" parameter:
version_ns=GIT ns_modules="nsconf nsstats" with_postgres=0 
with_postgres_driver=0 bash install-ns.sh build



by your parameter list, you will get the newest version from git 
which is the unreleased version of NaviServer 5. If one specifies the 
branch as well, different versions can be installed from git. E.g.,


    ... version_ns=GIT git_branch_ns=release/4.99 

installs the newest version from the release/4.99 branch, what you 
probably want.

But one can use as well arbitrary tags. For example

    ... version_ns=GIT git_branch_ns=naviserver-4.99.16  

will checkout the released version of NaviServer 4.99.16 via git.


Do you have an idea when this will be included in an official 
Naviserver release? Our Docker build relies on the official releases, 
so it would be good to have a timeframe on that.



When you use the docker image from oupfiz5, these are still based on 
4.99.24 (released on 2022-06-14, a year ago).



Anyhow, when do you need a release?


all the best

-g




thanks again for the amazing support,
Brian

--------
*From:* Gustaf Neumann 
*Sent:* Friday 18 August 2023 12:30 pm
*To:* naviserver-devel@lists.sourceforge.net 

*Subject:* Re: [naviserver-devel] Crashing on all versions >4.99.24 on 
Ubuntu

Hi Brian,

With your input, I could locate the source of the problem and fix this
in the repository. It was an ns_set but, that could only happen with the
output headers, when the code normalized the capitalization of the
header fields. There is now a test for this in the regression test suite
of NaviServer.

Many thanks for your patience and input!

-g

https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997 
<https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997>






___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel 
<https://lists.sourceforge.net/lists/listinfo/naviserver-devel>



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf Neumann]

Hi Brian,

With your input, I could locate the source of the problem and fix this 
in the repository. It was an ns_set but, that could only happen with the 
output headers, when the code normalized the capitalization of the 
header fields. There is now a test for this in the regression test suite 
of NaviServer.


Many thanks for your patience and input!

-g

https://bitbucket.org/naviserver/naviserver/commits/4f8cf8a548bc60f88756a30f33f6a5b589fc6997





___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf Neumann]

On 16.08.23 15:37, Brian Fenton wrote:
We don't issue any "ns_set cleanup" ourselves, but when I added your 
recommendation of tracing "ns_set", I saw plenty in the logs. The docs 
say that "This command is autoamtically executed by ns_cleanup, which 
runs after every request, freeing all sets created via *ns_set*", so I 
presume that is where the calls come from.


Hi Brian,

Yes, it is clear from the log that the crash happens in the automatic 
cleanup, but I just wanted to make sure, that the application has not 
tried the same earlier - this would be a good place to start debugging.


Is there something in particular I should be looking for in the trace 
output? There is an enormous amount of information the logs.


see below


I ran another backtrace and this time the error was a little bit 
different
as i said before, the messages one gets from the operating system are 
not always helpful to find the source of the problem.



("error: Cannot access memory at address 0x2" - see below).


This is actually not a message of the operating system, but a message 
from gdb about an unused C-structure, which is here not relevant.


... but we are getting closer.  The crash happens on an dynamic ns_set 
(in yesterday's mail in "d6").


#11 0x77f3e119 in NsTclSetObjCmd
(clientData=0x7fffdc035570, interp=0x7fffdc005250, objc=2,
objv=0x7fffdc245650) at tclset.c:330

        key = 0x7fffdc3aa070 "d6"

In this case, one has to watch out for operations on "d6". You might ask 
"What is a dynamic ns_set?".


Background: ns_sets might be

- C-only structures, or
- Tcl-exposed structures, which might be
   * dynamic (i.e., freed after every request, the Tcl-name of these 
starts with a "d"), or

   * static (the structures will be reused).

The Tcl-exposed ns_sets are "entered" (NaviServer terminology) into Tcl 
interpreters. It might be the case, that you do not find anything 
interesting in the log concerning the crashing ns_set (here "d6") in the 
Tcl trace output, ... since the ns_sets are as well manipulated from C.


In case, you find nothing revealing on "d6", I've added more debugging 
output for C-level ns_set operations. Get the newest version from the 
branch "release/4.99", activate debug output on Debug(nsset)


    ns_logctl severity Debug(nsset) on

and then you will see some more output, like e.g. the following:

[16/Aug/2023:19:53:13.016627][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [29] key 'item_id' value '1066' size -1
[16/Aug/2023:19:53:13.016630][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [30] key 'revision_id' value '163776' size -1
[16/Aug/2023:19:53:13.016633][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [31] key 'publish_date' value '2021-11-13 
13:35:27.423904+01' size -1
[16/Aug/2023:19:53:13.016637][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [32] key 'modifying_user' value '' size -1
[16/Aug/2023:19:53:13.016640][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [33] key 'last_modified' value '2017-08-08 
13:26:49.138414+02' size -1
[16/Aug/2023:19:53:13.016644][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [34] key 'modifying_ip' value '::1' size -1
[16/Aug/2023:19:53:13.016647][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [35] key 'modifying_user' value '704' size -1
[16/Aug/2023:19:53:13.016651][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetPut 0x62845600 [36] key 'parent_id' value '1064' size -1
[16/Aug/2023:19:53:13.016655][-conn:oacs-head:default:0:11-] Debug(nsset): 
EnterSet 0x62845600 with name 'd15'
[16/Aug/2023:19:53:13.016678][-conn:oacs-head:default:0:11-] Notice: trace: 
ns_set array d15

...
[16/Aug/2023:20:04:31.283342][-conn:oacs-head:default:0:11-] Debug(nsset): ns_set 
cleanup key  dynamic 1
[16/Aug/2023:20:04:31.283344][-conn:oacs-head:default:0:11-] Debug(nsset): 
Ns_SetFree 0x62845600 'db': elements 37
[16/Aug/2023:20:04:31.283346][-conn:oacs-head:default:0:11-] Debug(nsset): ... 0: key 
 value <::xowiki::Form>
[16/Aug/2023:20:04:31.283349][-conn:oacs-head:default:0:11-] Debug(nsset): ... 1: key 
 value <1060>
[16/Aug/2023:20:04:31.283351][-conn:oacs-head:default:0:11-] Debug(nsset): ... 2: key 
 value <704>
...

In this example output, there is a bunch of Ns_SetPut() operations on a 
set "0x62845600"  which is entered then to Tcl as set "d15" 
somewhere later it is freed via "ns_set cleanup".


Let me know if you need some help on getting/compiling a branch of 
NaviServer from Bitbucket.


all the best

-g



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf Neumann]

Brian,


many thanks, the backtrace gives some insights:


The problem happens in a Ns_SetFree operation if set "d8"
triggered by an "ns_set cleanup" during the cleanup of the request.


Something is broken with this nsset.
Can it be that your application package issues "ns_set cleanup" as well?


Do you have a recent version of OpenACS? If so, you can turn on tracing
of the "ns_set" command by adding it to the "traced_cmds".


-g



=
--- acs-tcl/tcl/tcltrace-init.tcl   27 Nov 2020 09:52:06 -  1.4.2.4
+++ acs-tcl/tcl/tcltrace-init.tcl   15 Aug 2023 17:13:51 -
@@ -32,6 +32,7 @@
 #set traced_cmds {::ns_setcookie ::ns_getcookie ::ns_deletecookie}
 #set traced_cmds {::ns_return ::ns_returnnotfound ::ns_returnfile 
::ns_returnmoved}
 #set traced_cmds [lsort [info commands ::ns_return*]]
+set traced_cmds {::ns_set}
 foreach cmd $traced_cmds {
 append trace "\ntrace add execution $cmd  enter {::tcltrace::before}"
 }
=



On 15.08.23 15:51, Brian Fenton wrote:

Hi

I reproduced the problem using the install-ns.sh script running under 
gdb. Here's the output of backtrace and bt full. I'm new to using gdb 
so please let me know if you'd like to see some other info.
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf Neumann]

On 11.08.23 20:15, Brian Fenton wrote:

Hi Gustaf

thanks for the response. I've been looking at this in more detail this 
afternoon and it does appear to be caused by something in the 
interaction of our OpenACS application with 4.99.27. As I previously 
mentioned, it has been running fine on 4.99.24 on the same Ubuntu 
version. I realise that I may not have been clear on this point on my 
previous email: this is Naviserver running on Ubuntu in a Docker 
container. The version of Naviserver is based on this Docker build 
https://github.com/oupfiz5/naviserver-s6 
<https://github.com/oupfiz5/naviserver-s6> which I have forked and 
updated to 4.99.27 (I may well have missed something in updating NS 
version - maybe I should have waited until oupfiz updates his build).


  * I can confirm that nsd-config.tcl runs fine with 4.99.27
  * Some good news: I am able to do an OpenACS clean install on Oracle
with 4.99.27. I then successfully installed our application using
the APM.


did you run at this state any Oracle queries?
did you recompile in the "clean install" also the oracle driver?


  * However, once I restart Naviserver the problems start.
  * I tried using the openacs-config.tcl that ships with 4.99.27 and
the problems are happening with that too.

you mean the crash happens in the plain openacs-config.tcl, with no 
additional drivers etc, no oracle involved?
this can get us closer to something i might be able to reproduce. My 
request in the last mail was to try to reproduce the problem with 
nsd-config.tcl (i.e. no OpenACS involved). If you can reproduce the 
crash, you should compile with debugging turned on and run nsd under gdb 
or lldb. First one should get he most simple case causing the crash.



What is odd is that it seems to be able to handle one request before 
crashing. Eg. I type in the URL, it shows the /register page but then 
crashes. After restarting, I enter my login details on the register 
page, press return. It then crashes. After restarting, it successfully 
logs me, then crashes again.
the memory errors or normally hinting on some buffer overflow, or a 
mixture between 32bit and 64bit compilation, etc.


There is no clear pattern in the logs. I thought it might be related 
to OCSP and disabled that, but the problems continued to occur.
if you suspect nsssl, then one potential problem might be a mixture 
during of different OpenSSL versions during compilation (when using 
install_ns.sh, this will not happen).
Turning on debug hasn't helped - but maybe there is so much 
information in the log that I have missed something important.


What drivers are you referring to in your question?


actually all naviserver modules you are using, including the db drivers 
(since you mentioned nsoracle, which is not part of the regular 
regression tests). To be on the safe side, all /usr/local/ns/bin/*.so 
files should be newly compiled.



all the best

-gn



thanks
Brian

----
*From:* Gustaf Neumann 
*Sent:* Thursday 10 August 2023 7:27 pm
*To:* naviserver-devel@lists.sourceforge.net 

*Subject:* Re: [naviserver-devel] Crashing on all versions >4.99.24 on 
Ubuntu


Hi Brian,


The new NaviServer versions are running fine on Ubuntu 22.04. Have you 
recompiled the drivers you are using with the updated version?



A good test for the NaviServer binary is to test it with one of the 
packaged configuration files, e.g. nsd-config.tcl.



all the best

-gn


On 10.08.23 18:23, Brian Fenton wrote:

Hello

we have been testing out our OpenACS application on Ubuntu 22.04.2 
LTS (previously we only ran on Windows). It was working great with 
Naviserver 4.99.24 but I have been getting constant crashes on more 
recent versions.


I get this error on 4.99.25, 4.99.26 and today I also got it on 
4.99.27. The server runs fine until I click on a page, then it 
immediately crashes.

The log has only the following error:
free(): invalid size

and today I got this one:
[10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] 
Fatal: received fatal signal 11


We have an Oracle application and are using the latest nsoracle 
driver, which might be a factor here.
We have been running it with a pretty old OpenACS config file, so I 
am currently looking to merge in all the latest changes to ensure 
that is not an issue.
Also note that I am running Naviserver on Docker on Windows, but as 
mentioned it was running great on 4.99.24.


thanks for any help
Brian




___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
___
naviserver-devel 

Re: [naviserver-devel] Crashing on all versions >4.99.24 on Ubuntu

[Gustaf Neumann]

Hi Brian,


The new NaviServer versions are running fine on Ubuntu 22.04. Have you 
recompiled the drivers you are using with the updated version?



A good test for the NaviServer binary is to test it with one of the 
packaged configuration files, e.g. nsd-config.tcl.



all the best

-gn


On 10.08.23 18:23, Brian Fenton wrote:

Hello

we have been testing out our OpenACS application on Ubuntu 22.04.2 LTS 
(previously we only ran on Windows). It was working great with 
Naviserver 4.99.24 but I have been getting constant crashes on more 
recent versions.


I get this error on 4.99.25, 4.99.26 and today I also got it on 
4.99.27. The server runs fine until I click on a page, then it 
immediately crashes.

The log has only the following error:
free(): invalid size

and today I got this one:
[10/Aug/2023:15:02:23][303.7fa3a64ee640][-conn:openacs:default:1:119-] 
Fatal: received fatal signal 11


We have an Oracle application and are using the latest nsoracle 
driver, which might be a factor here.
We have been running it with a pretty old OpenACS config file, so I am 
currently looking to merge in all the latest changes to ensure that is 
not an issue.
Also note that I am running Naviserver on Docker on Windows, but as 
mentioned it was running great on 4.99.24.


thanks for any help
Brian
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] nswebpush & "invalid JWT provided"

[Gustaf Neumann]

Many thanks, David, for figuring this out!

Many thanks, David, for figuring this out!

The change is incorporated in the nswebpush module on Bitbucket.
Against my own rules, I've updated the just released tar file for the 
modules


    naviserver-4.99.27-modules.tar.gz

to include this change.

all the best!

-g

On 09.08.23 12:19, David Osborne wrote:

Thanks Gustaf - replies inline...

On Wed, 9 Aug 2023 at 10:38, Gustaf Neumann  wrote:

Hi David,

We do not have nswebpush somewhere in production. Can you tell
more precisely, what "suddenly" means?

About lunchtime on 2nd Aug!

Does this mean, that you have not changed anything in your
environment, but google started to refuse it?

Yes exactly...

We've worked out what was angering Google - it was a version of this 
code in our case:

https://bitbucket.org/naviserver/nswebpush/src/1e412c76626b29a4573b595a069a8ea10feece8a/webpush-procs.tcl#lines-607

Construction of the json from the claim dict was treating "exp" as a 
string rather than numeric.
Just as an illustration, this quick hack makes the "make test" run 
cleanly in the nswebpush codebase:


    proc dictToJson {dict} {
        #
        # Serializes a Tcl dict to compact JSON.  No testing for
        # nested dicts or arrays, these will be simply added as a
        # string the JSON is in compact form, meaning no whitespaces
        # and newlines between keys/values.

        set pairs {}
        dict for {key value} $dict {
            regsub -all \" $key "\\\"" key
            regsub -all \" $value "\\\"" value
            if { $key eq "exp"} {
                lappend pairs [subst {"$key":$value}]
            } else {
                lappend pairs [subst {"$key":"$value"}]
            }
        }
        return "{[join $pairs ,]}"
    }




___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] NaviServer 4.99.27 available

[Gustaf Neumann]

Dear all,

I am glad to announce that the release of NaviServer 4.99.27 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users.

See below for a summary of the changes.

Furthermore, the online documentation at sourceforge

https://naviserver.sourceforge.io/n/toc.html

is now more mobile-friendly and contains a simple version switcher for the
stable release branch (4.99) and the main branch, which will be released 
as 5.0.


All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.27/

===
NaviServer 4.99.27, released 2023-08-09
===

 46 files changed, 739 insertions(+), 172 deletions(-)

New Features:
-
 - None

Bug Fixes:
--

 - Bug fix: fixed potential crash when fallbackCharset is not defined
   in the configuration file (using outdated configuration)

 - Bug fix for fastpath in connection with "ns_serverrootproc"

   Earlier versions of NaviServer initialized for fastpath the
   interpreter with the connection conditionally, and rather late.
   When "ns_serverrootproc" is configured, the interpreter with its
   linkage to the connection is needed very early in the request.  Now
   it is ensured, that the interpreted is registered for the
   connection when the callback is issued.

   NaviServer 5 will have more changes in this respect.

   Many thanks to Georg Lehner for reporting this bug and testing.

 - Fixed various spelling errors in source code and documentation

 - Documentation:
* fixed documentation bugs
* modernized examples

 - Ease debugging, how values for "ns_conn location" are determined

Configuration Files:


- Added sample section for charset mapping to sample-config.tcl

Modules:

The following list contains just bug fixes, new features will
be documented with NaviServer 5.0.

 - nsdbpg:
* Fixed potential crash in Tcl, when Database contains UTF-8
  characters invalid to Tcl 8.6. Crash was observed in
  "string tolower $var".
* Fixed potential crash, when bind variables are passed via
  explicit "-bind" option, but variable binding was missing

 - nsoracle:
* Reduced verbosity: For debug messages of the driver
  implementation, use again "Debug" severity to avoid too much
  verbosity, when looking for slow queries ("Debug(sql)" severity
  of nsdb).

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] nswebpush & "invalid JWT provided"

[Gustaf Neumann]

Hi David,

We do not have nswebpush somewhere in production. Can you tell more 
precisely, what "suddenly" means?
Does this mean, that you have not changed anything in your environment, 
but google started to refuse it?


The implementation in nswebpush uses for JWT the algorithm ES256 (based 
on elliptic curves), which seems not supported by google cloud 
endpoints, whereas [2] uses ES256, there is as well support in 
firebase/php-jwt [3]. Not sure, where to start to look for helping you.


-g

[1] 
https://cloud.google.com/endpoints/docs/frameworks/python/troubleshoot-jwt?hl=en
[2] 
https://cloud.google.com/iap/docs/signed-headers-howto?hl=en#securing_iap_headers

[3] https://github.com/firebase/php-jwt/blob/main/src/JWT.php

On 08.08.23 17:32, David Osborne wrote:

Hi there,

We have a chat implementation based on the Naviserver nswebpush module 
which recently stopped working with Google endpoints (eg. 
https://fcm.googleapis.com/fcm/send...).

Suddenly it's complaining about invalid JWTs.

We went back to reference the nswebpush code.
https://bitbucket.org/naviserver/nswebpush/src/main/
We installed it on a clean Debian Bullseye server with 
latest Naviserver from bitbucket.


When we ran the "make test" we also get a 403 from Google... more 
specifically, the reply was:
Webpush failed with reply status 403 time 0:88018 headers d8 body 
{invalid JWT provided } https {sslversion TLSv1.3 cipher 
TLS_AES_256_GCM_SHA384}


Is anyone else experiencing this or can make any suggestions as to 
what has changed?


--

*David
*
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Forthcoming OpenACS + EuroTcl/Tk conference

[Gustaf Neumann]

Dear all

At the forthcoming OpenACS + EuroTcl/Tk conference (details below),  we 
are considering a "Wishlist" session, where users can place their 
needs/demands, such we can consider this with higher priority in the 
development. This also covers NaviServer.


If you have something on your mind, please send a short email with such 
topic to me. We might discuss these with the community at the conference.


All the best

-gn


   June 30th, 2023: Registration ends
   July 19th, 2023: Meet & greet
   July 20th - July 21st, 2023:  Conference

For details, see:https://openacs.org/conf2023/

--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Gustaf Neumann]

On 17.06.23 19:08, Sassy Natan wrote:

Is there any way to keep it as it was?
So encoding convertfrom utf-8 will work like in the past?


The "encoding convertfrom" has not changed, but its output is now 
properly encoded.


The following will probably work for you: add the following section to your
configuration file (and for other extensions, when you need this).

   ns_section ns/encodings {
   ns_param .adp iso8859-1
   }

I am not sure, what this will mean for Tcl9 though.

all the best

-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NaviServer/4.99.16 vs NaviServer/4.99.25

[Gustaf Neumann]

Dear Sassy,

Why do you say that the output right next to text utf-8 is broken?

You simply do not need the extra [encoding convertfrom utf-8 ... ] in 
versions after 4.99.23 in text pages.The UTF-8 handling in earlier 
versions was broken.


The changes in 4.99.23 were triggered by requests of full emoji support 
(in UTF-8 multi-byte characters, up to 4 bytes). The changes effect  
also the DB-Interface, such that now such multi-byte characters are 
fully supported in display and storing. Look e.g.: 
https://openacs.org/xowiki/text-with-hebrew


> One more thing to note is that the log file which also has UTF-8 
characters is not translated. I see the escape characters instead of the 
data.


This is exactly the same answer as in you earlier question. There are 
several known attacks based on Unicode, which can as well be used to 
trick the person reading the log-file.


If you do not care about the log file sanitizing and you prefer the old 
behavior, turn sanitizing off.


all the best

-gn



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] NaviServer 4.99.26 available

[Gustaf Neumann]

Dear all,

I am glad to announce that the release of NaviServer 4.99.26 is 
available at SourceForge [1].  This release is a pure bug-fix and 
maintenance release, which fixes a bug annoying for some OpenACS users 
(wrong value when hostname is retrieved from the configuration values 
via "ns_config ...").

See below for a summary of the changes.

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.26/

===
NaviServer 4.99.26, released 2023-06-16
===

 18 files changed, 109 insertions(+), 64 deletions(-)

New Features:
-
 - None

Bug Fixes:
--

 - Bug fix: don't update the value of an ns_set entry using the
   identical string.

   Since the update of an ns_set entry requires a free operation of
   the old memory and a copy operation of the new value, in case the
   pointers are identical, the new value might be freed in the first
   step, leading to an invalid value. This condition can happen only
   from the C level API.  Unfortunately, this happened with the sample
   configuration file for OpenACS.

 - Increase portability for older compilers and windows.
   NaviServer 4.99.* uses now same idiom for NS_BITELEMENTS
   as in NaviServer 5

 - improved error message on socket errors (reported unknown error
   before)

Configuration Files:

 - Modernize recommended cipher lists in OpenSSL configuration
   files
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] NaviServer 4.99.25 available

[Gustaf Neumann]

Dear Sassy,

You are referring to a change that was released more than 3 years ago 
(4.99.19).


Log file sanitizing works as expected, namely it prints invisible 
characters in a hex representation (in your case tab characters). 
Sanitizing was required by security  audits, since otherwise, it would 
be possible to execute code by looking into the log file, one could 
obfuscate the log file, and confuse log file analyzer that alarm when 
suspect activities are noticed, etc.


Certainly, the harm caused by a tab character alone is limited, but when 
debugging and analyze problems, it is often important to distinguish 
between a tab and some spaces.


You can easily replace the hex notation in the log file via "sed", such 
as e.g.


   cat log/error.log | sed -En 's/\\x09/\t/p' | more

or the like.

all the best

-gn

On 13.06.23 20:51, Sassy Natan wrote:

Hi All,

After upgrading my server to the latest version, my log file is broken.

For example I see UTF-8 as special characters.

I check the readme but the sanitizelogfiles 2 doesn't seems to work as 
expected.

I have in my configuration:

set        debug                                true
ns_logctl  severity "Debug(sql)"                on

Any ideas?

Here is example:
:    ],is_break_pcols)) as hrs_wo_break
:    FROM wt_payroll_analysis_pp
:     join (select sid,agg_array(case when is_break_pcol=true then 
9 else 0 end) as is_break_pcols
:    \x09\x09\x09from ( select * from wt_et_cols_defs where sid=41 
 order by  pcol_number )pc_defs group by sid) pdefs using (sid)

:    \x09\x09\x09\x09WHERE sid=41 and employee_id =40599
:    \x09\x09\x09\x09and 
 to_date(pp_year||'-'||pp||'-01','-mm-dd')  between 
to_date('2023-6-01','-mm-dd')- interval '6 month'
:    \x09\x09\x09\x09\x09\x09\x09and ( 
to_date('2023-6-01','-mm-dd')  - interval '1 day' )::date

:    \x09\x09\x09\x09UNION
:    \x09\x09\x09\x09 select 
sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','-mm-dd') 
as day,null2zero(hrs_wo_break)*3600

:    \x09\x09\x09\x09                                from pp_pa_table
:    \x09\x09\x09\x09) ggg
:    \x09\x09\x09\x09group by  sid,employee_id,pp,pp_year


Thanks
Sassy

On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann  wrote:

Dear all,

I am glad to announce that the release of NaviServer 4.99.25 is
available at SourceForge [1].  This release is mostly a bug-fix
release. The forthcoming version 5.0 of NaviServer will contain
several new features omitted in this bug-fix branch. In case, you are
building NaviServer from the Bitbucket repository, please note that
the release 4.99.25 is in the branch release/4.99 (bug fix branch for
the NaviServer 4.99 family). New development happens in the "main"
branch of the repository, leading to NaviServer 5.*.

See below for a summary of the changes.

Many thanks to the contributors of this release:

 Andrew Piskorski
 Antonio Pisano
         Brian Fenton
 Gustaf Neumann
 Hector Romojaro
 Joe Oldak
 Khy Huang
 Oleg Oleinick
 Zoran Vasiljevic

    All the best!

-gustaf neumann

[1]
https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/

===
NaviServer 4.99.25, released 2023-05-01
===

  132 files changed, 3957 insertions(+), 2068 deletions(-)

New Features:
-

  - Added meta-information to configuration values

NaviServer can now report, what configuration values provided in
the configuration file were actually used, what their default
values are, and whether these values were specified or not (using
the default values). With this information, administration
(e.g. migration) becomes easier. The NaviServer module "nsstats"
shows this meta information via the web interface.

This functionality is provided via the new option "-filter" for the
command "ns_configsection ... section". When the "-filter" option
is used, different kinds of information about the parameters is
returned from the specified section.

 "-filter unread":

 Returns the parameters, which were set during configuration (i.e.,
 in the configuration file) but which were not read in the startup
 phase of the server. This option is useful to determine
 e.g. typographical errors of specified parameter names.

 "-filter defaulted":

 Returns the parameters, from which the default values were read
 (i.e., which were not explicitly set)

 "-filter defaults":

 Returns the defaults of the parameter. This is useful for
 contrasting the actual values with the default values for
   

[naviserver-devel] Joint OpenACS and EuroTcl Conference 2023

[Gustaf Neumann]

Dear all,

The deadline for submissions of abstracts for presentation at the 
forthcoming joint OpenACS / EuroTcl conference is in about one month. 
Presentations about NaviServer applications are very welcome.


Important dates:

    June 15th, 2023: Deadline for submissions of abstracts (max. 2 
pages, min. 250 words);

    June 19th, 2023: Notification of acceptance
    June 30th, 2023: Registration ends
    July 19th, 2023: Meet & greet
    July 20th - July 21st, 2023:  Conference

For details, see: https://openacs.org/conf2023/

Gustaf Neumann and Harald Oehlmann

--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] NaviServer 4.99.25 available

[Gustaf Neumann]

Dear all,

I am glad to announce that the release of NaviServer 4.99.25 is
available at SourceForge [1].  This release is mostly a bug-fix
release. The forthcoming version 5.0 of NaviServer will contain
several new features omitted in this bug-fix branch. In case, you are
building NaviServer from the Bitbucket repository, please note that
the release 4.99.25 is in the branch release/4.99 (bug fix branch for
the NaviServer 4.99 family). New development happens in the "main"
branch of the repository, leading to NaviServer 5.*.

See below for a summary of the changes.

Many thanks to the contributors of this release:

 Andrew Piskorski
 Antonio Pisano
 Brian Fenton
     Gustaf Neumann
 Hector Romojaro
 Joe Oldak
 Khy Huang
 Oleg Oleinick
 Zoran Vasiljevic

All the best!

-gustaf neumann

[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/


===
NaviServer 4.99.25, released 2023-05-01
===

 132 files changed, 3957 insertions(+), 2068 deletions(-)

New Features:
-

 - Added meta-information to configuration values

   NaviServer can now report, what configuration values provided in
   the configuration file were actually used, what their default
   values are, and whether these values were specified or not (using
   the default values). With this information, administration
   (e.g. migration) becomes easier. The NaviServer module "nsstats"
   shows this meta information via the web interface.

   This functionality is provided via the new option "-filter" for the
   command "ns_configsection ... section". When the "-filter" option
   is used, different kinds of information about the parameters is
   returned from the specified section.

"-filter unread":

Returns the parameters, which were set during configuration (i.e.,
in the configuration file) but which were not read in the startup
phase of the server. This option is useful to determine
e.g. typographical errors of specified parameter names.

"-filter defaulted":

Returns the parameters, from which the default values were read
(i.e., which were not explicitly set)

"-filter defaults":

Returns the defaults of the parameter. This is useful for
contrasting the actual values with the default values for
Parameters, e.g. in a web based interface.

 - ns_set reform (per default deactivated in 4.99, but activated in 5.*)

   The classical implementation for ns_sets uses separately malloced
   storage for every attribute name and attribute value. So, e.g., for
   1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000
   malloc/free operations, e.g., for a single db query! Although the
   malloc implementations have improved over the years, these will
   require many lock operations, especially under load, where many
   threads might perform many concurrent malloc operations. One other
   consequence is that the allocated memory will be scattered over
   address space, which has bad implications for CPU caching.

   The new implementation uses for one "ns_set" a single Tcl_DString
   keeping all attribute names and attribute values. This reduces the
   malloc operations and improves memory locality, such that cache
   hits will improve.

   One caveat of this change is that modules using "ns_set" have to be
   recompiled, since the full C-level data structure of the "ns_set"
   is exposed. Therefore, adding a member causes a binary
   incompatibility. One other potential problem is that C-level
   modules using the Ns_Set* API have to make sure that long-living
   string values are copied (this was necessary before as well, but
   was in many cases no problem, when the "ns_sets" were seldom
   updated).

   For high compatibility, this feature is deactivated per default in
   the 4.99.* series and can be activated by setting the compile-time
   C macro "NS_SET_DSTRING".



API changes:


   API extensions:
- Provide a new interface ending with *Sz to provide string sizes.
  This reduces the need of strlen() operations.
  * Ns_SetCreateSz()
  * Ns_SetIUpdateSz()
  * Ns_SetPutSz()
  * Ns_SetPutValueSz()
  * Ns_SetUpdateSz()

- New API calls for "ns_set" reform
  * Ns_SetClearValues(): clear the values for all keys
  * Ns_SetDataPrealloc(): creating ns_sets with preallocated values
to avoid resize operations
  * NsSetResize()
  * NsHeaderSetGet()

- Ns_ConfigSet(const char *section, const char *key, const char *name)
  The last argument is new and allows one to create named sets
  (previously, all such sets were unnamed)

- NsHexPrint(): Print the potentially binary content of a buffer
  in human-read

Re: [naviserver-devel] Some questions about Docker images for Naviserver

[Gustaf Neumann]

Here is some background, what's good about s6-overlay:
https://kreuzwerker.de/post/managing-multi-process-applications-in-containers-using-s6

In short: it is sufficiently simple to use and has a couple of useful 
utilities...


-g

On 31.03.23 14:23, Maksym Zinchenko wrote:
Hello, I'm using Docker Naviserver in production, but it's not a big 
one. I can recommend it. In my case I'm using only Naviserver, not 
OpenACS. I have some images on docker hub under siqsuruq/naviserver, 
it's not official, but they work for me. They are based on a custom 
ubuntu image I made with Tcl. They are kinda big, about 300 mb but it 
works for me. About S6 and non S6, difference is only on type of init 
system used in the image, but i'm not pro at this question. The S6 
version of Docker images uses an init system called s6-overlay, the 
non-S6 version of Docker images typically use a different init system, 
such as Systemd or SysVinit. In my case I'm using Postgres only, at 
host not in a Docker container. it looks like its working faster this way.
So, in general, you can use a Docker container with Naviserver without 
any problem.


Maksym

On Fri, 31 Mar 2023, 10:44 Brian Fenton, 
 wrote:


Hello all

We're exploring reviving the Linux version of our Naviserver
product. As part of this we'd like to investigate the feasibility
of using Docker, initially for ease of development, and
potentially later for production systems. I have a couple of
questions that would be good to get the community's responses to.

1. Is there anyone using Docker Naviserver in production? If so
would you recommend it, and are there any peculiarities that we
should know about?

2. There are quite a few Naviserver images on on
https://hub.docker.com, but I don't see an official one. It seems
from posts on OpenACS.org
<https://openacs.org/xowiki/setup-with-docker-s6>that Vlad V's
build has had a lot of community input, and seems to be the /de
facto /build. Would it be worth trying to get that version tagged
as an official Naviserver Docker
<https://docs.docker.com/docker-hub/official_images/> build to
give newcomers clarity? I must admit to some confusion regarding
the difference between the S6 and the non-S6 versions - should we
as a community pick one as the official one?

3. For those of us using Oracle, what would be involved in getting
Oracle driver support added as a build/run option? I'd be happy to
help with this in any way I can.

thanks in advance
Brian

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] Support for Argon2

[Gustaf Neumann]

Dear all,

For all security hungry NaviServer users:

NaviServer supports now Argon2, which is currently the best known
password hashing function (well more than this, it is a key derivation
algorithm). For details, see [1]. It may take still some time until
OpenSSL 3.2 is available in the main Linux distributions.

With this change, NaviServer provides direct support for the two most
recommended password hashing algorithms of the OWASP project [2],
namely Argon2 and scrypt, along with SCRAM-sha-256 (actually PBKDF2)
which is the most secure algorithm implemented in PostgreSQL.

All the best

-g

[1] 
https://bitbucket.org/naviserver/naviserver/commits/4d634d54b77d1ce6b61f07944871f3dcf1a330a5
[2] 
https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#password-hashing-algorithms




___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] nscp segfaults server when missing module/nscp/users

[Gustaf Neumann]

Thanks for the report. The problem could be repeated and is fixed in the
public repository [1]. The problem was introduced in 2022-07-11, but
not included in any released version.

all the best -g

[1] 
https://bitbucket.org/naviserver/naviserver/commits/168eebbe67d0d631df5948e1647a36e6e0d9baf0


On 18.03.23 20:07, Andrew Piskorski wrote:

Probably nscp shouldn't segfault if it's missing that section?
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] how to check if nsd has zlib?

[Gustaf Neumann]

On 15.03.23 19:50, Andrew Piskorski wrote:

On Windows, what's the best (simple?) way to check if my nsd is
actually linked correctly with zlib?


If zlib is not linked correctly, you would get errors during linking.

The easiest thing to test, whether compression via zlib works fine
is to test this via browser (e.g., browse the file
tests/testserver/pages/ns_adp_compress.adp)

Since the errors you get are from "nstest::http-0.9", i would
not be surprised, if this is related with the last problem you reported,
also using the old-style regression test interface.

The code in "nstest::http-0.9" is quite old, it has not been changed
since ages. Are you sure, these tests were working before? It can
also be that this the one-of-content detection was never correct
but happened to work due to different buffering in tcl/c-library/OS/

all the best

-g



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Windows test failures, http_chunked

[Gustaf Neumann]

On 15.03.23 19:39, Andrew Piskorski wrote:

But despite that change, I still get those same 4 failures on Windows.


The change your are citing has nothing to do with windows.

If i see correctly, all these tests use "nstest::http-0.9", which is the 
old-style
regression test interface using ns_sockopen and tcl/IO (fconfigure, 
read, ...).

I would expect an error there.

The newer regression test interface is based on ns_http.
The failing tests use the old interface, since one gets there the raw
output from the server, while ns_http returns the parsed output.

-gn
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf Neumann]

The problem with ns_crypto::aead::encrypt/decrypt test under
OpenSSL 1.1.1 (OpenSSL 1.1.1-1ubuntu2.1~18.04.21) on Ubuntu 18.04.4
is now fixed in the repositotry.

In short, the problem was that with this version of OpenSSL, setting
empty additional authenticated data (AAD) behaved differently from
other versions, namely it was clearing incorrectly the information
that the initialization vector (IV) was already set. An upgrade of
OpenSSL fixed the problem. However, with these changes, also the stock
version of OpenSSL can be used.

Fixing this was more tricky as already apprehend, but solving a riddle
is also rewarding.

all the best

-g



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf Neumann]

On 09.03.23 17:30, Andrew Piskorski wrote:

In the meantime, how widely used within NaviServer is aead::encrypt?
Is it necessary for basic serving of https pages, or just an extra API
programmers can optionally use?  Do even the latest versions of
OpenACS depend on it?  (In other words, I'm wondering if these two
aead::encrypt test failures actually matter for me.)

the aead::* functions are not used anywhere in OpenACS, these
are not used for serving pages via TLS/...  or other kind of "regular"
or "internal" usage.

The only public code that i am aware of is the nswebpush module
(optional naviserver module). But of course, every tailored
OpenACS application might use this...

all the best

-g

___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf Neumann]

Small update: with Ubuntu 18.04 +  "OpenSSL 1.1.1-1ubuntu2.1~18.04.21",
also older versions of naviserver do not work (went back until 4.99.20, 
jan 2021).

so, in case this is really needed, it requires detailed debugging, including
comparing the results of OpenSSL API calls.

-gn

On 09.03.23 12:27, Gustaf Neumann wrote:


On 08.03.23 21:52, Andrew Piskorski wrote:

Building the NaviServer head (latest code from 2023-03-02), I'm
getting two "make test" failures, both from aead::encrypt (below).
Any advice for me on what the problem might be,


My first suspicion is the version of OpenSSL in use. OpenSSL is a 
moving target.
If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
place for

Ubuntu 18.04 will all updates. It this what you are using?

Options:
- Upgrade of OpenSSL (i have just tested an installation with 18.04 + 
OpenSSL 3)

- Upgrade of Ubuntu (18.04 is not the youngest)
- Downgrade of NaviServer: there were changes between NaviServer 
4.99.23 and .24

   concerning support of OpenSSL 3.0 - using newer API calls - so maybe
   these newer API calls are available in this version of OpenSSL, but
   not fully functioning yet.

... and of course, provide a fix to "make it work" also in your 
combination.


-gn



or how I should
further track it down?  (Thanks!)

This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
tdom 0.9.3, and tcllib 1.20.


## Excerpts from "make test" output:

[08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: 
SSL_shutdown(33) has failed: error:14094123:SSL 
routines:ssl3_read_bytes:application data after close notify

ns_crypt.test
ns_crypto.test

 aead-1.0 aead::encrypt FAILED
 Result was:
bytes 0 tag 32
 Result should have been (exact matching):
bytes 22 tag 32
 aead-1.0 FAILED

 aead-1.1 aead::encrypt and decrypt FAILED
 Test generated error; Return code was: 1
 Return code should have been one of: 0 2
 aead-1.1 FAILED

Tests ended at Wed Mar 08 15:24:13 EST 2023
all.tcl:    Total    1998    Passed    1971    Skipped    25 Failed    2
Sourced 71 Test Files.
Files with failing tests: ns_driver.test
Number of tests skipped for each constraint:
19    !usingExternalToUtf
2    binaryMismatch
1    copyAliasBug
2    knownBug
1    stress
Makefile:236: recipe for target 'test' failed
make: *** [test] Error 130


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] aead::encrypt test failures

[Gustaf Neumann]

On 08.03.23 21:52, Andrew Piskorski wrote:

Building the NaviServer head (latest code from 2023-03-02), I'm
getting two "make test" failures, both from aead::encrypt (below).
Any advice for me on what the problem might be,


My first suspicion is the version of OpenSSL in use. OpenSSL is a moving 
target.
If i see correctly, there is a version "1.1.1-1ubuntu2.1~18.04.21" in 
place for

Ubuntu 18.04 will all updates. It this what you are using?

Options:
- Upgrade of OpenSSL (i have just tested an installation with 18.04 + 
OpenSSL 3)

- Upgrade of Ubuntu (18.04 is not the youngest)
- Downgrade of NaviServer: there were changes between NaviServer 4.99.23 
and .24

   concerning support of OpenSSL 3.0 - using newer API calls - so maybe
   these newer API calls are available in this version of OpenSSL, but
   not fully functioning yet.

... and of course, provide a fix to "make it work" also in your combination.

-gn



or how I should
further track it down?  (Thanks!)

This is on an old Ubuntu 18.04.4 LTS machine, using gcc 8.4.0, and Tcl
8.6.13.  My built-from-source Tcl includes nsf 2.4.0, Thread 2.8.9,
tdom 0.9.3, and tcllib 1.20.


## Excerpts from "make test" output:

[08/Mar/2023:15:23:16][11421.7f2097bfc700][-command-] Notice: SSL_shutdown(33) 
has failed: error:14094123:SSL routines:ssl3_read_bytes:application data after 
close notify
ns_crypt.test
ns_crypto.test

 aead-1.0 aead::encrypt FAILED
 Result was:
bytes 0 tag 32
 Result should have been (exact matching):
bytes 22 tag 32
 aead-1.0 FAILED

 aead-1.1 aead::encrypt and decrypt FAILED
 Test generated error; Return code was: 1
 Return code should have been one of: 0 2
 aead-1.1 FAILED

Tests ended at Wed Mar 08 15:24:13 EST 2023
all.tcl:Total   1998Passed  1971Skipped 25  Failed  2
Sourced 71 Test Files.
Files with failing tests: ns_driver.test
Number of tests skipped for each constraint:
19  !usingExternalToUtf
2   binaryMismatch
1   copyAliasBug
2   knownBug
1   stress
Makefile:236: recipe for target 'test' failed
make: *** [test] Error 130


--
Univ.Prof. Dr. Gustaf Neumann
Head of the Institute of Information Systems and New Media
of Vienna University of Economics and Business
Program Director of MSc "Information Systems"



___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Ns_conn location questions

[Gustaf Neumann]

On 29.01.23 19:36, Maksym Zinchenko wrote:
About the second question, I didn't notice this before :) Thank you 
for your information.


When you fetch a new version of the nsshell module, it supports now 
"ns_conn location" (it's an emulaton, but probably for most situations 
good enough).


all the best

-g
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Ns_conn location questions

[Gustaf Neumann]

> My first question is where this localhost comes from?

I would think, this comes from your configuration file and/of from the 
request.


If one starts e.g. with the sample configuration file nsd-config.tcl, 
one sees entries like:


[29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding 
virtual host entry for host  location:http://localhost:8080  
mapped to server: default ctx 0x0
[29/Jan/2023:16:35:24][54720.100490580][-main:default-] Notice: nssock:0: adding 
virtual host entry for host  
location:http://MacBook-Pro-6.local:8080  mapped to server: default ctx 0x0

These log-entries are coming e.g. from the following section

ns_section ns/module/nssock/servers {
ns_param defaultlocalhost
ns_param default[ns_info hostname]
}

from the configuration file that define the mapping of hostnames for the 
"default" server. When there are incoming requests following HTTP/1.1, 
these come with a "Host:" header field, which might be in your case as 
well "localhost".



> When I try to run ns_conn location in nsshell it gives me error 
bad_option "location".


This is a know limitation (see first line of "Current shortcomings" on
https://bitbucket.org/naviserver/nsshell/src/main/

The reason for this is that nsshell communicates with a "kernel" thread 
in the background that keeps the state of your nsshell session (e.g. to 
be able to obtain the variable value for later requests, if one types 
"set x 1" in nsshell). Since this kernel runs in the background, it is 
no connection thread, and has no connection information available.


Theoretically, one could stretch the limits of nsshell further, but it 
is quite hard to make the background job look completely like the a 
connection thread.


-gn
___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] ns_adp_registerproc and attributes

[Gustaf Neumann]

Dear Wolfgang,

After Maksym Zinchenko, we found as well a problem with the modified ADP 
parser (inside an embedded minified JavaScript code)


The situation is more tricky than expected, since we have
- a language mix in ADP files (Tcl, HTML, JavaScript) and special ADP 
tags and semantics

- Some of these problems are addressed by HTML, which treats 
   context as CDATA (e.g. tags inside JavaScript)