thank you On Wed, Jun 14, 2023 at 1:19 PM Gustaf Neumann <neum...@wu.ac.at> wrote:
> Dear Sassy, > > You are referring to a change that was released more than 3 years ago > (4.99.19). > > Log file sanitizing works as expected, namely it prints invisible > characters in a hex representation (in your case tab characters). > Sanitizing was required by security audits, since otherwise, it would be > possible to execute code by looking into the log file, one could obfuscate > the log file, and confuse log file analyzer that alarm when suspect > activities are noticed, etc. > > Certainly, the harm caused by a tab character alone is limited, but when > debugging and analyze problems, it is often important to distinguish > between a tab and some spaces. > > You can easily replace the hex notation in the log file via "sed", such as > e.g. > > cat log/error.log | sed -En 's/\\x09/\t/p' | more > > or the like. > > all the best > > -gn > On 13.06.23 20:51, Sassy Natan wrote: > > Hi All, > > After upgrading my server to the latest version, my log file is broken. > > For example I see UTF-8 as special characters. > > I check the readme but the sanitizelogfiles 2 doesn't seems to work as > expected. > I have in my configuration: > > set debug true > ns_logctl severity "Debug(sql)" on > > Any ideas? > > Here is example: > : ],is_break_pcols)) as hrs_wo_break > : FROM wt_payroll_analysis_pp > : join (select sid,agg_array(case when is_break_pcol=true then 99999 > else 0 end) as is_break_pcols > : \x09\x09\x09from ( select * from wt_et_cols_defs where sid=410000 > order by pcol_number )pc_defs group by sid) pdefs using (sid) > : \x09\x09\x09\x09WHERE sid=410000 and employee_id =40599 > : \x09\x09\x09\x09and to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd') > between to_date('2023-6-01','yyyy-mm-dd')- interval '6 month' > : \x09\x09\x09\x09\x09\x09\x09and ( to_date('2023-6-01','yyyy-mm-dd') > - interval '1 day' )::date > : \x09\x09\x09\x09UNION > : \x09\x09\x09\x09 select > sid,employee_id,pp,pp_year,ppid,to_date(pp_year||'-'||pp||'-01','yyyy-mm-dd') > as day,null2zero(hrs_wo_break)*3600 > : \x09\x09\x09\x09 from pp_pa_table > : \x09\x09\x09\x09) ggg > : \x09\x09\x09\x09group by sid,employee_id,pp,pp_year > > > Thanks > Sassy > > On Mon, May 1, 2023 at 6:55 PM Gustaf Neumann <neum...@wu.ac.at> wrote: > >> Dear all, >> >> I am glad to announce that the release of NaviServer 4.99.25 is >> available at SourceForge [1]. This release is mostly a bug-fix >> release. The forthcoming version 5.0 of NaviServer will contain >> several new features omitted in this bug-fix branch. In case, you are >> building NaviServer from the Bitbucket repository, please note that >> the release 4.99.25 is in the branch release/4.99 (bug fix branch for >> the NaviServer 4.99 family). New development happens in the "main" >> branch of the repository, leading to NaviServer 5.*. >> >> See below for a summary of the changes. >> >> Many thanks to the contributors of this release: >> >> Andrew Piskorski >> Antonio Pisano >> Brian Fenton >> Gustaf Neumann >> Hector Romojaro >> Joe Oldak >> Khy Huang >> Oleg Oleinick >> Zoran Vasiljevic >> >> All the best! >> >> -gustaf neumann >> >> [1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.25/ >> >> ======================================= >> NaviServer 4.99.25, released 2023-05-01 >> ======================================= >> >> 132 files changed, 3957 insertions(+), 2068 deletions(-) >> >> New Features: >> ------------- >> >> - Added meta-information to configuration values >> >> NaviServer can now report, what configuration values provided in >> the configuration file were actually used, what their default >> values are, and whether these values were specified or not (using >> the default values). With this information, administration >> (e.g. migration) becomes easier. The NaviServer module "nsstats" >> shows this meta information via the web interface. >> >> This functionality is provided via the new option "-filter" for the >> command "ns_configsection ... section". When the "-filter" option >> is used, different kinds of information about the parameters is >> returned from the specified section. >> >> "-filter unread": >> >> Returns the parameters, which were set during configuration (i.e., >> in the configuration file) but which were not read in the startup >> phase of the server. This option is useful to determine >> e.g. typographical errors of specified parameter names. >> >> "-filter defaulted": >> >> Returns the parameters, from which the default values were read >> (i.e., which were not explicitly set) >> >> "-filter defaults": >> >> Returns the defaults of the parameter. This is useful for >> contrasting the actual values with the default values for >> Parameters, e.g. in a web based interface. >> >> - ns_set reform (per default deactivated in 4.99, but activated in 5.*) >> >> The classical implementation for ns_sets uses separately malloced >> storage for every attribute name and attribute value. So, e.g., for >> 1000 ns_sets with 20 members each, this means 1,000*20*2 = 40,000 >> malloc/free operations, e.g., for a single db query! Although the >> malloc implementations have improved over the years, these will >> require many lock operations, especially under load, where many >> threads might perform many concurrent malloc operations. One other >> consequence is that the allocated memory will be scattered over >> address space, which has bad implications for CPU caching. >> >> The new implementation uses for one "ns_set" a single Tcl_DString >> keeping all attribute names and attribute values. This reduces the >> malloc operations and improves memory locality, such that cache >> hits will improve. >> >> One caveat of this change is that modules using "ns_set" have to be >> recompiled, since the full C-level data structure of the "ns_set" >> is exposed. Therefore, adding a member causes a binary >> incompatibility. One other potential problem is that C-level >> modules using the Ns_Set* API have to make sure that long-living >> string values are copied (this was necessary before as well, but >> was in many cases no problem, when the "ns_sets" were seldom >> updated). >> >> For high compatibility, this feature is deactivated per default in >> the 4.99.* series and can be activated by setting the compile-time >> C macro "NS_SET_DSTRING". >> >> >> >> API changes: >> ------------ >> >> API extensions: >> - Provide a new interface ending with *Sz to provide string sizes. >> This reduces the need of strlen() operations. >> * Ns_SetCreateSz() >> * Ns_SetIUpdateSz() >> * Ns_SetPutSz() >> * Ns_SetPutValueSz() >> * Ns_SetUpdateSz() >> >> - New API calls for "ns_set" reform >> * Ns_SetClearValues(): clear the values for all keys >> * Ns_SetDataPrealloc(): creating ns_sets with preallocated values >> to avoid resize operations >> * NsSetResize() >> * NsHeaderSetGet() >> >> - Ns_ConfigSet(const char *section, const char *key, const char *name) >> The last argument is new and allows one to create named sets >> (previously, all such sets were unnamed) >> >> - NsHexPrint(): Print the potentially binary content of a buffer >> in human-readable form. >> >> - Ns_RelativeTime(Ns_Time *relTimePtr, Ns_Time *timePtr) >> This call implements the inverse operation of Ns_AbsoluteTime(), >> and is used mostly to make debug messages eye-friendly. >> >> >> Performance Improvements: >> ------------------------- >> >> - Replaced malloc operation per log entry by thread local variable in >> system log implementation. >> >> - When NaviServer 4.99.25 is compiled with NS_SET_DSTRING supportm >> the following preliminary performance results were measured from >> the "ns_set" reform (see above). The tests were performed on >> openacs.org (Xeon Gold 6226R CPU @ 2.90GHz, 32 cores, >> hyper-threading enabled). The test executes the SQL query >> >> select * from acs_objects limit 1000 >> >> 100 times in sequence. This test is run in 1 to 30 concurrent >> threads. With 30 threads, 3mio tuples are retrieved, and 72 mio >> malloc/free operations are needed alone for the retrieved values. >> >> Before (classical ns_set with many mallocs): >> >> threads 1 total 4606.787 ms avg 3285.25 ms >> threads 5 total 4595.358 ms avg 3493.07 ms >> threads 10 total 4804.193 ms avg 3755.93 ms >> threads 20 total 6279.524 ms avg 4569.16 ms >> threads 30 total 8966.427 ms avg 6618.58 ms >> >> After reform (using common Tcl_DString per tuple): >> >> threads 1 total 4524.645 ms avg 3242.54 ms >> threads 5 total 4251.266 ms avg 3450.09 ms >> threads 10 total 4656.795 ms avg 3665.31 ms >> threads 20 total 5934.105 ms avg 4671.38 ms >> threads 30 total 7384.591 ms avg 5642.76 ms >> >> To summarize, the improvement increases under higher load (with >> more parallel threads). E.g., with 30 threads, the total time >> improved by 17%.... leading also to a smaller RSS. These tests were >> not performed under "clinical" conditions. >> >> The new Tcl API call "ns_set size" can be used to pre-allocate >> larger ns_sets, such that the usual Tcl_DString growing policy does >> not kick in, reducing further realloc() operations. >> >> >> Bug Fixes: >> ---------- >> >> - Fixed potential crash in "ns_accesslog extendedheaders XXXX". >> Setting extended headers via configuration file was correct, but >> changing it dynamically via "ns_accesslog extendedheaders .." was >> broken. (Issue https://sourceforge.net/p/naviserver/bugs/91/) >> >> - "ns_conn location": Fixed potential race condition >> >> It was possible that "ns_conn location" could return inconsistent >> results in a single request, when the underlying sockPtr was >> aggressively reused. Now, the value of the location member is >> copied to the connection structure instead of being shared with >> the socket structure (as before). >> >> - "ns_cache_eval -force": Fixed potential race condition >> >> There was a problem with "ns_cache_eval -force", where the system >> relied on the existence of a pre-existing entry, but in case the >> entry was flushed in the meantime problems a crash might have >> happened. Now the value during the "-force" call is cleared exactly >> like in the case of an unset operation. The null-value operations >> are already protected until these are finished in various places >> in the code. >> >> - "ns_socknread": Fix potentially wrong result for buffered channels >> >> - Bug fixes for problems showing up under MS Windows: >> >> * Make sure that the output variable of Ns_ObjvIndex() is always an >> integer. Previously, the output variable was in two places a >> character variable, causing crashes under MS Windows. >> >> * Handle incompatibility in setlocale() under MS Windows. Under >> MS Windows, later calls to setlocale() overwrite the string >> returned by former calls. So, it is necessary to copy of the >> returned string of a setlocale() call under MS Windows. >> >> * Handling linking problems: MS Windows requires explicit handling >> when importing symbols from .dll files (Ns_LogSqlDebug, >> NS_intTypePtr) >> >> * Pass error codes from low-level function SockRecv() and >> SockSend() via variables. This change makes sure the real error >> code (immediately after the I/O operation) is passed to the >> caller to avoid missed error cases and weird error message >> (e.g. under windows). >> >> >> - Bug fixes for ADP parser: >> >> * Support for greater than sign ">" inside attribute values. >> >> Previously, NaviServer determined the terminating end-of-tag >> character as literally the first greater than sign, no matter if >> this was used as attribute values between single or double >> quotes. >> >> The new version supports such values, since the "Living Standard >> of HTML" [1] allows the use of less than "<" and greater than >> ">" signs inside attribute values as long these are between >> single or double quotes. The guide [2] just recommends using >> character escapes for "<", ">" and "&". >> >> [1] >> https://html.spec.whatwg.org/multipage/syntax.html#syntax-attribute-value >> [2] https://www.w3.org/International/questions/qa-escapes >> >> This problem was reported by Wolfgang Winkler >> >> * Clear ADP flags in case of errors (this error was present for >> many years) >> >> Previously, The following ADP page could lead to a full >> breakdown of the server, since the error states of the server >> were never cleared, and subsequent requests served be the same >> interpreter could lead to the old error states. >> >> Many thanks to Oleg Oleinick for reporting and the great test >> cases. >> >> - Provide better HTTP status code for early errors: When the driver >> terminates already a request, it might be the case that the error >> flags were not read out. In such cases, we provide now more >> specific status codes rather than 400. >> >> - ns_http: >> >> * Improved robustness with domain names resolving against many IP >> addresses: When a domain name is resolved against many IP >> addresses, and all these IP addresses block (connect returns "in >> progress") then the old code might have looped infinitely. Now >> the code respects the provided timeout (default set to 5s) per >> resolved IP address. >> >> Many thanks to Joe Oldak for pointing out the problem and >> leading us to the solution. >> >> * Fixed timeout handling during TLS handshake: This problem could >> appear, when the TCP connection to a server succeeded quickly, >> but the TLS handshake was taking a long time, without a raising >> a timeout exception, although the timeout time has expired. >> >> - Set the default server before the init-scripts is called to make it >> accessible from there >> >> - Provide compatibility between in-memory and file-based form-data >> handling for invalid characters: Since the file based >> implementation uses a fallback-charset of iso8859-1 when parsing >> form data in POST requests (in order to be able to extract >> "_charset_" data), the in-memory based variant does now the >> same. The decision of which parser is used is taken based on the >> size of the form data.... and should therefore be consistent. >> >> - nscp: Fixed problem, when the nscp module is activated but no >> "users" section is specified. Many thanks to Andrew Piskorski for >> reporting the problem. >> >> - Security Improvement: Added protection against sneaking in fake >> NAME.tmpfile entries into form fields, when performing file upload >> operations. >> >> >> Documentation improvements: >> --------------------------- >> >> - Improved the following man pages: >> >> doc/src/manual/adp-overview.man >> doc/src/manual/tcl-lib-nstrace.man >> doc/src/naviserver/commandlist.man >> doc/src/naviserver/ns_adp.man >> doc/src/naviserver/ns_adp_ctl.man >> doc/src/naviserver/ns_config.man >> doc/src/naviserver/ns_conn.man >> doc/src/naviserver/ns_connchan.man >> doc/src/naviserver/ns_cookie.man >> doc/src/naviserver/ns_crypto.man >> doc/src/naviserver/ns_http.man >> doc/src/naviserver/ns_ictl.man >> doc/src/naviserver/ns_locationproc.man >> doc/src/naviserver/ns_parsehostport.man >> doc/src/naviserver/ns_return.man >> doc/src/naviserver/ns_set.man >> doc/src/naviserver/ns_setprivileges.man >> >> >> Configuration Files: >> -------------------- >> >> - New feature for the sample configuration file openacs-config.tcl >> >> The following variables (with prefix oacs_) can be taken from the >> shell variables. This makes it easier to run similar variants of >> NaviServer instances from a single configuration file, while >> providing variables from the command line. This is e.g. useful for >> Docker or cluster setups. >> >> CookieNamespace >> bindir >> cachingmode >> db_host >> db_name >> db_port >> db_user >> homedir >> hostname >> httpport >> httpsport >> ipaddress >> logroot >> nscpport >> server >> serverroot >> smtpdport >> >> One can use e.g.the following command to change some ports and the >> log file during startup >> >> oacs_httpport=8000 oacs_httpsport=8443 oacs_smtpdport=2526 \ >> oacs_logroot=/var/www/XXXX/log-node1/ \ >> /usr/local/ns/bin/nsd -t /var/www/XXXX/etc/openacs-config.tcl ... >> >> - further updates for openacs-config.tcl: >> * Added sample nssmtpd configuration >> * Added placeholder for ClusterSecret >> * Reflect recent Oracle requirements (tested with Oracle 19c) >> * Added documentation for "StaticCSP", "CookieNamespace", >> "NsShutdownWithNonZeroExitCode", "LogIncludeUserId" >> >> - Updated all sample configuration files >> >> >> Code Changes: >> ------------- >> >> - Added and updated predefined MIME types based on >> https://www.iana.org/assignments/media-types/media-types.xhtml >> >> - Added compile time macro NS_VERBOSE_MALLOC to identify frequent >> *alloc operations >> >> - Print version of zlib during startup >> >> - OpenSSL maintenance: >> >> * Improved robustness for OCSP with OpenSSL 3.* >> >> The previous version crashed with OpenSSL 3.*, when OCSP was >> tried on self-signed certificates. Aside of the fact that OCSP >> does not make sense for self-signed certificates, the server >> should not crash in such situations. >> >> * Fixed ns_crypto::aead::encrypt/decrypt test under OpenSSL 1.1.1 >> (as shipped per default, e.g. on Ubuntu 18.04.4) >> >> In short, the problem was that with this version of OpenSSL, >> setting empty additional authenticated data (AAD) behaved >> differently from other versions, namely it was clearing >> incorrectly (forgetting) the information that the initialization >> vector (IV) was already set. An upgrade of OpenSSL fixed the >> problem. However, with these changes, also the stock version of >> OpenSSL can be used. As a byproduct, better error messages are >> now produced, the code received more cleanup (e.g. explicit >> initialization, etc.) >> >> >> - Automated testing: >> >> * Setup if Bitbucket + GitHub pipelines for automated regression >> testing with multiple versions of components >> >> For NaviServer 4.99 the current setup performs tests with gcc-10 >> + gcc-11, Tcl 8.6.13 + 8.7a5, NSF 2.4,0 + 2.4.0, tDOM 0.9.1 + >> 0.9.3, extra modules: nsdbpg nsdbi nsdbipg nsudp nscoap nssmtpd. >> >> https://github.com/nm-wu/naviserver-mirror/actions >> >> * Improved robustness of regression test when running with the >> docker networking setup >> >> * Force nonzero exit code when regression test fails >> >> * Added parameter "-timeout" to call of regression test cases >> Previously, the timeout was hard-wired to 3 seconds. One can now >> call a test with e.g. "nstest::http -timeout 1s ... GET ..." >> >> * adp_compress.test: removed trailing newline to ease >> cross-platform regression tests >> >> * Prefer standard Tcl test constraint "macOrUnix" over own solution >> >> * Extended regression tests with more test cases >> >> - Code management: >> * Changed name of branch from "master" to "main" >> >> - Code Cleanup >> * Improved type cleanness >> * Removed deprecated calls to "sprintf" >> * Improved portability for Tcl 8.7* (handling of binary data) >> >> - Improved comments, fixed typos >> >> >> Changes in NaviServer Modules: >> ============================== >> >> 39 files changed, 9658 insertions(+), 1781 deletions(-) >> >> General: >> Adjust to necessary API changes in NaviServer (contains as well >> support for the forthcoming release of NaviServer 5) >> >> >> nsdbpg: >> ------- >> >> - new pg-driver specific command: ns_pg_prepare /sql/ >> >> Return a dict building a prepared statement for the passed-in SQL >> statement. The dict contains the keys "sql" and "args". The >> function is used by e.g. OpenACS to generate prepared statements >> from SQL commands with bind variables. >> >> - Raise exception when a value for a bind variable contains a NUL >> character. >> This value is explicitly forbidden in text strings passed to PostgreSQL. >> >> - Let "ns_pg" report available subcommands even when handle is not >> specified. This makes the command compatible with the "icanuse" >> feature in OpenACS. >> >> >> nsstats: >> -------- >> >> - HTTP client log analysis: >> * Provide charts for performance (using highcharts via CDN) >> * Provide charts on request frequency (using highcharts via CDN) >> * Provide a summary table for HTTP client requests >> * Improved robustness against invalid URLs (containing unescaped >> spaces) >> * Added support for selection of different HTTP client log files >> via web interface >> >> - "Process" page: >> * Added percentage of request distribution over connection pools >> * Added information about the connected client >> * Added more detailed version information >> >> - Added cache configuration to output when looking at a single cache >> >> - Improved "log file" analysis >> * Automated stripping of color codes >> >> * Added filter option. The filter can be used to grep for (ID) >> strings in both the system and access logs, providing a summary >> for the traces of a request in the access log and system log on >> a single place. >> >> - Added default and usage information to "Config Parameters" page >> >> >> nsoracle: >> --------- >> >> - Fixed bug when streaming LOB content to connection. The old code >> did not distinguish between binary and non-binary content. This >> bug was discussed in >> https://openacs.org/forums/message-view?message_id=5693661 >> >> Bumped version number to 2.9 >> >> - switched to plain Debug handling for debugging the driver >> >> The handling of Ns_LogSqlDebug is performed inside nsdb, including >> also the printout of (long) SQL statements. Previously, the driver >> was too chatty when Debug(sql) was turned on. >> >> - Added support for output columns of type SQLT_TIMESTAMP or >> SQLT_TIMESTAMP_TZ >> >> This change fixes a bug, where SQL queries of the form >> >> SELECT TO_TIMESTAMP(sysdate) FROM dual >> >> lead to errors for the form: >> >> Database operation "getrow" failed (exception 1406, >> "nsoracle.c:3659:Ns_OracleGetRow: >> error in `OCIStmtFetch ()': ORA-01406: fetched column value was >> truncated >> >> The driver needs for several output types special rules, where the >> timestamp cases were not supported so far. It is also recommended >> to set the according environment variables specifying the output >> format in the configuration server of NaviServer, such as e.g. >> >> set ::env(NLS_TIMESTAMP_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6" >> set ::env(NLS_TIMESTAMP_TZ_FORMAT) "YYYY-MM-DD HH24:MI:SS.FF6 >> TZH:TZM" >> >> For testing in you local Oracle installation, you might test the >> output formats (and the required sizes with the following snippet >> for sqlplus: >> >> COLUMN localtimestamp format a40 >> COLUMN systimestamp format a40 >> COLUMN ts_bytes format a80 >> >> alter session set nls_timestamp_format='YYYY-MM-DD HH24:MI:SS.FF6'; >> select localtimestamp, length(localtimestamp), dump(localtimestamp) >> ts_bytes from dual; >> >> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 >> TZH:TZM'; >> select systimestamp, length(systimestamp), dump(systimestamp) >> ts_bytes from dual; >> >> alter session set nls_timestamp_tz_format='YYYY-MM-DD HH24:MI:SS.FF6 >> TZR'; >> select systimestamp, length(systimestamp), dump(systimestamp) >> ts_bytes from dual; >> >> >> letsencrypt: >> ------------ >> >> - Added option to produce certificates with ECDSA: >> >> Prior to this change, all certificates were using RSA keys. >> Since a while, keys based on elliptic curves became the preference >> of letsencrypt. >> >> >> nsshell: >> -------- >> >> - Fixed a bug in "ns_conn" emulation, when the "kernel" was not correctly >> identified >> >> >> _______________________________________________ >> naviserver-devel mailing list >> naviserver-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/naviserver-devel >> > > > -- > Regards, > > Sassy Natan > 972-(0)54-2203702 > > > _______________________________________________ > naviserver-devel mailing > listnaviserver-devel@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/naviserver-devel > > -- > Univ.Prof. Dr. Gustaf Neumann > Head of the Institute of Information Systems and New Media > of Vienna University of Economics and Business > Program Director of MSc "Information Systems" > > _______________________________________________ > naviserver-devel mailing list > naviserver-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/naviserver-devel > -- Regards, Sassy Natan 972-(0)54-2203702
_______________________________________________ naviserver-devel mailing list naviserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/naviserver-devel