Hello.
So i am doing this, onto v2-9-2b.
Please some useless headache rants first, thank you..
, and i hate autotools, it took me two hours alone to get this
going vs fancy curses and lynx defining getbegx() and
- SCREEN *oldscreen = LYscreen;
+ SCREEN *oldscreen = (SCREEN*)LYscreen;
also delete_screen() is unknown.
I had to edit configure because i dunno because of
$ac_cv_sys_file_offset_bits being empty in
configure:
CPPFLAGS="${CPPFLAGS}-D_FILE_OFFSET_BITS=$ac_cv_sys_file_offset_bits"
In file included from conftest.c:27:
In file included from /usr/include/limits.h:26:
In file included from /usr/include/bits/libc-header-start.h:33:
/usr/include/features.h:390:52: error: invalid token at start of a
preprocessor expression
390 | #if defined _FILE_OFFSET_BITS && _FILE_OFFSET_BITS == 64
| ^
but again, i have no idea. Sigh. Shall i ever do anything lynx
again then only on a release ball, really. Too late again. Hm.
"My lynx", fwiw, is
./configure --disable-color-style --disable-config-info \
--disable-dired --disable-echo --disable-gopher \
--disable-included-msgs --disable-menu-options \
--disable-prettysrc --disable-scrollbar \
--disable-session-cache --disable-sessions \
--disable-trace --enable-default-colors --enable-gzip-help \
--enable-ipv6 --enable-nested-tables \
--enable-underlines --enable-widec --with-ssl \
--with-zlib --with-zstd
:-)
Ok, so i did that *i think*. It needs more careful testing.
It must be said that i think the Brotli addition was not truly
tested without other *Z*, i do not think this will do then -- some
fixes for that are included in the patch below.
I am an autotools idiot. Btw i see
...
checking for zError... yes
checking if you want to use brotli decompression... yes
configure: WARNING: Cannot find brotlidec library
checking if you want to use zstd decompression... yes
...
checking for bzip2... /usr/bin/bzip2
checking for brotli... no
checking for brotli... no
checking for zstd... /usr/bin/zstd
...
I am a day late, because i lost several hours due to myriads of
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.0 200 OK
Data transfer complete
Alert!: Error uncompressing temporary file!
lynx: Start file could not be found or is not text/html or text/plain
Exiting...
errors that i did not understand. By sheer despair i then changed
in HTFWriter_free() the first
#ifdef USE_ZLIB
to
#ifdef xUSE_ZLIB
and convinced myself that the code path doing content-encoding
decompression via gzip(1) is broken, thus "also", and thus i could
not help that code path for now:
ACCEPT ENCODING IS gzip, bzip2, zstd
(
CODING IS zstd
| CODING IS gzip
CONTEN-TYPE_IST text/html
(
NOW THI SIS NOT UTST: text/html
(
CODING IS zstd
| CODING IS gzip
FOUND AND SET ZSTD /usr/bin/zstd --rm -d %s
| FOUND AND SET GZIP /bin/gzip -d --no-name %s
AFTER FIRST IF
(
HTCOMPRESSED ANCHORT CT is text/html
(
TEMP is .html.zst
| TEMP is .html.gz
PROGRAM SET RECEIVING COMPRESSED /tmp/lynxXXXX8vtgxq/L18273-1TMP.html.zst
.html.zst | PROGRAM SET RECEIVING COMPRESSED
/tmp/lynxXXXXOFtvdP/L18191-1TMP.html.gz .html.gz
RETURN HTCOMRPESSED() /tmp/lynxXXXX8vtgxq/L18273-1TMP.html.zst
| RETURN HTCOMRPESSED() /tmp/lynxXXXXOFtvdP/L18191-1TMP.html.gz
WRITER_FREE CT zstd
| WRITER_FREE CT gzip
COMPRESSION FORMET www/compressed check=2
(
ISCF GZIP
(
COMPRESSION FORMET www/compressed check=5
(
ISCF BROTLI
(
COMPRESSION FORMET www/compressed check=6
(
ISCF ZSTD
(
DECOMPRESS, ELSE www/compressed
(
SKIP_LOADFILE=0 viewer=
(
==YEAH ZSTD
| ==YEAH GZUP
PATH REMOVE ZSTD IS /tmp/lynxXXXX8vtgxq/L18273-1TMP.html
| PATH REMOVE GZIP IS /tmp/lynxXXXXOFtvdP/L18191-1TMP.html
!use_zread
(
FOUND <> uncompress failed
(
(In my experience the "PATH REMOVE" file *is already* plain HTML,
ie, somehow, magically, uncompressed. I have the debug soiled
code around still..)
Ciao!
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
From 6fa4291903326c823f00c0cb9e98223114b7ebcd Mon Sep 17 00:00:00 2001
Message-ID: <6fa4291903326c823f00c0cb9e98223114b7ebcd.1720043570.git.stef...@sdaoden.eu>
From: Steffen Nurpmeso <stef...@sdaoden.eu>
Date: Wed, 3 Jul 2024 23:52:02 +0200
Subject: [PATCH] zstd encoding support
---
INSTALLATION | 10 +-
PACKAGE/lynx.spec | 5 +-
WWW/Library/Implementation/HTFTP.c | 3 +
WWW/Library/Implementation/HTFile.c | 72 +++++++++-
WWW/Library/Implementation/HTFile.h | 2 +
WWW/Library/Implementation/HTFormat.c | 181 +++++++++++++++++++++++++-
WWW/Library/Implementation/HTFormat.h | 19 ++-
WWW/Library/Implementation/HTTP.c | 3 +
aclocal.m4 | 21 +++
config.hin | 2 +
configure.in | 14 ++
lynx.cfg | 6 +
src/HTFWriter.c | 157 ++++++++++++++++++++++
src/HTInit.c | 2 +
src/LYReadCFG.c | 1 +
src/LYrcFile.c | 3 +
src/LYrcFile.h | 1 +
17 files changed, 496 insertions(+), 6 deletions(-)
diff --git a/INSTALLATION b/INSTALLATION
index 68acdbbb4e..ca590f5dc0 100644
--- a/INSTALLATION
+++ b/INSTALLATION
@@ -300,7 +300,7 @@ II. Compile instructions -- UNIX
Affected commands include
chmod compress cp gzip install mkdir mv rm tar touch gunzip
- unzip brotli bzip2 uudecode zcat zip telnet tn3270 rlogin
+ unzip brotli bzip2 uudecode zcat zip zstd telnet tn3270 rlogin
(Not all commands are used on all systems or in all configurations.)
This option makes Lynx simpler to install, but potentially less secure,
@@ -787,6 +787,14 @@ II. Compile instructions -- UNIX
or one level above. In either case, the corresponding header files
are assumed to be in the parallel "include" directory.
+ --with-zstd[=XXX] (define USE_ZSTD)
+ Use zstd decompression.
+
+ The optional value XXX specifies the directory in which the library
+ can be found, and may be either the path of the "lib" directory,
+ or one level above. In either case, the corresponding header files
+ are assumed to be in the parallel "include" directory.
+
1d. Environment variables
The configure script looks for programs and libraries in known/standard
locations. You can override the behavior of the script by presetting
diff --git a/PACKAGE/lynx.spec b/PACKAGE/lynx.spec
index 858171dbe7..da93c4cb8c 100644
--- a/PACKAGE/lynx.spec
+++ b/PACKAGE/lynx.spec
@@ -22,7 +22,7 @@ BuildRequires: libidn-devel
# BuildRequires: libopenssl-1_1-devel, or
# BuildRequires: libopenssl-3-devel
-Requires: brotli, gzip, bzip2, tar, zip, unzip
+Requires: brotli, gzip, bzip2, tar, zip, unzip, zstd
%description
Lynx is a fully-featured World Wide Web (WWW) client for users running
@@ -102,6 +102,9 @@ strip $RPM_BUILD_ROOT%{_bindir}/%{name}
%changelog
+* Sat Jul 03 2024 Thomas E. Dickey
+- add zstd compression
+
* Thu Mar 14 2024 Thomas E. Dickey
- trim redundant options
diff --git a/WWW/Library/Implementation/HTFTP.c b/WWW/Library/Implementation/HTFTP.c
index a08701cfb2..ee854ff78b 100644
--- a/WWW/Library/Implementation/HTFTP.c
+++ b/WWW/Library/Implementation/HTFTP.c
@@ -4095,6 +4095,9 @@ int HTFTPLoad(const char *name,
case cftBrotli:
StrAllocCopy(anchor->content_encoding, "x-brotli");
break;
+ case cftZstd:
+ StrAllocCopy(anchor->content_encoding, "zstd");
+ break;
case cftNone:
break;
}
diff --git a/WWW/Library/Implementation/HTFile.c b/WWW/Library/Implementation/HTFile.c
index 9b6c54f674..d5db0bd9e9 100644
--- a/WWW/Library/Implementation/HTFile.c
+++ b/WWW/Library/Implementation/HTFile.c
@@ -1283,6 +1283,11 @@ CompressFileType HTCompressFileType(const char *filename,
&& StrChr(dots, ftype[-4]) != 0) {
result = cftBzip2;
ftype -= 4;
+ } else if ((len > 4)
+ && !strcasecomp((ftype - 3), "zst")
+ && StrChr(dots, ftype[-4]) != 0) {
+ result = cftZstd;
+ ftype -= 4;
} else if ((len > 3)
&& !strcasecomp((ftype - 2), "gz")
&& StrChr(dots, ftype[-3]) != 0) {
@@ -1335,6 +1340,9 @@ const char *HTCompressTypeToSuffix(CompressFileType method)
case cftBrotli:
result = ".br";
break;
+ case cftZstd:
+ result = ".zst";
+ break;
}
return result;
}
@@ -1366,6 +1374,9 @@ const char *HTCompressTypeToEncoding(CompressFileType method)
case cftBrotli:
result = "brotli";
break;
+ case cftZstd:
+ result = "zstd";
+ break;
}
return result;
}
@@ -1377,6 +1388,7 @@ const char *HTCompressTypeToEncoding(CompressFileType method)
* compress
* deflate
* as well as "identity" (but that does nothing).
+ * RFC 8878 gives zstd.
*/
CompressFileType HTEncodingToCompressType(const char *coding)
{
@@ -1400,6 +1412,8 @@ CompressFileType HTEncodingToCompressType(const char *coding)
!strcasecomp(coding, "x-br") ||
!strcasecomp(coding, "brotli")) { /* expected */
result = cftBrotli;
+ } else if (!strcasecomp(coding, "zstd")) {
+ result = cftZstd;
}
return result;
}
@@ -1426,6 +1440,8 @@ CompressFileType HTContentTypeToCompressType(const char *ct)
!strncasecomp(ct, "application/x-br", 16) ||
!strncasecomp(ct, "application/brotli", 18)) {
method = cftBrotli;
+ } else if (!strncasecomp(ct, "application/zstd", 16)) {
+ method = cftZstd;
}
return method;
}
@@ -1459,6 +1475,9 @@ BOOL IsCompressionFormat(HTAtom *format, CompressFileType check)
format == HTAtom_for("application/x-br") ||
format == HTAtom_for("application/brotli"));
break;
+ case cftZstd:
+ result = (format == HTAtom_for("application/zstd"));
+ break;
case cftNone:
break;
}
@@ -2498,7 +2517,11 @@ static int decompressAndParse(HTParentAnchor *anchor,
#ifdef USE_BROTLI
FILE *brfp = 0;
#endif /* USE_BROTLI */
-#if defined(USE_ZLIB) || defined(USE_BZLIB)
+#ifdef USE_ZSTD
+ FILE *zfp = 0;
+#endif
+#if defined(USE_ZLIB) || defined(USE_BZLIB) ||\
+ defined(USR_BROTLI) || defined(USE_ZSTD)
CompressFileType internal_decompress = cftNone;
BOOL failed_decompress = NO;
#endif
@@ -2556,7 +2579,8 @@ static int decompressAndParse(HTParentAnchor *anchor,
* this is a compressed file, no need to look at the filename
* again. - kw
*/
-#if defined(USE_ZLIB) || defined(USE_BZLIB)
+#if defined(USE_ZLIB) || defined(USE_BZLIB) ||\
+ defined(USE_BROTLI) || defined(USE_ZSTD)
CompressFileType method = HTEncodingToCompressType(HTAtom_name(myEncoding));
#endif
@@ -2607,6 +2631,16 @@ static int decompressAndParse(HTParentAnchor *anchor,
internal_decompress = cftBrotli;
} else
#endif /* USE_BROTLI */
+#ifdef USE_ZSTD
+ if (isDOWNLOAD(cftZstd)) {
+ fclose(fp);
+ fp = 0;
+ zfp = fopen(localname, BIN_R);
+ CTRACE((tfp, "HTLoadFile: fopen of `%s' gives %p\n",
+ localname, (void *) zfp));
+ internal_decompress = cftZstd;
+ } else
+#endif /* USE_ZSTD */
{
StrAllocCopy(anchor->content_type, format->name);
StrAllocCopy(anchor->content_encoding, HTAtom_name(myEncoding));
@@ -2701,6 +2735,21 @@ static int decompressAndParse(HTParentAnchor *anchor,
format = HTAtom_for("www/compressed");
#endif /* USE_BROTLI */
break;
+ case cftZstd:
+ StrAllocCopy(anchor->content_encoding, "zstd");
+#ifdef USE_ZSTD
+ if (strcmp(format_out->name, STR_DOWNLOAD) != 0) {
+ fclose(fp);
+ fp = 0;
+ zfp = fopen(localname, BIN_R);
+ CTRACE((tfp, "HTLoadFile: fopen of `%s' gives %p\n",
+ localname, (void *) zfp));
+ internal_decompress = cftZstd;
+ }
+#else /* USE_ZSTD */
+ format = HTAtom_for("www/compressed");
+#endif /* USE_ZSTD */
+ break;
case cftNone:
break;
}
@@ -2726,6 +2775,11 @@ static int decompressAndParse(HTParentAnchor *anchor,
case cftBrotli:
failed_decompress = (BOOLEAN) (brfp == NULL);
break;
+#endif
+#ifdef USE_ZSTD
+ case cftZstd:
+ failed_decompress = (BOOLEAN) (zfp == NULL);
+ break;
#endif
default:
failed_decompress = YES;
@@ -2758,6 +2812,12 @@ static int decompressAndParse(HTParentAnchor *anchor,
if (sugfname && *sugfname)
StrAllocCopy(anchor->SugFname, sugfname);
FREE(sugfname);
+#ifdef USE_ZSTD
+ if (zfp)
+ *statusp = HTParseZstdFile(format, format_out,
+ anchor,
+ zfp, sink);
+#endif
#ifdef USE_BROTLI
if (brfp)
*statusp = HTParseBrFile(format, format_out,
@@ -3055,6 +3115,9 @@ int HTLoadFile(const char *addr,
case cftBrotli:
atomname = "application/x-brotli";
break;
+ case cftZstd:
+ atomname = "application/zstd";
+ break;
case cftNone:
break;
}
@@ -3356,6 +3419,11 @@ void HTInitProgramPaths(BOOL init)
for (n = (int) ppUnknown + 1; n < (int) pp_Last; ++n) {
switch (code = (ProgramPaths) n) {
+#ifdef ZSTD_PATH
+ case ppZSTD:
+ path = ZSTD_PATH;
+ break;
+#endif
#ifdef BROTLI_PATH
case ppBROTLI:
path = BROTLI_PATH;
diff --git a/WWW/Library/Implementation/HTFile.h b/WWW/Library/Implementation/HTFile.h
index 9340963773..652f065b16 100644
--- a/WWW/Library/Implementation/HTFile.h
+++ b/WWW/Library/Implementation/HTFile.h
@@ -214,6 +214,7 @@ extern "C" {
,cftBzip2
,cftDeflate
,cftBrotli
+ ,cftZstd
} CompressFileType;
/*
@@ -336,6 +337,7 @@ extern "C" {
,ppUUDECODE
,ppZCAT
,ppZIP
+ ,ppZSTD
,pp_Last
} ProgramPaths;
diff --git a/WWW/Library/Implementation/HTFormat.c b/WWW/Library/Implementation/HTFormat.c
index ddb981cee3..f6780e43f9 100644
--- a/WWW/Library/Implementation/HTFormat.c
+++ b/WWW/Library/Implementation/HTFormat.c
@@ -61,6 +61,10 @@ static float HTMaxSecs = 1e10; /* No effective limit */
#include <brotli/decode.h>
#endif
+#ifdef USE_ZSTD
+# include <zstd.h>
+#endif
+
BOOL HTOutputSource = NO; /* Flag: shortcut parser to stdout */
/* this version used by the NetToText stream */
@@ -1481,7 +1485,108 @@ static int HTBrFileCopy(FILE *brfp, HTStream *sink)
return rv;
#undef THIS_FUNC
}
-#endif /* USE_BZLIB */
+#endif /* USE_BROTLI */
+
+#ifdef USE_ZSTD
+/* Push data from a zstd file pointer down a stream
+ * -------------------------------------
+ *
+ * This routine is responsible for creating and PRESENTING any
+ * graphic (or other) objects described by the file.
+ *
+ *
+ * State of file and target stream on entry:
+ * ZFILE (zfp) assumed open (should have zstd content),
+ * target (sink) assumed valid.
+ *
+ * Return values:
+ * HT_INTERRUPTED Interruption after some data read.
+ * HT_PARTIAL_CONTENT Error after some data read.
+ * -1 Error before any data read.
+ * HT_LOADED Normal end of file indication on reading.
+ *
+ * State of file and target stream on return:
+ * always zfp still open, target stream still valid.
+ */
+static int HTZstdFileCopy(FILE *zfp, HTStream *sink)
+{
+ off_t bytes;
+ HTStreamClass targetClass;
+ ZSTD_DStream *izsp;
+ void *ibp, *obp;
+ size_t ibl, obl;
+ int rv;
+
+ rv = -1;
+
+ ibl = ZSTD_DStreamInSize();
+ ibp = malloc(ibl);
+ if (ibp == NULL)
+ goto jout0;
+
+ obl = ZSTD_DStreamOutSize();
+ obp = malloc(obl);
+ if (obp == NULL)
+ goto jout1;
+
+ if ((izsp = ZSTD_createDStream()) == NULL)
+ goto jout2;
+ ZSTD_initDStream(izsp);
+
+ /* Push the data down the stream */
+ targetClass = *(sink->isa); /* Copy pointers to procedures */
+
+ /* read and inflate file, and push binary down sink */
+ HTReadProgress(bytes = 0, (off_t) 0);
+
+ for (;;) {
+ ZSTD_outBuffer ob;
+ ZSTD_inBuffer ib;
+ size_t r, w;
+
+ r = fread(ibp, 1, ibl, zfp);
+ if (r == 0) {
+ if (feof(zfp))
+ rv = HT_LOADED;
+ else if (bytes > 0)
+ rv = HT_PARTIAL_CONTENT;
+ break;
+ }
+
+ ib.src = ibp;
+ ib.size = r;
+ ib.pos = 0;
+ ob.dst = obp;
+ ob.size = obl;
+ ob.pos = 0;
+
+ r = ZSTD_decompressStream(izsp, &ob, &ib);
+ if (ZSTD_isError(r)) {
+ if (bytes > 0)
+ rv = HT_PARTIAL_CONTENT;
+ break;
+ }
+
+ (*targetClass.put_block) (sink, ob.dst, ob.pos);
+ bytes += ob.pos;
+ HTReadProgress(bytes, (off_t) -1);
+ HTDisplayPartial();
+
+ if (HTCheckForInterrupt()) {
+ _HTProgress(TRANSFER_INTERRUPTED);
+ rv = HT_INTERRUPTED;
+ break;
+ }
+ } /* next bufferload */
+
+ ZSTD_freeDStream(izsp);
+jout2: free(obp);
+jout1: free(ibp);
+jout0:
+ HTFinishDisplayPartial();
+ return rv;
+}
+#endif /* USE_ZSTD */
/* Push data from a socket down a stream STRIPPING CR
* --------------------------------------------------
@@ -2048,6 +2153,80 @@ int HTParseBrFile(HTFormat rep_in,
}
#endif /* USE_BROTLI */
+#ifdef USE_ZSTD
+/* HTParseZstdFile
+ *
+ * State of file and target stream on entry:
+ * FILE* (zfp) assumed open,
+ * target (sink) usually NULL (will call stream stack).
+ *
+ * Return values:
+ * -501 Stream stack failed (cannot present or convert).
+ * -1 Download cancelled.
+ * HT_NO_DATA Error before any data read.
+ * HT_PARTIAL_CONTENT Interruption or error after some data read.
+ * HT_LOADED Normal end of file indication on reading.
+ *
+ * State of file and target stream on return:
+ * always zfp closed; target freed, aborted, or NULL.
+ */
+int HTParseZstdFile(HTFormat rep_in,
+ HTFormat format_out,
+ HTParentAnchor *anchor,
+ FILE *zfp,
+ HTStream *sink)
+{
+ HTStream *stream;
+ HTStreamClass targetClass;
+ int rv;
+ int result;
+
+ stream = HTStreamStack(rep_in, format_out, sink, anchor);
+
+ if (!stream || !stream->isa) {
+ char *buffer = 0;
+
+ fclose(zfp);
+ if (LYCancelDownload) {
+ LYCancelDownload = FALSE;
+ result = -1;
+ } else {
+ HTSprintf0(&buffer, CANNOT_CONVERT_I_TO_O,
+ HTAtom_name(rep_in), HTAtom_name(format_out));
+ CTRACE((tfp, "HTFormat(in HTParseZstdFile): %s\n", buffer));
+ rv = HTLoadError(sink, 501, buffer);
+ FREE(buffer);
+ result = rv;
+ }
+ } else {
+ /*
+ * Push the data down the stream
+ *
+ * @@ Bug: This decision ought to be made based on "encoding" rather than
+ * on content-type. @@@ When we handle encoding. The current method
+ * smells anyway.
+ */
+ targetClass = *(stream->isa); /* Copy pointers to procedures */
+ rv = HTZstdFileCopy(zfp, stream);
+ if (rv == -1 || rv == HT_INTERRUPTED) {
+ (*targetClass._abort) (stream, NULL);
+ } else {
+ (*targetClass._free) (stream);
+ }
+
+ fclose(zfp);
+ if (rv == -1) {
+ result = HT_NO_DATA;
+ } else if (rv == HT_INTERRUPTED || (rv > 0 && rv != HT_LOADED)) {
+ result = HT_PARTIAL_CONTENT;
+ } else {
+ result = HT_LOADED;
+ }
+ }
+ return result;
+}
+#endif /* USE_ZSTD */
+
/* Converter stream: Network Telnet to internal character text
* -----------------------------------------------------------
*
diff --git a/WWW/Library/Implementation/HTFormat.h b/WWW/Library/Implementation/HTFormat.h
index 835daef3f0..6015748791 100644
--- a/WWW/Library/Implementation/HTFormat.h
+++ b/WWW/Library/Implementation/HTFormat.h
@@ -238,11 +238,13 @@ The HTPresentation and HTConverter types
,encodingCOMPRESS = 4
,encodingBZIP2 = 8
,encodingBROTLI = 16
+ ,encodingZSTD = 32
,encodingALL = (encodingGZIP
+ encodingDEFLATE
+ encodingCOMPRESS
+ encodingBZIP2
- + encodingBROTLI)
+ + encodingBROTLI
+ + encodingZSTD)
} AcceptEncoding;
/*
@@ -549,6 +551,21 @@ HTParseBrFile: Parse a brotli'ed File through a file pointer
#endif /* USE_BROTLI */
+#ifdef USE_ZSTD
+/*
+HTParseZstdFile: Parse a zstd'ed File through a file pointer
+
+ This routine is called by protocols modules to load an object. uses
+ HTStreamStack and HTZstdFileCopy. Returns HT_LOADED if successful, can also
+ return HT_PARTIAL_CONTENT, HT_NO_DATA, or other <0 for failure.
+ */
+ extern int HTParseZstdFile(HTFormat format_in,
+ HTFormat format_out,
+ HTParentAnchor *anchor,
+ FILE *brfp,
+ HTStream *sink);
+#endif /* USE_ZSTD */
+
/*
HTNetToText: Convert Net ASCII to local representation
diff --git a/WWW/Library/Implementation/HTTP.c b/WWW/Library/Implementation/HTTP.c
index c47d446b71..f53c4c3fcb 100644
--- a/WWW/Library/Implementation/HTTP.c
+++ b/WWW/Library/Implementation/HTTP.c
@@ -713,6 +713,9 @@ static BOOL acceptEncoding(int code)
case encodingBROTLI:
program = HTGetProgramPath(ppBROTLI);
break;
+ case encodingZSTD:
+ program = HTGetProgramPath(ppZSTD);
+ break;
default:
break;
}
diff --git a/aclocal.m4 b/aclocal.m4
index 471d935931..617ad11d6a 100644
--- a/aclocal.m4
+++ b/aclocal.m4
@@ -7161,6 +7161,27 @@ AC_DEFUN([CF_WITH_BZLIB],[
],bz2,,,bzlib)
])dnl
dnl ---------------------------------------------------------------------------
+dnl CF_WITH_ZSTD version: 1 updated: 2024/06/29 21:30:00
+dnl --------------
+dnl Check for Zstd decoder library
+dnl
+dnl $1 = optional path for headers/library
+AC_DEFUN([CF_WITH_ZSTD],[
+ CF_ADD_OPTIONAL_PATH($1, [zstd library])
+
+ CF_FIND_LINKAGE([
+#include <zstd.h>
+],[
+
+ ZSTD_DStream *izsp = ZSTD_createDStream();
+ ZSTD_inBuffer ib;
+ ZSTD_outBuffer ob;
+ ZSTD_initDStream(izsp);
+ ZSTD_decompressStream(izsp, &ob, &ib);
+ ZSTD_freeDStream(izsp);
+],zstd,,,zstd)
+])dnl
+dnl ---------------------------------------------------------------------------
dnl CF_WITH_CURSES_DIR version: 4 updated: 2021/01/02 19:22:58
dnl ------------------
dnl Wrapper for AC_ARG_WITH to specify directory under which to look for curses
diff --git a/config.hin b/config.hin
index 764636144a..fbad64e492 100644
--- a/config.hin
+++ b/config.hin
@@ -293,6 +293,7 @@
#undef USE_SYSV_UTMP /* CF_UTMP */
#undef USE_X509_SUPPORT /* CF_GNUTLS, CF_SSL */
#undef USE_ZLIB /* AC_ARG_WITH(zlib) */
+#undef USE_ZSTD /* AC_ARG_WITH(zstd) */
#undef UTF8 /* CF_SLANG_CPPFLAGS */
#undef UTMPX_FOR_UTMP /* use <utmpx.h> since <utmp.h> not found */
#undef UUDECODE_PATH /* CF_PATH_PROG(uudecode) */
@@ -302,6 +303,7 @@
#undef XCURSES /* CF_PDCURSES_X11 */
#undef ZCAT_PATH /* CF_PATH_PROG(zcat) */
#undef ZIP_PATH /* CF_PATH_PROG(zip) */
+#undef ZSTD_PATH /* CF_PATH_PROG(zstd) */
#undef _WINDOWS_NSL /* CF_ARG_ENABLE(nsl-fork) */
#undef inline /* AC_C_INLINE */
#undef intptr_t /* AC_CHECK_TYPE(intptr_t,...) */
diff --git a/configure.in b/configure.in
index 475c544e59..832fa8b814 100644
--- a/configure.in
+++ b/configure.in
@@ -1326,6 +1326,19 @@ if test ".$use_brotli" != ".no" ; then
test "x$cf_cv_find_linkage_brotlidec" = "xyes" && AC_DEFINE(USE_BROTLI,1,[Define to 1 if you want to use libbrotli decompression])
fi
+dnl --------------------------------------------------------------------------
+AC_MSG_CHECKING(if you want to use zstd decompression)
+AC_ARG_WITH(zstd,
+[ --with-zstd use zstd decompression],
+ [use_zstd=$withval],
+ [use_zstd=yes])
+AC_MSG_RESULT($use_zstd)
+
+if test ".$use_zstd" != ".no" ; then
+ CF_WITH_ZSTD($use_zstd)
+ test "x$cf_cv_find_linkage_zstd" = "xyes" && AC_DEFINE(USE_ZSTD,1,[Define to 1 if you want to use libzstd decompression])
+fi
+
dnl --------------------------------------------------------------------------
CF_HELP_MESSAGE(
Other Network Services:)
@@ -1474,6 +1487,7 @@ CF_PATH_PROG(UNCOMPRESS,gunzip)
CF_PATH_PROG(UNZIP, unzip)
CF_PATH_PROG(BZIP2, bzip2)
CF_PATH_PROG(BROTLI, brotli)
+CF_PATH_PROG(ZSTD, zstd)
CF_PATH_PROG(TAR, tar, pax gtar gnutar bsdtar star)
CF_TAR_OPTIONS($TAR)
diff --git a/lynx.cfg b/lynx.cfg
index aa9ef975c1..18ad988f99 100644
--- a/lynx.cfg
+++ b/lynx.cfg
@@ -2262,6 +2262,7 @@ MINIMAL_COMMENTS:TRUE
#SUFFIX:.Z::compress
#SUFFIX:.gz::gzip
#SUFFIX:.bz2:application/x-bzip2
+#SUFFIX:.zst:application/zstd
#SUFFIX:.zip:application/zip
#SUFFIX:.lzh:application/x-lzh
#SUFFIX:.lha:application/x-lha
@@ -2407,6 +2408,7 @@ MINIMAL_COMMENTS:TRUE
# COMPRESS For compress
# BZIP2 For bzip2
# BROTLI For brotli
+# ZSTD For zstd
# ALL All of the above.
#PREFERRED_ENCODING:all
@@ -3574,6 +3576,10 @@ COLOR:6:brightred:black
# This is the path used for DIRED mode to create a zip-archive from one or more
# files, e.g., the program "unzip".
+.h2 ZSTD_PATH
+# This is the path used for DIRED mode and web connections to compress a file
+# to ".zst", e.g., the Unix command "zstd".
+
.h1 Interaction
.h2 FORCE_SSL_PROMPT
diff --git a/src/HTFWriter.c b/src/HTFWriter.c
index 0adaa80d5f..d8cd6017f2 100644
--- a/src/HTFWriter.c
+++ b/src/HTFWriter.c
@@ -49,6 +49,10 @@
#include <brotli/decode.h>
#endif
+#ifdef USE_ZSTD
+# include <zstd.h>
+#endif
+
/* contains the name of the temp file which is being downloaded into */
char *WWW_Download_File = NULL;
BOOLEAN LYCancelDownload = FALSE; /* exported to HTFormat.c in libWWW */
@@ -362,6 +366,123 @@ static void decompress_br(HTStream *me)
}
}
+static void decompress_zstd(HTStream *me) /* XXX too small a buffer; EINTR? */
+{
+ char *in_name = me->anchor->FileCache;
+ char copied[LY_MAXPATH];
+ FILE *ofp = LYOpenTemp(copied, ".tmp.zst", BIN_W);
+
+ if (ofp != NULL) {
+#ifdef USE_ZSTD
+ off_t dsize;
+ FILE *ifp;
+ ZSTD_DStream *izsp;
+ void *ibp, *obp;
+ size_t ibl, obl;
+
+ CTRACE((tfp, "decompressing '%s'\n", in_name));
+
+ ibl = ZSTD_DStreamInSize();
+ ibp = malloc(ibl);
+ if (ibp == NULL)
+ goto jout0;
+
+ obl = ZSTD_DStreamOutSize();
+ obp = malloc(obl);
+ if (obp == NULL)
+ goto jout1;
+
+ if ((izsp = ZSTD_createDStream()) == NULL)
+ goto jout2;
+ ZSTD_initDStream(izsp);
+
+ if ((ifp = fopen(in_name, BIN_R)) == NULL)
+ goto jout3;
+ CTRACE((tfp, "...opened '%s'\n", copied));
+
+ for (dsize = 0;;) {
+ ZSTD_inBuffer ib;
+ size_t r;
+
+ r = fread(ibp, 1, ibl, ifp);
+ if (r == 0)
+ break;
+ ib.src = ibp;
+ ib.size = r;
+ ib.pos = 0;
+
+ do {
+ ZSTD_outBuffer ob;
+ size_t w;
+
+ ob.dst = obp;
+ ob.size = obl;
+ ob.pos = 0;
+ r = ZSTD_decompressStream(izsp, &ob, &ib);
+ if (ZSTD_isError(r))
+ goto jout4;
+
+ for (w = 0; w < ob.pos;) {
+ size_t x;
+
+ x = fwrite(&((char*)obp)[w], 1, ob.pos - w, ofp);
+ if (x == 0)
+ goto jout4;
+ w += x;
+ }
+ dsize += w;
+ } while (ib.pos < ib.size);
+ }
+
+ LYCloseTempFP(ofp);
+ CTRACE((tfp, "...decompress %" PRI_off_t " to %" PRI_off_t "\n",
+ CAST_off_t (me->anchor->actual_length), CAST_off_t (dsize)));
+ if (LYRenameFile(copied, in_name) == 0)
+ me->anchor->actual_length = dsize;
+ (void) LYRemoveTemp(copied);
+
+jout4: fclose(ifp);
+jout3: ZSTD_freeDStream(izsp);
+jout2: free(obp);
+jout1: free(ibp);
+jout0:;
+
+#else
+# define FMT "%s -d --rm %s"
+ const char *program;
+
+ if ((program = HTGetProgramPath(ppZSTD)) == NULL) {
+ HTAlert(ERROR_UNCOMPRESSING_TEMP);
+ } else if (LYCopyFile(in_name, copied) == 0) {
+ char expanded[LY_MAXPATH];
+ char *command = NULL;
+
+ HTAddParam(&command, FMT, 1, program);
+ HTAddParam(&command, FMT, 2, copied);
+ HTEndParam(&command, FMT, 2);
+ if (LYSystem(command) == 0) {
+ struct stat stat_buf;
+
+ strcpy(expanded, copied);
+ *strrchr(expanded, '.') = '\0';
+ if (LYRenameFile(expanded, in_name) != 0) {
+ CTRACE((tfp, "rename failed %s to %s\n", expanded, in_name));
+ } else if (stat(in_name, &stat_buf) != 0) {
+ CTRACE((tfp, "stat failed for %s\n", in_name));
+ } else {
+ me->anchor->actual_length = stat_buf.st_size;
+ }
+ } else {
+ CTRACE((tfp, "command failed: %s\n", command));
+ }
+ free(command);
+ (void) LYRemoveTemp(copied);
+ }
+# undef FMT
+#endif
+ }
+}
+
/* Free an HTML object
* -------------------
*
@@ -404,6 +525,11 @@ static void HTFWriter_free(HTStream *me)
&& IsCompressionFormat(me->input_format, cftBrotli)
&& !strcmp(me->anchor->content_encoding, "br")) {
decompress_br(me);
+ } else if (me->anchor->FileCache != NULL
+ && me->anchor->no_content_encoding == FALSE
+ && IsCompressionFormat(me->input_format, cftZstd)
+ && !strcmp(me->anchor->content_encoding, "zstd")) {
+ decompress_zstd(me);
}
#ifdef VMS
else if (0 == strcmp(me->end_command, "SaveVMSBinaryFile")) {
@@ -469,6 +595,16 @@ static void HTFWriter_free(HTStream *me)
path[len - 3] = '\0';
(void) remove(path);
}
+ } else if (len > 4 && !strcasecomp(&path[len - 3], "zst")) {
+#ifdef USE_ZSTD
+ if (!skip_loadfile) {
+ use_zread = YES;
+ } else
+#endif /* USE_ZSTD */
+ {
+ path[len - 4] = '\0';
+ (void) remove(path);
+ }
} else if (len > 2 && !strcasecomp(&path[len - 1], "Z")) {
path[len - 2] = '\0';
(void) remove(path);
@@ -1323,6 +1459,13 @@ HTStream *HTCompressed(HTPresentation *pres,
compress_suffix = "br";
}
break;
+ case cftZstd:
+ if ((program = HTGetProgramPath(ppZSTD)) != NULL) {
+ StrAllocCopy(uncompress_mask, program);
+ StrAllocCat(uncompress_mask, " --rm -d %s");
+ compress_suffix = "zst";
+ }
+ break;
case cftCompress:
if ((program = HTGetProgramPath(ppUNCOMPRESS)) != NULL) {
/*
@@ -1500,6 +1643,20 @@ HTStream *HTCompressed(HTPresentation *pres,
StrAllocCopy(me->end_command, "");
} else
#endif /* USE_ZLIB */
+#ifdef USE_ZSTD
+ if (compress_suffix[0] == 'z' /* must be gzip */
+ && compress_suffix[1] == 's'
+ && compress_suffix[2] == 't'
+ && compress_suffix[3] == '\0'
+ && !me->viewer_command) {
+ /*
+ * We won't call gzip or compress externally, so we don't need to
+ * supply a command for it.
+ */
+ StrAllocCopy(me->end_command, "");
+ } else
+#endif /* USE_ZLIB */
+
{
me->end_command = NULL;
HTAddParam(&(me->end_command), uncompress_mask, 1, fnam);
diff --git a/src/HTInit.c b/src/HTInit.c
index 460bd1f9db..363e941913 100644
--- a/src/HTInit.c
+++ b/src/HTInit.c
@@ -1183,6 +1183,8 @@ void HTFileInit(void)
SET_SUFFIX1(".xz", "application/x-xz", "binary");
+ SET_SUFFIX1(".zst", "application/zstd", "binary");
+
SET_SUFFIX1(".lz", "application/x-lzip", "binary");
SET_SUFFIX1(".lzma", "application/x-lzma", "binary");
diff --git a/src/LYReadCFG.c b/src/LYReadCFG.c
index 24efbf51c6..acb69b71d6 100644
--- a/src/LYReadCFG.c
+++ b/src/LYReadCFG.c
@@ -1790,6 +1790,7 @@ static Config_Type Config_Table [] =
PARSE_SET(RC_XHTML_PARSING, LYxhtml_parsing),
PARSE_PRG(RC_ZCAT_PATH, ppZCAT),
PARSE_PRG(RC_ZIP_PATH, ppZIP),
+ PARSE_PRG(RC_ZSTD_PATH, ppZSTD),
PARSE_NIL
};
diff --git a/src/LYrcFile.c b/src/LYrcFile.c
index b8cef3776a..dee3e86083 100644
--- a/src/LYrcFile.c
+++ b/src/LYrcFile.c
@@ -122,6 +122,9 @@ Config_Enum tbl_preferred_encoding[] = {
#endif
#if defined(USE_BROTLI) || defined(BROTLI_PATH)
{ "br", encodingBROTLI },
+#endif
+#if defined(USE_ZSTD) || defined(ZSTD_PATH)
+ { "zstd", encodingZSTD },
#endif
{ "all", encodingALL },
{ NULL, -1 }
diff --git a/src/LYrcFile.h b/src/LYrcFile.h
index 1285a21666..fa7ac580e1 100644
--- a/src/LYrcFile.h
+++ b/src/LYrcFile.h
@@ -294,6 +294,7 @@
#define RC_XLOADIMAGE_COMMAND "xloadimage_command"
#define RC_ZCAT_PATH "zcat_path"
#define RC_ZIP_PATH "zip_path"
+#define RC_ZSTD_PATH "zstd_path"
extern Config_Enum tbl_cookie_version[];
extern Config_Enum tbl_force_prompt[];
--
2.45.2
