--- Begin Message ---
Package: ckermit
Version: 416~beta12-1
Severity: grave
Tags: patch upstream
Justification: causes non-serious data loss
ckermit has a system that is designed to automatically convert files when
transferring between different platforms.
Unfortunately, this system is enabled by default and it most often does the
wrong thing in a modern context.
There are two settings that impact this:
SET FILE TYPE BINARY
SET TRANSFER MODE MANUAL
Fortunately, the file type is already binary. However, according to the help
text:
When TRANSFER MODE is AUTOMATIC (as it is by default), various automatic
methods (depending on the platform) are used to determine whether a file
is transferred in text or binary mode; these methods (which might include
content scan (see SET FILE SCAN below), filename pattern matching (SET FILE
PATTERNS), client/server "kindred-spirit" recognition, or source file
record format) supersede the FILE TYPE setting but can, themselves, be
superseded by including a /BINARY or /TEXT switch in the SEND, GET, or
RECEIVE command.
When TRANSFER MODE is MANUAL, the automatic methods are skipped for sending
files; the FILE TYPE setting is used instead, which can be superseded on
a per-command basis with a /TEXT or /BINARY switch.
Unfortunately, transfer mode is indeed automatic by default.
When sending between Unix platforms, the Kermits recognize that they are similar
and always use binary mode (this uses a "sysid" system).
However, when sending between Unix and other platforms (Windows, etc.), the
Kermits recognize they are different and the automatic methods are used. The
default file type is binary, but the automatic methods can override it.
The enclosed patch will set the transfer mode to manual by default. There is a
small chance this may change behavior for certain legacy scripts; they can
always add "set transfer mode automatic" to the start. This automatic behavior
has not been static throughout the life of ckermit, however, and this would not
be the first time changes have been made.
Prior to this change, "show file" shows, among other things:
Transfer mode: automatic
File patterns: automatic (SHOW PATTERNS for list)
Default file type: binary
With this change:
Transfer mode: manual
File patterns: automatic (but disabled by TRANSFER-MODE MANUAL)
File type: binary
The modern assumption is a byte-accurate transfer of files. We have had a
proliferation of file types, extensions, and complicating circumstances since
the earlier days of Kermit.
By changing this default, we disable the heuristic for attempting to guess the
type of files, and convert the existing binary default into a binary setting.
This can always be changed by the user, but the idea is to not violate the
principle of least surprise. If the user asks to transfer a file, we assume the
user wants an exact transfer of the file unless stated otherwise.
Additionally, some platforms (eg, HP48 calculators) have wildly different
behavior depending on whether a text or binary transfer is requested. By
defaulting to manual mode, the user is in charge and further surprises that may
be caused by "set file type" being ignored can be avoided.
It should be noted that the retiring head of the ckermit project, Frank da Cruz,
was reported to have said: "The default transfer mode is "auto", where it uses
an time-tested algorithm to look at the contents of each file to see if it
contains only text characters, in which case the xfer mode is text, otherwise
it's binary. Text is sent as ascii lines terminated by CRLF, and all other
Kermits understand this and convert to their own text format, e.g. Unix (CR
instead of CRLF), IBM OS/MVT (which also converts from ASCII to EBCDIC (did I
forget how to spell that?)); this scheme has worked smoothly for decades and
changing it would be nasty surprise for most users, would break countless
scripts and init files, etc.
It's especially important to keep this behavior when sending a group
of files where some are binary and others are text. Kermit automatically
switches mode for each file. (I'm sure you know all this, so I'm just
being careful.)" via
https://github.com/KermitProject/ckermit/pull/15#issuecomment-3560129029
While I appreciate Frank's perspective, I believe we are in a different era now
than when ckermit was initially written. Performing those changes, silently, by
default, is interpreted as corruption by modern systems even though, in a
different era, it was helpful. Cryptographic signatures will fail to verify,
hashes will change, etc.
Yes, even if it's a .txt or .c file or whatever, changing the file size and
bytes that make it up will be interpreted as corruption by lots of systems. A
property where files that are transferred by sftp, zip, tar, 7z, etc. have one
content, and those sent by kermit have another, is also strongly undesirable for
systems that do deduplication (backup systems and filesystems like btrfs and
zfs) and those that do content-based hardlinking.
My point here is that even if it contains only text characters, changing the
content of the file is a big deal on modern systems and with modern security
systems.
I would absolutely think this change is something to be highlighted in the
release notes. Fortunately it is very easy to workaround with a single line
consisting of SET TRANSFER MODE AUTO in .kermrc or other scripts.
The fact that even I did not realize that SET FILE TYPE BINARY was insufficient
to prevent corruption even after years of using kermit is telling.
Additionally, it looks like behavior around this has changed in the past.
https://www.kermitproject.org/faq-c-bix.html says "Both Kermit and ftp transfer
files in text mode by default." That's not the case in the current Kermit 10
betas; I don't know when it changed.
That page also says "You can tell Kermit to skip all conversions and transfer
the file literally, as-is, with the command SET FILE TYPE BINARY", which is not
correct because the trransfer mode can override that by default.
This has clearly been a source of confusion for many people for many years.
I confirmed just now, that using the latest Kermit 95 to the latest ckermit on
Linux, sending a text file with a .txt extension -- even with the file type set
to binary -- produces output that has fewer bytes on Linux than it did on
Windows.
I myself have appreciated the text transfer mode and of course this doesn't take
it away. My HP-48GX calculator did something wildly different when in text vs.
binary mode (basically, sending the source code of a program vs. bytecode of it)
and of course sometimes the automatic CRLF conversion is helpful. It shouldn't
go away! Just that the defaults should be suitable for a modern environment,
recognizing that the passage of time has changed the default-conversion approach
from helpful to corruption.
I also observed that with the default settings, a file saved in UTF-16 on
Windows had its BOM stripped and was converted to an 8-bit encoding (unclear of
it was UTF-8 or latin-1 or what, since it didn't use any non-ASCII characters)
on Linux.
When transferred back to Windows, it was not converted back to UTF-16. The
round-trip, therefore, was also lossy.
SET FILE TRANSFER MODE MANUAL prevents this issue as well.
Let me back up a moment to the big picture.
To assume auto transfer mode enabled by default is a good idea, we have to
believe all of these propositions:
1. The primary purpose of sending files to a different platform is to interact
with them on that platform
2. The identity of the sending and receiving platforms is sufficient to
determine the proper line-ending encodings
3. Interaction with text files from a different platform requires assistance in
transforming them
4. The file transfer tool is the appropriate place for this, and default-on
does not violate the principle of least surprise
5. The benefit of these outweighs the harm
None of these are clearly true, and most of them are clearly false.
THE PRIMARY PURPOSE OF SENDING FILES
In many cases, the primary purpose of sending text files between platforms is
not to interact with them on the target. For instance, if you are running
Windows and have mounted a network share, that network share might be hosted on
Windows or Linux (samba), and the files may never ever be accessed except on
Windows clients.
Likewise if ckermit is used to send text files to a file server, or a backup
server, or whatever, it is quite possible -- and, these days, probably even
likely -- that the intent is not to interact with them on that system, but
simply to store them on that system. For instance, I transfer files to a hosting
directories for web, Gopher, and Gemini on a server. I never look at that on
that server itself; they are only viewed by clients (which may be of many
platforms)
Most transfer and synchronization software tries to be platform-independent and
preserve your content regardless of the client and server platforms.
DETERMINING LINE-ENDING AND CHARACTER SET ENCODINGS
ckermit currently appears to assess what line endings to use based on what
platforms are at the sending and receiving end. This is incorrect, because any
given system may contain files that originated on any other given system, and
moreover may contain files intended for any other given system, in any
combination.
My file server contains text files originating on DOS, modern Windows, MacOS,
and *nix systems. It would be an impossibility for ckermit to correctly
determine what encoding to use for a file placed in a given directory. It may be
able to guess the encoding of a file by examining it, but even that gets dicey
with character sets.
INTERACTION WITH TEXT FILES REQUIRES ASSISTANCE
Every modern editor for text files, from vim on up, supports autodetection and
preservation of line endings. Granted, cat on Unix may do the wrong thing with a
file from Windows or Mac, but these issues are both routine and easily solved
these days.
THE APPROPRIATE PLACE FOR THIS, AND THE PRINCIPLE OF LEAST SURPRISE
I submit that the transfer tool is the wrong place for this (by default). Nobody
expects that these days. Here is a partial list of software commonly used to
exchange files that does not perform this translation:
rsync
x/y/zmodem
UUCP
NNCP
scp
sftp
SMB/samba/CIFS
NFS
Dropbox
Syncthing
Nextcloud
Google Drive
Amazon S3
tar
dar
xz
rdiff
unzip (has optional text file conversion, off by default)
curl, wget, other http clients (curl supports ASCII mode for FTP but it
defaults off)
gzip, bzip2, zstd, etc.
In fact, the only other one I can think of that does this by default in some
circumstances is some very old FTP clients. Modern ones like lftp and ncftp use
binary mode by default.
So this /definitely/ violates the principle of least surprise. Tools like
dos2unix are widely understood to perform the conversion for those cases where
it's needed, but it rarely is anymore.
THE BENEFIT VS. THE HARM
The harm is significant.
It will cause severe breakage to:
content-addressible (hashing) systems, such as git-annex
Cryptographic signature verification tools
Tools that create or apply binary deltas (rdiff, xdelta, dar)
Some version control systems
rsync (will cause it to perceive a file to be different and generally
require a full retransmit since the typical line size is less than the
typical rsync block size)
syncthing
many other syncing tools may perceive a conflict
There is utility to this conversion, especially for very old systems or ones
that don't have what amounts to the typical notion of a file or one of the big
three line-ending approaches (Microsoft, Mac, and Unix). But I don't see much
utility, and a significant amount of harm, to it being enabled by default. As
someone that's had to get data off an AS/400, which neither uses ASCII nor has a
standard notion of a file, I get it. But I'd rather request that mode explicitly
than have an algorithm decide it for me -- especially when that algorithm may
change between releases.
As for an extension map, I think that is a losing battle. If that is truly the
desire, libmagic is probably a better bet (it's been around for a long time but
may not be on all platforms)
This is not binding on Debian, but as a side note, the long-time upstream Kermit
maintainer is retiring from the project and there may be a need for me to step
up and maintain kermit in a more direct way. One former Kermit programmer, from
some decades back, has registered a ckermit project on Github but has been
hostile to most suggestions, including this one. His repos were dormant for
years before I started asking about the future of Kermit after Frank's
retirement, and have gone dormant again after I decided his bikeshedding was not
worth the time. However, for completeness, I note that I initially reported
this issue at https://github.com/KermitProject/ckermit/pull/15 and you can find
Jeffrey's comments there as well.
As ckermit maintainer in Debian, my conscience is troubled with the idea of
allowing software to continue to exist with known data corruption issues in the
default configuration. Therefore I intend to patch this in Debian, especially
since any compatibility issues are likely minimal and easily addressed.
-- System Information:
Debian Release: 13.2
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.12.57+deb13-amd64 (SMP w/16 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages ckermit depends on:
ii debconf [debconf-2.0] 1.5.91
ii libc6 2.41-12
ii libncurses6 6.5+20250216-2
ii libpam0g 1.7.0-5
ii libssl3t64 3.5.4-1~deb13u1
ii libtinfo6 6.5+20250216-2
Versions of packages ckermit recommends:
ii openssh-client [ssh-client] 1:10.0p1-7
Versions of packages ckermit suggests:
pn openbsd-inetd | inet-superserver <none>
-- debconf information excluded
commit 3b4af6b057ca1f1f17e5b262cf76b7f9b1394779
Author: John Goerzen <[email protected]>
Date: Tue Nov 18 01:36:49 2025 +0000
Disable auto transfer mode
Prior to this change, "show file" shows, among other things:
Transfer mode: automatic
File patterns: automatic (SHOW PATTERNS for list)
Default file type: binary
With this change:
Transfer mode: manual
File patterns: automatic (but disabled by TRANSFER-MODE MANUAL)
File type: binary
The modern assumption is a byte-accurate transfer of files. We have had
a proliferation of file types, extensions, and complicating
circumstances since the earlier days of Kermit.
By changing this default, we disable the heuristic for attempting to
guess the type of files, and convert the existing binary default into a
binary setting.
This can always be changed by the user, but the idea is to not violate
the principle of least surprise. If the user asks to transfer a file,
we assume the user wants an exact transfer of the file unless stated
otherwise.
Additionally, some platforms (eg, HP48 calculators) have wildly
different behavior depending on whether a text or binary transfer is
requested. By defaulting to manual mode, the user is in charge and
further surprises that may be caused by "set file type" being ignored
can be avoided.
diff --git a/ckcftp.c b/ckcftp.c
index 7780cdd..f4f813f 100644
--- a/ckcftp.c
+++ b/ckcftp.c
@@ -951,7 +951,7 @@ int ftp_log = 1; /* FTP Auto-login */
int sav_log = -1;
int ftp_action = 0; /* FTP action from command line */
int ftp_dates = 1; /* Set file dates from server */
-int ftp_xfermode = XMODE_A; /* FTP-specific transfer mode */
+int ftp_xfermode = XMODE_M; /* FTP-specific transfer mode */
char ftp_reply_str[FTP_BUFSIZ] = ""; /* Last line of previous reply */
char ftp_srvtyp[SRVNAMLEN] = { NUL, NUL }; /* Server's system type */
diff --git a/ckcmai.c b/ckcmai.c
index a84b9cb..8c08087 100644
--- a/ckcmai.c
+++ b/ckcmai.c
@@ -1408,7 +1408,7 @@ int deblog = 0, /* Debug log is
open */
cursor_save = -1, /* Cursor state */
#endif /* OS2 */
- xfermode = XMODE_A, /* Transfer mode, manual or auto */
+ xfermode = XMODE_M, /* Transfer mode, manual or auto */
xfiletype = -1, /* Transfer only text (or binary) */
recursive = 0, /* Recursive directory traversal */
nolinks = 2, /* Don't follow symbolic links */
--- End Message ---
pgp9TjdyTkTsW.pgp
