Joao Matos via GNU coreutils Bug Reports <[email protected]> writes:
> We found a bug in csplit that loses data when the input file has the
> name of one of the output files (xx00, xx01, ... by default). csplit
> creates (and so truncates) each output file before reading the part of
> the input that goes into it. If the input is one of these files and
> has not yet been read to the end, the part not yet read is lost.
>
> For xx00 this is all of the input, since nothing has been read yet.
> For the later output files, how far csplit has read is the previous
> split point rounded up to its read buffer, so whether data is lost
> depends on where the input is split, not only on its size. After the
> truncation, csplit sees an early end of file, and usually exits with
> status 0. If a later pattern then fails, csplit removes the output
> files it has created, the input among them.
>
> With regular expressions, both how far csplit has read and which
> output files are created depend on the input's contents: the split
> points are where the expressions match, and with a repeat count of {*}
> the number of output files is the number of matches. A leading %REGEX%
> only delays the creation of xx00 until the skipped lines have been
> read.
>
> We reproduced this with versions 8.32 (Ubuntu 22.04.5 LTS), 9.4
> (Ubuntu 24.04) and 9.7 (Debian 13). The results below are the same in
> all three except where noted; byte counts are from 9.4 and 9.7 (8.32
> differs by one byte).
>
> This bug was found by João Matos (joaorsmatos02) and Frederico Ramos
> (frediramos), as part of a research project at Instituto Superior
> Técnico, University of Lisbon (https://tecnico.ulisboa.pt/) and
> Carnegie Mellon University (https://www.cmu.edu/).
>
> ## Environment
>
> - Coreutils version: GNU coreutils 9.7
> - Operating system: Debian GNU/Linux 13
> - Architecture: x86_64
> - Locale: C.UTF8 (LANG=C.UTF-8, LC_ALL unset)
>
> ## Steps to reproduce
>
> ### Case 1: the input is named xx00
>
> $ seq 10 > xx00 && csplit xx00 5
>
> 8.32 and 9.4: "csplit: input disappeared", exit status 1, xx00 is left
> empty. 9.7: "csplit: '5': line number out of range", exit status 1,
> xx00 is deleted. Where the input is split does not matter.
>
> ### Case 2: a later output file, silent loss
>
> $ seq 300000 > xx01 && csplit xx01 2
> 2
> 8188
>
> Exit status 0, no message. xx01 now holds 8188 bytes; the input had 1988895.
>
> ### Case 3: a later pattern fails, and the input is deleted
>
> $ seq 300000 > xx01 && csplit xx01 2 50000
> csplit: '50000': line number out of range
> 8188
>
> Exit status 1. Both xx00 and xx01 are removed: nothing is left of the
> input. The same happens with regular expressions:
>
> $ seq 300000 > xx01 && csplit xx01 '/^1000/' '/299990/'
> csplit: '/^299990$/': match not found
> 3888
> csplit: '/299990/': match not found
> 4302
>
> ### Case 4: {*}: which files are at risk depends on the contents
>
> $ seq 300000 > xx02 && csplit xx02 '/0000$/' '{*}'
>
> Exit status 0. Only xx00 to xx02 are created, and xx02 holds 5766
> bytes. The same command on a file named xx31 loses nothing: there are
> 30 matches, so only xx00 to xx30 are created.
>
> ### Case 5: a leading %REGEX%
>
> $ seq 300000 > xx00 && csplit xx00 '%^5$%' 10
> 10
> 8172
>
> Exit status 0. xx00 holds lines 5 to 9 as expected, but xx01, which
> should hold the rest of the input, holds only 8172 bytes.
>
> ## Fix
>
> We have a patch (along with a new test) that does what split already
> does: opens each output file without O_TRUNC, compares it with the
> input (same device and inode), and fails with "'xx00' would overwrite
> input; aborting" before writing anything. The input is left intact.
I'm a bit confused by this. I'm not aware of any csplit implementation
that doesn't overwrite files like this, opening with O_TRUNC. We also
document in the manual that xx00 - xx99 are the file names created by
default, and that they will be removed if 'csplit' runs into an error.
So, by using the same file name as the input file name, I feel that the
user is acknowledging the possibility of data loss here. It is similar
to an invocation like this:
$ echo a > a; cat a > a; wc -l < a
0
Also note that you can use --prefix (-f) to change the prefix from the
default of "xx". Or use --keep-files (-k) to preserve the output files
even when an error occurs. Those would solve any issues.
Am I missing something here?
Collin
