Re: editing a binary file
well, one of my colleagues pointed out a feature of fortran 2003, which I, being an idiot, have missed. YOu have access='stream' in f2003, which is all I need. No record separators, just data. many thanks for all your help and advice. anton -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Editing a binary file
On Fri, Dec 18, 2009 at 09:33:49AM -0700, Warren Block wrote:
per...@pluto.rain.com wrote:
Greg Larkin glar...@freebsd.org wrote:
...
truncate -4 myfile should get rid of the last four bytes. Maybe
there's a similar efficient way to truncate the start of a file.
This should do it:
dd if=oldfile of=newfile bs=1 skip=4
Or, perhaps marginally more efficient:
dd if=oldfile of=newfile bs=4 skip=1
It would be nice to avoid the file copy, but maybe there's no way to do
that. The small buffer size for dd will probably make copies of
multi-gig files slow. This might be faster:
tail -c +5 myfile outfile
truncate -4 outfile
yes, quite. On 1.5GHz ia64, on 1GB binary file tail takes about 25 s,
but dd.. I killed after 25 min (!) and it had only done 1/3 of the file.
But even tail is too slow.
So I'll probably have to write a C I/O routine and avoid fortran I/O
alltogether, so I write straight away just my data.
I'm a ksh partisan, so I tried it this way:
{ dd bs=4 count=1 of=/dev/null ; cat ; } oldfile newfile
I ran this on a 640M file residing on a 10K rpm SCSI disk on an old 5.4 system.
(Yes, I'm trying to upgrade but the ports are killing me; I may have q?s
later.) It took 111 seconds wall time. Not great, not bad for 640M in the
file system. Both files were on the same disk, which was buzzing along at
about 120 tps.
I'm sure this is possible in csh, though I'd have to spend some man page time
to get the syntax right.
Yes, a custom program will be faster if you go through stdio or C++'s iostreams
AND OPEN THE FILE EXPLICITLY because they do the read via mmap, saving one
copy. If you do the read via read(2) it won't be that much faster. I suspect
(but have not bothered to prove) that in this case cat(1) used simple reads.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Thu, Dec 17, 2009 at 06:38:04PM -0700, Warren Block wrote: On Fri, 18 Dec 2009, Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. thank you, that works fine: hexdump -C fort.10 01 00 00 00 01 01 00 00 00 |.| 0009 truncate -s -4 fort.10 hexdump -C fort.10 01 00 00 00 01|.| 0005 -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Thu, Dec 17, 2009 at 08:45:02PM -0500, Greg Larkin wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Warren Block wrote: On Fri, 18 Dec 2009, Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. -Warren Block * Rapid City, South Dakota USA This should do it: dd if=oldfile of=newfile bs=1 skip=4 yes, this is very useful: hexdump -C fort.10 01 00 00 00 08 01 00 00 00 |.| 0009 truncate -s -4 fort.10 hexdump -C fort.10 01 00 00 00 08|.| 0005 dd if=fort.10 of=zzz bs=1 skip=4 1+0 records in 1+0 records out 1 bytes transferred in 0.94 secs (10645 bytes/sec) hexdump -C zzz 08|.| 0001 many thanks anton -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Fri, Dec 18, 2009 at 09:33:49AM -0700, Warren Block wrote: per...@pluto.rain.com wrote: Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 It would be nice to avoid the file copy, but maybe there's no way to do that. The small buffer size for dd will probably make copies of multi-gig files slow. This might be faster: tail -c +5 myfile outfile truncate -4 outfile yes, quite. On 1.5GHz ia64, on 1GB binary file tail takes about 25 s, but dd.. I killed after 25 min (!) and it had only done 1/3 of the file. But even tail is too slow. So I'll probably have to write a C I/O routine and avoid fortran I/O alltogether, so I write straight away just my data. many thanks anton -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
Anton Shterenlikht wrote:
On Fri, Dec 18, 2009 at 02:09:58AM +0100, Rolf Nielsen wrote:
Anton Shterenlikht wrote:
I'm creating binary files in fortran.
Fortran adds 4 byte record delimiters at the beginning
and the end of each record, which, in the case of a binary
file, is just at the beginning and at the end of the file.
I need to delete these record delimiters, because the
software I use to visualise the binary files interprets
them as data. But I don't know how. I've looked at
hexdump and od, but those are only dumping (I think)
file contents, and I cannot see how to edit a file with them.
Any advice?
many thanks
anton
Hello Anton,
My bet would be /usr/ports/editors/hexedit. Been a while since I've used
it, but AFAIR, it has a curses or a curses like interface, and it's
fairly simple to use, yet sufficiently powerful for most normal binary
editing. If you want a GUI, I believe gnome (and probably KDE as well)
has its own hex editor.
thank you. hexedit does the job on small files, but is quite
clunky. If I've a xGB file and I need to delete the first and
the last record, this becomes quite hard, if at all possible.
I didn't appreciate it's not that simple.
Perhaps I can read a file with C and write back? I can't
remember if C supports binary files, and whether it
also writes some record delimiters.
Sure, you can write a fairly short C program to do this. In fact,
it's pretty easy in perl too:
#!/usr/bin/perl -w
use Fcntl;
use constant BUFFSIZ = 4096;
for my $file (@ARGV) {
my $buffer = '';
my $bytes_read = 0;
sysopen INFILE, $file, O_RDONLY
or die Failed to open file $file for reading -- $!\n;
sysseek INFILE, 4, 0; # skip first 4 bytes
sysopen OUTFILE, ${file}.out, O_WRONLY|O_CREAT
or die Failed to open file ${file}.out for writing -- $!\n;
while ( $bytesread = sysread INFILE, $buffer, BUFFSIZ, 0 ) {
# If we don't read 4096 bytes, try a second read: if this
# returns zero, then we're at EOF
if ( $bytes_read BUFFSIZ ) {
my $offset = $bytes_read;
$bytes_read = sysread INFILE, $buffer, BUFFSIZ, $offset;
if ( $bytes_read == 0 ) {
# Trim the last 4 bytes
substr ($buffer, -4) = ''; # Trim off last 4 bytes
}
}
syswrite OUTFILE, $buffer;
}
close INFILE;
close OUTFILE;
}
Untested, and needs more error checking around those sysread()s and syswrite()s,
but it should give you the general idea.
Cheers,
Matthew
--
Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard
Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
Kent, CT11 9PW
signature.asc
Description: OpenPGP digital signature
Re: editing a binary file
On Fri, 18 Dec 2009 01:29:18 +, Anton Shterenlikht me...@bristol.ac.uk
wrote:
My bet would be /usr/ports/editors/hexedit. Been a while since I've
used it, but AFAIR, it has a curses or a curses like interface, and
it's fairly simple to use, yet sufficiently powerful for most normal
binary editing. If you want a GUI, I believe gnome (and probably KDE
as well) has its own hex editor.
thank you. hexedit does the job on small files, but is quite
clunky. If I've a xGB file and I need to delete the first and the last
record, this becomes quite hard, if at all possible.
I didn't appreciate it's not that simple.
Perhaps I can read a file with C and write back? I can't remember if C
supports binary files, and whether it also writes some record
delimiters.
Yes, C supports binary files and does not insert spurious 'record
delimiters' unless you instruct it to do so. It may even be possible to
use one of the scripting languages (Perl or Python) to do the same work.
It's often easier to hack together a solution if you don't have to worry
about some of the details C will require.
I don't know how your record delimiters look like, but here's a small
sample of how Python can read a binary file of 32 bytes and strip the
last 2 bytes of each 16-byte record:
A binary file of two 16-byte records may look like this:
keram...@kobe:/tmp$ hd binfile
b6 b0 fc 58 96 48 56 d5 e9 10 f0 55 55 67 87 5d |...X.HVUUg.]|
0010 b0 c9 8b 49 db 53 26 28 57 d6 62 0d d5 1b c4 dc |...I.S(W.b.|
0020
Reading the file in chunks of 16 bytes and stripping the last 2 bytes of
each record from Python is only a few lines of code:
keram...@kobe:/tmp$ python
Python 2.6.4 (r264:75706, Dec 3 2009, 23:31:07)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd9
Type help, copyright, credits or license for more information.
ifp = file('binfile') # open input file for reading
ofp = file('outfile', 'w') # open output file for writing
for rec in range(2):# we'll transfer 2 records
... bytes = ifp.read(16)# of 16 bytes each
... obytes = bytes[0:14]# strip the last two bytes of each
record
... ofp.write(obytes) # push to the output file
...
ifp.close() # close input
ofp.close() # close output
The output file now looks like this:
keram...@kobe:/tmp$ hd outfile
b6 b0 fc 58 96 48 56 d5 e9 10 f0 55 55 67 b0 c9 |...X.HVUUg..|
0010 8b 49 db 53 26 28 57 d6 62 0d d5 1b |.I.S(W.b...|
001c
This is 4 bytes smaller than the original file, and the last two bytes
of each 16-byte record are gone. Bingo!
Now this example is really a very small and contrived sample of what you
can do. This script lacks serious error-checking too, and it may be
slightly more involved if you have variable record sizes. But the
general idea is that it *is* possible to hack together something that
loads and processes binary data. As long as you know the on-disk format
of the records you are reading, anything goes.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
If your Fortran file has the same word size and enddian-ness as your
C, this simple program convert.c will strip all the record length
fields. I just knocked it up now, no warranty, etc, but it works for
me. Use as a pipe:
$ ls
convert.c test.f
$ gcc -Wall -Werror -ansi -pedantic convert.c -o convert
$ gfortran44 test.f -o test
$ ./test
$ ls -l test-output
-rw-r--r-- 1 nb nb 2460 Dec 18 11:17 test-output
$ ./convert test-output test-converted
$ ls -l test-converted
-rw-r--r-- 1 nb nb 2420 Dec 18 11:18 test-converted
$
The code does a fair amount of checking; if you get one of the error
messages, let us know. The most obvious unchecked problem is a short
read, which will complain about mismatched lengths.
If your Fortran has different word sizes or enddian-ness (e.g. most of
the Fortran output files I use on the Clear Climate Code project
http://clearclimatecode.org/ are generated on big-endian machines),
you will need to add code to tweak the 'size' value after reading it,
and when checking the record-end marker.
Nick B
/* convert.c: remove record length fields from Fortran output file. */
/* Nick Barnes, Ravenbrook Limited, 2009-12-18 */
#include stdlib.h
#include stdio.h
#include unistd.h
#include assert.h
int main(void)
{
long size;
char *buf;
ssize_t bytes;
assert(sizeof(size) == 4);
while(bytes = read(0, (void*)size, sizeof(size))) {
if (bytes 0) {
fprintf(stderr, read() returned %ld\n, bytes);
exit(1);
}
if (size = 0) {
fprintf(stderr, Read bad record length %ld\n, size);
exit(1);
}
buf = (char*)malloc(size + sizeof(size));
if (!buf) {
fprintf(stderr, Couldn't allocate buffer of %ld
bytes\n,
size + sizeof(size));
exit(1);
}
bytes = read(0, buf, size + sizeof(size));
if (bytes = 0) {
fprintf(stderr, read() returned %ld\n, bytes);
exit(1);
}
if ((*(long*)(buf+size)) != size) {
fprintf(stderr, Mismatched record lengths: %ld, %ld\n,
size, *(long*)(buf+size));
exit(1);
}
write(1, buf, size);
free(buf);
}
return 0;
}
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
If you don't mind, I would add fixed buffer processing to your program.
For some really huge files (or any other type of stream) which have
really huge records, reading entire records into memory would get the
box down.
Markiyan
/* convert.c: remove record length fields from Fortran output file. */
/* Nick Barnes, Ravenbrook Limited, 2009-12-18 */
#include stdlib.h
#include stdio.h
#include unistd.h
#include assert.h
#define BUFLEN 1024*1024
static char buf[BUFLEN];
int main(void)
{
long size, size_in, size_out;
ssize_t bytes;
assert(sizeof(size) == 4);
int buflen;
while((bytes = read(0, (void*)size, sizeof(size {
if (bytes 0) {
fprintf(stderr, read() returned %d\n, bytes);
exit(1);
}
if (size = 0) {
fprintf(stderr, Read bad record
length %ld\n, size);
exit(1);
}
size_in = size;
buflen = size_in BUFLEN ? BUFLEN : size_in;
while (size_in 0
(bytes = read(0, buf, buflen)) 0) {
write(1, buf, bytes);
size_in -= bytes;
if (size_in BUFLEN) {
/* the last chunk is less than BUFLEN */
buflen = size_in;
}
}
if (bytes == 0) {
/* EOF */
if (size_in 0) {
fprintf(stderr, premature end of
file: %ld bytes unread.\n,
size_in);
exit(1);
}
break;
} else if (bytes = 0) {
fprintf(stderr, read() returned %d\n, bytes);
exit(1);
}
if (read(0, (void *)size_out, sizeof(size_out)) 0) {
if (size_out != (size)) {
fprintf(stderr,
Mismatched record lengths:
%ld, %ld\n, size, size_out);
exit(1);
}
}
}
return 0;
}
Nick Barnes wrote:
If your Fortran file has the same word size and enddian-ness as your
C, this simple program convert.c will strip all the record length
fields. I just knocked it up now, no warranty, etc, but it works for
me. Use as a pipe:
$ ls
convert.c test.f
$ gcc -Wall -Werror -ansi -pedantic convert.c -o convert
$ gfortran44 test.f -o test
$ ./test
$ ls -l test-output
-rw-r--r-- 1 nb nb 2460 Dec 18 11:17 test-output
$ ./convert test-output test-converted
$ ls -l test-converted
-rw-r--r-- 1 nb nb 2420 Dec 18 11:18 test-converted
$
The code does a fair amount of checking; if you get one of the error
messages, let us know. The most obvious unchecked problem is a short
read, which will complain about mismatched lengths.
If your Fortran has different word sizes or enddian-ness (e.g. most of
the Fortran output files I use on the Clear Climate Code project
http://clearclimatecode.org/ are generated on big-endian machines),
you will need to add code to tweak the 'size' value after reading it,
and when checking the record-end marker.
Nick B
/* convert.c: remove record length fields from Fortran output file. */
/* Nick Barnes, Ravenbrook Limited, 2009-12-18 */
#include stdlib.h
#include stdio.h
#include unistd.h
#include assert.h
int main(void)
{
long size;
char *buf;
ssize_t bytes;
assert(sizeof(size) == 4);
while(bytes = read(0, (void*)size, sizeof(size))) {
if (bytes 0) {
fprintf(stderr, read() returned %ld\n, bytes);
exit(1);
}
if (size = 0) {
fprintf(stderr, Read bad record length %ld\n, size);
exit(1);
}
buf = (char*)malloc(size + sizeof(size));
if (!buf) {
fprintf(stderr, Couldn't allocate buffer of %ld
bytes\n,
size + sizeof(size));
exit(1);
}
bytes = read(0, buf, size + sizeof(size));
if (bytes = 0) {
fprintf(stderr, read() returned %ld\n, bytes);
exit(1);
}
if ((*(long*)(buf+size)) != size) {
fprintf(stderr, Mismatched record lengths: %ld, %ld\n,
size, *(long*)(buf+size));
exit(1);
}
write(1,
Re: editing a binary file
per...@pluto.rain.com wrote: Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 It would be nice to avoid the file copy, but maybe there's no way to do that. The small buffer size for dd will probably make copies of multi-gig files slow. This might be faster: tail -c +5 myfile outfile truncate -4 outfile (Has anyone mentioned that you can edit binary files interactively with vi yet? No? Well, it's horrific and surely has interesting failure modes. And there are probably disadvantages also.) -Warren Block * Rapid City, South Dakota USA ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
At 2009-12-18 16:33:49+, Warren Block writes: per...@pluto.rain.com wrote: Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 It would be nice to avoid the file copy, but maybe there's no way to do that. The small buffer size for dd will probably make copies of multi-gig files slow. This might be faster: tail -c +5 myfile outfile truncate -4 outfile (Has anyone mentioned that you can edit binary files interactively with vi yet? No? Well, it's horrific and surely has interesting failure modes. And there are probably disadvantages also.) All very interesting, but the OP is wanting to lose all the Fortran record markers, not just the first (and last) four bytes of the file. The record markers precede and follow each record, and give the record's length. The size and enddian-ness of the record marker itself depends on the Fortran implementation. Nick B ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Fri, Dec 18, 2009 at 09:33:49AM -0700, Warren Block wrote: per...@pluto.rain.com wrote: Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 It would be nice to avoid the file copy, but maybe there's no way to do that. The small buffer size for dd will probably make copies of multi-gig files slow. This might be faster: tail -c +5 myfile outfile truncate -4 outfile (Has anyone mentioned that you can edit binary files interactively with vi yet? No? Well, it's horrific and surely has interesting failure modes. And there are probably disadvantages also.) Vim, yes. I tried, but failed. At the moment dd/truncate combination seems the most appealing. But I'll look at C/perl/python proposed solutions as well. many thanks -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Fri, Dec 18, 2009 at 04:38:16PM +, Nick Barnes wrote: At 2009-12-18 16:33:49+, Warren Block writes: per...@pluto.rain.com wrote: Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 It would be nice to avoid the file copy, but maybe there's no way to do that. The small buffer size for dd will probably make copies of multi-gig files slow. This might be faster: tail -c +5 myfile outfile truncate -4 outfile (Has anyone mentioned that you can edit binary files interactively with vi yet? No? Well, it's horrific and surely has interesting failure modes. And there are probably disadvantages also.) All very interesting, but the OP is wanting to lose all the Fortran record markers, not just the first (and last) four bytes of the file. The record markers precede and follow each record, and give the record's length. The size and enddian-ness of the record marker itself depends on the Fortran implementation. actually the file consists of just one (potentially very long) record, so erasing the first and the last 4 bytes is all I need. I haven't had a chance to look at the proposed solutions, but many thanks for all your advice. -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
Nick Barnes wrote: All very interesting, but the OP is wanting to lose all the Fortran record markers, not just the first (and last) four bytes of the file. The record markers precede and follow each record, and give the record's length. The size and enddian-ness of the record marker itself depends on the Fortran implementation. He did say his binary files consisted of a single record... Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: editing a binary file
Anton Shterenlikht wrote: On Fri, Dec 18, 2009 at 09:33:49AM -0700, Warren Block wrote: per...@pluto.rain.com wrote: Greg Larkin glar...@freebsd.org wrote: ... truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. This should do it: dd if=oldfile of=newfile bs=1 skip=4 Or, perhaps marginally more efficient: dd if=oldfile of=newfile bs=4 skip=1 It would be nice to avoid the file copy, but maybe there's no way to do that. The small buffer size for dd will probably make copies of multi-gig files slow. This might be faster: tail -c +5 myfile outfile truncate -4 outfile (Has anyone mentioned that you can edit binary files interactively with vi yet? No? Well, it's horrific and surely has interesting failure modes. And there are probably disadvantages also.) Vim, yes. I tried, but failed. At the moment dd/truncate combination seems the most appealing. But I'll look at C/perl/python proposed solutions as well. many thanks ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
editing a binary file
I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. Any advice? many thanks anton -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. Any advice? many thanks anton Hello Anton, My bet would be /usr/ports/editors/hexedit. Been a while since I've used it, but AFAIR, it has a curses or a curses like interface, and it's fairly simple to use, yet sufficiently powerful for most normal binary editing. If you want a GUI, I believe gnome (and probably KDE as well) has its own hex editor. Good luck, Rolf ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Fri, Dec 18, 2009 at 02:09:58AM +0100, Rolf Nielsen wrote: Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. Any advice? many thanks anton Hello Anton, My bet would be /usr/ports/editors/hexedit. Been a while since I've used it, but AFAIR, it has a curses or a curses like interface, and it's fairly simple to use, yet sufficiently powerful for most normal binary editing. If you want a GUI, I believe gnome (and probably KDE as well) has its own hex editor. thank you. hexedit does the job on small files, but is quite clunky. If I've a xGB file and I need to delete the first and the last record, this becomes quite hard, if at all possible. I didn't appreciate it's not that simple. Perhaps I can read a file with C and write back? I can't remember if C supports binary files, and whether it also writes some record delimiters. many thanks anton -- Anton Shterenlikht Room 2.6, Queen's Building Mech Eng Dept Bristol University University Walk, Bristol BS8 1TR, UK Tel: +44 (0)117 331 5944 Fax: +44 (0)117 929 4423 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
Anton Shterenlikht wrote: On Fri, Dec 18, 2009 at 02:09:58AM +0100, Rolf Nielsen wrote: Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. Any advice? many thanks anton Hello Anton, My bet would be /usr/ports/editors/hexedit. Been a while since I've used it, but AFAIR, it has a curses or a curses like interface, and it's fairly simple to use, yet sufficiently powerful for most normal binary editing. If you want a GUI, I believe gnome (and probably KDE as well) has its own hex editor. thank you. hexedit does the job on small files, but is quite clunky. If I've a xGB file and I need to delete the first and the last record, this becomes quite hard, if at all possible. I didn't appreciate it's not that simple. Perhaps I can read a file with C and write back? I can't remember if C supports binary files, and whether it also writes some record delimiters. many thanks anton How about one of these then? http://www.freebsd.org/cgi/url.cgi?ports/editors/bless/pkg-descr Main Features - * Efficient editing of large data files. * Multilevel undo - redo operations. * Customizable data views. * Fast data rendering on screen. http://www.freebsd.org/cgi/url.cgi?ports/editors/lfhex/pkg-descr Features: - Low memory usage with respect to filesize. Opening a 2gig file requires only ~1.4megs of additional memory. - Fast load times. - Fast save times. - Infinite undo/redo. - Conversion dialog - Search function. - Shows modified regions in alternate color. - Scalable working area. Resize can use as much screen as you give it. - Multiple editing modes (can switch on the fly) - Runtime configurable bytes per column. - binary comparison user interface I haven't tried either of them myself, but they do look promising. Rolf ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
On Fri, 18 Dec 2009, Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. -Warren Block * Rapid City, South Dakota USA ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: editing a binary file
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Warren Block wrote: On Fri, 18 Dec 2009, Anton Shterenlikht wrote: I'm creating binary files in fortran. Fortran adds 4 byte record delimiters at the beginning and the end of each record, which, in the case of a binary file, is just at the beginning and at the end of the file. I need to delete these record delimiters, because the software I use to visualise the binary files interprets them as data. But I don't know how. I've looked at hexdump and od, but those are only dumping (I think) file contents, and I cannot see how to edit a file with them. truncate -4 myfile should get rid of the last four bytes. Maybe there's a similar efficient way to truncate the start of a file. -Warren Block * Rapid City, South Dakota USA This should do it: dd if=oldfile of=newfile bs=1 skip=4 Hope that helps, Greg - -- Greg Larkin http://www.FreeBSD.org/ - The Power To Serve http://www.sourcehosting.net/ - Ready. Set. Code. http://twitter.com/sourcehosting/ - Follow me, follow you -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFLKt6e0sRouByUApARAt0uAJ9CfG3DmsUbrUMg7hX2dIT+FOZ1sACfQtaD b7z5vvm/+vohelNIch1/ME8= =Zp5i -END PGP SIGNATURE- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
