Re: Help finding file-analysis tool?
On Mon, 2 May 2011 17:50:48 -0400 (EDT)
Dave Anderson wrote:
> Sorry to bother you all, but I'm failing miserably at searching for a
> tool to help analyze the structure of arbitrary files (prefereably one
> which runs on OpenBSD).
>
> I've got a device which exports data in a undocumented format and the
> only program available to use that data doesn't do what I need, so I
> need to figure out the file formats so I can communicate with the device
> the way I need to.
>
> What I'm looking for is an interactive program which makes it easy to
> look at selected parts of a file (individual items, sets of items
> located at regular intervals, sets of items linked by pointers or
> offsets, etc) in any of many formats (ascii, unicode, int, double float,
> etc) and either endianness, store comments about items or sets of items
> in an aux file, store names for various values in particular items and
> display those items values using those names, search for patterns at
> regular intervals or linked by pointers or offsets, etc, etc, etc; all
> those things which make it easier to discover and keep track of the
> structure of an unknown file.
>
> It's hard to believe that nobody has ever written such a program, but
> I've been unable to find one. Any suggestions for effective searches or
> for suitable programs would be appreciated.
>
> Thanks,
>
> Dave
>
> --
> Dave Anderson
>
>
Never heard of such a program.
I would use /usr/ports/editors/bvi, a hex editor, and Python, a very
high-level scripting language in which you can perform various
operations on data pretty easily.
f = open('myfile.dat', 'rb')
bytes = f.read(4)
msg = 'first 4 bytes in hex: '
for x in bytes:
msg += hex(ord(x))[2:].upper()
print msg
You could create a .py file with various useful functions, then start
up Python interpreter, import this file and explore the file
interactively by calling functions.
Conceive a theory by trying to think how would you create a file for
these purposes and by trying to see patterns in the file, test the
theory by changing the file one thing at a time, observe behavior,
repeat.
Re: Help finding file-analysis tool?
On Tue, 3 May 2011, Erik wrote: >Op 3-5-2011 16:51, Dave Anderson schreef: >> On Tue, 3 May 2011, Joachim Gwoke wrote: >> >>> Ever visit the people at http://www.woodmann.com? They might offer >>> some more answers. > >Alternately you might have a look at the coroners toolkit and its >successors, such as sleuthkit or Autopsy. Maybe these programs can do a >little bit of what you want. Thanks for the suggestion. I'll take a look. Dave -- Dave Anderson
Re: Help finding file-analysis tool?
Op 3-5-2011 16:51, Dave Anderson schreef: On Tue, 3 May 2011, Joachim Gwoke wrote: Ever visit the people at http://www.woodmann.com? They might offer some more answers. Alternately you might have a look at the coroners toolkit and its successors, such as sleuthkit or Autopsy. Maybe these programs can do a little bit of what you want.
Re: Help finding file-analysis tool?
On Tue, 3 May 2011, Joachim Gwoke wrote: >Ever visit the people at http://www.woodmann.com? They might offer >some more answers. No, I wasn't aware of them. Thanks for the pointer. Dave >On 5/3/11, Alexander Hall wrote: >> On 05/02/11 23:50, Dave Anderson wrote: >>> Sorry to bother you all, but I'm failing miserably at searching for a >>> tool to help analyze the structure of arbitrary files (prefereably one >>> which runs on OpenBSD). >>> >>> I've got a device which exports data in a undocumented format and the >>> only program available to use that data doesn't do what I need, so I >>> need to figure out the file formats so I can communicate with the device >>> the way I need to. >>> >>> What I'm looking for is an interactive program which makes it easy to >>> look at selected parts of a file (individual items, sets of items >>> located at regular intervals, sets of items linked by pointers or >>> offsets, etc) in any of many formats (ascii, unicode, int, double float, >>> etc) and either endianness, store comments about items or sets of items >>> in an aux file, store names for various values in particular items and >>> display those items values using those names, search for patterns at >>> regular intervals or linked by pointers or offsets, etc, etc, etc; all >>> those things which make it easier to discover and keep track of the >>> structure of an unknown file. >>> >>> It's hard to believe that nobody has ever written such a program, but >>> I've been unable to find one. Any suggestions for effective searches or >>> for suitable programs would be appreciated. >> >> Without a terribly skilled mathematician and tons of luck I would expect >> such a program to be close to impossible to create, or at least require >> tons of CPU time and data to perform the observations on, to come up >> with a reasonably reliable result. However, since I am not a terribly >> skilled matematician myself, I may be totally wrong. >> >> Meanwhile, file(1) comes to mind. :-) >> >> $ file /etc/pwd.db >> /etc/pwd.db: Berkeley DB 1.85 (Hash, version 2, native byte-order) >> >>> Thanks, >>> >>> Dave > -- Dave Anderson
Re: Help finding file-analysis tool?
On Tue, 3 May 2011, Alexander Hall wrote: >On 05/02/11 23:50, Dave Anderson wrote: >> Sorry to bother you all, but I'm failing miserably at searching for a >> tool to help analyze the structure of arbitrary files (prefereably one >> which runs on OpenBSD). >> >> I've got a device which exports data in a undocumented format and the >> only program available to use that data doesn't do what I need, so I >> need to figure out the file formats so I can communicate with the device >> the way I need to. >> >> What I'm looking for is an interactive program which makes it easy to >> look at selected parts of a file (individual items, sets of items >> located at regular intervals, sets of items linked by pointers or >> offsets, etc) in any of many formats (ascii, unicode, int, double float, >> etc) and either endianness, store comments about items or sets of items >> in an aux file, store names for various values in particular items and >> display those items values using those names, search for patterns at >> regular intervals or linked by pointers or offsets, etc, etc, etc; all >> those things which make it easier to discover and keep track of the >> structure of an unknown file. >> >> It's hard to believe that nobody has ever written such a program, but >> I've been unable to find one. Any suggestions for effective searches or >> for suitable programs would be appreciated. > >Without a terribly skilled mathematician and tons of luck I would expect >such a program to be close to impossible to create, or at least require >tons of CPU time and data to perform the observations on, to come up >with a reasonably reliable result. However, since I am not a terribly >skilled matematician myself, I may be totally wrong. If the program is expected to do the analysis on its own, I'm sure you're correct. What I'm looking for is one which automates a lot of the tedious parts of a human-directed analysis; the intelligence (or lack thereof) in what to look for, where to look for it, and what it really means would come from the user. >Meanwhile, file(1) comes to mind. :-) > >$ file /etc/pwd.db >/etc/pwd.db: Berkeley DB 1.85 (Hash, version 2, native byte-order) Unfortunately, that doesn't go nearly far enough. In particular (judging from the man page) it doesn't do anything with structured binary files unless they're one of the particular formats it recognizes -- and the files I have won't be. Thanks anyway, Dave -- Dave Anderson
Re: Help finding file-analysis tool?
Ever visit the people at http://www.woodmann.com? They might offer some more answers. regards Joachim On 5/3/11, Alexander Hall wrote: > On 05/02/11 23:50, Dave Anderson wrote: >> Sorry to bother you all, but I'm failing miserably at searching for a >> tool to help analyze the structure of arbitrary files (prefereably one >> which runs on OpenBSD). >> >> I've got a device which exports data in a undocumented format and the >> only program available to use that data doesn't do what I need, so I >> need to figure out the file formats so I can communicate with the device >> the way I need to. >> >> What I'm looking for is an interactive program which makes it easy to >> look at selected parts of a file (individual items, sets of items >> located at regular intervals, sets of items linked by pointers or >> offsets, etc) in any of many formats (ascii, unicode, int, double float, >> etc) and either endianness, store comments about items or sets of items >> in an aux file, store names for various values in particular items and >> display those items values using those names, search for patterns at >> regular intervals or linked by pointers or offsets, etc, etc, etc; all >> those things which make it easier to discover and keep track of the >> structure of an unknown file. >> >> It's hard to believe that nobody has ever written such a program, but >> I've been unable to find one. Any suggestions for effective searches or >> for suitable programs would be appreciated. > > Without a terribly skilled mathematician and tons of luck I would expect > such a program to be close to impossible to create, or at least require > tons of CPU time and data to perform the observations on, to come up > with a reasonably reliable result. However, since I am not a terribly > skilled matematician myself, I may be totally wrong. > > Meanwhile, file(1) comes to mind. :-) > > $ file /etc/pwd.db > /etc/pwd.db: Berkeley DB 1.85 (Hash, version 2, native byte-order) > >> Thanks, >> >> Dave
Re: Help finding file-analysis tool?
On 05/02/11 23:50, Dave Anderson wrote: > Sorry to bother you all, but I'm failing miserably at searching for a > tool to help analyze the structure of arbitrary files (prefereably one > which runs on OpenBSD). > > I've got a device which exports data in a undocumented format and the > only program available to use that data doesn't do what I need, so I > need to figure out the file formats so I can communicate with the device > the way I need to. > > What I'm looking for is an interactive program which makes it easy to > look at selected parts of a file (individual items, sets of items > located at regular intervals, sets of items linked by pointers or > offsets, etc) in any of many formats (ascii, unicode, int, double float, > etc) and either endianness, store comments about items or sets of items > in an aux file, store names for various values in particular items and > display those items values using those names, search for patterns at > regular intervals or linked by pointers or offsets, etc, etc, etc; all > those things which make it easier to discover and keep track of the > structure of an unknown file. > > It's hard to believe that nobody has ever written such a program, but > I've been unable to find one. Any suggestions for effective searches or > for suitable programs would be appreciated. Without a terribly skilled mathematician and tons of luck I would expect such a program to be close to impossible to create, or at least require tons of CPU time and data to perform the observations on, to come up with a reasonably reliable result. However, since I am not a terribly skilled matematician myself, I may be totally wrong. Meanwhile, file(1) comes to mind. :-) $ file /etc/pwd.db /etc/pwd.db: Berkeley DB 1.85 (Hash, version 2, native byte-order) > Thanks, > > Dave
Help finding file-analysis tool?
Sorry to bother you all, but I'm failing miserably at searching for a tool to help analyze the structure of arbitrary files (prefereably one which runs on OpenBSD). I've got a device which exports data in a undocumented format and the only program available to use that data doesn't do what I need, so I need to figure out the file formats so I can communicate with the device the way I need to. What I'm looking for is an interactive program which makes it easy to look at selected parts of a file (individual items, sets of items located at regular intervals, sets of items linked by pointers or offsets, etc) in any of many formats (ascii, unicode, int, double float, etc) and either endianness, store comments about items or sets of items in an aux file, store names for various values in particular items and display those items values using those names, search for patterns at regular intervals or linked by pointers or offsets, etc, etc, etc; all those things which make it easier to discover and keep track of the structure of an unknown file. It's hard to believe that nobody has ever written such a program, but I've been unable to find one. Any suggestions for effective searches or for suitable programs would be appreciated. Thanks, Dave -- Dave Anderson
