Re: Unique Files Verification Software (Mac)

[Robert ListMail via 4D_Tech]

Hi Kirk, I'll take a deeper look but at first glance... well, I didn't think I 
was looking for any forensic tools. Remember it's all about revealing unique 
data on a source drive such that something labeled "temp" or "backup" or an 
obvious drive that was in use before the contents were cloned to a higher 
capacity drive is not lost. 

Thanks,

R

Sent from my iPhone

> On Mar 14, 2017, at 11:54 PM, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> forensic programs for
> comparing disk files":

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Kirk Brooks via 4D_Tech]

Robert,
On Tue, Mar 14, 2017 at 2:56 PM, Robert ListMail via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Actually, I’ve looked for years and have never found anything that will do
> this. It’s not impossible to do what needs to be done manually, it’s just a
> royal pain and prone to user errors that could result in data loss. Do you
> have a tool in mind?

​Im afraid not. The only forensic work I've done had to do with financial
data and that was simply (simply!) a matter of working with the actual
pieces of paper or copies - I didn't have to analyze or compare disk files.

Here's the top hit that came up when I googled "forensic programs for
comparing disk files": link
.
​At least gives you an idea of what they are called. I notice they all seem
to be windows based which supports the notion that only nice people or
smart crooks use Macs.

-- 
Kirk Brooks
San Francisco, CA
===
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Robert ListMail via 4D_Tech]

I’m not looking to identify all duplicates. I want to know what unique data 
might be on a disk labeled “backup"…. To determine this the software (rsync?) 
must query an index of ALL specified drives where the original pathname may be 
very different for each an every file. Can rsync perform the equivalent of a 
Finder query for each and every file on the source drive (the drive labeled 
“Backup”)?  The goal is to not recycle a disk unless it is confirmed that all 
files are represented elsewhere. Does that clear things up?  I’m not aware of 
any utility that offers this feature. 

Robert

> On Mar 14, 2017, at 5:33 PM, Keisuke Miyako via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> for simply synchronising two directories possibly on a separate volume,
> rsync has been around for quite some time.
> 
> https://en.wikipedia.org/wiki/Rsync
> 
> but I may be getting wrong the "this" in "anything that will do this".
> 
>> 2017/03/15 6:56、Robert ListMail via 4D_Tech <4d_tech@lists.4D.com> のメール：
>> Actually, I’ve looked for years and have never found anything that will do 
>> this. It’s not impossible to do what needs to be done manually, it’s just a 
>> royal pain and prone to user errors that could result in data loss. Do you 
>> have a tool in mind?  I’m not looking to reinvent the wheel.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Chip Scheide via 4D_Tech]

You can try this link :
http://download.cnet.com/s/duplicate-file-finder/mac/

> I guess Git does a pretty decent job at tracking changes and 
> movements in a designated directory,
> but if you feel 4D give you the strength and flexibility to do 
> exactly what you want,
> I have nothing against it.
> 
>> 2017/03/15 8:46、Robert ListMail via 4D_Tech <4d_tech@lists.4D.com> 
>> のメール：
>> Chip, thanks for your input but no traditional rsync or clone tools 
>> or backup software is up to the task…. Since the pathnames are 
>> guaranteed to be different…
> 
> 
> 
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

Hell is other people 
 Jean-Paul Sartre
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Chip Scheide via 4D_Tech]

I came across a tool :
zsDuplicateHunter

I haven't used it for a long time - so not clue as to how (or if) it is has 
been updated



> Hi Kirk!
> 
>> On Mar 14, 2017, at 11:23 AM, Kirk Brooks via 4D_Tech 
>> <4d_tech@lists.4d.com> wrote:
>> Robert,
>> It sounds like you are doing some really interesting stuff.
> 
> Why yes, is there any other way?  :)
> 
>> kinds of files where surreptitious data are easily hidden. Then we get to
>> the thousands of .dll files on a typical windows machine.
> 
> The requirement will be for Mac only and would exclude system files…
> . I’m mostly interested in media files… ProRes, Quicktime, TIFF, 
> etc.
> 
>> These are the sorts of things already worked out and tools built to do
>> them. I think it would be a large wheel to reinvent within 4D.
> 
> Actually, I’ve looked for years and have never found anything that 
> will do this. It’s not impossible to do what needs to be done 
> manually, it’s just a royal pain and prone to user errors that could 
> result in data loss. Do you have a tool in mind?  I’m not looking to 
> reinvent the wheel.
> 
> R
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

Hell is other people 
 Jean-Paul Sartre
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Keisuke Miyako via 4D_Tech]

I guess Git does a pretty decent job at tracking changes and movements in a 
designated directory,
but if you feel 4D give you the strength and flexibility to do exactly what you 
want,
I have nothing against it.

> 2017/03/15 8:46、Robert ListMail via 4D_Tech <4d_tech@lists.4D.com> のメール：
> Chip, thanks for your input but no traditional rsync or clone tools or backup 
> software is up to the task…. Since the pathnames are guaranteed to be 
> different…




**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Robert ListMail via 4D_Tech]

Chip, thanks for your input but no traditional rsync or clone tools or backup 
software is up to the task….  Since the pathnames are guaranteed to be 
different…

R

> On Mar 14, 2017, at 5:33 PM, Keisuke Miyako via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> for simply synchronising two directories possibly on a separate volume,
> rsync has been around for quite some time.
> 
> https://en.wikipedia.org/wiki/Rsync
> 
> but I may be getting wrong the "this" in "anything that will do this".
> 
>> 2017/03/15 6:56、Robert ListMail via 4D_Tech <4d_tech@lists.4D.com> のメール：
>> Actually, I’ve looked for years and have never found anything that will do 
>> this. It’s not impossible to do what needs to be done manually, it’s just a 
>> royal pain and prone to user errors that could result in data loss. Do you 
>> have a tool in mind?  I’m not looking to reinvent the wheel.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Keisuke Miyako via 4D_Tech]

for simply synchronising two directories possibly on a separate volume,
rsync has been around for quite some time.

https://en.wikipedia.org/wiki/Rsync

but I may be getting wrong the "this" in "anything that will do this".

> 2017/03/15 6:56、Robert ListMail via 4D_Tech <4d_tech@lists.4D.com> のメール：
> Actually, I’ve looked for years and have never found anything that will do 
> this. It’s not impossible to do what needs to be done manually, it’s just a 
> royal pain and prone to user errors that could result in data loss. Do you 
> have a tool in mind?  I’m not looking to reinvent the wheel.




**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Robert ListMail via 4D_Tech]

Thanks Tim, I’ll have a look.

R

> On Mar 14, 2017, at 2:56 PM, Timothy Penner via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> It's been a few years since I looked at it but I think the "HASH Examples in 
> 4D" tech note includes a sample database that had as a proof of concept the 
> ability to "find duplicate files on a hard drive"
> http://kb.4d.com/assetid=76130 
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Robert ListMail via 4D_Tech]

Hi Kirk!

> On Mar 14, 2017, at 11:23 AM, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> Robert,
> It sounds like you are doing some really interesting stuff.

Why yes, is there any other way?  :)

> kinds of files where surreptitious data are easily hidden. Then we get to
> the thousands of .dll files on a typical windows machine.

The requirement will be for Mac only and would exclude system files…. I’m 
mostly interested in media files… ProRes, Quicktime, TIFF, etc.

> These are the sorts of things already worked out and tools built to do
> them. I think it would be a large wheel to reinvent within 4D.

Actually, I’ve looked for years and have never found anything that will do 
this. It’s not impossible to do what needs to be done manually, it’s just a 
royal pain and prone to user errors that could result in data loss. Do you have 
a tool in mind?  I’m not looking to reinvent the wheel.

R
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

RE: Unique Files Verification Software (Mac)

[Timothy Penner via 4D_Tech]

It's been a few years since I looked at it but I think the "HASH Examples in 
4D" tech note includes a sample database that had as a proof of concept the 
ability to "find duplicate files on a hard drive"
http://kb.4d.com/assetid=76130

Tech Note: Hash Examples in 4D
PRODUCT: 4D | VERSION: 12 | PLATFORM: Mac & Win
Published On: July 16, 2010
This Technical Note discusses hash functions including some of their common 
uses; the concept of salt is also discussed. Included with the Technical Note 
is a sample database that demonstrates using hash functions for encrypting 
passwords, file integrity, file fingerprinting, and finding duplicate files on 
a hard drive.


-Tim



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Kirk Brooks via 4D_Tech]

Robert,
It sounds like you are doing some really interesting stuff. It also sounds
like you might want to be looking for forensic tools already been built for
this sort of work.

Alex is right about hashing the file blob to develop a unique identifier
for exact matches regardless of name but that also means 'exact'. I'm
thinking about an excel spreasheet opened with numbers or googlesheets and
resaved. Copy two may be perfectly fine as far as the programs and anything
else are concerned but not be 'exact' if a single character was replaced,
changed, added or removed anywhere in the file. With image files it's even
easier for something to get shifted or changed and those are precisely the
kinds of files where surreptitious data are easily hidden. Then we get to
the thousands of .dll files on a typical windows machine.

These are the sorts of things already worked out and tools built to do
them. I think it would be a large wheel to reinvent within 4D.

On Mon, Mar 13, 2017 at 11:56 PM, Robert ListMail via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> I need a utility that can scan a backup drive (or index) and identify
> what’s unique to the backup volume without expecting identical pathnames on
> the other drives... So, the routine would have to query (effectively a
> Finder Search for each file) all specified drives looking for each file and
> reporting those that are missing... Basically, I need to know which data on
> this given backup drive is truly unique and therefore potentially valuable.
>
> Might there be a 4D solution?  Have you dealt with large directories or
> many directories from the file system? If there is a utility already built
> I’m open to that as well.
>
> Thanks,
>
> Robert
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **




-- 
Kirk Brooks
San Francisco, CA
===
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Herr Alexander Heintz via 4D_Tech]

You're welcome.
I need this as we store all attached files of e-mails in our DMS and here we 
find truckloads of duplicates (logos in mail signatures, etc.)
In order to avoid blowing up storage we calculate a hash of every file and only 
store uniques, and then link them to the appropriate source documents.
Of course, when we switched to this system we had to cleanup all the existing 
storage, hence we did somethings similar.
Using the hash has proven to be the best and most secure approach as ist is 
guaranteed to identify identical files no matter what they are named, it even 
finds them if someone mistakenly uses a different suffix (i.e. jpeg instead of 
jpg).

Cheers
Alex

> Am 14.03.2017 um 10:11 schrieb Robert ListMail via 4D_Tech 
> <4d_tech@lists.4d.com>:
> 
> Alex, thanks for the input. I thought it might be a good task for 4D.  So how 
> or why do you need such a tool?
> 
> Btw: I don't really need the hash comparison if I had other file attributes. 
> I'll look at this again tomorrow. 
> 
> Thanks,
> 
> Robert
> 
> Sent from my iPhone
> 
>> On Mar 14, 2017, at 3:25 AM, Herr Alexander Heintz via 4D_Tech 
>> <4d_tech@lists.4d.com> wrote:
>> 
>> 
>> I use a similar Algorithm for optimizing document storage.
>> Pretty simple actually:
>> just troll through all directories recursively and store each file in a 
>> record.
> 
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Robert ListMail via 4D_Tech]

Alex, thanks for the input. I thought it might be a good task for 4D.  So how 
or why do you need such a tool?

Btw: I don't really need the hash comparison if I had other file attributes. 
I'll look at this again tomorrow. 

Thanks,

Robert

Sent from my iPhone

> On Mar 14, 2017, at 3:25 AM, Herr Alexander Heintz via 4D_Tech 
> <4d_tech@lists.4d.com> wrote:
> 
> 
> I use a similar Algorithm for optimizing document storage.
> Pretty simple actually:
> just troll through all directories recursively and store each file in a 
> record.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Unique Files Verification Software (Mac)

[Herr Alexander Heintz via 4D_Tech]

Hi,

I use a similar Algorithm for optimizing document storage.
Pretty simple actually:
just troll through all directories recursively and store each file in a record.
You just need the path and the file hash which you can create with

DOCUMENT TO BLOB($t_DocPath;$x_Content)
$t_FileHash:=Generate digest($x_Content;MD5 digest)
SET BLOB SIZE($x_Content;0)

Then just check for unique hashes and voila!
Using the hash will also find identical files that have different filenames.
The chances of 2 different files generating the same hash are so close to 0 
they are for all practivcal reasons 0.
Now write something that moves unigue data somewehere else or deletes 
duplicates.

Whole thing is quickly written, I guess some 100 lines of code.
120 with progress bars :-)

As for running it, well that will take some time, don’t do it on you main work 
machine, it might be tied up for a while.

Hope that helped.

Cheers
Alex

> Am 14.03.2017 um 07:56 schrieb Robert ListMail via 4D_Tech 
> <4d_tech@lists.4d.com>:
> 
> I need a utility that can scan a backup drive (or index) and identify what’s 
> unique to the backup volume without expecting identical pathnames on the 
> other drives... So, the routine would have to query (effectively a Finder 
> Search for each file) all specified drives looking for each file and 
> reporting those that are missing... Basically, I need to know which data on 
> this given backup drive is truly unique and therefore potentially valuable.
> 
> Might there be a 4D solution?  Have you dealt with large directories or many 
> directories from the file system? If there is a utility already built I’m 
> open to that as well.
> 
> Thanks,
> 
> Robert
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**