Re: [darktable-user] Philosophical question

[Bernhard]

German ... that's why I translated the second part of that page and 
added to my mail.


The basic concept is: darktable treats all files as _read-only_ and thus 
they are supposed to never change. So if they change something bad had 
happened.
So I create a list of checksums of all files in a given directory and 
check them from time to time - be it automatically (cronjob or something 
like that) or a remark in a calender or by accident - that's left to the 
user.


I also have a copy of my archive on at least two other locations - and 
there I also check those checksums from time to time.


If something happens at one location I can copy the correct file from 
one of the other locations ... but then I should also check if the drive 
itself goes eol and replace it.


For me that's a simple concept which I understand - and I like those 
transparent concepts :-).


--

regards
Bernhard

https://www.bilddateien.de

Timothy Spear schrieb am 30.08.2018 um 15:58:
Using Google Translate (I am struggling to learn French, no way will I 
take on German also!), I think I have a pretty good idea on how you 
did it.
Nice solution. Not sure which way I will go, from overkill using 
Tripwire, to a custom shell script or a Java/.Net application. I just 
know I am not going to try and learn Lua.


Tim

On Thu, Aug 30, 2018 at 2:04 AM, Bernhard 
mailto:darkta...@intervalsignals.org>> 
wrote:




Timothy Spear schrieb am 12.08.2018 um 18:42:

The primary reason I have converted from CR2 and JPEG to DNG
was for the inherit hash embedded in the file which can help
prevent/notify you of bit rot. Since I have previously
experience with bit rot I consider this worth it.

Which reminds me, I need to put in a new ticket issue to
suggest a validation function to dt which will store a hash
for both the XMP and the original image, and can run in a
background process.


That's what I am doing currently with my .NEF and .ORF files:
https://www.bilddateien.de/fotografie/bildbearbeitung/foto-backup.html


Translation of the important part:

Checking data consistency
What does "data consistency check" mean?

For digital data, so-called checksums can be determined by
mathematical methods, which represent a kind of "fingerprint"
for a file. Changing the file, either by processing or by a
data error that has crept in over time, results in a change to
the checksum.

If you now determine their respective checksums for all files
to be backed up and save them, you can check at any time later
whether the files are still in their original state. To do
this, the checksums are determined again and compared with the
values originally calculated and saved. Deviations indicate
file errors.
prerequisites

On the basis of the "non destructive imaging" technique
described above, the consistency check results:

There are a lot of files in the camera raw formats, but
possibly also uncompressed in.tif, which take over the
function of the analog "negative" or "slide" as original
files, can only be read and should therefore not change anymore.
These - and only these - are the subject of the test!
And then there are at least as many of these xml files (2 edit
versions of the same image give 1 raw file and two xml files)
in the same folders, whose characteristic is the change.
Pictures and associated xml-files are typically located in
year folders and there in subordinate project folders.

The check (both the original generation of the checksums and
the subsequent comparisons) should be performed automatically
by software.
Such a software results in a requirement profile:

    Program with GUI (graphical user interface - no command
line stress)
    Recursive creation and checking in directory trees
(subdirectories are also searched)
    Filtering of unwanted file types (here in the concrete
example: xmp (of Darktable), pp3 (of RawTherapee), ...)
    possibly different hash functions (mathematical methods
for checksum determination) for selection (crc32 or md5)
    if possible cross-platform software (there are versions
for Windows as well as Linux)

An example of a suitable software is Checksum Control.

Translated with www.DeepL.com/Translator



-- 


regards
Bernhard

https://www.bilddateien.de


darktable user mailing list
to unsubscribe send a mail to
darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Philosophical question

[Timothy Spear]

Using Google Translate (I am struggling to learn French, no way will I take
on German also!), I think I have a pretty good idea on how you did it.
Nice solution. Not sure which way I will go, from overkill using Tripwire,
to a custom shell script or a Java/.Net application. I just know I am not
going to try and learn Lua.

Tim

On Thu, Aug 30, 2018 at 2:04 AM, Bernhard 
wrote:

>
>
> Timothy Spear schrieb am 12.08.2018 um 18:42:
>
>> The primary reason I have converted from CR2 and JPEG to DNG was for the
>> inherit hash embedded in the file which can help prevent/notify you of bit
>> rot. Since I have previously experience with bit rot I consider this worth
>> it.
>>
>> Which reminds me, I need to put in a new ticket issue to suggest a
>> validation function to dt which will store a hash for both the XMP and the
>> original image, and can run in a background process.
>>
>>
> That's what I am doing currently with my .NEF and .ORF files:
> https://www.bilddateien.de/fotografie/bildbearbeitung/foto-backup.html
>
> Translation of the important part:
>
>> Checking data consistency
>> What does "data consistency check" mean?
>>
>> For digital data, so-called checksums can be determined by mathematical
>> methods, which represent a kind of "fingerprint" for a file. Changing the
>> file, either by processing or by a data error that has crept in over time,
>> results in a change to the checksum.
>>
>> If you now determine their respective checksums for all files to be
>> backed up and save them, you can check at any time later whether the files
>> are still in their original state. To do this, the checksums are determined
>> again and compared with the values originally calculated and saved.
>> Deviations indicate file errors.
>> prerequisites
>>
>> On the basis of the "non destructive imaging" technique described above,
>> the consistency check results:
>>
>> There are a lot of files in the camera raw formats, but possibly also
>> uncompressed in.tif, which take over the function of the analog "negative"
>> or "slide" as original files, can only be read and should therefore not
>> change anymore.
>> These - and only these - are the subject of the test!
>> And then there are at least as many of these xml files (2 edit versions
>> of the same image give 1 raw file and two xml files) in the same folders,
>> whose characteristic is the change.
>> Pictures and associated xml-files are typically located in year folders
>> and there in subordinate project folders.
>>
>> The check (both the original generation of the checksums and the
>> subsequent comparisons) should be performed automatically by software.
>> Such a software results in a requirement profile:
>>
>> Program with GUI (graphical user interface - no command line stress)
>> Recursive creation and checking in directory trees (subdirectories
>> are also searched)
>> Filtering of unwanted file types (here in the concrete example: xmp
>> (of Darktable), pp3 (of RawTherapee), ...)
>> possibly different hash functions (mathematical methods for checksum
>> determination) for selection (crc32 or md5)
>> if possible cross-platform software (there are versions for Windows
>> as well as Linux)
>>
>> An example of a suitable software is Checksum Control.
>>
>> Translated with www.DeepL.com/Translator
>>
>
> --
>
> regards
> Bernhard
>
> https://www.bilddateien.de
>
> 
> 
> darktable user mailing list
> to unsubscribe send a mail to darktable-user+unsubscribe@lis
> ts.darktable.org
>
>


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Philosophical question

[Bernhard]

Timothy Spear schrieb am 12.08.2018 um 18:42:
The primary reason I have converted from CR2 and JPEG to DNG was for 
the inherit hash embedded in the file which can help prevent/notify 
you of bit rot. Since I have previously experience with bit rot I 
consider this worth it.


Which reminds me, I need to put in a new ticket issue to suggest a 
validation function to dt which will store a hash for both the XMP and 
the original image, and can run in a background process.




That's what I am doing currently with my .NEF and .ORF files:
https://www.bilddateien.de/fotografie/bildbearbeitung/foto-backup.html

Translation of the important part:

Checking data consistency
What does "data consistency check" mean?

For digital data, so-called checksums can be determined by 
mathematical methods, which represent a kind of "fingerprint" for a 
file. Changing the file, either by processing or by a data error that 
has crept in over time, results in a change to the checksum.


If you now determine their respective checksums for all files to be 
backed up and save them, you can check at any time later whether the 
files are still in their original state. To do this, the checksums are 
determined again and compared with the values originally calculated 
and saved. Deviations indicate file errors.

prerequisites

On the basis of the "non destructive imaging" technique described 
above, the consistency check results:


There are a lot of files in the camera raw formats, but possibly also 
uncompressed in.tif, which take over the function of the analog 
"negative" or "slide" as original files, can only be read and should 
therefore not change anymore.

These - and only these - are the subject of the test!
And then there are at least as many of these xml files (2 edit 
versions of the same image give 1 raw file and two xml files) in the 
same folders, whose characteristic is the change.
Pictures and associated xml-files are typically located in year 
folders and there in subordinate project folders.


The check (both the original generation of the checksums and the 
subsequent comparisons) should be performed automatically by software.

Such a software results in a requirement profile:

    Program with GUI (graphical user interface - no command line stress)
    Recursive creation and checking in directory trees (subdirectories 
are also searched)
    Filtering of unwanted file types (here in the concrete example: 
xmp (of Darktable), pp3 (of RawTherapee), ...)
    possibly different hash functions (mathematical methods for 
checksum determination) for selection (crc32 or md5)
    if possible cross-platform software (there are versions for 
Windows as well as Linux)


An example of a suitable software is Checksum Control.

Translated with www.DeepL.com/Translator


--

regards
Bernhard

https://www.bilddateien.de


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Philosophical question

[Timothy Spear]

In theory, the data is converted to a standard, the unknown elements are
supposed to be added as maker's notes.
You have to look at the specific DNG conversion software to determine how
good a job it will do.

The second most common reason to not convert is because it adds another
step to the process flow and uses computing power; which may affect you
depending on your workflow. Since I import at the end of the day/trip and
rarely start editing the same day I have not had an issue letting the
computer chug away converting images while I go eat, sleep, check email
Since I take pictures for personal/hobby I am not time constrained and the
extra step does not bother me.

The third most popular reason I have seen is that at some point a brilliant
idea in image processing will occur, and it depends on the original raw
files.

The most common reasons to convert are to deal with longevity of the data;
since we are already seeing problems recovering/viewing data in other
industries in many other fields. However, I am not concerned about this one
yet. Since I generally lag on updating my software except for security
patches, when support for an older format is dropped I expect will have
time to convert any file I have not done so. I would also expect Adobe,
Microsoft, Apple, dt etc... to provide a warning message that the next time
we upgrade it will no longer be able to read X. Lastly, worst case, I just
load a VM with an older version of the software, do the conversion and
re-import the data.

The primary reason I have converted from CR2 and JPEG to DNG was for the
inherit hash embedded in the file which can help prevent/notify you of bit
rot. Since I have previously experience with bit rot I consider this worth
it.

Which reminds me, I need to put in a new ticket issue to suggest a
validation function to dt which will store a hash for both the XMP and the
original image, and can run in a background process.

Tim

On Sun, Aug 12, 2018 at 12:39 AM, Mike Strock 
wrote:

> What do I lose converting from CR2 RAW files to DNG? I always shoot raw
> but is there a benefit or a degradation to converting to DNG for future
> proofing my images?
>
> If this is off topic I apologize.
>
> Mike.
>
> Sent from ProtonMail Mobile
> 
> darktable user mailing list to unsubscribe send a mail to
> darktable-user+unsubscr...@lists.darktable.org


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Philosophical question

[I. Ivanov]

What do I lose converting from CR2 RAW files to DNG? I always
shoot raw but is there a benefit or a degradation to converting to
DNG for future proofing my images?

What I recall reading on the list is generally against conversion to 
DNG. My understanding is that a) the format is being changed - so there 
is a chance something to not work as intended and b) some tags in the 
original RAW that are not used "now" may become useful in the future. 
Converting to a different format would loose these tags. I personally 
don't use DNG so I can't say anything about possible benefits.


Regards,
B


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Philosophical question

[Timur Davletshin]

Personally, I'm OK with converted CR2 → DNG, it works fine from my
experience. But I have problems with NEF → DNG, there is some problem with
maker notes interpretation in exiv2 (used by dt) which leads to problems in
lens detection (hence no automatic lens correction). I know that some
developers
 strongly oblige against conversion, but I find it useful on big
collections — 15-20% size reduction does matter for me.

Timur.

On Sun, Aug 12, 2018 at 7:39 AM Mike Strock 
wrote:

> What do I lose converting from CR2 RAW files to DNG? I always shoot raw
> but is there a benefit or a degradation to converting to DNG for future
> proofing my images?
>
> If this is off topic I apologize.
>
> Mike.
>
> Sent from ProtonMail Mobile
> 
> darktable user mailing list to unsubscribe send a mail to
> darktable-user+unsubscr...@lists.darktable.org


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org