Re: Reconcile between library and xmp files (Re: [darktable-user] Question regarding sidecar files)

[Guillermo Rozas]

>
> > Besides the lack of album and search, one important point is that some
> > information is never stored in the XMP file, only in the database.
>
> Are any _processing_ parameters not stored in the XMP file?  In other
> words, are the resulting pixels identical regardless of whether the
> information comes from the database or from the XMP file?
>

 I would say yes: all processing parameters are stored in the XMP file, and
both database and XMP file processed images are identical. It should be the
case as darktable-cli can take the XMP as the input. However, one of the
developers should chip in at this point to confirm it.

Regards,
Guillermo


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: Reconcile between library and xmp files (Re: [darktable-user] Question regarding sidecar files)

[Robert Krawitz]

On Thu, 11 Oct 2018 07:55:36 -0300, Guillermo Rozas wrote:
> Besides the lack of album and search, one important point is that some
> information is never stored in the XMP file, only in the database. This is
> the case at least with the time offset from the GPS module, I think there
> was another one. So you lose that info when you close Darktable.

Are any _processing_ parameters not stored in the XMP file?  In other
words, are the resulting pixels identical regardless of whether the
information comes from the database or from the XMP file?  I don't
care about metadata such as GPS; processing is another story.  I have
no use for the database, since I prefer to use a different tool for
indexing my images.
-- 
Robert Krawitz 

***  MIT Engineers   A Proud Tradition   http://mitathletics.com  ***
Member of the League for Programming Freedom  --  http://ProgFree.org
Project lead for Gutenprint   --http://gimp-print.sourceforge.net

"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton

darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Bernhard]

Romano Giannetti schrieb am 11.10.2018 um 11:09:

On 10/10/18 15:08, Willy Williams wrote:


I have three laptops on which Darktable is installed; two Windows 10 
and one Ubuntu Linux.  I generally sync the two Windows computers' 
photos folders, including JPEGs, RAW files and XMP sidecar files.  
I've begun to wonder if I'm shooting myself in the foot by doing so.  
The question is this - are the sidecar files closely tied to an 
internal Darktable database on the computer where the work was done?  
Am I making the dog's breakfast of things by syncing files and in 
doing so, inadvertently getting the sidecar files out of sync with 
the unique database associated with each computer?





I use synchronized photo folders too; the important thing is to have 
set the option (in core option group) "look for updated xmp files on 
startup". When you open you darktable on the other machine, it will 
ask if you want to reload the XMPs or rewrite them; answer "reload" 
and voilá, all is going well.



not really. As long as you work on the fixed pool of images - yes. But 
if you add images or delete some of them - darktable on the other 
computers will not know about those images/absence of the images.

So you should also sync the database.

In wrote an article about this (German language) here: 
https://www.bilddateien.de/blog/2015-11-bildarchiv-auf-2-rechnern-synchronisation-der-darktable-konfiguration.html


deepL translation:
Image archive on 2 computers - Synchronization of the darktable 
configuration


published on 12.11.2015

Those who edit and especially manage their images in a workflow 
software will sooner or later face the problem: Which 
files/directories have to be saved in order to transfer all edits, 
settings and keywords etc. to a new computer?


But it can also be useful "in daily operation" to know about it. I 
myself use two computers at the same time:


    a powerful desktop computer: I don't want to have to do without 
the comfort of a large screen, good computing power, a good 
keyboard/mouse when I work longer on the computer. I could do the same 
with a laptop and a docking station, but


    I use a laptop for travelling. This laptop is not suitable as a 
replacement for the above-mentioned computer, not even with a docking 
station, because it is not "bred" for high computing power, because I 
am often on the road for a long time without mains supply, and then 
electricity consumption is an important issue.


Back to the data: When using darktable as workflow software, the files 
have to "fit together" at 3 locations in the computer and therefore 
have to be transferred together and completely when changing from one 
computer to another:


    of course the image files themselves: RAW files (or, if there are 
no RAW files, the corresponding jpg or tif files) together with the 
corresponding .xmp files


    the configuration directory of the program, which contains two things:

    Database: Contains first the same information about the 
individual images as they are stored in the .xmp files, but 
additionally also information that is not stored in a single image, 
e.g. groupings of photos.


    Program settings: Everything you can set yourself, such as 
export settings and directories, presets for individual development 
modules, default settings for metadata (copyright notices, etc.), 
automatically applied default settings when importing images, etc.


    the cache directory: here the preview images are stored for fast 
display at the light table.


This results in the following 3 paths whose contents have to be saved 
and transferred:


/home/[username]/.cache/darktable
/home/[username]/.config/darktable
/home/[username]/[path/to/the/pictures]

For the synchronization itself I use FreeFileSync.

(... and by the way the parallel data storage on two different 
computers results in a first step to prevent data loss :). …).


Translated with www.DeepL.com/Translator

If you do like this ANY change  will be known in any darktable installation.

Problem in your case is:
The darktable database stores the location of all the images in the 
filesystem.
With different operating systems the locations / path are not compatible 
between the different computers - so in your case this will most 
probably NOT work.


--

regards
Bernhard

https://www.bilddateien.de


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Bernhard]

Romano Giannetti schrieb am 11.10.2018 um 11:09:

On 10/10/18 15:08, Willy Williams wrote:


I have three laptops on which Darktable is installed; two Windows 10 
and one Ubuntu Linux.  I generally sync the two Windows computers' 
photos folders, including JPEGs, RAW files and XMP sidecar files.  
I've begun to wonder if I'm shooting myself in the foot by doing so.  
The question is this - are the sidecar files closely tied to an 
internal Darktable database on the computer where the work was done?  
Am I making the dog's breakfast of things by syncing files and in 
doing so, inadvertently getting the sidecar files out of sync with 
the unique database associated with each computer?





I use synchronized photo folders too; the important thing is to have 
set the option (in core option group) "look for updated xmp files on 
startup". When you open you darktable on the other machine, it will 
ask if you want to reload the XMPs or rewrite them; answer "reload" 
and voilá, all is going well.



not really. As long as you work on the fixed pool of images - yes. But 
if you add images or delete some of them - darktable on the other 
computers will not know about those images/absence of the images.

So you should also sync the database.

In wrote an article about this (German language) here: 
https://www.bilddateien.de/blog/2015-11-bildarchiv-auf-2-rechnern-synchronisation-der-darktable-konfiguration.html


deepL translation:
Image archive on 2 computers - Synchronization of the darktable 
configuration


published on 12.11.2015

Those who edit and especially manage their images in a workflow 
software will sooner or later face the problem: Which 
files/directories have to be saved in order to transfer all edits, 
settings and keywords etc. to a new computer?


But it can also be useful "in daily operation" to know about it. I 
myself use two computers at the same time:


    a powerful desktop computer: I don't want to have to do without 
the comfort of a large screen, good computing power, a good 
keyboard/mouse when I work longer on the computer. I could do the same 
with a laptop and a docking station, but


    I use a laptop for travelling. This laptop is not suitable as a 
replacement for the above-mentioned computer, not even with a docking 
station, because it is not "bred" for high computing power, because I 
am often on the road for a long time without mains supply, and then 
electricity consumption is an important issue.


Back to the data: When using darktable as workflow software, the files 
have to "fit together" at 3 locations in the computer and therefore 
have to be transferred together and completely when changing from one 
computer to another:


    of course the image files themselves: RAW files (or, if there are 
no RAW files, the corresponding jpg or tif files) together with the 
corresponding .xmp files


    the configuration directory of the program, which contains two things:

    Database: Contains first the same information about the 
individual images as they are stored in the .xmp files, but 
additionally also information that is not stored in a single image, 
e.g. groupings of photos.


    Program settings: Everything you can set yourself, such as 
export settings and directories, presets for individual development 
modules, default settings for metadata (copyright notices, etc.), 
automatically applied default settings when importing images, etc.


    the cache directory: here the preview images are stored for fast 
display at the light table.


This results in the following 3 paths whose contents have to be saved 
and transferred:


/home/[username]/.cache/darktable
/home/[username]/.config/darktable
/home/[username]/[path/to/the/pictures]

For the synchronization itself I use FreeFileSync.

(... and by the way the parallel data storage on two different 
computers results in a first step to prevent data loss :). …).


Translated with www.DeepL.com/Translator

If you do like this ANY change  will be known in any darktable installation.

Problem in your case is:
The darktable database stores the location of all the images in the 
filesystem.
With different operating systems the locations / path are not compatible 
between the different computers - so in your case this will most 
probably NOT work.


--

regards
Bernhard

https://www.bilddateien.de


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

re: [darktable-user] Question regarding sidecar files

[Jean-Luc CECCOLI]

> Message du 11/10/18 01:02
> De : "Willy Williams" 
> A : darktable-user@lists.darktable.org
> Copie à : 
> Objet : [darktable-user] Question regarding sidecar files
> 
>
> I have three laptops on which Darktable is installed; two Windows 10 and one 
> Ubuntu Linux.  I generally sync the two Windows computers' photos folders, 
> including JPEGs, RAW files and XMP sidecar files.  I've begun to wonder if 
> I'm shooting myself in the foot by doing so.  The question is this - are the 
> sidecar files closely tied to an internal Darktable database on the computer 
> where the work was done?  Am I making the dog's breakfast of things by 
> syncing files and in doing so, inadvertently getting the sidecar files out of 
> sync with the unique database associated with each computer?  
>

The only issue would be if your workflow runs from Windows to Linux : as Linux 
is case-sensitive, it might expect to find lowercase sidecars and, since those 
created by Windows are upercase, create its own ones, which would not match 
those imported from Windows.
Though I never tested this with DT, it happened to me with ASP files on a HD 
that I used with both Windows and Linux, and I ended up with 2 different 
versions of each sidecar per photo, one upercase and the other lowercase.
I did not notice that first, and going back to Windows I got some strange 
results, then pluging the disk on Linux again I could delete the lc sidecars.

Regards,

J.-Luc

darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Romano Giannetti]

On 11/10/18 13:12, Willy Williams wrote:


Thank you!!!  This should do it for me.

For those that asked, I'm using FreeFileSync 
(https://freefilesync.org/) to sync files between various systems.


Willy

Thanks, interesting link. I use Unison 
 (also 
http://www.cis.upenn.edu/~bcpierce/unison/ )(which uses ssh and rsync in 
a very transparent way), and I am quite happy with it.


Romano

--
Romano Giannetti
http://www.rgtti.com/



darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Willy Williams]

Thank you!!!  This should do it for me.

For those that asked, I'm using FreeFileSync (https://freefilesync.org/) 
to sync files between various systems.


Willy

**
On 10/11/2018 at 05:09, Romano Giannetti wrote:

On 10/10/18 15:08, Willy Williams wrote:


I have three laptops on which Darktable is installed; two Windows 10 
and one Ubuntu Linux.  I generally sync the two Windows computers' 
photos folders, including JPEGs, RAW files and XMP sidecar files.  
I've begun to wonder if I'm shooting myself in the foot by doing so.  
The question is this - are the sidecar files closely tied to an 
internal Darktable database on the computer where the work was done?  
Am I making the dog's breakfast of things by syncing files and in 
doing so, inadvertently getting the sidecar files out of sync with 
the unique database associated with each computer?





I use synchronized photo folders too; the important thing is to have 
set the option (in core option group) "look for updated xmp files on 
startup". When you open you darktable on the other machine, it will 
ask if you want to reload the XMPs or rewrite them; answer "reload" 
and voilá, all is going well.


Screenshot of the option (I hope it goes through):


--
--
Romano Giannetti
http://www.rgtti.com/  

 
darktable user mailing list to unsubscribe send a mail to 
darktable-user+unsubscr...@lists.darktable.org




darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: Reconcile between library and xmp files (Re: [darktable-user] Question regarding sidecar files)

[Guillermo Rozas]

>
> Darktable maintains a copy of the data in its library from the XMP files
> to allow finding pictures from the criteria in the XMP without
> re-reading all the XMP.
>
> Besides, darktable updates XMP files whenever you end an edit (switching
> to another image, or switching from dark room back to light table), so
> they are up-to-date with respect to the library which can be regarded as
> a cache.
>
> Still, darktable notices when XMP files are newer.
>
> I've seen darktable ask me a number of times whether to import new
> information from the XMP files. On virtually all occurrences, I clicked
> on "check all" (or similar) and confirmed because XMP were indeed newer.
>

This only happens if the option "look for XMP file changes on startup" is
enabled. If not, Darktable will silently overwrite the files. The database
is not a cache, is the real information. The XMP is, by default, an
external backup of the database.

Anyway, since the per-image information in darktable library is actually
> useless to me, for about a year I've run darktable with --library
> :memory: option.
>
> This way, per-image information in darktable library is empty at each
> start. I can't search per album of whatever but I don't care, I just
> open darktable with the directory I'm interested in as argument.


Besides the lack of album and search, one important point is that some
information is never stored in the XMP file, only in the database. This is
the case at least with the time offset from the GPS module, I think there
was another one. So you lose that info when you close Darktable.

One alternative option: is it possible to sync the entire database? I'd
there any drawback on doing that? (with the caveat of "same Darktable
version on all computers")

Best regards,
Guillermo


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Guillermo Rozas]

One point I forgot: sometimes Darktable's version of the database and XMP
file format changes and this change might not be backward compatible. So
make sure to have the same Darktable's version on all computers (probably
with major version is enough).
Best regards,
Guillermo

On Thu, Oct 11, 2018, 07:34 Guillermo Rozas  wrote:

> Just to complement Romano's answer: if that option is NOT selected, the
> first time you open one of those pictures the corresponding XMP file will
> be overwritten with the local database info (I don't remember if on
> opening, editing or closing), without asking or telling you. On Darktable
> by default the truth is in the database, the XMP is just an external backup.
> Best regards,
> Guillermo
>
> On Thu, Oct 11, 2018, 07:03 Romano Giannetti  wrote:
>
>> On 10/10/18 15:08, Willy Williams wrote:
>>
>> I have three laptops on which Darktable is installed; two Windows 10 and
>> one Ubuntu Linux.  I generally sync the two Windows computers' photos
>> folders, including JPEGs, RAW files and XMP sidecar files.  I've begun to
>> wonder if I'm shooting myself in the foot by doing so.  The question is
>> this - are the sidecar files closely tied to an internal Darktable database
>> on the computer where the work was done?  Am I making the dog's breakfast
>> of things by syncing files and in doing so, inadvertently getting the
>> sidecar files out of sync with the unique database associated with each
>> computer?
>>
>>
>>


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Guillermo Rozas]

Just to complement Romano's answer: if that option is NOT selected, the
first time you open one of those pictures the corresponding XMP file will
be overwritten with the local database info (I don't remember if on
opening, editing or closing), without asking or telling you. On Darktable
by default the truth is in the database, the XMP is just an external backup.
Best regards,
Guillermo

On Thu, Oct 11, 2018, 07:03 Romano Giannetti  wrote:

> On 10/10/18 15:08, Willy Williams wrote:
>
> I have three laptops on which Darktable is installed; two Windows 10 and
> one Ubuntu Linux.  I generally sync the two Windows computers' photos
> folders, including JPEGs, RAW files and XMP sidecar files.  I've begun to
> wonder if I'm shooting myself in the foot by doing so.  The question is
> this - are the sidecar files closely tied to an internal Darktable database
> on the computer where the work was done?  Am I making the dog's breakfast
> of things by syncing files and in doing so, inadvertently getting the
> sidecar files out of sync with the unique database associated with each
> computer?
>
>
> I use synchronized photo folders too; the important thing is to have set
> the option (in core option group) "look for updated xmp files on startup".
> When you open you darktable on the other machine, it will ask if you want
> to reload the XMPs or rewrite them; answer "reload" and voilá, all is going
> well.
>
> Screenshot of the option (I hope it goes through):
>
>
> --
> --
> Romano Giannettihttp://www.rgtti.com/
>
>
> 
> darktable user mailing list to unsubscribe send a mail to
> darktable-user+unsubscr...@lists.darktable.org
>


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Reconcile between library and xmp files (Re: [darktable-user] Question regarding sidecar files)

[Stéphane Gourichon]

Le 2018-10-10 à 15:08, Willy Williams a écrit :


I have three laptops on which Darktable is installed; two Windows 10 
and one Ubuntu Linux.  I generally sync the two Windows computers' 
photos folders, including JPEGs, RAW files and XMP sidecar files.  
I've begun to wonder if I'm shooting myself in the foot by doing so.  
The question is this - are the sidecar files closely tied to an 
internal Darktable database on the computer where the work was done?  
Am I making the dog's breakfast of things by syncing files and in 
doing so, inadvertently getting the sidecar files out of sync with the 
unique database associated with each computer?




I think it should go without any problem. I would probably make sure 
darktable or the sync program is fully closed before running the other, 
just in case.


What sync program do you use ?


## Closely tied ? No fear, darktable has a sane model and reconcile 
capabilities


Darktable maintains a copy of the data in its library from the XMP files 
to allow finding pictures from the criteria in the XMP without 
re-reading all the XMP.


Besides, darktable updates XMP files whenever you end an edit (switching 
to another image, or switching from dark room back to light table), so 
they are up-to-date with respect to the library which can be regarded as 
a cache.


Still, darktable notices when XMP files are newer.

I've seen darktable ask me a number of times whether to import new 
information from the XMP files. On virtually all occurrences, I clicked 
on "check all" (or similar) and confirmed because XMP were indeed newer.


You may have a similar experience. I've seen no important ill-effect.

Still you might be interested in a similar case, read on.


## One solution to avoid any sync issue

Actually, my collection is so big that most of it is not mounted at the 
time I run darktable. It is split among several hard drives.
And even part of it is mounted, the path to the storage change with 
time. For example, a disk is sometimes mounted via USB and sometimes via 
network, thus the mount point are different.


As a result, the per-image information in darktable library is full of 
stale/unsynced copies from various mount points.


This is actually not much of a problem. It mostly causes any view of the 
library to show duplicate entries and lots of skulls in lighttable.


Anyway, since the per-image information in darktable library is actually 
useless to me, for about a year I've run darktable with --library 
:memory: option.


This way, per-image information in darktable library is empty at each 
start. I can't search per album of whatever but I don't care, I just 
open darktable with the directory I'm interested in as argument. No 
duplicate, no skulls, only the directory I'm working on.


This way, darktable never asks me about which version to keep (because 
the "memory" library is just empty so never conflicting), and things go 
smooth.


Here is the executable script sitting in my ~/bin/darktable :


#!/bin/bash
exec /usr/bin/darktable --library :memory: "$@"


Open to any comment, observation, similar experience or other solutions.

--
Stéphane Gourichon


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Romano Giannetti]

On 10/10/18 15:08, Willy Williams wrote:


I have three laptops on which Darktable is installed; two Windows 10 
and one Ubuntu Linux.  I generally sync the two Windows computers' 
photos folders, including JPEGs, RAW files and XMP sidecar files.  
I've begun to wonder if I'm shooting myself in the foot by doing so.  
The question is this - are the sidecar files closely tied to an 
internal Darktable database on the computer where the work was done?  
Am I making the dog's breakfast of things by syncing files and in 
doing so, inadvertently getting the sidecar files out of sync with the 
unique database associated with each computer?





I use synchronized photo folders too; the important thing is to have set 
the option (in core option group) "look for updated xmp files on 
startup". When you open you darktable on the other machine, it will ask 
if you want to reload the XMPs or rewrite them; answer "reload" and 
voilá, all is going well.


Screenshot of the option (I hope it goes through):


--
--
Romano Giannetti
http://www.rgtti.com/



darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Question regarding sidecar files

[Patrick Shanahan]

* Willy Williams  [10-10-18 19:05]:
> I have three laptops on which Darktable is installed; two Windows 10 and one
> Ubuntu Linux.  I generally sync the two Windows computers' photos folders,
> including JPEGs, RAW files and XMP sidecar files.  I've begun to wonder if
> I'm shooting myself in the foot by doing so.  The question is this - are the
> sidecar files closely tied to an internal Darktable database on the computer
> where the work was done?  Am I making the dog's breakfast of things by
> syncing files and in doing so, inadvertently getting the sidecar files out
> of sync with the unique database associated with each computer?

if you move files to a different computer (and library) from where they
were generated, just have the new computer (darktable) read the xmp file
accompanying the images and the library will then match.

-- 
(paka)Patrick Shanahan   Plainfield, Indiana, USA  @ptilopteri
http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri
Registered Linux User #207535@ http://linuxcounter.net
Photos: http://wahoo.no-ip.org/piwigo   paka @ IRCnet freenode

darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org