Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Christian Kanzian]

Hi,

A small comment as ingenious user:
Does darktable create XMP files if I only plugin a USB stick? Not that 
I'm aware of. I need to import the images, or not?


I have the impression that darktable is used as a kind of filemanager or 
imageviewer. If I only want to edit a view images, I always can open 
them one by one from a filemanager. There is no need for a time 
consuming import, messing up the database with someones images.


There would also be the option to start darktable with the option 
--configdir  --library  for cases 
where I want to turn off XMP creation.

Please correct my if I am wrong.

BTW sometimes I mass tag and rate my images before editing.

Christian

Am 2016-11-22 16:48, schrieb Stéphane Gourichon:

Le 22/11/2016 à 12:30, Tobias Ellinghaus a écrit :

Am Montag, 21. November 2016, 18:19:30 CET schrieb Stéphane Gourichon:
What happens if you import your folder, cull the images, give ratings 
and
color labels but never open any in darkroom? You wouldn't have any of 
that
saved in sidecar files even though you invested hours in that. Sounds 
like a
bad idea, doesn't it? Same is true for putting images on the map, 
tagging or

any other operation on lighttable.


Thank you for mentioning those operations, they are obviously good
points. I naively thought they raised the "changed" flag. After
checking the source I see they are not.

Let's define what we want with a rationale:

"Any operation that would cause a XMP sidecar to *reflect any user
work (action/input) on this picture* should trigger creating it:
rating, labels, history (including applying styles, rotation),
metadata, tagging, geotagging, export".

I believe that this definition make it clear that a solution must
cover what you mentioned.

Do we miss some more user actions?

Thank you for any enlightenment.


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Stéphane Gourichon]

Le 22/11/2016 à 12:30, Tobias Ellinghaus a écrit :

Am Montag, 21. November 2016, 18:19:30 CET schrieb Stéphane Gourichon:
What happens if you import your folder, cull the images, give ratings and
color labels but never open any in darkroom? You wouldn't have any of that
saved in sidecar files even though you invested hours in that. Sounds like a
bad idea, doesn't it? Same is true for putting images on the map, tagging or
any other operation on lighttable.


Thank you for mentioning those operations, they are obviously good 
points. I naively thought they raised the "changed" flag. After checking 
the source I see they are not.


Let's define what we want with a rationale:

"Any operation that would cause a XMP sidecar to *reflect any user work 
(action/input) on this picture* should trigger creating it: rating, 
labels, history (including applying styles, rotation), metadata, 
tagging, geotagging, export".


I believe that this definition make it clear that a solution must cover 
what you mentioned.


Do we miss some more user actions?

Thank you for any enlightenment.

--
Stéphane Gourichon


darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Tobias Ellinghaus]

Am Montag, 21. November 2016, 18:19:30 CET schrieb Stéphane Gourichon:

[...]

> Given the expected semantic of a "changed" tag, it looks efficient,
> simple and natural to write something like:
> 
> if ( (no XMP file already exists) && (image has no tag "changed") )  {
> skip xmp creation };
> 
> That way, XMP indeed gets created when relevant.

What happens if you import your folder, cull the images, give ratings and 
color labels but never open any in darkroom? You wouldn't have any of that 
saved in sidecar files even though you invested hours in that. Sounds like a 
bad idea, doesn't it? Same is true for putting images on the map, tagging or 
any other operation on lighttable.

[...]

> (2) If someone offered a clean patch performing this exact change "just
> opening a file does not create a XMP, an edit causes it", would you
> merge it?  If not, can you explain the rationale?

Yes, if the logic is convincing and really dealing with all the corner cases. 
However, dealing with all corner cases will (in general) end up in writing an 
XMP file for every image.

> Thank you.

Tobias

signature.asc
Description: This is a digitally signed message part.

Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Stéphane Gourichon]

Le 21/11/2016 à 18:49, Patrick Shanahan a écrit :

* Stéphane Gourichon  [11-21-16 12:21]:


Also, option write_sidecar_files is not the wished behavior, as XMP files
are definitely useful when they contain not only generated information, cf.
8.2. Core options | user manual | darktable


there is an option to "write sidecar file for each image".  turn it "off".


I already covered this case, see citation above.

Don't get me wrong, XMP sidecar is necessary, I even consider it the 
primary storage for two reasons:


(1) I love darktable's ability to move my HD to any machine and open 
files from there with all their edits even if local database does not 
know them (also via network mount)
(2) Several times within recent years I had darktable *lose* editing 
work in database *and* overwrite XMP files with default content, which 
is not good. Now I version control XMP files to protect against this. My 
1.3Tb, 260k+ photo+video collection is now in git-annex: XMP sidecars as 
regular git files, RAW and JPG in the annex. Efficiently replicated by 
git-annex. Should be safe and somehow future-proof.


Yet, generating wads of XMP any time darktable is used to view a 
pristine directory on other people's drives and network mounts feels 
clutter to me and others.  It also happened that I lost editing work 
removing unwanted XMPs to clean up the clutter... ಠ_ಠ


--
Stéphane Gourichon



darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Stéphane Gourichon]

TL;DR: darktable gets better over time, so version-specific defaults 
have a chance to get better, too, right? Why force *current* defaults 
even for unedited unexported files?



Le 21/11/2016 à 18:47, Roman Lebedev a écrit :


If we do not write XMP when doing any export based on raw data (not counting
embedded thumbs), next time we do the same export, the result may be
different. And modulo bugs, that is*not*  the intended result.


Good point. In most cases, the user will edit somehow before exporting, 
writing XMP so we're safe.
In some cases the user may be satisfied with defaults *and* expect exact 
reproducibility.
That's a reason to say that exporting a picture should trigger 
unconditional write of a XMP sidecar file.


if ( (no XMP file already exists) && (image has no tag "changed" or 
"exported") )  { skip

xmp creation };


So what happens if you open image and then immediately close it?


In that case it's not "changed", it's not "exported".

So, it's a case of picture *viewing*. I see no reason to generate a XMP 
file with parameters *specific to current version of darktable* just 
because I happened to view it with *that* version.





Concrete case:
* You have a dir full of RAW. To figure out which one you want to edit, 
you open directory in darktable.
* You find and edit *one* picture, export it. XMP sidecar for that 
picture is definitely needed.
* Now, consider all other files. What's the reason to generate XMP files 
for *all* those files? Is the XMP generated *now* better than a XMP 
generated by a future, improved version of darktable?


I'm trying to understand your position.  Do you consider that whatever 
behavior *current* darktable exhibits on first it sees a picture (not 
even editing or exporting it) should be cast in stone and frozen in 
time? Or did I misunderstand ?


Thank you for enlightening me.

Sincerely,

--
Stéphane Gourichon



darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Patrick Shanahan]

* Stéphane Gourichon  [11-21-16 12:21]:
> Le 20/11/2016 à 19:24, Tobias Ellinghaus a écrit :
> > 
> > That is not feasible as we rewrite the XMP files all the time. Not just when
> > you edit the image in darkroom.
> 
> IMHO this behavior pops up too regularly in discussions.
> 
> 
> # Context: user feedback + usual reply
> 
> This is from memory, I might be wrong.
> 
> User: "darktable (writes an uninteresting XMP file | clobbers interesting
> XMP file) for *each and any* raw or jpeg file that it encounters at startup.
> This (slows down|clutters directories|loses minor information|loses
> important work|feels wrong)."
> 
> Reply (from memory): (due to the way we organized things|we just write XMP
> at some points in time and which causes overall behavior just like you
> wrote|it's too difficult to change that behavior|just remove them if
> undesired|make dir read-only)
> 
> Spirit of Free Software whispers : "Use the source Luke."
> 
> 
> # Zooming out to get the big picture
> 
> From an outside perspective, current behavior still doesn't feel right.
> 
> From a functional point of view, there's no good reason to write a XMP file
> for a file that was not even opened in darkroom.
> 
> Sure, darktable has default processing options, and might use them to
> display thumbnails (in some configurations).
> But by definition these options are *generated defaults*, so there's no
> point in saving them.
> Saving them "pins" those default to the particular version of darktable just
> run, is there added value ever in that?
> 
> 
> # Related information
> 
> Does current code causes an inclination for bugs like this one? Bug #10012:
> Darktable overwrites existing XMP files
> 
> 
> Also, option write_sidecar_files is not the wished behavior, as XMP files
> are definitely useful when they contain not only generated information, cf.
> 8.2. Core options | user manual | darktable
> 
> 
> 
> # Guessing an implementation
> 
> Quick look at ./src/common/image.c void dt_image_write_sidecar_file(int
> imgid):
> 
> >  // TODO: compute hash and don't write if not needed!
> 
> Darktable *already features some tags "changed" and "exported"* in database,
> searchable with the "collect images" structures filtering module.
> 
> Given the expected semantic of a "changed" tag, it looks efficient, simple
> and natural to write something like:
> 
> if ( (no XMP file already exists) && (image has no tag "changed") )  { skip
> xmp creation };
> 
> That way, XMP indeed gets created when relevant.
> 
> 
> # Benefit and conclusion
> 
> Current implementations feels like darktable has "dirty hands" or goes "like
> a snail/slob/slug": wherever it goes you see its trails that you have to
> clean if they're undesired.
> 
> * With new behavior, user can open in darktable any dir (USB key from a
> friend, whatever) and not clutter it.
> * Only edited pictures (if any) will get a sidecar file, which makes it
> easier to spot them e.g. for copy-paste back in external file manager.
> * Opening wrong directory will not clutter it.
> 
> All those use case happened to me. I'm not covering all cases for the sake
> of brevity, but I don't get why this was not approved earlier.
> 
> (1) Can a fellow savvy darktable wizard elaborate/explain?
> 
> (2) If someone offered a clean patch performing this exact change "just
> opening a file does not create a XMP, an edit causes it", would you merge
> it?  If not, can you explain the rationale?
> 

there is an option to "write sidecar file for each image".  turn it "off".
-- 
(paka)Patrick Shanahan   Plainfield, Indiana, USA  @ptilopteri
http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri
Photos: http://wahoo.no-ip.org/gallery2  Registered Linux User #207535  
  
Photos: http://wahoo.no-ip.org/piwigo@ http://linuxcounter.net

darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org

Re: [darktable-user] Creating XMP file only when makes sense: rationale?

[Roman Lebedev]

On Mon, Nov 21, 2016 at 8:19 PM, Stéphane Gourichon
 wrote:
> Le 20/11/2016 à 19:24, Tobias Ellinghaus a écrit :
>
>
> That is not feasible as we rewrite the XMP files all the time. Not just when
> you edit the image in darkroom.
>
>
> IMHO this behavior pops up too regularly in discussions.
>
>
> # Context: user feedback + usual reply
>
> This is from memory, I might be wrong.

> User: "darktable (writes an uninteresting XMP file
That is not true, see more detailed reply later on.

> clobbers interesting XMP file)
???

>for *each and any* raw or jpeg file that it encounters at startup.
Only if "don't use embedded preview JPEG but half-size raw" setting is enabled.

> This (slows down
If writing one XMP slows down your machine, how are you using DT on it?

>|clutters directories
Well, disable XMP altogether then, or  ls -lah | grep -iv .xmp$  :)

>|loses minor information|loses important work
Care to elaborate?
Unless you enable crawler, and make sure to *never* have more than one
program that might modify XMP opened at the same time, there should be
no loss.

>|feels wrong)."
So is editing raw files (read: editing the files, the metadata in
them), yet people
does not understand it :)

> Reply (from memory): (due to the way we organized things|we just write XMP
> at some points in time and which causes overall behavior just like you
> wrote|it's too difficult to change that behavior|just remove them if
> undesired|make dir read-only)
>
> Spirit of Free Software whispers : "Use the source Luke."
>
>
> # Zooming out to get the big picture
>
> From an outside perspective, current behavior still doesn't feel right.

> From a functional point of view, there's no good reason to write a XMP file
> for a file that was not even opened in darkroom.
Yes, there is.

> Sure, darktable has default processing options, and might use them to
> display thumbnails (in some configurations).
> But by definition these options are *generated defaults*, so there's no
> point in saving them.
> Saving them "pins" those default to the particular version of darktable just
> run, is there added value ever in that?
https://redmine.darktable.org/issues/10345

If we do not write XMP when doing any export based on raw data (not counting
embedded thumbs), next time we do the same export, the result may be
different. And modulo bugs, that is *not* the intended result.

> # Related information
>
> Does current code causes an inclination for bugs like this one? Bug #10012:
> Darktable overwrites existing XMP files
Bugs happen, that is the nature of the code :)

> Also, option write_sidecar_files is not the wished behavior, as XMP files
> are definitely useful when they contain not only generated information, cf.
> 8.2. Core options | user manual | darktable
>
>
> # Guessing an implementation
>
> Quick look at ./src/common/image.c void dt_image_write_sidecar_file(int
> imgid):
>
>>  // TODO: compute hash and don't write if not needed!
>
> Darktable *already features some tags "changed" and "exported"* in database,
> searchable with the "collect images" structures filtering module.
>
> Given the expected semantic of a "changed" tag, it looks efficient, simple
> and natural to write something like:
>
> if ( (no XMP file already exists) && (image has no tag "changed") )  { skip
> xmp creation };
>
> That way, XMP indeed gets created when relevant.
>
>
> # Benefit and conclusion
>
> Current implementations feels like darktable has "dirty hands" or goes "like
> a snail/slob/slug": wherever it goes you see its trails that you have to
> clean if they're undesired.
>
> * With new behavior, user can open in darktable any dir (USB key from a
> friend, whatever) and not clutter it.
> * Only edited pictures (if any) will get a sidecar file, which makes it
> easier to spot them e.g. for copy-paste back in external file manager.
> * Opening wrong directory will not clutter it.
>
> All those use case happened to me. I'm not covering all cases for the sake
> of brevity, but I don't get why this was not approved earlier.

> (1) Can a fellow savvy darktable wizard elaborate/explain?
> (2) If someone offered a clean patch performing this exact change "just
> opening a file does not create a XMP, an edit causes it"
So what happens if you open image and then immediately close it?
If you export it, and then thing like https://redmine.darktable.org/issues/10345
will happen, and you export it again, the two exports will be
radically different.
That way, dt's history stack will no longer result in the same export every time
(* currently, it still does not, but that is a bug)

>, would you merge
> it?  If not, can you explain the rationale?
(this mail is in dt-user)
Nope, i would not merge it, because then i'd have stale commit in my
git history, and
i would have to workaround it each time i want to push something upstream.

> Thank you.
>
> --
> Stéphane Gourichon
Roman.

Also, almost, tl:dr;

> _

[darktable-user] Creating XMP file only when makes sense: rationale?

[Stéphane Gourichon]

Le 20/11/2016 à 19:24, Tobias Ellinghaus a écrit :


That is not feasible as we rewrite the XMP files all the time. Not just when
you edit the image in darkroom.


IMHO this behavior pops up too regularly in discussions.


# Context: user feedback + usual reply

This is from memory, I might be wrong.

User: "darktable (writes an uninteresting XMP file | clobbers 
interesting XMP file) for *each and any* raw or jpeg file that it 
encounters at startup. This (slows down|clutters directories|loses minor 
information|loses important work|feels wrong)."


Reply (from memory): (due to the way we organized things|we just write 
XMP at some points in time and which causes overall behavior just like 
you wrote|it's too difficult to change that behavior|just remove them if 
undesired|make dir read-only)


Spirit of Free Software whispers : "Use the source Luke."


# Zooming out to get the big picture

From an outside perspective, current behavior still doesn't feel right.

From a functional point of view, there's no good reason to write a XMP 
file for a file that was not even opened in darkroom.


Sure, darktable has default processing options, and might use them to 
display thumbnails (in some configurations).
But by definition these options are *generated defaults*, so there's no 
point in saving them.
Saving them "pins" those default to the particular version of darktable 
just run, is there added value ever in that?



# Related information

Does current code causes an inclination for bugs like this one? Bug 
#10012: Darktable overwrites existing XMP files 



Also, option write_sidecar_files is not the wished behavior, as XMP 
files are definitely useful when they contain not only generated 
information, cf. 8.2. Core options | user manual | darktable 




# Guessing an implementation

Quick look at ./src/common/image.c void dt_image_write_sidecar_file(int 
imgid):


>  // TODO: compute hash and don't write if not needed!

Darktable *already features some tags "changed" and "exported"* in 
database, searchable with the "collect images" structures filtering module.


Given the expected semantic of a "changed" tag, it looks efficient, 
simple and natural to write something like:


if ( (no XMP file already exists) && (image has no tag "changed") )  { 
skip xmp creation };


That way, XMP indeed gets created when relevant.


# Benefit and conclusion

Current implementations feels like darktable has "dirty hands" or goes 
"like a snail/slob/slug": wherever it goes you see its trails that you 
have to clean if they're undesired.


* With new behavior, user can open in darktable any dir (USB key from a 
friend, whatever) and not clutter it.
* Only edited pictures (if any) will get a sidecar file, which makes it 
easier to spot them e.g. for copy-paste back in external file manager.

* Opening wrong directory will not clutter it.

All those use case happened to me. I'm not covering all cases for the 
sake of brevity, but I don't get why this was not approved earlier.


(1) Can a fellow savvy darktable wizard elaborate/explain?

(2) If someone offered a clean patch performing this exact change "just 
opening a file does not create a XMP, an edit causes it", would you 
merge it?  If not, can you explain the rationale?


Thank you.

--
Stéphane Gourichon



darktable user mailing list
to unsubscribe send a mail to darktable-user+unsubscr...@lists.darktable.org