Re: [darktable-dev] Re: XML parser → Huge sidecar files
BTW, thanks for pointing to the "cleanup unused shapes" solution! On Tue, 2018-04-17 at 18:27 +0200, Ulrich Pegelow wrote: > You don't need to do this manually. Just go into the mask manager > (left > hand panel in the darkroom view), press the right mouse button. On > the > menu that appears you select "cleanup unused shapes". > > In your case the huge number of invisible shapes comes from the spot > removal tool. The root cause lies in a combination of two facts. > darktable does not delete shapes automatically and spot removal > shapes > do not appear in the mask manager. So even if you reset the spot > removal > tool all shapes created will still exist in the data set and you do > not > even know because the mask manager does not tell you. We should > consider > how to change this behavior. > > In the time being you can manually clean up unused shapes with the > method described above. > > ulrich > > > Am 16.04.2018 um 10:55 schrieb Timur Irikovich Davletshin: > > Well, I was able to edit them with sed script to remove all masks, > > but > > parsing XML to find used/unused masks is beyond my skills (I'm not > > familiar with XMP standards). I believe future DT releases should > > sort > > this out in some way. At least for this issue is very critical, it > > slows down opening/closing files in my library multiple times. > > > > On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: > > > Hello, > > > > > > I can confirm this for the latest version in the darktable-2.4.x > > > branch > > > (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar > > > files > > > seem to contain an awful lot of masks, but I can't seem to see > > > which > > > module even uses them? > > > > > > cheers, > > > Simon > > > > > > > > > On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: > > > > It looks like I was able to find what action causes this > > > > problem. > > > > > > > > Steps to reproduce: > > > > > > > > 1. Download, unpack and import as folder — https://drive.google > > > > .com > > > > /ope > > > > n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to > > > > content > > > > and settings) > > > > 2. Choose second picture and click (in lighttable mode) copy > > > > history > > > > stack and choose, let's say, 'shadows and highlights' settings. > > > > 3. Apply it (paste) to first picture, open it, compress history > > > > stack > > > > and close DT. > > > > 4. Now compare XMP files, first one is twice bigger than second > > > > one > > > > (in > > > > my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history > > > > stack > > > > looks the same. > > > > > > > > Can anyone comment, what is going on here? > > > > > > _ > > > > > _ > __ > darktable developer mailing list > to unsubscribe send a mail to darktable-dev+unsubscribe@lists.darktab > le.org > ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
Actually I didn't get why masks are getting copied when I ask only particular module settings. But anyway I hope it will change soon, opening every file (especially when they are so slow) is just so time consuming. It should be 'all at once' action :) On Tue, 2018-04-17 at 18:27 +0200, Ulrich Pegelow wrote: > You don't need to do this manually. Just go into the mask manager > (left > hand panel in the darkroom view), press the right mouse button. On > the > menu that appears you select "cleanup unused shapes". > > In your case the huge number of invisible shapes comes from the spot > removal tool. The root cause lies in a combination of two facts. > darktable does not delete shapes automatically and spot removal > shapes > do not appear in the mask manager. So even if you reset the spot > removal > tool all shapes created will still exist in the data set and you do > not > even know because the mask manager does not tell you. We should > consider > how to change this behavior. > > In the time being you can manually clean up unused shapes with the > method described above. > > ulrich > > > Am 16.04.2018 um 10:55 schrieb Timur Irikovich Davletshin: > > Well, I was able to edit them with sed script to remove all masks, > > but > > parsing XML to find used/unused masks is beyond my skills (I'm not > > familiar with XMP standards). I believe future DT releases should > > sort > > this out in some way. At least for this issue is very critical, it > > slows down opening/closing files in my library multiple times. > > > > On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: > > > Hello, > > > > > > I can confirm this for the latest version in the darktable-2.4.x > > > branch > > > (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar > > > files > > > seem to contain an awful lot of masks, but I can't seem to see > > > which > > > module even uses them? > > > > > > cheers, > > > Simon > > > > > > > > > On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: > > > > It looks like I was able to find what action causes this > > > > problem. > > > > > > > > Steps to reproduce: > > > > > > > > 1. Download, unpack and import as folder — https://drive.google > > > > .com > > > > /ope > > > > n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to > > > > content > > > > and settings) > > > > 2. Choose second picture and click (in lighttable mode) copy > > > > history > > > > stack and choose, let's say, 'shadows and highlights' settings. > > > > 3. Apply it (paste) to first picture, open it, compress history > > > > stack > > > > and close DT. > > > > 4. Now compare XMP files, first one is twice bigger than second > > > > one > > > > (in > > > > my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history > > > > stack > > > > looks the same. > > > > > > > > Can anyone comment, what is going on here? > > > > > > _ > > > > > _ > __ > darktable developer mailing list > to unsubscribe send a mail to darktable-dev+unsubscribe@lists.darktab > le.org > ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
Hi David, only for the image currently opened in the darkroom. Ulrich Am 17.04.2018 um 18:33 schrieb David Vincent-Jones: Hi Ulrich does the 'cleanup' work globally or is this just for the current image? David On 04/17/2018 09:27 AM, Ulrich Pegelow wrote: You don't need to do this manually. Just go into the mask manager (left hand panel in the darkroom view), press the right mouse button. On the menu that appears you select "cleanup unused shapes". In your case the huge number of invisible shapes comes from the spot removal tool. The root cause lies in a combination of two facts. darktable does not delete shapes automatically and spot removal shapes do not appear in the mask manager. So even if you reset the spot removal tool all shapes created will still exist in the data set and you do not even know because the mask manager does not tell you. We should consider how to change this behavior. In the time being you can manually clean up unused shapes with the method described above. ulrich Am 16.04.2018 um 10:55 schrieb Timur Irikovich Davletshin: Well, I was able to edit them with sed script to remove all masks, but parsing XML to find used/unused masks is beyond my skills (I'm not familiar with XMP standards). I believe future DT releases should sort this out in some way. At least for this issue is very critical, it slows down opening/closing files in my library multiple times. On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: Hello, I can confirm this for the latest version in the darktable-2.4.x branch (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar files seem to contain an awful lot of masks, but I can't seem to see which module even uses them? cheers, Simon On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: It looks like I was able to find what action causes this problem. Steps to reproduce: 1. Download, unpack and import as folder — https://drive.google.com /ope n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to content and settings) 2. Choose second picture and click (in lighttable mode) copy history stack and choose, let's say, 'shadows and highlights' settings. 3. Apply it (paste) to first picture, open it, compress history stack and close DT. 4. Now compare XMP files, first one is twice bigger than second one (in my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history stack looks the same. Can anyone comment, what is going on here? ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
Hi Ulrich does the 'cleanup' work globally or is this just for the current image? David On 04/17/2018 09:27 AM, Ulrich Pegelow wrote: > You don't need to do this manually. Just go into the mask manager (left > hand panel in the darkroom view), press the right mouse button. On the > menu that appears you select "cleanup unused shapes". > > In your case the huge number of invisible shapes comes from the spot > removal tool. The root cause lies in a combination of two facts. > darktable does not delete shapes automatically and spot removal shapes > do not appear in the mask manager. So even if you reset the spot removal > tool all shapes created will still exist in the data set and you do not > even know because the mask manager does not tell you. We should consider > how to change this behavior. > > In the time being you can manually clean up unused shapes with the > method described above. > > ulrich > > > Am 16.04.2018 um 10:55 schrieb Timur Irikovich Davletshin: >> Well, I was able to edit them with sed script to remove all masks, but >> parsing XML to find used/unused masks is beyond my skills (I'm not >> familiar with XMP standards). I believe future DT releases should sort >> this out in some way. At least for this issue is very critical, it >> slows down opening/closing files in my library multiple times. >> >> On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: >>> Hello, >>> >>> I can confirm this for the latest version in the darktable-2.4.x >>> branch >>> (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar files >>> seem to contain an awful lot of masks, but I can't seem to see which >>> module even uses them? >>> >>> cheers, >>> Simon >>> >>> >>> On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: It looks like I was able to find what action causes this problem. Steps to reproduce: 1. Download, unpack and import as folder — https://drive.google.com /ope n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to content and settings) 2. Choose second picture and click (in lighttable mode) copy history stack and choose, let's say, 'shadows and highlights' settings. 3. Apply it (paste) to first picture, open it, compress history stack and close DT. 4. Now compare XMP files, first one is twice bigger than second one (in my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history stack looks the same. Can anyone comment, what is going on here? >>> >>> _ > ___ > darktable developer mailing list > to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org > ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
You don't need to do this manually. Just go into the mask manager (left hand panel in the darkroom view), press the right mouse button. On the menu that appears you select "cleanup unused shapes". In your case the huge number of invisible shapes comes from the spot removal tool. The root cause lies in a combination of two facts. darktable does not delete shapes automatically and spot removal shapes do not appear in the mask manager. So even if you reset the spot removal tool all shapes created will still exist in the data set and you do not even know because the mask manager does not tell you. We should consider how to change this behavior. In the time being you can manually clean up unused shapes with the method described above. ulrich Am 16.04.2018 um 10:55 schrieb Timur Irikovich Davletshin: Well, I was able to edit them with sed script to remove all masks, but parsing XML to find used/unused masks is beyond my skills (I'm not familiar with XMP standards). I believe future DT releases should sort this out in some way. At least for this issue is very critical, it slows down opening/closing files in my library multiple times. On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: Hello, I can confirm this for the latest version in the darktable-2.4.x branch (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar files seem to contain an awful lot of masks, but I can't seem to see which module even uses them? cheers, Simon On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: It looks like I was able to find what action causes this problem. Steps to reproduce: 1. Download, unpack and import as folder — https://drive.google.com /ope n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to content and settings) 2. Choose second picture and click (in lighttable mode) copy history stack and choose, let's say, 'shadows and highlights' settings. 3. Apply it (paste) to first picture, open it, compress history stack and close DT. 4. Now compare XMP files, first one is twice bigger than second one (in my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history stack looks the same. Can anyone comment, what is going on here? _ ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
* Timur Irikovich Davletshin [04-16-18 04:56]: > Well, I was able to edit them with sed script to remove all masks, but > parsing XML to find used/unused masks is beyond my skills (I'm not > familiar with XMP standards). I believe future DT releases should sort > this out in some way. At least for this issue is very critical, it > slows down opening/closing files in my library multiple times. > > On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: > > Hello, > > > > I can confirm this for the latest version in the darktable-2.4.x > > branch > > (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar files > > seem to contain an awful lot of masks, but I can't seem to see which > > module even uses them? > > > > cheers, > > Simon > > > > > > On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: > > > It looks like I was able to find what action causes this problem. > > > > > > Steps to reproduce: > > > > > > 1. Download, unpack and import as folder — https://drive.google.com > > > /ope > > > n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to > > > content > > > and settings) > > > 2. Choose second picture and click (in lighttable mode) copy > > > history > > > stack and choose, let's say, 'shadows and highlights' settings. > > > 3. Apply it (paste) to first picture, open it, compress history > > > stack > > > and close DT. > > > 4. Now compare XMP files, first one is twice bigger than second one > > > (in > > > my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history > > > stack > > > looks the same. > > > > > > Can anyone comment, what is going on here? > > > > _ > > __ > > darktable developer mailing list > > to unsubscribe send a mail to darktable-dev+unsubscribe@lists.darktab > > le.org > > > ___ > darktable developer mailing list > to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org I don't know that it would make any difference, but all my xmp files have at least 14 masks where I have not employed any masks and most of my xmp files range from 5k to 17k and doesn't appear to depend on which body employed. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri Registered Linux User #207535@ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
Well, I was able to edit them with sed script to remove all masks, but parsing XML to find used/unused masks is beyond my skills (I'm not familiar with XMP standards). I believe future DT releases should sort this out in some way. At least for this issue is very critical, it slows down opening/closing files in my library multiple times. On Mon, 2018-04-16 at 10:19 +0200, sturmflut wrote: > Hello, > > I can confirm this for the latest version in the darktable-2.4.x > branch > (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar files > seem to contain an awful lot of masks, but I can't seem to see which > module even uses them? > > cheers, > Simon > > > On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: > > It looks like I was able to find what action causes this problem. > > > > Steps to reproduce: > > > > 1. Download, unpack and import as folder — https://drive.google.com > > /ope > > n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to > > content > > and settings) > > 2. Choose second picture and click (in lighttable mode) copy > > history > > stack and choose, let's say, 'shadows and highlights' settings. > > 3. Apply it (paste) to first picture, open it, compress history > > stack > > and close DT. > > 4. Now compare XMP files, first one is twice bigger than second one > > (in > > my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history > > stack > > looks the same. > > > > Can anyone comment, what is going on here? > > _ > __ > darktable developer mailing list > to unsubscribe send a mail to darktable-dev+unsubscribe@lists.darktab > le.org > ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
Re: [darktable-dev] Re: XML parser → Huge sidecar files
Hello, I can confirm this for the latest version in the darktable-2.4.x branch (commit edf1168371be288f071986d93a47fb3e082573de). The sidecar files seem to contain an awful lot of masks, but I can't seem to see which module even uses them? cheers, Simon On 15.04.2018 20:28, Timur Irikovich Davletshin wrote: > It looks like I was able to find what action causes this problem. > > Steps to reproduce: > > 1. Download, unpack and import as folder — https://drive.google.com/ope > n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to content > and settings) > 2. Choose second picture and click (in lighttable mode) copy history > stack and choose, let's say, 'shadows and highlights' settings. > 3. Apply it (paste) to first picture, open it, compress history stack > and close DT. > 4. Now compare XMP files, first one is twice bigger than second one (in > my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history stack > looks the same. > > Can anyone comment, what is going on here? ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
[darktable-dev] Re: XML parser → Huge sidecar files
It looks like I was able to find what action causes this problem. Steps to reproduce: 1. Download, unpack and import as folder — https://drive.google.com/ope n?id=14sZLgnpZSV5W3pw1K_8owHW29Heq9EWz (don't pay attention to content and settings) 2. Choose second picture and click (in lighttable mode) copy history stack and choose, let's say, 'shadows and highlights' settings. 3. Apply it (paste) to first picture, open it, compress history stack and close DT. 4. Now compare XMP files, first one is twice bigger than second one (in my case ~700kB vs ~1400kB). Meanwhile in darkroom mode history stack looks the same. Can anyone comment, what is going on here? On Sat, 2018-04-07 at 19:22 +0300, Timur Irikovich Davletshin wrote: > Hello everybody! > > I'm sorry but I will address this question again (posted earlier as > user question) because this issue just irritates me quite often. > > I have a lot of files imported in 2014-15 which have problem I > describe > a bit later. May be it will give some clues but files imported in > 2012- > 13 and in 2016 (and later) are not affected (years started to use > DT). > I open files randomly time to time so DT updates old saidecar files > if > it likes. So affected files are not original files created in 2014- > 15. > > I don't know since which DT version it started but all files which > are > affected now have huge sidecar files — 10-25 megabytes. So as > expected > XML parser (which is single threaded I believe and not hardware > accelerated) takes very long time to load and write sidecar files. > Meanwhile editing is not any slower. > > I tried little trick — I just removed all history stack by clicking > "original" and compressing it. I expected sidecar to shrink but it > remained nearly the same. I tried opening this file and it is full of > old tags related to masks, spot removal, etc... which were used long > time ago and no longer present in history stack or mask manager. > Looks > like DT doesn't handle (or couldn't handle in earlier versions) > importing of files created in previous DT versions. > > Thanks is advance, > > Timur. > > P.S. Sample RAW + XMP (plus XMP after removing history stack): > https:// > drive.google.com/open?id=1RwMA7Aq5PiF3iORSGsN5xpkYEv-IEHQ0 ___ darktable developer mailing list to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org
