Re: [darktable-dev] Tagging module: What the...

[Patrick Shanahan]

* August Schwerdfeger  [08-05-17 11:17]:
> I compiled the new version (revision 79e8763) and tried it out on the test
> images and also on a copy of my main database. The speed improvement is
> considerable; none of the tagging operations I tried took more than seven
> seconds to complete. I am looking forward to using this in a stable release.

I have been using the "development" (I wouldn't call it that) version for
several years with only two or three ?hickups? which were easy to recover
and during high-school and college soccer season, I process up to 4k a
week.  think you are safe staying with 79e8763 and greater.

-- 
(paka)Patrick Shanahan   Plainfield, Indiana, USA  @ptilopteri
http://en.opensuse.orgopenSUSE Community Memberfacebook/ptilopteri
Registered Linux User #207535@ http://linuxcounter.net
Photos: http://wahoo.no-ip.org/piwigo   paka @ IRCnet freenode
___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Tagging module: What the...

[Tobias Ellinghaus]

Am Samstag, 5. August 2017, 10:14:29 CEST schrieb August Schwerdfeger:
> I compiled the new version (revision 79e8763) and tried it out on the test
> images and also on a copy of my main database. The speed improvement is
> considerable; none of the tagging operations I tried took more than seven
> seconds to complete. I am looking forward to using this in a stable release.

Good, thanks for testing and confirming that it works.

> I have experienced the same kind of delays with tags in the collect module,
> but, oddly enough, not with this version -- neither selecting the "tag"
> option, nor selecting a tag to collect by, nor applying a color label to an
> image in a tag-based collection, took more than two seconds with any
> database I tried.

That's because I fixed that one, too, bringing it from O(n^2) to O(n). :-)

> --
> August Schwerdfeger
> aug...@schwerdfeger.name

Tobias

signature.asc
Description: This is a digitally signed message part.

Re: [darktable-dev] Tagging module: What the...

[August Schwerdfeger]

I compiled the new version (revision 79e8763) and tried it out on the test
images and also on a copy of my main database. The speed improvement is
considerable; none of the tagging operations I tried took more than seven
seconds to complete. I am looking forward to using this in a stable release.

I have experienced the same kind of delays with tags in the collect module,
but, oddly enough, not with this version -- neither selecting the "tag"
option, nor selecting a tag to collect by, nor applying a color label to an
image in a tag-based collection, took more than two seconds with any
database I tried.

--
August Schwerdfeger
aug...@schwerdfeger.name


On Thu, Aug 3, 2017 at 4:49 AM, Tobias Ellinghaus  wrote:

> Am Donnerstag, 3. August 2017, 09:58:01 CEST schrieb Tobias Ellinghaus:
> > Am Dienstag, 1. August 2017, 23:45:54 CEST schrieb August Schwerdfeger:
> > > I managed to reproduce the problem on a smaller scale using a generated
> > > set
> > > of 5,000 1x1 PNG images, each annotated with a certain number of tags,
> > > with
> > > a total of 6,111 across the set. (I will send you these images in a
> > > separate message.)
> >
> > Thanks for that, I was able to reproduce some glacial speeds with that.
>
> This should be fixed now in master. Some parts are still really slow with
> that
> many tags though. For example selecting "tag" in the collect module takes
> minutes just to show up for me.
>
> > > When I imported the first 250 of these images into a fresh database in
> > > Darktable 2.2.4, attaching one new tag to the lot took about two
> seconds.
> > > When I imported all 5,000 into another fresh database, attaching the
> same
> > > tag to the same 250 images took about 10 seconds.
> > >
> > > I also carried out this same test in Darktable 2.0.7. Importing the
> 5,000
> > > images gave me a database file of about 500 megabytes (as opposed to
> four
> > > with Darktable 2.2.4), and the same tag-attachment operation took four
> and
> > > a half minutes. When, however, I repeated the operation with the name
> of a
> > > nonexistent tag in the tagging module text box, it took 10 seconds as
> in
> > > Darktable 2.2.4.
> >
> > What I tried was attaching one new tag to the whole lot. It wasn't fun. I
> > will investigate, but from a first peek it seems that we are spending a
> lot
> > of time in sqlite3. So maybe some SQL queries need to be revised, stuff
> > might have to go into a transaction or not be done at all. I will see.
> >
> > > In light of this result, I should mention that the databases I am
> having
> > > this trouble with were all originally created by Darktable 1.4.2 and
> are
> > > over 75 megabytes in size.
> >
> > As it's also slow with a fresh new database we can probably ignore that
> > part.
> > > --
> > > August Schwerdfeger
> > > aug...@schwerdfeger.name
> >
> > Tobias
> >
> > [...]
>
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Tagging module: What the...

[Tobias Ellinghaus]

Am Donnerstag, 3. August 2017, 09:58:01 CEST schrieb Tobias Ellinghaus:
> Am Dienstag, 1. August 2017, 23:45:54 CEST schrieb August Schwerdfeger:
> > I managed to reproduce the problem on a smaller scale using a generated
> > set
> > of 5,000 1x1 PNG images, each annotated with a certain number of tags,
> > with
> > a total of 6,111 across the set. (I will send you these images in a
> > separate message.)
> 
> Thanks for that, I was able to reproduce some glacial speeds with that.

This should be fixed now in master. Some parts are still really slow with that 
many tags though. For example selecting "tag" in the collect module takes 
minutes just to show up for me.

> > When I imported the first 250 of these images into a fresh database in
> > Darktable 2.2.4, attaching one new tag to the lot took about two seconds.
> > When I imported all 5,000 into another fresh database, attaching the same
> > tag to the same 250 images took about 10 seconds.
> > 
> > I also carried out this same test in Darktable 2.0.7. Importing the 5,000
> > images gave me a database file of about 500 megabytes (as opposed to four
> > with Darktable 2.2.4), and the same tag-attachment operation took four and
> > a half minutes. When, however, I repeated the operation with the name of a
> > nonexistent tag in the tagging module text box, it took 10 seconds as in
> > Darktable 2.2.4.
> 
> What I tried was attaching one new tag to the whole lot. It wasn't fun. I
> will investigate, but from a first peek it seems that we are spending a lot
> of time in sqlite3. So maybe some SQL queries need to be revised, stuff
> might have to go into a transaction or not be done at all. I will see.
> 
> > In light of this result, I should mention that the databases I am having
> > this trouble with were all originally created by Darktable 1.4.2 and are
> > over 75 megabytes in size.
> 
> As it's also slow with a fresh new database we can probably ignore that
> part.
> > --
> > August Schwerdfeger
> > aug...@schwerdfeger.name
> 
> Tobias
> 
> [...]



signature.asc
Description: This is a digitally signed message part.

Re: [darktable-dev] Tagging module: What the...

[Tobias Ellinghaus]

Am Dienstag, 1. August 2017, 23:45:54 CEST schrieb August Schwerdfeger:
> I managed to reproduce the problem on a smaller scale using a generated set
> of 5,000 1x1 PNG images, each annotated with a certain number of tags, with
> a total of 6,111 across the set. (I will send you these images in a
> separate message.)

Thanks for that, I was able to reproduce some glacial speeds with that.

> When I imported the first 250 of these images into a fresh database in
> Darktable 2.2.4, attaching one new tag to the lot took about two seconds.
> When I imported all 5,000 into another fresh database, attaching the same
> tag to the same 250 images took about 10 seconds.
> 
> I also carried out this same test in Darktable 2.0.7. Importing the 5,000
> images gave me a database file of about 500 megabytes (as opposed to four
> with Darktable 2.2.4), and the same tag-attachment operation took four and
> a half minutes. When, however, I repeated the operation with the name of a
> nonexistent tag in the tagging module text box, it took 10 seconds as in
> Darktable 2.2.4.

What I tried was attaching one new tag to the whole lot. It wasn't fun. I will 
investigate, but from a first peek it seems that we are spending a lot of time 
in sqlite3. So maybe some SQL queries need to be revised, stuff might have to 
go into a transaction or not be done at all. I will see.

> In light of this result, I should mention that the databases I am having
> this trouble with were all originally created by Darktable 1.4.2 and are
> over 75 megabytes in size.

As it's also slow with a fresh new database we can probably ignore that part.

> --
> August Schwerdfeger
> aug...@schwerdfeger.name

Tobias

[...]

signature.asc
Description: This is a digitally signed message part.

Re: [darktable-dev] Tagging module: What the...

[August Schwerdfeger]

I managed to reproduce the problem on a smaller scale using a generated set
of 5,000 1x1 PNG images, each annotated with a certain number of tags, with
a total of 6,111 across the set. (I will send you these images in a
separate message.)

When I imported the first 250 of these images into a fresh database in
Darktable 2.2.4, attaching one new tag to the lot took about two seconds.
When I imported all 5,000 into another fresh database, attaching the same
tag to the same 250 images took about 10 seconds.

I also carried out this same test in Darktable 2.0.7. Importing the 5,000
images gave me a database file of about 500 megabytes (as opposed to four
with Darktable 2.2.4), and the same tag-attachment operation took four and
a half minutes. When, however, I repeated the operation with the name of a
nonexistent tag in the tagging module text box, it took 10 seconds as in
Darktable 2.2.4.

In light of this result, I should mention that the databases I am having
this trouble with were all originally created by Darktable 1.4.2 and are
over 75 megabytes in size.

--
August Schwerdfeger
aug...@schwerdfeger.name

On Tue, Aug 1, 2017 at 10:09 AM, August Schwerdfeger <
aug...@schwerdfeger.name> wrote:

> For privacy reasons, I cannot send the full list of tags from my main
> database, but I will try to get together a set that reproduces the problem.
> I am also prepared for any needed custom compilations of Darktable.
>
> I have used a range of GTK versions since I started noticing this issue,
> but currently I am using GTK 3.22 on Fedora and whatever version is bundled
> with Darktable 2.0.7 for OS X.
>
> I think that import with tag application has been slowing down as well.
> Lately when I import images Darktable just seizes up and only becomes
> responsive again some time later, when the import is complete.
>
> --
> August Schwerdfeger
> aug...@schwerdfeger.name
>
>
> On Tue, Aug 1, 2017 at 4:02 AM, Tobias Ellinghaus  wrote:
>
>> Am Montag, 31. Juli 2017, 11:25:02 CEST schrieb August Schwerdfeger:
>> > I have used nothing but the Ctrl+T method for some time (I experience
>> lags
>> > of about 10 seconds between typing a tag name into the module and its
>> > actual appearance in the text box).
>> >
>> > For a while, I could get the Ctrl+T tagging to go faster by keeping the
>> > name of a non-existent tag (e.g., "abcdefg") in the module text box, but
>> > now even that does not work very well.
>>
>> That all sounds strange. I just tried with >2k tags in the library and
>> everything worked smoothly.
>> Could you export the tag list in lighttable and send it to me? I don't
>> think
>> that the format of the tags makes any difference, but you never know.
>> Also, what version of GTK do you use?
>> As a last thing you could try: how quick is it when you import an image
>> and
>> set a tag to be applied during import (in the bottom of the import
>> dialog)? Is
>> that quick? The reason I ask is that I want to know if it's dt's tagging
>> that
>> is slow or if it's the GTK side (selecting from a list, auto completion/
>> suggestions, ...). The import module uses our code directly while the
>> lighttable parts also depend on GTK.
>>
>> If all of that doesn't resolve the issue I fear that you have to compile
>> darktable yourself so that we can add soem debug prints for you to time
>> things.
>>
>> > --
>> > August Schwerdfeger
>> > aug...@schwerdfeger.name
>>
>> Tobias
>>
>> [...]
>
>
>

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Tagging module: What the...

[August Schwerdfeger]

For privacy reasons, I cannot send the full list of tags from my main
database, but I will try to get together a set that reproduces the problem.
I am also prepared for any needed custom compilations of Darktable.

I have used a range of GTK versions since I started noticing this issue,
but currently I am using GTK 3.22 on Fedora and whatever version is bundled
with Darktable 2.0.7 for OS X.

I think that import with tag application has been slowing down as well.
Lately when I import images Darktable just seizes up and only becomes
responsive again some time later, when the import is complete.

--
August Schwerdfeger
aug...@schwerdfeger.name

On Tue, Aug 1, 2017 at 4:02 AM, Tobias Ellinghaus  wrote:

> Am Montag, 31. Juli 2017, 11:25:02 CEST schrieb August Schwerdfeger:
> > I have used nothing but the Ctrl+T method for some time (I experience
> lags
> > of about 10 seconds between typing a tag name into the module and its
> > actual appearance in the text box).
> >
> > For a while, I could get the Ctrl+T tagging to go faster by keeping the
> > name of a non-existent tag (e.g., "abcdefg") in the module text box, but
> > now even that does not work very well.
>
> That all sounds strange. I just tried with >2k tags in the library and
> everything worked smoothly.
> Could you export the tag list in lighttable and send it to me? I don't
> think
> that the format of the tags makes any difference, but you never know.
> Also, what version of GTK do you use?
> As a last thing you could try: how quick is it when you import an image and
> set a tag to be applied during import (in the bottom of the import
> dialog)? Is
> that quick? The reason I ask is that I want to know if it's dt's tagging
> that
> is slow or if it's the GTK side (selecting from a list, auto completion/
> suggestions, ...). The import module uses our code directly while the
> lighttable parts also depend on GTK.
>
> If all of that doesn't resolve the issue I fear that you have to compile
> darktable yourself so that we can add soem debug prints for you to time
> things.
>
> > --
> > August Schwerdfeger
> > aug...@schwerdfeger.name
>
> Tobias
>
> [...]

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Tagging module: What the...

[Tobias Ellinghaus]

Am Montag, 31. Juli 2017, 11:25:02 CEST schrieb August Schwerdfeger:
> I have used nothing but the Ctrl+T method for some time (I experience lags
> of about 10 seconds between typing a tag name into the module and its
> actual appearance in the text box).
> 
> For a while, I could get the Ctrl+T tagging to go faster by keeping the
> name of a non-existent tag (e.g., "abcdefg") in the module text box, but
> now even that does not work very well.

That all sounds strange. I just tried with >2k tags in the library and 
everything worked smoothly.
Could you export the tag list in lighttable and send it to me? I don't think 
that the format of the tags makes any difference, but you never know.
Also, what version of GTK do you use?
As a last thing you could try: how quick is it when you import an image and 
set a tag to be applied during import (in the bottom of the import dialog)? Is 
that quick? The reason I ask is that I want to know if it's dt's tagging that 
is slow or if it's the GTK side (selecting from a list, auto completion/
suggestions, ...). The import module uses our code directly while the 
lighttable parts also depend on GTK.

If all of that doesn't resolve the issue I fear that you have to compile 
darktable yourself so that we can add soem debug prints for you to time 
things.

> --
> August Schwerdfeger
> aug...@schwerdfeger.name

Tobias

[...]

signature.asc
Description: This is a digitally signed message part.

Re: [darktable-dev] Tagging module: What the...

[August Schwerdfeger]

I have used nothing but the Ctrl+T method for some time (I experience lags
of about 10 seconds between typing a tag name into the module and its
actual appearance in the text box).

For a while, I could get the Ctrl+T tagging to go faster by keeping the
name of a non-existent tag (e.g., "abcdefg") in the module text box, but
now even that does not work very well.

--
August Schwerdfeger
aug...@schwerdfeger.name

On Mon, Jul 31, 2017 at 3:39 AM, Tobias Ellinghaus  wrote:

> Am Samstag, 29. Juli 2017, 22:14:09 CEST schrieb August Schwerdfeger:
> > Darktable 1.4.2, 1.6.9, 2.0.7, and 2.2.4, on Fedora, CentOS, and OS X.
> >
> > There is something seriously wrong with Darktable's tagging module.
> >
> > As I have used successive versions of Darktable with databases containing
> > an increasing number of tags, the tagging module has gradually become so
> > slow in its operation as to be essentially unusable.
> >
> > For example, recently, on a database containing ~2500 tags, I attempted
> to
> > attach a single tag to ~250 images. This operation took approximately
> three
> > minutes, during which time Darktable was unresponsive and consistently
> used
> > about 100% of two cores.
>
> That sounds bad. How did you tag the images? By selecting the tag in the
> tagging module and clicking the "attach" button, or using the quick tag
> feature (ctrl-t, type, enter)? Could you please try the latter if you
> didn't
> do so before and tell me if that's faster?
>
> > By way of comparison, I accessed a copy of the database file directly and
> > used an INSERT INTO command to apply the same tag to the same images.
> This
> > took about 20ms.
> >
> > Since actually attaching the tag is clearly not taking six minutes of
> > processor time, what else is the tagging module doing that could possibly
> > be causing these delays (and, what is more to the point, how do I work
> > around it so I can actually use the thing again)?
> >
> > --
> > August Schwerdfeger
> > aug...@schwerdfeger.name
>
> Tobias

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org

Re: [darktable-dev] Tagging module: What the...

[Tobias Ellinghaus]

Am Samstag, 29. Juli 2017, 22:14:09 CEST schrieb August Schwerdfeger:
> Darktable 1.4.2, 1.6.9, 2.0.7, and 2.2.4, on Fedora, CentOS, and OS X.
> 
> There is something seriously wrong with Darktable's tagging module.
> 
> As I have used successive versions of Darktable with databases containing
> an increasing number of tags, the tagging module has gradually become so
> slow in its operation as to be essentially unusable.
> 
> For example, recently, on a database containing ~2500 tags, I attempted to
> attach a single tag to ~250 images. This operation took approximately three
> minutes, during which time Darktable was unresponsive and consistently used
> about 100% of two cores.

That sounds bad. How did you tag the images? By selecting the tag in the 
tagging module and clicking the "attach" button, or using the quick tag 
feature (ctrl-t, type, enter)? Could you please try the latter if you didn't 
do so before and tell me if that's faster?

> By way of comparison, I accessed a copy of the database file directly and
> used an INSERT INTO command to apply the same tag to the same images. This
> took about 20ms.
> 
> Since actually attaching the tag is clearly not taking six minutes of
> processor time, what else is the tagging module doing that could possibly
> be causing these delays (and, what is more to the point, how do I work
> around it so I can actually use the thing again)?
> 
> --
> August Schwerdfeger
> aug...@schwerdfeger.name

Tobias

signature.asc
Description: This is a digitally signed message part.

Re: [darktable-dev] Tagging module: What the...

[Joel Brunetti]

I'm guessing the tags are getting written to the photo's xmp files.

Open one up and you can see the hierarchicalSubject and subject tags. These
both contain tags I use.

Some file systems aren't very efficient writing a small amount of data to
lots of files though I will admit this seems excessive.

Full disclosure, I'm not a developer. Just a lurker/user on the mailing
list.

On 29 July 2017 at 21:14, August Schwerdfeger 
wrote:

> Darktable 1.4.2, 1.6.9, 2.0.7, and 2.2.4, on Fedora, CentOS, and OS X.
>
> There is something seriously wrong with Darktable's tagging module.
>
> As I have used successive versions of Darktable with databases containing
> an increasing number of tags, the tagging module has gradually become so
> slow in its operation as to be essentially unusable.
>
> For example, recently, on a database containing ~2500 tags, I attempted to
> attach a single tag to ~250 images. This operation took approximately three
> minutes, during which time Darktable was unresponsive and consistently used
> about 100% of two cores.
>
> By way of comparison, I accessed a copy of the database file directly and
> used an INSERT INTO command to apply the same tag to the same images. This
> took about 20ms.
>
> Since actually attaching the tag is clearly not taking six minutes of
> processor time, what else is the tagging module doing that could possibly
> be causing these delays (and, what is more to the point, how do I work
> around it so I can actually use the thing again)?
>
> --
> August Schwerdfeger
> aug...@schwerdfeger.name
>
> ___
> darktable developer mailing list to unsubscribe send a mail to
> darktable-dev+unsubscr...@lists.darktable.org
>



-- 
Joel Brunetti
c: 403.808.6208
joelbrune...@gmail.com

___
darktable developer mailing list
to unsubscribe send a mail to darktable-dev+unsubscr...@lists.darktable.org