[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594544#comment-16594544
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Stefan, I'll start a release asap. I'll just check a few things in Fortress
that seems to have trouble with LDAP API 2.0.
We are soon going to look like Eclipse with its train of releases ;-)
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594543#comment-16594543
]
Stefan Seelmann commented on DIRSTUDIO-1174:
Great news. [~elecharny] feel free to cut an LDAP API release, I'll need anyway
some more days to fix some test failures and Mac testing.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594425#comment-16594425
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
Cloned the LDAP API repo, ran a build (not sure I did that properly, but it
made my CPU warm up for a while so it seemed good enough), and then copied JARs
until I found this one copied by this command to where my newly-built Directory
Studio install was waiting:
{{cp ./ldap/model/target/api-ldap-model-2.0.0.AM2-SNAPSHOT.jar
/home/ab/Desktop/ApacheDirectoryStudio/plugins/org.apache.directory.api.ldap.model_2.0.0.AM1.jar}}
Summary: Twenty-seven (27) seconds came down to seven (7), which is only a
couple seconds longer than a fresh install with no connections defined. If it
gets better than this, I'll be surprised. Great, wonderful work! I think
people who use Directory Studio at all will notice this difference since we're
going from a one-second penalty per connection with non-default schema, to
something around a 0.05 second penalty (don't check my math).
I think we can close this bug now, assuming the build is updated to use the new
LDAP API jar. I know we probably need some time to test this new SchemaObject
class definition, but I cannot wait to see this released. M14 (or however the
next release is labeled) deserves to be downloaded today.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593448#comment-16593448
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
So I have done some more tests using the schema you provided, this time in
isolation (ie, I have a test class that does not start LdapStudio at all, so
it's pure Java, no UI).
The file loads in 61ms. It would tale 6 seconds to load 100 schema like this
one. This is quite long. I have ran another Profiling session on this, and I
found some atrocious error : all the schema objects have a hashCode which value
is {{0}} !!! That leads to some expensive {{Set}}/{{Map}} operations (a
{{Tree}} is being used internally because of the collisions). This is due to
[DIRAPI-318|https://issues.apache.org/jira/browse/DIRAPI-318].
Making so the schema object have a valid hash code speed up the schema loading
by a factor 4 (ie, I can now load the schema in 15ms instead of 61ms).
Now, to leverage this change, a new LDAP API version will have to be released
and used in Studio. I'll see with Stefan if this is an option for the next
Studio version - it will take 3 days to cut such a release... -.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592825#comment-16592825
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
OTOH, the results I see in a profiling tool are necessarily biased...
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592815#comment-16592815
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
Rebuilding with the latest commits I see very little change in performance from
the previous nightly version I used, which surprises me. I see the changes in
the files you made, namely swapping out the regexes with a method that iterates
over the char array directly, and changing the comparison function to be
simpler, but I still get the same times I had after I removed folding from the
files manually via 'sed' in the previous comment. I suppose having unfolded
files means that the logic does not need to do much, but I would not expect it
to perform worse on an unfolded file than on a folded one based on the current
logic. Perhaps I will watch for a nightly build to see if that shows any
difference.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755,
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592805#comment-16592805
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
The replaceAll with newline followed by carriage return followed by a space is,
in my opinion, an invalid case since no OS does that (or, as far as I know, has
ever done that):
s = s.replaceAll( "\n\r ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
The carriage only line is valid for Mac OS 9 and earlier, but I am guessing
almost nobody runs that anymore since OS X debuted in 2001, and if they do they
probably cannot get Directory Studio on there. Still, it's theoretically
possible somebody could have an old file from there sent to somebody else with
Directory Studio. If that is deemed too much of an unlikely scenario, then we
can take out this line:
s = s.replaceAll( "\r ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
That leaves the windows carriage return followed by newline abomination, and
the Linux/Unix/MacOSX/etc. case of a simple newline. Since all of these calls
use the String object which is immutable, all of those calls basically recreate
the String each time, and while the regex part is probably the slow part, the
recreation of strings of this size probably does not help much either. It
would be interesting to see which of the following performed best:
s = s.replaceAll( "\r?\n ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
vs.
s = s.replaceAll( "\r", "" ); //$NON-NLS-1$ //$NON-NLS-2$
s = s.replaceAll( "\n ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
vs.
s = s.replaceAll( "(?:\r\n)|(?:\n) ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
Also, is there a reason we fold the lines in the schema files saved out by
Directory Studio? If that is stopped, then a method to read schema files
without trying to unfold them could be used. I suspect folding for internal
use is not that helpful (I personally think folding is not that helpful in
general unless you really hate long lines, but this isn't meant for humans as
much as computers), though I am sure we need the current methods to properly
handle unfolding when getting files from outside Directory Studio.
I went into my
~/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core
directory and unfolded the schema files manually and the load time decreased a
little (two (2) to three (3) seconds), but since the replaceAll calls are still
in there I would expect even better performance with the changes suggested
above:
{quote}{{for onefile in *.ldif; do sed -i -n '1 \{h; $ !d}; $ \{x; s/\n //g;
p}; /^ / \{H; d}; /^ /! \{x; s/\n //g; p}' "${onefile}"; done}}
{quote}
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108]
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592798#comment-16592798
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Having ran another profiling session on Studio, the {{arseSchemaRecord}} method
is now down from 81% to 5% in the hot spots list.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592793#comment-16592793
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Fixes applied with commit fadb2fd53777e28662a7d95a14b9c69c5c178e6e and
ba20ecd330173b100b49a648c3508fcd48500d8c.
There is certainly much to do, but that should already have a visible impact.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592790#comment-16592790
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Ok, I have profiled the schema loading, and it seems there is a lot of room for
improvement, but not some that can be whipped in a few hours ...
The way we process schema is quite convoluted, as we construct a inner
representation of the LDIF file which is supposedly going to be used in the
LDIF editor.
However, there are easy things that can be done to improve the overall speed.
The
{{org.apache.directory.studio.ldapbrowser.core.model.schema.Schema.parseSchemaRecord(LdifContentRecord)
}} function is called as many times as we have schemas, and 22% of the
processing time is spent in a function called 28% of the time. Here is the
function :
{code:java}
protected static String unfold( String s )
{
s = s.replaceAll( "\n\r ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
s = s.replaceAll( "\r\n ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
s = s.replaceAll( "\n ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
s = s.replaceAll( "\r ", "" ); //$NON-NLS-1$ //$NON-NLS-2$
return s;
}
{code}
It's *really* ineficient.
Another place that needs to be improved, the
{{org.apache.directory.studio.ldapbrowser.core.model.schema.SchemaUtils.nameAndOidComparator}}
field declaration can be much simpler (and faster):
{code:java}
private static final Comparator nameAndOidComparator = new
Comparator()
{
public int compare( String s1, String s2 )
{
if ( s1.matches( "[0-9\\.]+" ) && !s2.matches( "[0-9\\.]+" ) )
//$NON-NLS-1$ //$NON-NLS-2$
{
return 1;
}
else if ( !s1.matches( "[0-9\\.]+" ) && s2.matches( "[0-9\\.]+" ) )
//$NON-NLS-1$ //$NON-NLS-2$
{
return -1;
}
else
{
return s1.compareToIgnoreCase( s2 );
}
}
};
{code}
There is no reason this function should not be :
{code:java}
private static final Comparator nameAndOidComparator = new
Comparator()
{
public int compare( String s1, String s2 )
{
return s1.compareToIgnoreCase( s2 );
}
};
{code}
(why would we check if a string is an OID while the other is not at all ? It's
enough to compare them)
I do believe that fixing those two methods would already speed up the schema
loading quite much.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588376#comment-16588376
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
Attaching a new export of times on the schema files; the times are still there
overall, but nothing seems to take quite as long to load (previously fourteen
(14) seconds, this time more like one (1) second). Also managed to load the
process with the YourKit agent so I was able to see the whole startup, which
may be useful for comparison with other systems:
{{/usr/bin/java
-agentpath:/home/ab/apps/yjp-12.0.6/bin/linux-x86-64/libyjpagent.so
-Dosgi.requiredJavaVersion=1.8 -jar
/home/ab/Desktop/ApacheDirectoryStudio//plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar
-os linux -ws gtk -arch x86_64 -showsplash -launcher
/home/ab/Desktop/ApacheDirectoryStudio/ApacheDirectoryStudio -name
ApacheDirectoryStudio --launcher.library
/home/ab/Desktop/ApacheDirectoryStudio//plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.700.v20180518-1200/eclipse_1705.so
-startup
/home/ab/Desktop/ApacheDirectoryStudio//plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar
--launcher.overrideVmargs -exitdata 4ea803f
/studio-rcp/resources/icons/linux/studio.xpm -vm /usr/bin/java -vmargs
-Dosgi.requiredJavaVersion=1.8 -jar
/home/ab/Desktop/ApacheDirectoryStudio//plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar}}
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> 20180821-schema-analysis-b.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108]
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588362#comment-16588362
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
I have attached the 20180821-schema-analysis-a.csv.bz2 file which is a CSV
using a comma as a separator. Notably some of the schema files take much
longer to load than others, one taking fifteen (15) seconds, another fourteen
(14), another twelve (12). I could potentially look at these files to see if
they can be shared, but many are from clients who would probably NOT like to
have the data shared ,and cleaning them up is probably going to be a lot of
work. If anybody has a neat anonymizer, I'd be happy to investigate that.
I've found one file I can attach, which took longer than most, to use as a
sample in tests, but at only 173 KiB it is far from the slowest (about 1/4 of
the slowest, and 1/6 of the largest), though t took a similar amount of time to
open as the slowest.
If there are specific other bits I can get from YourKit I'd be happy to try
that; I do not have a current YourKit version, but I'm on 12-something.
After looking a bit more, the time taken to load these files seems to vary
significantly, making me wonder if YourKit is on drugs, or if CPU scheduling
alone causes some threads to take a long time, though in all tests so far the
aggregate load time is consistent. I'll look for more files to share,
preferably the big ones I suppose, or at least the long-loading ones.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2,
> schema-9060594b-7c28-4123-b574-35fe09727283.ldif.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588357#comment-16588357
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
So it seems some LDIF files take 15 seconds to be processed...
Would it be an option to attach one of those schema to this ticket ?
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch, 20180821-schema-analysis-a.csv.bz2
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found DIRSTUDIO-1027 which may be
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588295#comment-16588295
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Interesting to see that the speedup is clear. I would have expected it to be
more important though, and 28s is still slow as a dog :/
Is it possible for you to investigate where the time is eaten ? (YourKit or
whatever tool could help).
Thanks !
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588102#comment-16588102
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
It is certainly much better; the load time on my box is around twenty-eight
(28) seconds, as opposed to sixty-four seconds (64) for my current install of
whatever is current, an also compared to about four (4) seconds to load a new
instance with no existing preferences. Doing much better may not be realistic,
and perhaps with the new build I can figure out how to delay loading of schema
files in general, but I have not looked into that for a while. I was able to
delay loading of schema files entirely, but then it broke things like the
schema browser when not yet connected, so more work needed to be done.
For now I think this makes a big enough difference to at least get out the
door, but either a new enhancement to delay loading of schema files in general,
or else leaving this bug for that work is still valid.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378,
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587901#comment-16587901
]
Stefan Seelmann commented on DIRSTUDIO-1174:
So should we close this issue?
[~dajoker] maybe you can test if the problem is fixed for you with the latest
snapshot ([https://builds.apache.org/view/D/view/Directory/job/dir-studio/)] ?
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found DIRSTUDIO-1027
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482293#comment-16482293
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
I have rewritten the schema parser. On my preliminary tests, starting Studio is
now considerably faster. We just have to run more tests and verify that the new
parser is correctly covering all the use cases
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441608#comment-16441608
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Antlr is not bad. For a standard implementation, I would say that {{antlr}} is
around 2.5 times slower than an hand written parser.
The problem here is that we are invoking {{antlr}} from {{antlr}} to parse
sub-elements, and it costs a hell lot of CPU.
That being said, my hand written parser is not yet covering all what the
existing parser covers (typically, it's quite strict in what it accepts, while
the {{antlr}} parser allows less strict rules, to be able to handle schema from
'exotic' LDAP servers... However, I can parse pretty much all forms for
{{AttributeType}} and {{ObjectClass}}. What is missing is the
{{objectIdentifier}}, which is a prefix used in the {{OID}} (that is pretty
simple to implement) and the proper handling of {{qdescr}}/{{qdstring}} (the
quoted description/quoted string).
I may push a limited version of the parser tomorrow, up to the caller to use
it, and default to the slower version if an exception is raised (best case : it
gets the data fine and fast, worse case, the job is done twice but it's almost
invisible).
I'll update the ticket tomorrow, atm I'm trying to have the existing tests
passing with the fast parser.
No matter what, your work is certainly useful. I'm pretty sure it will be a
valuable addition to the project, in teh long run.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441228#comment-16441228
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
That is quite an improvement. I was starting to work on how to cleanup the
loading of schema when needed, but if the loading of schema is so fast that it
takes one second instead of one minute, I am not sure how beneficial it would
be to potentially introduce other bugs to make the system load connection
schema as-needed instead of all at the start. Maybe it is still worthwhile to
do that work, but there may also be bigger issues to tackle.
My idea to make this work is not complete, but basically I was thinking of
having an alternate constructor that could be told to do the schema loading so
most o the current code does not try to (or does, depending on the default),
but in the cases where it needs to be loaded from filesystem it could be tried,
and modified in those (few?) places. Modifying the
OpenSchemaBrowserAction.getConnection() method seemed like a good place to do
the check and try to load schema. I have not figured out what to change in
BrowserConnectionWidgetContributionItem but I have not looked hard yet.
Is it worth it for me to continue trying to optimize the loading of schema file
in general, or should I stop and wait for an updated parser? I agree the
current performance is terrible, but that may be less because antlr is doing
poorly and more because of what antlr needs to do based on how it is
configured/programmed in the Directory Studio code. If antlr adds some great
value somehow, improving its configuration in Directory Studio may be in order,
but if its value can be replaced by something simpler/smaller/faster, then
maybe this can also make the build simpler and smaller.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378,
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440928#comment-16440928
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
FTR, I have drafted a {{FastOpenLdapSchemaParser}} which proceeds the same
schema files, without using {{Antlr}}. It's not completely finished, but when
it comes to parse a file like {{core.schema}} (620 lines long, containing 52
AttributeTypes and 27 objectClasses, and is 21Kb big), it's around 50 times
faster !
o antlr parser : 10,925s to parse the file 1,000 times
o hand-written parser : 23.568 to parse the file... 100,000 times
Quite an improvement :-)
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439926#comment-16439926
]
Stefan Seelmann commented on DIRSTUDIO-1174:
The patch looks good, but two things come to my mind:
1. Your change loads the schema from the LDIF whenever opening the connection.
So when closing and opening the connection the LDIF is parsed again. I think
that can be avoided by checking if the set schema is the default schema.
2. Without opening the connection it is possible to open the "Schema Browser"
(right-click the connection). With your change I assume it would always show
the default schema. Currently it shows the cached schema that was loaded on
startup. I don't know how often that feature is used. But after quickly
browsing the code I didn't find a good way how to solve that, at least it needs
to be at two places: In OpenSchemaBrowserAction when opening the schema
browser, and in BrowserConnectionWidgetContributionItem when choosing another
connection from withing the the schema browser.
Regarding how to commit the code: You don't have (yet) permission to directly
push to the git repo, for that you first need to be voted as committer. Either
I commit your patch, or you create a Github pull request against
[https://github.com/apache/directory-studio] then we can simply merge it.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch,
> 20180416-dirstudio-1174-fix-a.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439388#comment-16439388
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
I think you were right, but I think the problem was because I had coded that
last change incorrectly; I was setting an empty Schema object, but not setting
the schema to Schema.DEFAULT_SCHEMA, so the forceReload() method did not see a
need to reload; my bad.
I am attaching another fix which, thanks to your help, I think may be complete.
It seems to load the LDIF on a new connection (if it exists), but if it does
not exist then it properly pulls (and writes) the schema from the directory.
On a reload of Directory Studio, as well as a subsequent connection, the cached
schema seems to be used, as it does not request schema repeatedly from the
directory.
Could somebody let me know if this is good enough, and maybe point me in the
direction of how to commit this? I have the git repo setup, and I have an
apache.org account of course, but right now my git repository is setup
anonymously. If I commit this and try to push, are my credentials sufficient
to get the code checked in?
Thank-you for the help so far. It is nice fixing my own things, and much
easier with guidance on where to look as has been provided here.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439010#comment-16439010
]
Stefan Seelmann commented on DIRSTUDIO-1174:
Loading of the schema from the LDAP server happens in
ReloadSchemaRunnable.java, method reloadSchema(). That is called when opening
the connection or when forcing the reload of the schema.
Your are right, with your change the schema is always loaded from the server
(because the modifyTimestamp of an empyt Schema object is null). However
looking into the mustReload() method I think the logic there is wrong an if the
cached schema is null or empty the schema is not loaded from the server.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438987#comment-16438987
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
For what it is worth, I managed to get the maven build to work on my laptop and
while my normal startup time is about one (1) minute, it's closer to five (5)
seconds when I make a tiny change in one file (see attached patch file), which
is probably as expected. After making this change I was still able to make
LDAP connections to stored connections and those did not seem to be slower than
usual, but I'm still new to this code repository. If somebody could tell me
where the loading of the actual connection happens, I'd be happy to go a step
further and add the lazy loading of the schema file (if available) and see how
performance differs at that time, though since my very-basic testing has not
found errors, I almost wonder if the tool is pulling schema on a new connection
automatically since I do not have schema defined at startup time anymore.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
> Attachments: 20180415-no-load-schema-ldif-by-default.patch
>
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438648#comment-16438648
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
I'm conducting some tests on the schema parser. Parsing 1000 times a schema
which is 620 lines long, containing 52 AttributeTypes and 27 objectClasses, and
is 21Kb big takes 11 seconds on my laptop. This is around 10ms per round. That
also means loading all the default schemas (around 500 schema objects) takes
around 100ms.
For a 1Mb file, that would take 11/(1000/50) = 0,5 seconds, more or less. It's
slow, and can be improved. Of course, if you have tens of such files to parse
on startup, the snowiness will start to be visible...
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs,
Re: [jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
Stefan Seelmann wrote: > Does Studio recreate them next time when opening a > > connection? > > Yes, every time the connection is opened and the schema is loaded it is > written to disk. So a simple pre-start script could simulate non-caching. I guess I'll give that a try and see how it works for me. > In theory when opening the connection Studio checks if the > remote schema was updated (based on modifyTimeStamp) In that case, why even bother loading the cached schemas at startup? Only to speed up connection time? Or can you see other reasons? > but it was multiple times reported that this doesn't work reliable. So if we fix this, we could simply stop loading schema at startup? > Do you want to give it a try? Sounds interesting, but I'm on vacation the next two weeks and will hava a few pretty busy weeks afterwards. So it'll take some time... > Let me know if you need pointers to the code. Definitely needed. If oyu post them here, maybe someone else will even jump at it quicker than I can.
Re: [jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
On 03/25/2018 01:00 PM, Lothar Haeger wrote: > Stefan Seelmann (JIRA) wrote: > >> I think it's easier to change the code to load the schema from LDIF lazily >> when using it the first time. > > While I think that's a good move, what exactly happens when I delete the > schema-*ldif files? Does Studio recreate them next time when opening a > connection? Yes, every time the connection is opened and the schema is loaded it is written to disk. > If so, I'd really like to see an option to not cache schema between sessions > and thus force reading it on every (re-)connect, simply because in my use case > (Identity Management with NetIQ IDM & Edirectory) we do a lot of schema > extensions and have to refresh it all the time to be sure looking at the > current version (which might have been extended by someone else in the mean > time). > > This bug would not bother anyone runnign Studio in > do-not-cache-schema-between-sessions mode as a side effect. Good point. In theory when opening the connection Studio checks if the remote schema was updated (based on modifyTimeStamp) but it was multiple times reported that this doesn't work reliable. Do you want to give it a try? Let me know if you need pointers to the code. Kind Regards, Stefan PS: Unfortunately those replies aren't picked up by Jira...
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413088#comment-16413088
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
I'm not sure antlr4 brings a lot of performance improvement, but nevertheless,
it would worth it to switch to this version. I think we could also merge the 3
schema grammars into one, to avoid switch from one grammar to the other.
Also note that it will imply some other grammar migration :
* ACIItem.g
* ACIItemChecker.g
* TriggerSpecification.g
* distinguishedName.g
* schema-extension.g
* schema-qdstring.g
* schema-value.g
* schema.g
* subtree-specification.g
* SubtreeSpecificationChecker.g
* schemaConverter.g
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1
Re: [jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
Stefan Seelmann (JIRA) wrote: > I think it's easier to change the code to load the schema from LDIF lazily > when using it the first time. While I think that's a good move, what exactly happens when I delete the schema-*ldif files? Does Studio recreate them next time when opening a connection? If so, I'd really like to see an option to not cache schema between sessions and thus force reading it on every (re-)connect, simply because in my use case (Identity Management with NetIQ IDM & Edirectory) we do a lot of schema extensions and have to refresh it all the time to be sure looking at the current version (which might have been extended by someone else in the mean time). This bug would not bother anyone runnign Studio in do-not-cache-schema-between-sessions mode as a side effect.
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412959#comment-16412959
]
Stefan Seelmann commented on DIRSTUDIO-1174:
Regarding antlr, we are still using version 2.7.7, the latest version is 4.7.1
so two major upgrades. I have no idea how hard it is to upgrade and if it would
increase performance or if we would need to rewrite the grammar files...
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found DIRSTUDIO-1027 which may be related.
> During startup of Directory Studio one of
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412953#comment-16412953
]
Stefan Seelmann commented on DIRSTUDIO-1174:
I think it's easier to change the code to load the schema from LDIF lazily when
using it the first time.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found DIRSTUDIO-1027 which may be related.
> During startup of Directory Studio one of my laptop's eight cores is fully
> utilized, which makes me think this may be more about processing the LDIF
> than just
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412946#comment-16412946
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Thanks Stefan.
Knowing that, trying to fix DIRSTUDIO-1027 atm would not address the main
issue. I ran Yorukit on my Studio instance, and I also saw that the antlr was
slow as a dog.
Now :
* should we focus on speeding up the antlr parser
* or should we try to avoid parsing schema at startup, differing the work to
when a connection is used for the first time ?
*
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412940#comment-16412940
]
Stefan Seelmann commented on DIRSTUDIO-1174:
Yes, we parse all cached schema at startup. This happens in
{code:java}
org.apache.directory.studio.ldapbrowser.core.BrowserConnectionManager.loadBrowserConnections()
org.apache.directory.studio.ldapbrowser.core.BrowserConnectionManager.BrowserConnectionManager()
org.apache.directory.studio.ldapbrowser.core.BrowserCorePlugin.start(BundleContext{code}
I added a test to isolate the schema parsing:
[https://github.com/apache/directory-studio/commit/f85cd15360ff1e6cc3c04b18323840b2221d5579].
In this test the default schema (which is around 50kB) is used, 100 iterations
take 5.5 seconds, 20% of time is spent in the actual LDIF parsing
(DIRSTUDIO-1027), the majority of time is spent in the actual parsing of the
schema string via our antlr based schema parsers.
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412920#comment-16412920
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Technically speaking, and 'benefiting' from the fact Studio takes 1 min to
starts, what you should do is to attach Yourkit to the running Studio process
and capture a performance trace (CPU profiling), not a memory trace. Be sure to
configure it to capture every method (Tracing), not sampling, for a better
accuracy.
You can attach the snapshot (I hope I can read it, as I'm using the 2017
version).
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412851#comment-16412851
]
Aaron Burgemeister commented on DIRSTUDIO-1174:
---
I have an old version (12.0.6) of Yourkit which I just tried pointing to
Studio; I am not sure what all I should see that may help, other than the
couple dozen garbage collections as Studio starts, possibly because of
inefficient memory use related to schema parsing, or possibly just because the
tool is starting up. I have spent more time with Eclipse Memory Analyzer in the
past when doing memory stuff, but if there is anything I can specifically check
I am willing to do that.
I believe your understanding is correct in that the schema LDIFs are parsed and
something about that process takes a fair bit of time. I can probably delete
these files to recover performance, but I imagine there is a good reason for
them, like perhaps not needing to grab schema every time a connection is opened
since the cached copy can be referenced instead. Parsing schema LDIFs at
Studio startup seems like a less-great idea than parsing them at connection
creation time, since the latter situation may only need to parse one, or a few,
at a time, but I definitely do not understand all inner workings of Studio to
know the potential impact of that kind of change. On the other hand, taking a
second or two to parse the schema may also be high, so if we can find out how
to optimize that it may not be a huge deal to parse them during startup
(assuming there is significant value in doing so all at once).
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
>
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412687#comment-16412687
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Stefan, AFAIU, we will parse every schema associated with any configured
connections. With hundred of such connections, that might take a while. A I
assuming correctly that is what's going on on startup?
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found DIRSTUDIO-1027 which may be related.
> During startup of Directory Studio one of my laptop's eight cores
[jira] [Commented] (DIRSTUDIO-1174) Directory Studio startup very slow due to schema LDIF processing
[
https://issues.apache.org/jira/browse/DIRSTUDIO-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412686#comment-16412686
]
Emmanuel Lecharny commented on DIRSTUDIO-1174:
--
Interesting findings...
Would you be able to run a profiler on Studio, like YourKit ?
> Directory Studio startup very slow due to schema LDIF processing
>
>
> Key: DIRSTUDIO-1174
> URL: https://issues.apache.org/jira/browse/DIRSTUDIO-1174
> Project: Directory Studio
> Issue Type: Bug
> Components: studio-connection
>Affects Versions: 2.0.0-M13
> Environment: openSUSE Linux (installed on my laptop)
> Sun/Oracle Java 1.8.0_111 (previously 1.7 with same issue)
> Apache Directory Studio 2.0.0 M12 and M13, plus earlier milestones too
>Reporter: Aaron Burgemeister
>Priority: Major
> Labels: LDIF, schema, startup-time
>
> For the past couple years startup of Apache Directory Studio has slowed down
> to the point where it takes more than a minute on my not-a-slouch laptop to
> start. Other systems, VMs with new installs, start much faster, even on the
> same laptop, implying something other than the base product is at fault. As
> a result, I had suspected maybe Directory Studio slowed down precipitously
> due to the number of stored connections, but never confirmed the same.
> Today I connected strace to the 'java' process as it started and noticed the
> following:
>
> [pid 30108] *1521902717*.154740
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-ba001fb7-4b83-4dca-be44-517c14139f4b.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.154906
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.154948
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-95e1202e-9a67-418c-afe9-b02f4e7c06df.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155019
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155053
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-687f43f6-9d05-4d08-b159-35b0e76dc95a.ldif",
> O_RDONLY) = *-1 ENOENT (No such file or directory)*
> [pid 30108] *1521902717*.155120
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902717*.155154
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-d62d0e10-c81e-4477-81a2-ac2c9e5c7169.ldif",
> O_RDONLY) = *121*
> [pid 30108] *1521902718*.698702
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902718*.698800
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-7b6a9a7c-2192-4b24-8874-1378e5b1b30c.ldif",
> O_RDONLY) = *126*
> [pid 30108] *1521902719*.770570
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
> [pid 30108] *1521902719*.770660
> open("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core/schema-b3b02838-067f-4f24-bf92-6bf3fccdbc52.ldif",
> O_RDONLY) = *127*
> [pid 30108] *1521902721*.198417
> stat("/home/ab/.ApacheDirectoryStudio/.metadata/.plugins/org.apache.directory.studio.ldapbrowser.core",
> \{st_mode=S_IFDIR|0755, st_size=5378, ...}) = 0
>
> Notice the timestamps (bolded near beginning of line) and how they change
> based on whether or not a schema LDIF file was found (bolded near end of
> line) and, presumably, processed. When a file is not found, subsequent files
> are sought immediately without significantly delaying startup.
> These schema files are all under 1 MiB in size, but most of them are several
> hundred KiBs, approaching the 1 MiB size, so depending on what Directory
> Studio is doing as it reads and processes these files, it would seem that
> this introduces the slowness when a file is found.
> Looking for an existing issue I found DIRSTUDIO-1027 which may be related.
> During startup of Directory Studio one of my laptop's eight cores is fully
> utilized, which makes me think this may be more about processing the LDIF
> than just swapping memory due
