Re: build monday from trunk very slow for ldif import
Le 10/10/12 4:55 AM, carlo.acco...@ibs-ag.com a écrit : Hi, I'm still trying finish the testing of all the Password policy work Karin did over the weekend but I have another issue that's come up. Ldif imports are extremely slow. During our testing, we often delete the entire partition directory to start fresh. When the server starts, it lays down the partition and .db files as defined config.ldif. Anyway, we used to import (via studio) an ldif file with 80k entries and it would load about 90 entries per second. That was great! With this build it's going at about 4-5 entries per second. Hmmm. This is very slow. How many indexed attributes do you have ? I have done some tests locally, and I'm able to get up to 200 add/s, but with a simpler entry. We noticed is that previously, the partition .db files would not change (on disk) until after the ldif import was complete. Then when we stopped the server, it was like the entire import got flushed to the disk at once. The files would go from 20K to 400MB. With this build, it seems to be updating the files as it goes. Could this be the reason? It may be because we flush on disk for every single write. You could turn the ads-partitionSyncOnWrite flag to FALSE, so that the data are flush only ever ads-dsSyncPeriodMillis (default to 15 seconds) Also, this is the first build I noticed the .lg files in the partition directory. I think they're there for journaling but don't know if that's an option something new? It's a JDBM file which is created beside the db files, AFAIR. I have to double check that. We removed all the password policy Attributes from my ldif file thinking that was slowing it down but it's essentially the same performance. Below is my partition and all the indexes are set like the one I included. Any changes that would affect this in the last few weeks. Anyone else seeing this? Thanks! dn: ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=default,ou=config objectclass: top objectClass: ads-base objectclass: ads-partition objectclass: ads-jdbmPartition ads-indexes: apacheRdn ads-indexes: apacheSubLevel ads-indexes: apachePresence ads-indexes: apacheOneLevel ads-indexes: apacheOneAlias ads-indexes: apacheSubAlias ads-indexes: apacheAlias ads-indexes: entryCSN ads-indexes: krb5PrincipalName ads-indexes: objectClass ads-indexes: ou ads-indexes: uid ads-indexes: employeeNumber ads-indexes: displayName ads-indexes: cn ads-indexes: mail ads-indexes: roomNumber ads-indexes: pwdPolicySubEntry ads-indexes: member ads-indexes: description ads-indexes: givenName ads-indexes: sn ads-indexes: administrativeRole ads-partitionSuffix: o=cpro ads-jdbmpartitionoptimizerenabled: TRUE ads-partitioncachesize: 100 ads-partitionsynconwrite: TRUE ads-partitionid: cpro ads-enabled: TRUE #index example, they're all like this..HasReverse=FALSE dn: ads-indexAttributeId=uid,ou=indexes,ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=default,ou=config ads-indexattributeid: uid ads-indexHasReverse: FALSE ads-indexcachesize: 100 objectclass: ads-index objectclass: ads-jdbmIndex objectclass: ads-base objectclass: top ads-enabled: TRUE -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
RE: build monday from trunk very slow for ldif import
ads-partitionSyncOnWrite=FALSE did the trick! Back to 80 adds/sec, Thank you!! Regards, Carlo Accorsi -Original Message- From: Emmanuel Lécharny [mailto:elecha...@gmail.com] Sent: Wednesday, October 10, 2012 3:16 AM To: users@directory.apache.org Subject: Re: build monday from trunk very slow for ldif import Le 10/10/12 4:55 AM, carlo.acco...@ibs-ag.com a écrit : Hi, I'm still trying finish the testing of all the Password policy work Karin did over the weekend but I have another issue that's come up. Ldif imports are extremely slow. During our testing, we often delete the entire partition directory to start fresh. When the server starts, it lays down the partition and .db files as defined config.ldif. Anyway, we used to import (via studio) an ldif file with 80k entries and it would load about 90 entries per second. That was great! With this build it's going at about 4-5 entries per second. Hmmm. This is very slow. How many indexed attributes do you have ? I have done some tests locally, and I'm able to get up to 200 add/s, but with a simpler entry. We noticed is that previously, the partition .db files would not change (on disk) until after the ldif import was complete. Then when we stopped the server, it was like the entire import got flushed to the disk at once. The files would go from 20K to 400MB. With this build, it seems to be updating the files as it goes. Could this be the reason? It may be because we flush on disk for every single write. You could turn the ads-partitionSyncOnWrite flag to FALSE, so that the data are flush only ever ads-dsSyncPeriodMillis (default to 15 seconds) Also, this is the first build I noticed the .lg files in the partition directory. I think they're there for journaling but don't know if that's an option something new? It's a JDBM file which is created beside the db files, AFAIR. I have to double check that. We removed all the password policy Attributes from my ldif file thinking that was slowing it down but it's essentially the same performance. Below is my partition and all the indexes are set like the one I included. Any changes that would affect this in the last few weeks. Anyone else seeing this? Thanks! dn: ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=default,ou=c onfig objectclass: top objectClass: ads-base objectclass: ads-partition objectclass: ads-jdbmPartition ads-indexes: apacheRdn ads-indexes: apacheSubLevel ads-indexes: apachePresence ads-indexes: apacheOneLevel ads-indexes: apacheOneAlias ads-indexes: apacheSubAlias ads-indexes: apacheAlias ads-indexes: entryCSN ads-indexes: krb5PrincipalName ads-indexes: objectClass ads-indexes: ou ads-indexes: uid ads-indexes: employeeNumber ads-indexes: displayName ads-indexes: cn ads-indexes: mail ads-indexes: roomNumber ads-indexes: pwdPolicySubEntry ads-indexes: member ads-indexes: description ads-indexes: givenName ads-indexes: sn ads-indexes: administrativeRole ads-partitionSuffix: o=cpro ads-jdbmpartitionoptimizerenabled: TRUE ads-partitioncachesize: 100 ads-partitionsynconwrite: TRUE ads-partitionid: cpro ads-enabled: TRUE #index example, they're all like this..HasReverse=FALSE dn: ads-indexAttributeId=uid,ou=indexes,ads-partitionId=cpro,ou=partitions ,ads-directoryServiceId=default,ou=config ads-indexattributeid: uid ads-indexHasReverse: FALSE ads-indexcachesize: 100 objectclass: ads-index objectclass: ads-jdbmIndex objectclass: ads-base objectclass: top ads-enabled: TRUE -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: build monday from trunk very slow for ldif import
Le 10/10/12 12:03 PM, carlo.acco...@ibs-ag.com a écrit : ads-partitionSyncOnWrite=FALSE did the trick! Back to 80 adds/sec, Thank you!! Cool. We may have to set this flag to false by default... -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: build monday from trunk very slow for ldif import
we should also change the script to start JVM with -Xms512m and -server options On Wed, Oct 10, 2012 at 4:22 PM, Emmanuel Lécharny elecha...@gmail.com wrote: Le 10/10/12 12:03 PM, carlo.acco...@ibs-ag.com a écrit : ads-partitionSyncOnWrite=FALSE did the trick! Back to 80 adds/sec, Thank you!! Cool. We may have to set this flag to false by default... -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com -- Kiran Ayyagari http://keydap.com
build monday from trunk very slow for ldif import
Hi, I'm still trying finish the testing of all the Password policy work Karin did over the weekend but I have another issue that's come up. Ldif imports are extremely slow. During our testing, we often delete the entire partition directory to start fresh. When the server starts, it lays down the partition and .db files as defined config.ldif. Anyway, we used to import (via studio) an ldif file with 80k entries and it would load about 90 entries per second. That was great! With this build it's going at about 4-5 entries per second. We noticed is that previously, the partition .db files would not change (on disk) until after the ldif import was complete. Then when we stopped the server, it was like the entire import got flushed to the disk at once. The files would go from 20K to 400MB. With this build, it seems to be updating the files as it goes. Could this be the reason? Also, this is the first build I noticed the .lg files in the partition directory. I think they're there for journaling but don't know if that's an option something new? We removed all the password policy Attributes from my ldif file thinking that was slowing it down but it's essentially the same performance. Below is my partition and all the indexes are set like the one I included. Any changes that would affect this in the last few weeks. Anyone else seeing this? Thanks! dn: ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=default,ou=config objectclass: top objectClass: ads-base objectclass: ads-partition objectclass: ads-jdbmPartition ads-indexes: apacheRdn ads-indexes: apacheSubLevel ads-indexes: apachePresence ads-indexes: apacheOneLevel ads-indexes: apacheOneAlias ads-indexes: apacheSubAlias ads-indexes: apacheAlias ads-indexes: entryCSN ads-indexes: krb5PrincipalName ads-indexes: objectClass ads-indexes: ou ads-indexes: uid ads-indexes: employeeNumber ads-indexes: displayName ads-indexes: cn ads-indexes: mail ads-indexes: roomNumber ads-indexes: pwdPolicySubEntry ads-indexes: member ads-indexes: description ads-indexes: givenName ads-indexes: sn ads-indexes: administrativeRole ads-partitionSuffix: o=cpro ads-jdbmpartitionoptimizerenabled: TRUE ads-partitioncachesize: 100 ads-partitionsynconwrite: TRUE ads-partitionid: cpro ads-enabled: TRUE #index example, they're all like this..HasReverse=FALSE dn: ads-indexAttributeId=uid,ou=indexes,ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=default,ou=config ads-indexattributeid: uid ads-indexHasReverse: FALSE ads-indexcachesize: 100 objectclass: ads-index objectclass: ads-jdbmIndex objectclass: ads-base objectclass: top ads-enabled: TRUE
