RE: Update 2.0.0-M7 and indexes

2012-04-18 Thread Carlo.Accorsi 
Checked out and built revision 1327587 and it's working like a charm.. 
Server starts up immediately. No issues searching and is super-fast.. Thanks 
Emmanuel !!

Regards,
Carlo Accorsi

-Original Message-
From: Emmanuel Lécharny [mailto:elecha...@gmail.com] 
Sent: Wednesday, April 18, 2012 1:14 PM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/18/12 3:01 PM, Emmanuel Lécharny a écrit :
> Le 4/18/12 12:16 AM, Emmanuel Lécharny a écrit :
>> Le 4/17/12 11:18 PM, carlo.acco...@ibs-ag.com a écrit :
>>> Was away all day today. From the debug below, i'm using these memory 
>>> settings. -Xms512m -Xmx1024m
>>>
>>> However i've tried 1024 / 1300 and the same thing occurs. I'll set 
>>> up a 64bit wrapper later tonight and will assign 2GB. I'll let you 
>>> know if this makes any difference. thanks.
>>
>> It's strange that the command line does not show the -XmxNNN value 
>> you have set. Injecting 80K entris should not eat more than 300Mo, so 
>> 512M should be just plain perfect.
>>
>> I'm fighting with verious versions of Linux on Virtual box to set up 
>> a configuration close to what you have. Wasted 3 hours at least with 
>> Ubuntu (man, those guys have lost their grinta...) and Fedora. Trying 
>> Mint now...
>>
>>
> Ok, making progress...
>
> First, there is a missing option in the wrapper.conf file :
>
> wrapper.startup.timeout=0
>
> It will not exit after 30 seconds (the default value) if the server is 
> not started.
>
> Second, there is still a problem when you restart the server (even if 
> you have created the index, stopped, restarted the server, injected 
> the 80K entries, stopped and restarted the server) : the index are 
> fully rebuilt, which is just plain wrong.
>
> I'll investigate this point and I'll probably be able to get a fix 
> working for this issue today.

Ok, the issue has been fixed with
http://svn.apache.org/viewvc?rev=1327580&view=rev

I have tested the server with this scenario :
- create a brand new server with nothing in it
- added a context entry for dc=example,dc=com
- added index for sn, cn and displayName
- stopped the server
- re-started the server, index are now present, but empty
- injected 80 000 entries
- the index are full of data
- stopped the server
- re-started the server

the server was up and running in 15 seconds, with all the data present, and 
index working.

I think we are done with those nasty bugs...

pfewww... :)

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

RE: Update 2.0.0-M7 and indexes

2012-04-18 Thread Carlo.Accorsi 
Kool!! Ok, will try it out and get back to the group. Thanks for your help. 

Regards,
Carlo Accorsi

-Original Message-
From: Emmanuel Lécharny [mailto:elecha...@gmail.com] 
Sent: Wednesday, April 18, 2012 1:14 PM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/18/12 3:01 PM, Emmanuel Lécharny a écrit :
> Le 4/18/12 12:16 AM, Emmanuel Lécharny a écrit :
>> Le 4/17/12 11:18 PM, carlo.acco...@ibs-ag.com a écrit :
>>> Was away all day today. From the debug below, i'm using these memory 
>>> settings. -Xms512m -Xmx1024m
>>>
>>> However i've tried 1024 / 1300 and the same thing occurs. I'll set 
>>> up a 64bit wrapper later tonight and will assign 2GB. I'll let you 
>>> know if this makes any difference. thanks.
>>
>> It's strange that the command line does not show the -XmxNNN value 
>> you have set. Injecting 80K entris should not eat more than 300Mo, so 
>> 512M should be just plain perfect.
>>
>> I'm fighting with verious versions of Linux on Virtual box to set up 
>> a configuration close to what you have. Wasted 3 hours at least with 
>> Ubuntu (man, those guys have lost their grinta...) and Fedora. Trying 
>> Mint now...
>>
>>
> Ok, making progress...
>
> First, there is a missing option in the wrapper.conf file :
>
> wrapper.startup.timeout=0
>
> It will not exit after 30 seconds (the default value) if the server is 
> not started.
>
> Second, there is still a problem when you restart the server (even if 
> you have created the index, stopped, restarted the server, injected 
> the 80K entries, stopped and restarted the server) : the index are 
> fully rebuilt, which is just plain wrong.
>
> I'll investigate this point and I'll probably be able to get a fix 
> working for this issue today.

Ok, the issue has been fixed with
http://svn.apache.org/viewvc?rev=1327580&view=rev

I have tested the server with this scenario :
- create a brand new server with nothing in it
- added a context entry for dc=example,dc=com
- added index for sn, cn and displayName
- stopped the server
- re-started the server, index are now present, but empty
- injected 80 000 entries
- the index are full of data
- stopped the server
- re-started the server

the server was up and running in 15 seconds, with all the data present, and 
index working.

I think we are done with those nasty bugs...

pfewww... :)

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-18 Thread Emmanuel Lécharny 

Le 4/18/12 3:01 PM, Emmanuel Lécharny a écrit :

Le 4/18/12 12:16 AM, Emmanuel Lécharny a écrit :

Le 4/17/12 11:18 PM, carlo.acco...@ibs-ag.com a écrit :
Was away all day today. From the debug below, i'm using these memory 
settings. -Xms512m -Xmx1024m


However i've tried 1024 / 1300 and the same thing occurs. I'll set 
up a 64bit wrapper later tonight and will assign 2GB. I'll let you 
know if this makes any difference. thanks.


It's strange that the command line does not show the -XmxNNN value 
you have set. Injecting 80K entris should not eat more than 300Mo, so 
512M should be just plain perfect.


I'm fighting with verious versions of Linux on Virtual box to set up 
a configuration close to what you have. Wasted 3 hours at least with 
Ubuntu (man, those guys have lost their grinta...) and Fedora. Trying 
Mint now...




Ok, making progress...

First, there is a missing option in the wrapper.conf file :

wrapper.startup.timeout=0

It will not exit after 30 seconds (the default value) if the server is 
not started.


Second, there is still a problem when you restart the server (even if 
you have created the index, stopped, restarted the server, injected 
the 80K entries, stopped and restarted the server) : the index are 
fully rebuilt, which is just plain wrong.


I'll investigate this point and I'll probably be able to get a fix 
working for this issue today.


Ok, the issue has been fixed with 
http://svn.apache.org/viewvc?rev=1327580&view=rev


I have tested the server with this scenario :
- create a brand new server with nothing in it
- added a context entry for dc=example,dc=com
- added index for sn, cn and displayName
- stopped the server
- re-started the server, index are now present, but empty
- injected 80 000 entries
- the index are full of data
- stopped the server
- re-started the server

the server was up and running in 15 seconds, with all the data present, 
and index working.


I think we are done with those nasty bugs...

pfewww... :)

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-18 Thread Emmanuel Lécharny 

Le 4/18/12 12:16 AM, Emmanuel Lécharny a écrit :

Le 4/17/12 11:18 PM, carlo.acco...@ibs-ag.com a écrit :
Was away all day today. From the debug below, i'm using these memory 
settings. -Xms512m -Xmx1024m


However i've tried 1024 / 1300 and the same thing occurs. I'll set up 
a 64bit wrapper later tonight and will assign 2GB. I'll let you know 
if this makes any difference. thanks.


It's strange that the command line does not show the -XmxNNN value you 
have set. Injecting 80K entris should not eat more than 300Mo, so 512M 
should be just plain perfect.


I'm fighting with verious versions of Linux on Virtual box to set up a 
configuration close to what you have. Wasted 3 hours at least with 
Ubuntu (man, those guys have lost their grinta...) and Fedora. Trying 
Mint now...




Ok, making progress...

First, there is a missing option in the wrapper.conf file :

wrapper.startup.timeout=0

It will not exit after 30 seconds (the default value) if the server is 
not started.


Second, there is still a problem when you restart the server (even if 
you have created the index, stopped, restarted the server, injected the 
80K entries, stopped and restarted the server) : the index are fully 
rebuilt, which is just plain wrong.


I'll investigate this point and I'll probably be able to get a fix 
working for this issue today.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

RE: Update 2.0.0-M7 and indexes

2012-04-18 Thread Carlo.Accorsi 
Emanuel.. values are there in the debug , just after 
GracefulDisconnectFactory.. I just arrived in the office but I don't have my 
64bit setup done. 
Are you still able to meet online today? I will have my configuration done in 
~20 mins. Then we can speak? I'll send a webex invitation in a few mins. 


Regards,
Carlo Accorsi

Carlo Accorsi | Technical Services
IBSAmerica, Inc. | www.ibs-us.com | Visit IBS on LinkedIn
24 Hartwell Ave | Lexington | Massachusetts | 02421
+1-781-862-9002 (Office) | +1-781-676-8129 (Direct) | +1-781-862-9003 (Fax) 

Sign up for the IBS Blog today!

IBS provides integrated management solutions that help companies improve 
quality and lower costs.


-Original Message-
From: Emmanuel Lécharny [mailto:elecha...@gmail.com] 
Sent: Tuesday, April 17, 2012 6:17 PM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/17/12 11:18 PM, carlo.acco...@ibs-ag.com a écrit :
> Was away all day today. From the debug below, i'm using these memory 
> settings. -Xms512m -Xmx1024m
>
> However i've tried 1024 / 1300 and the same thing occurs. I'll set up a 64bit 
> wrapper later tonight and will assign 2GB. I'll let you know if this makes 
> any difference. thanks.

It's strange that the command line does not show the -XmxNNN value you have 
set. Injecting 80K entris should not eat more than 300Mo, so 512M should be 
just plain perfect.

I'm fighting with verious versions of Linux on Virtual box to set up a 
configuration close to what you have. Wasted 3 hours at least with Ubuntu (man, 
those guys have lost their grinta...) and Fedora. Trying Mint now...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-17 Thread Emmanuel Lécharny 

Le 4/17/12 11:18 PM, carlo.acco...@ibs-ag.com a écrit :

Was away all day today. From the debug below, i'm using these memory settings. 
-Xms512m -Xmx1024m

However i've tried 1024 / 1300 and the same thing occurs. I'll set up a 64bit 
wrapper later tonight and will assign 2GB. I'll let you know if this makes any 
difference. thanks.


It's strange that the command line does not show the -XmxNNN value you 
have set. Injecting 80K entris should not eat more than 300Mo, so 512M 
should be just plain perfect.


I'm fighting with verious versions of Linux on Virtual box to set up a 
configuration close to what you have. Wasted 3 hours at least with 
Ubuntu (man, those guys have lost their grinta...) and Fedora. Trying 
Mint now...



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

RE: Update 2.0.0-M7 and indexes

2012-04-17 Thread Carlo.Accorsi 
Was away all day today. From the debug below, i'm using these memory settings. 
-Xms512m -Xmx1024m 

However i've tried 1024 / 1300 and the same thing occurs. I'll set up a 64bit 
wrapper later tonight and will assign 2GB. I'll let you know if this makes any 
difference. thanks. 



From: Emmanuel Lécharny [elecha...@gmail.com]
Sent: Tuesday, April 17, 2012 6:32 AM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/17/12 12:05 AM, carlo.acco...@ibs-ag.com a écrit :
> Hi, I just updated the JIRA.
> https://issues.apache.org/jira/browse/DIRSERVER-1711?focusedCommentId=13255111#comment-13255111
>
> Thanks for all you've done but I'm experiencing the same issue. I've rebuilt 
> the server from scratch from the trunk today at noon eastern time.
> Again, I create a partition with indexes, restart. Import my o/ou branches, 
> restart. Import 80k users. Once it's complete, searching of users is fast.
> Once I restart the server, the issue below occurs.
I'll investigate the issue. It's pretty strange ... What if you set up
the JVM tu use 512M ?

>
>
> D:\Program Files\ApacheDSM7\bin>wrapper.exe ..\conf\wrapper.conf 
> "set.INSTANCE_DIRECTORY=D:\Program files\apachedsm7\in
> ances\default" "set.INSTANCE=default"
> wrapper  | -->  Wrapper Started as Console
> wrapper  | Using tick timer.
> wrapperp | server listening on port 32000.
> wrapper  | Launching a JVM...
> wrapper  | command: "C:\Program Files (x86)\Java\jdk1.6.0_31\bin\java.exe" 
> -Dlog4j.configuration="file:///D:\Program fi
> s\apachedsm7\instances\default/conf/log4j.properties" 
> -Dapacheds.var.dir="D:\Program files\apachedsm7\instances\default
> -Dapacheds.log.dir="D:\Program files\apachedsm7\instances\default/log" 
> -Dapacheds.run.dir="D:\Program files\apachedsm7\
> stances\default/run" -Dapacheds.instance="default" 
> -Ddefault.controls="org.apache.directory.shared.ldap.codec.controls.
> scade.CascadeFactory,org.apache.directory.shared.ldap.codec.controls.manageDsaIT.ManageDsaITFactory,org.apache.director
> shared.ldap.codec.controls.search.entryChange.EntryChangeFactory,org.apache.directory.shared.ldap.codec.controls.search
> agedSearch.PagedResultsFactory,org.apache.directory.shared.ldap.codec.controls.search.persistentSearch.PersistentSearch
> ctory,org.apache.directory.shared.ldap.codec.controls.search.subentries.SubentriesFactory"
>  -Dextra.controls="org.apache
> irectory.shared.ldap.extras.controls.ppolicy_impl.PasswordPolicyFactory,org.apache.directory.shared.ldap.extras.control
> syncrepl_impl.SyncDoneValueFactory,org.apache.directory.shared.ldap.extras.controls.syncrepl_impl.SyncInfoValueFactory,
> g.apache.directory.shared.ldap.extras.controls.syncrepl_impl.SyncRequestValueFactory,org.apache.directory.shared.ldap.e
> ras.controls.syncrepl_impl.SyncStateValueFactory" 
> -Ddefault.extendedOperation.requests="org.apache.directory.shared.lda
> extras.extended.ads_impl.cancel.CancelFactory,org.apache.directory.shared.ldap.extras.extended.ads_impl.certGeneration.
> rtGenerationFactory,org.apache.directory.shared.ldap.extras.extended.ads_impl.gracefulShutdown.GracefulShutdownFactory,
> g.apache.directory.shared.ldap.extras.extended.ads_impl.storedProcedure.StoredProcedureFactory"
>  -Ddefault.extendedOpera
> on.responses="org.apache.directory.shared.ldap.extras.extended.ads_impl.gracefulDisconnect.GracefulDisconnectFactory"
>  -
> erver -Xms512m -Xmx1024m -Djava.library.path="../lib" -classpath 
> "../lib/apacheds-all-2.0.0-M7-SNAPSHOT.jar;../lib/apac
> ds-service-2.0.0-M7-SNAPSHOT.jar;../lib/apacheds-wrapper-2.0.0-M7-SNAPSHOT.jar;../lib/wrapper-3.2.3.jar"
>  -Dwrapper.key=
> v_49BY1PMOWCwTs" -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 
> -Dwrapper.jvm.port.max=31999 -Dwrapper.debug="TRUE"
> wrapper.pid=7280 -Dwrapper.version="3.2.3" -Dwrapper.native_library="wrapper" 
> -Dwrapper.cpu.timeout="10" -Dwrapper.jvmi
> 1 org.apache.directory.server.wrapper.ApacheDsTanukiWrapper "D:\Program 
> files\apachedsm7\instances\default"
> wrapper  | JVM started (PID=3132)
> jvm 1| WrapperManager class initialized by thread: main  Using 
> classloader: sun.misc.Launcher$AppClassLoader@f4a24a
> jvm 1| Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
> jvm 1|   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
> jvm 1|
> jvm 1| Wrapper Manager: JVM #1
> jvm 1| Running a 32-bit JVM.
> jvm 1| Wrapper Manager: Registering shutdown hook
> jvm 1| Wrapper Manager: Using wrapper
> jvm 1| Load native library.

Re: Update 2.0.0-M7 and indexes

2012-04-17 Thread Emmanuel Lécharny 
tarted.
jvm 1| Open socket to wrapper...Wrapper-Connection
jvm 1| Failed attempt to bind using local port 31000
jvm 1| Opened Socket from 31001 to 32000
jvm 1| Send a packet KEY : 9v_49BY1PMOWCwTs
jvm 1| handleSocket(Socket[addr=/127.0.0.1,port=32000,localport=31001])
wrapperp | accepted a socket from 127.0.0.1 on port 31001
wrapperp | read a packet KEY : 9v_49BY1PMOWCwTs
wrapper  | Got key from JVM: 9v_49BY1PMOWCwTs
wrapperp | send a packet LOW_LOG_LEVEL : 1
wrapperp | send a packet PING_TIMEOUT : 24000
wrapperp | send a packet PROPERTIES : (Property Values)
wrapper  | Start Application.
wrapperp | send a packet START : start
jvm 1| Received a packet LOW_LOG_LEVEL : 1
jvm 1| Wrapper Manager: LowLogLevel from Wrapper is 1
jvm 1| Received a packet PING_TIMEOUT : 24000
jvm 1| PingTimeout from Wrapper is 2400
jvm 1| Received a packet PROPERTIES : (Property Values)
jvm 1| Received a packet START : start
jvm 1| calling WrapperListener.start()
jvm 1| Waiting for WrapperListener.start runner thread to complete.
jvm 1| WrapperListener.start runner thread started.
wrapper  | Startup failed: Timed out waiting for signal from JVM.
wrapper  | JVM did not exit on request, terminated
wrapperp | server listening on port 32000.
wrapper  | JVM was only running for 30 seconds leading to a failed restart 
count of 1.
wrapper  | There were 1 failed launches in a row, each lasting less than 300 
seconds.  Giving up.
wrapper  |   There may be a configuration problem: please check the logs.
wrapper  |<-- Wrapper Stopped

D:\Program Files\ApacheDSM7\bin>pause

Regards,
Carlo Accorsi


Sign up for the IBS Blog today!

IBS provides integrated management solutions that help companies improve 
quality and lower costs.


-Original Message-
From: Emmanuel Lécharny [mailto:elecha...@gmail.com]
Sent: Sunday, April 15, 2012 6:33 AM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/14/12 5:12 AM, carlo.acco...@ibs-ag.com a écrit :

Hi, regarding the server startup issue I'm having after loading a
custom partition with 80k users, the problem seems to be the size of master.db 
For my partition. The file is 450 MB. If I delete my partition directory (and 
let to recreate on startup) the server starts fine.

Here are some other things we tried, none of which worked.

Don't waste your time doing experiments, tehre is a clear problem in the index 
construction when the server is restarted. I'm on it this afternoon.

Seems like the index creation actually works, when we have a few hundreds of 
entries, but we have serious performance issues with 80 000 entries.

I'm not sure it's a bug, but certainly some sub-optimal process.

Keep tuned...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

RE: Update 2.0.0-M7 and indexes

2012-04-16 Thread Carlo.Accorsi 
 31001 to 32000
jvm 1| Send a packet KEY : 9v_49BY1PMOWCwTs
jvm 1| handleSocket(Socket[addr=/127.0.0.1,port=32000,localport=31001])
wrapperp | accepted a socket from 127.0.0.1 on port 31001
wrapperp | read a packet KEY : 9v_49BY1PMOWCwTs
wrapper  | Got key from JVM: 9v_49BY1PMOWCwTs
wrapperp | send a packet LOW_LOG_LEVEL : 1
wrapperp | send a packet PING_TIMEOUT : 24000
wrapperp | send a packet PROPERTIES : (Property Values)
wrapper  | Start Application.
wrapperp | send a packet START : start
jvm 1| Received a packet LOW_LOG_LEVEL : 1
jvm 1| Wrapper Manager: LowLogLevel from Wrapper is 1
jvm 1| Received a packet PING_TIMEOUT : 24000
jvm 1| PingTimeout from Wrapper is 2400
jvm 1| Received a packet PROPERTIES : (Property Values)
jvm 1| Received a packet START : start
jvm 1| calling WrapperListener.start()
jvm 1| Waiting for WrapperListener.start runner thread to complete.
jvm 1| WrapperListener.start runner thread started.
wrapper  | Startup failed: Timed out waiting for signal from JVM.
wrapper  | JVM did not exit on request, terminated
wrapperp | server listening on port 32000.
wrapper  | JVM was only running for 30 seconds leading to a failed restart 
count of 1.
wrapper  | There were 1 failed launches in a row, each lasting less than 300 
seconds.  Giving up.
wrapper  |   There may be a configuration problem: please check the logs.
wrapper  | <-- Wrapper Stopped

D:\Program Files\ApacheDSM7\bin>pause

Regards,
Carlo Accorsi


Sign up for the IBS Blog today!

IBS provides integrated management solutions that help companies improve 
quality and lower costs.


-Original Message-
From: Emmanuel Lécharny [mailto:elecha...@gmail.com] 
Sent: Sunday, April 15, 2012 6:33 AM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/14/12 5:12 AM, carlo.acco...@ibs-ag.com a écrit :
> Hi, regarding the server startup issue I'm having after loading a 
> custom partition with 80k users, the problem seems to be the size of 
> master.db For my partition. The file is 450 MB. If I delete my partition 
> directory (and let to recreate on startup) the server starts fine.
>
> Here are some other things we tried, none of which worked.
Don't waste your time doing experiments, tehre is a clear problem in the index 
construction when the server is restarted. I'm on it this afternoon.

Seems like the index creation actually works, when we have a few hundreds of 
entries, but we have serious performance issues with 80 000 entries.

I'm not sure it's a bug, but certainly some sub-optimal process.

Keep tuned...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

RE: Update 2.0.0-M7 and indexes

2012-04-16 Thread Carlo.Accorsi 
Ok Great! Just so you know, i've always created indexes before importing users. 
Here are the steps i take:: Create partition and indexes then restart. ( the 
attribute.db files show up in the file system with size 20KB). Then i import 
the base dn and branches, restarting once more before importing users. When i 
import the users and monitor the filesysyem the indexes increase in size as the 
ldif is imported as one would expect. If the indexes are being re-created at 
startup you might want to look as to how the logic determines their existance. 
After i completely load the system and shut it down,  the attribute.db files 
are still there and full size and so are the definitions of the indexes in the 
config.ldif. Thank you. 


From: Emmanuel Lécharny [elecha...@gmail.com]
Sent: Monday, April 16, 2012 6:36 AM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Making progress...

The index were deleted due to some regression introduced last year when
using alias and not OIDs when creating index.

Another issue is that the master tabl, containing all the entries, was
read fully for each index to create, instead of reading it only once,
and adding the read entries in all the indexes in one operation.

I'm running some more tests before committing the changes.

One suggestion : declare the index, stop the server, restart the server,
then inject the entries. The index will be present and the server won't
try to recreate them when starting, as they will already be present when
you will start to inject the entries. You'll pay the price of injecting
the entries and creating the indexes only once.

More to come...

Le 4/15/12 1:42 PM, carlo.acco...@ibs-ag.com a écrit :
> ok much appreciaed. Thank you!
> 
> From: Emmanuel Lécharny [elecha...@gmail.com]
> Sent: Sunday, April 15, 2012 6:32 AM
> To: users@directory.apache.org
> Subject: Re: Update 2.0.0-M7 and indexes
>
> Le 4/14/12 5:12 AM, carlo.acco...@ibs-ag.com a écrit :
>> Hi, regarding the server startup issue I'm having after loading a custom 
>> partition with 80k users, the problem seems to be the size of master.db
>> For my partition. The file is 450 MB. If I delete my partition directory 
>> (and let to recreate on startup) the server starts fine.
>>
>> Here are some other things we tried, none of which worked.
> Don't waste your time doing experiments, tehre is a clear problem in the
> index construction when the server is restarted. I'm on it this afternoon.
>
> Seems like the index creation actually works, when we have a few
> hundreds of entries, but we have serious performance issues with 80 000
> entries.
>
> I'm not sure it's a bug, but certainly some sub-optimal process.
>
> Keep tuned...
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-16 Thread Emmanuel Lécharny 

Making progress...

The index were deleted due to some regression introduced last year when 
using alias and not OIDs when creating index.


Another issue is that the master tabl, containing all the entries, was 
read fully for each index to create, instead of reading it only once, 
and adding the read entries in all the indexes in one operation.


I'm running some more tests before committing the changes.

One suggestion : declare the index, stop the server, restart the server, 
then inject the entries. The index will be present and the server won't 
try to recreate them when starting, as they will already be present when 
you will start to inject the entries. You'll pay the price of injecting 
the entries and creating the indexes only once.


More to come...

Le 4/15/12 1:42 PM, carlo.acco...@ibs-ag.com a écrit :

ok much appreciaed. Thank you!

From: Emmanuel Lécharny [elecha...@gmail.com]
Sent: Sunday, April 15, 2012 6:32 AM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/14/12 5:12 AM, carlo.acco...@ibs-ag.com a écrit :

Hi, regarding the server startup issue I'm having after loading a custom 
partition with 80k users, the problem seems to be the size of master.db
For my partition. The file is 450 MB. If I delete my partition directory (and 
let to recreate on startup) the server starts fine.

Here are some other things we tried, none of which worked.

Don't waste your time doing experiments, tehre is a clear problem in the
index construction when the server is restarted. I'm on it this afternoon.

Seems like the index creation actually works, when we have a few
hundreds of entries, but we have serious performance issues with 80 000
entries.

I'm not sure it's a bug, but certainly some sub-optimal process.

Keep tuned...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

RE: Update 2.0.0-M7 and indexes

2012-04-15 Thread Carlo.Accorsi 
ok much appreciaed. Thank you!

From: Emmanuel Lécharny [elecha...@gmail.com]
Sent: Sunday, April 15, 2012 6:32 AM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/14/12 5:12 AM, carlo.acco...@ibs-ag.com a écrit :
> Hi, regarding the server startup issue I'm having after loading a custom 
> partition with 80k users, the problem seems to be the size of master.db
> For my partition. The file is 450 MB. If I delete my partition directory (and 
> let to recreate on startup) the server starts fine.
>
> Here are some other things we tried, none of which worked.
Don't waste your time doing experiments, tehre is a clear problem in the
index construction when the server is restarted. I'm on it this afternoon.

Seems like the index creation actually works, when we have a few
hundreds of entries, but we have serious performance issues with 80 000
entries.

I'm not sure it's a bug, but certainly some sub-optimal process.

Keep tuned...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-15 Thread Emmanuel Lécharny 

Le 4/14/12 5:12 AM, carlo.acco...@ibs-ag.com a écrit :

Hi, regarding the server startup issue I'm having after loading a custom 
partition with 80k users, the problem seems to be the size of master.db
For my partition. The file is 450 MB. If I delete my partition directory (and 
let to recreate on startup) the server starts fine.

Here are some other things we tried, none of which worked.
Don't waste your time doing experiments, tehre is a clear problem in the 
index construction when the server is restarted. I'm on it this afternoon.


Seems like the index creation actually works, when we have a few 
hundreds of entries, but we have serious performance issues with 80 000 
entries.


I'm not sure it's a bug, but certainly some sub-optimal process.

Keep tuned...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-13 Thread Carlo.Accorsi 
Hi, regarding the server startup issue I'm having after loading a custom 
partition with 80k users, the problem seems to be the size of master.db
For my partition. The file is 450 MB. If I delete my partition directory (and 
let to recreate on startup) the server starts fine.

Here are some other things we tried, none of which worked.
Removing all the .db files of my user added indexes and restarting,  The 
indexes are recreated empty, but the same error occurs.

In the config.ldif I tried setting these individually and both false, same 
thing.
ads-partitionsynconwrite: FALSE
ads-jdbmpartitionoptimizerenabled: FALSE

When the server crashes, the next messages expected on the console it the one 
that warns the default password was not reset.
I changed the password but the server still does not start.

Also we are using password polices. I thought maybe the system is trying to 
evaluate the age of every user password at startup?
In config.ldif,  for all policies, I set ads-enabled=FALSE  The same behavior 
occurs.

Is there any case where we should be adding an index for an operational AT? 
Specifically pwdPolicySubEntry?
I'll probably try that next if I don't hear back. Thanks!


unused index file D:\Program 
files\apachedsm7\instances\default\partitions\system\1.3.6.1.4.1.18060.0.4.1.2.4.db
jvm 1| [22:39:06] WARN 
[org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmPartition] - 
Failed to delete
unused index file D:\Program 
files\apachedsm7\instances\default\partitions\system\2.5.4.11.db
wrapper  | Startup failed: Timed out waiting for signal from JVM.
wrapper  | JVM did not exit on request, terminated
wrapperp | server listening on port 32000.
wrapper  | JVM was only running for 30 seconds leading to a failed restart 
count of 1.
wrapper  | There were 1 failed launches in a row, each lasting less than 300 
seconds.  Giving up.
wrapper  |   There may be a configuration problem: please check the logs.
wrapper  | <-- Wrapper Stopped

D:\Program Files\ApacheDSm7\bin>pause

Regards,
Carlo Accorsi

Carlo Accorsi | Technical Services
IBSAmerica, Inc. | www.ibs-us.com | Visit IBS on 
LinkedIn
24 Hartwell Ave | Lexington | Massachusetts | 02421
+1-781-862-9002 (Office) | +1-781-676-8129 (Direct) | +1-781-862-9003 (Fax)

Sign up for the IBS Blog today!

IBS provides integrated management solutions that help companies improve 
quality and lower costs.

RE: Update 2.0.0-M7 and indexes

2012-04-13 Thread Carlo.Accorsi 
OK, thanks for all your effort! Understood this a work in progress. Yes, 
performance is very good, until we restart :) 
If I could just get the server to restart I'd call it success! Thanks! 

Regards,
Carlo Accorsi
-Original Message-
From: Emmanuel Lécharny [mailto:elecha...@gmail.com] 
Sent: Friday, April 13, 2012 5:06 PM
To: users@directory.apache.org
Subject: Re: Update 2.0.0-M7 and indexes

Le 4/13/12 10:03 PM, carlo.acco...@ibs-ag.com a écrit :
> Hi Folks, we've made a lot of progress but it wouldn't be fun if there wasn't 
> one more gotcha.

:)
>
> We Built 2.0.0-M7 SHAPSHOT from the trunk and it installs and runs 
> fine with 1GB memory / 32 bit JVM  (Thanks Kiran for pointers last 
> night) I create a partition and index several attributes. When the server 
> restarts the user attribute  DBM databases appear ! (Thanks Emmanuel) I load 
> my ldif file with 80 k users. When the ldif is loaded, I can see all 80k 
> users, everything seems fine, the indexes are populated I can search and it 
> all works.

Performances are good, I hope...
>
> Once I shut down the server, it will not restart. The wrapper fails with the 
> error below. The size of the JVM goes from zero to 1.3GB, pauses and exists.
> I'm sure there's an OutOfMemory exception occurring but even with Debug it's 
> not appearing in the logs.
>
> The question I have is with a partition this size, with this number of users, 
> what can I do to make the jvm as small as possible to start?
>
> I've reset all my attribute indexes to 100, partition cache is set to 1000 
> BEFORE I loaded all the users. Still having same result.
> Any ideas?  Many thanks.

grmblgtmbl :/ I need to check what's going on when the serve ris stopped and 
restarted. This is clearly a bug, and it should be easy to fix.

The thing is that this is a work in progress, and we have modified many things 
in the index area recently. This should not too much time to get a clue about 
what's going on and to fix the issue. I'l give it a try this week-end.

At least, we are making progress !

Thanks for the info and the follow-up, this is really helpful for us.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Update 2.0.0-M7 and indexes

2012-04-13 Thread Emmanuel Lécharny 

Le 4/13/12 10:03 PM, carlo.acco...@ibs-ag.com a écrit :

Hi Folks, we've made a lot of progress but it wouldn't be fun if there wasn't 
one more gotcha.


:)


We Built 2.0.0-M7 SHAPSHOT from the trunk and it installs and runs fine with 
1GB memory / 32 bit JVM  (Thanks Kiran for pointers last night)
I create a partition and index several attributes. When the server restarts the 
user attribute  DBM databases appear ! (Thanks Emmanuel)
I load my ldif file with 80 k users. When the ldif is loaded, I can see all 80k 
users, everything seems fine, the indexes are populated I can search and it all 
works.


Performances are good, I hope...


Once I shut down the server, it will not restart. The wrapper fails with the 
error below. The size of the JVM goes from zero to 1.3GB, pauses and exists.
I'm sure there's an OutOfMemory exception occurring but even with Debug it's 
not appearing in the logs.

The question I have is with a partition this size, with this number of users, 
what can I do to make the jvm as small as possible to start?

I've reset all my attribute indexes to 100, partition cache is set to 1000 
BEFORE I loaded all the users. Still having same result.
Any ideas?  Many thanks.


grmblgtmbl :/ I need to check what's going on when the serve ris stopped 
and restarted. This is clearly a bug, and it should be easy to fix.


The thing is that this is a work in progress, and we have modified many 
things in the index area recently. This should not too much time to get 
a clue about what's going on and to fix the issue. I'l give it a try 
this week-end.


At least, we are making progress !

Thanks for the info and the follow-up, this is really helpful for us.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Update 2.0.0-M7 and indexes

2012-04-13 Thread Carlo.Accorsi 
Hi Folks, we've made a lot of progress but it wouldn't be fun if there wasn't 
one more gotcha.

We Built 2.0.0-M7 SHAPSHOT from the trunk and it installs and runs fine with 
1GB memory / 32 bit JVM  (Thanks Kiran for pointers last night)
I create a partition and index several attributes. When the server restarts the 
user attribute  DBM databases appear ! (Thanks Emmanuel)
I load my ldif file with 80 k users. When the ldif is loaded, I can see all 80k 
users, everything seems fine, the indexes are populated I can search and it all 
works.

Once I shut down the server, it will not restart. The wrapper fails with the 
error below. The size of the JVM goes from zero to 1.3GB, pauses and exists.
I'm sure there's an OutOfMemory exception occurring but even with Debug it's 
not appearing in the logs.

The question I have is with a partition this size, with this number of users, 
what can I do to make the jvm as small as possible to start?

I've reset all my attribute indexes to 100, partition cache is set to 1000 
BEFORE I loaded all the users. Still having same result.
Any ideas?  Many thanks.

BTW, those index delete warnings were there before I put any users in the 
system.


D:\Program Files\ApacheDSm7\bin>wrapper.exe ..\conf\wrapper.conf 
"set.INSTANCE_DIRECTORY=D:\Program files\apachedsm7\inst
ances\default" "set.INSTANCE=default"
wrapper  | --> Wrapper Started as Console
wrapper  | Using tick timer.
wrapperp | server listening on port 32000.
wrapper  | Launching a JVM...
wrapper  | command: "C:\Program Files (x86)\Java\jdk1.6.0_31\bin\java.exe" 
-Dlog4j.configuration="file:///D:\Program file
s\apachedsm7\instances\default/conf/log4j.properties" 
-Dapacheds.var.dir="D:\Program files\apachedsm7\instances\default"
-Dapacheds.log.dir="D:\Program files\apachedsm7\instances\default/log" 
-Dapacheds.run.dir="D:\Program files\apachedsm7\in
stances\default/run" -Dapacheds.instance="default" 
-Ddefault.controls="org.apache.directory.shared.ldap.codec.controls.ca
scade.CascadeFactory,org.apache.directory.shared.ldap.codec.controls.manageDsaIT.ManageDsaITFactory,org.apache.directory.
shared.ldap.codec.controls.search.entryChange.EntryChangeFactory,org.apache.directory.shared.ldap.codec.controls.search.p
agedSearch.PagedResultsFactory,org.apache.directory.shared.ldap.codec.controls.search.persistentSearch.PersistentSearchFa
ctory,org.apache.directory.shared.ldap.codec.controls.search.subentries.SubentriesFactory"
 -Dextra.controls="org.apache.d
irectory.shared.ldap.extras.controls.ppolicy_impl.PasswordPolicyFactory,org.apache.directory.shared.ldap.extras.controls.
syncrepl_impl.SyncDoneValueFactory,org.apache.directory.shared.ldap.extras.controls.syncrepl_impl.SyncInfoValueFactory,or
g.apache.directory.shared.ldap.extras.controls.syncrepl_impl.SyncRequestValueFactory,org.apache.directory.shared.ldap.ext
ras.controls.syncrepl_impl.SyncStateValueFactory" 
-Ddefault.extendedOperation.requests="org.apache.directory.shared.ldap.
extras.extended.ads_impl.cancel.CancelFactory,org.apache.directory.shared.ldap.extras.extended.ads_impl.certGeneration.Ce
rtGenerationFactory,org.apache.directory.shared.ldap.extras.extended.ads_impl.gracefulShutdown.GracefulShutdownFactory,or
g.apache.directory.shared.ldap.extras.extended.ads_impl.storedProcedure.StoredProcedureFactory"
 -Ddefault.extendedOperati
on.responses="org.apache.directory.shared.ldap.extras.extended.ads_impl.gracefulDisconnect.GracefulDisconnectFactory"
 -Xm
s1024m -Xmx1300m -Djava.library.path="../lib" -classpath 
"../lib/apacheds-all-2.0.0-M7-SNAPSHOT.jar;../lib/apacheds-servi
ce-2.0.0-M7-SNAPSHOT.jar;../lib/apacheds-wrapper-2.0.0-M7-SNAPSHOT.jar;../lib/wrapper-3.2.3.jar"
 -Dwrapper.key="idVWuG6KW
XIDWvWZ" -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 
-Dwrapper.jvm.port.max=31999 -Dwrapper.debug="TRUE" -Dwrapper.
pid=2056 -Dwrapper.version="3.2.3" -Dwrapper.native_library="wrapper" 
-Dwrapper.cpu.timeout="10" -Dwrapper.jvmid=1 org.ap
ache.directory.server.wrapper.ApacheDsTanukiWrapper "D:\Program 
files\apachedsm7\instances\default"
wrapper  | JVM started (PID=8520)
jvm 1| WrapperManager class initialized by thread: main  Using classloader: 
sun.misc.Launcher$AppClassLoader@f4a24a
jvm 1| Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
jvm 1|   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
jvm 1|
jvm 1| Wrapper Manager: JVM #1
jvm 1| Running a 32-bit JVM.
jvm 1| Wrapper Manager: Registering shutdown hook
jvm 1| Wrapper Manager: Using wrapper
jvm 1| Load native library.  One or more attempts may fail if platform 
specific libraries do not exist.
jvm 1| Loaded native library: wrapper-windows-x86-32.dll
jvm 1| Calling native initialization method.
jvm 1| Initializing WrapperManager native library.
jvm 1| Java Executable: C:\Program Files (x86)\Java\jdk1.6.0_31\bin\java.exe
jvm 1| Windows version: 6.1.7601
jvm 1| Java Version   : 1.6.0_31-b05 Java HotSpot(TM) Client VM
jvm 1| Ja