Re: Toward 2.0...
On Thu, Mar 21, 2013 at 2:09 PM, Emmanuel Lécharny elecha...@gmail.comwrote: Hi guys, it's now 2 years we are working on 2.0, adding milestones after milestones. 1.0 has been released nearly 6 years ago, and 1.5 was just an intermediate version. Basically, we can say that we are working on 2.0 since april 2007... I think it's about time to get a 2.0 out now. The big bugs we had in JDBM have been fixed, replication is petty much working, the Kerberos server is also delivering tickets, This sentence below makes perfect sense. I don't really know what we can add in a milstone that would make 2.0 very different than what we have. Of course, we have some important things to work on : - Mavibot as a replacement for JDBM : this is an ongoing effort, but in any case, it's just a new Partition. It can be added in a 2.1 - MINA 3 switch : again, this is a work in progress. It will bring better performances on the network side, but nothing that can't want for a 2.1 - Triggers/SP : not critical atm - Replication : we currently support MMR with Syncrepl, we would like to add delat-syncrepl, but this is not urgent (2.1) Eveything else are just new features of improvements, that can wait for minor releases. True. Otherwise, we have a bunch of pending bugs that need to be reviewed and fixed, and most important, we need a better documentation. All in all, I think we are pretty much ready, and we can get a 2.0 done by end of april or mid may. thoughts ? In complete agreement. -- Best Regards, -- Alex
Re: Toward 2.0...
My +1! I think it's time to do it. I'll probably propose the same thing for Studio. We've been running in milestones for too long now. How about the Apache Directory LDAP API ? Should we also consider a 1.0.0 release soon? AFAIK, the API has proven to be working pretty great since thousands of people are already using it (without even knowing) via Studio and we've not heard of many complaints. Regards, Pierre-Arnaud On 21 mars 2013, at 13:09, Emmanuel Lécharny elecha...@gmail.com wrote: Hi guys, it's now 2 years we are working on 2.0, adding milestones after milestones. 1.0 has been released nearly 6 years ago, and 1.5 was just an intermediate version. Basically, we can say that we are working on 2.0 since april 2007... I think it's about time to get a 2.0 out now. The big bugs we had in JDBM have been fixed, replication is petty much working, the Kerberos server is also delivering tickets, I don't really know what we can add in a milstone that would make 2.0 very different than what we have. Of course, we have some important things to work on : - Mavibot as a replacement for JDBM : this is an ongoing effort, but in any case, it's just a new Partition. It can be added in a 2.1 - MINA 3 switch : again, this is a work in progress. It will bring better performances on the network side, but nothing that can't want for a 2.1 - Triggers/SP : not critical atm - Replication : we currently support MMR with Syncrepl, we would like to add delat-syncrepl, but this is not urgent (2.1) Eveything else are just new features of improvements, that can wait for minor releases. Otherwise, we have a bunch of pending bugs that need to be reviewed and fixed, and most important, we need a better documentation. All in all, I think we are pretty much ready, and we can get a 2.0 done by end of april or mid may. thoughts ? -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: Toward 2.0...
If I correctly understood the need from the ADS point of view (and not from a pure technology point of view Spring vs ), what the ADS team is looking for is a solution for configuring through an XML file with a simple syntax a set of classes and assembling them in a simple way (JavaBeans convention, no AOP, no IOC). So why not use commons digester (http://commons.apache.org/digester/) ? It is being used by Tomcat for a long time ago now and seems to fill the requirements ? Regards Jeff MAURY On Wed, Mar 18, 2009 at 2:20 AM, Howard Chu h...@symas.com wrote: David Boreham wrote: Graham Leggett wrote: That said, both Fedora and OpenLDAP use the DIT for config, so that may be an XML free option :) FDS doesn't exactly use the DIT. We had the same argument that you're having, in 1997 or so, and decided to write an 'ldif back end' for the server, specifically for config (there had been an earlier flat-file back end that provided some inspiration and possibly some code). Heh. Sounds like the same discussion we had in 2000, it took us a while to get out of the weeds of LDAPv3 before we could see it. ;) This replaced the original UMich text config file (which I believe OpenLDAP still used until recently). So while the config manifests as LDAP entries, and can be read/written over-the-wire, it is stored in a text file as LDIF. If it were in the primary database, you'd have a chicken/egg problem. Right. We wound up with an LDIF backend as well; it can still be configured as a primary database (but that would be stupid) but its main point is that it is so simple to configure (just needs a directory path to store its files) that its bootstrap can be hardcoded, thus giving us the initial egg. I also experimented with using back-ldap as the cn=config backing store; i.e., running a server with a configuration proxied from (and thus identical to) a separate server. But it's more reliable to just replicate the master's config to a local LDIF copy. There's no reason we couldn't provide hardcoded parameters for any other backend type as well, but back-ldif has the virtue of being human-readable/editable in case Something Went Wrong... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- La mélancolie c’est communiste Tout le monde y a droit de temps en temps La mélancolie n’est pas capitaliste C’est même gratuit pour les perdants La mélancolie c’est pacifiste On ne lui rentre jamais dedans La mélancolie oh tu sais ça existe Elle se prend même avec des gants La mélancolie c’est pour les syndicalistes Il faut juste sa carte de permanent Miossec (2006) http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.lastfm.fr/listen/user/jeffmaury/personal
Re: Toward 2.0...
I think this whole configuration stuff should go into its own thread. Towards 2.0 is more general view thread. On Wed, Mar 18, 2009 at 11:06, Jeff MAURY jeffma...@gmail.com wrote: If I correctly understood the need from the ADS point of view (and not from a pure technology point of view Spring vs ), what the ADS team is looking for is a solution for configuring through an XML file with a simple syntax a set of classes and assembling them in a simple way (JavaBeans convention, no AOP, no IOC). So why not use commons digester (http://commons.apache.org/digester/) ? It is being used by Tomcat for a long time ago now and seems to fill the requirements ? Regards Jeff MAURY On Wed, Mar 18, 2009 at 2:20 AM, Howard Chu h...@symas.com wrote: David Boreham wrote: Graham Leggett wrote: That said, both Fedora and OpenLDAP use the DIT for config, so that may be an XML free option :) FDS doesn't exactly use the DIT. We had the same argument that you're having, in 1997 or so, and decided to write an 'ldif back end' for the server, specifically for config (there had been an earlier flat-file back end that provided some inspiration and possibly some code). Heh. Sounds like the same discussion we had in 2000, it took us a while to get out of the weeds of LDAPv3 before we could see it. ;) This replaced the original UMich text config file (which I believe OpenLDAP still used until recently). So while the config manifests as LDAP entries, and can be read/written over-the-wire, it is stored in a text file as LDIF. If it were in the primary database, you'd have a chicken/egg problem. Right. We wound up with an LDIF backend as well; it can still be configured as a primary database (but that would be stupid) but its main point is that it is so simple to configure (just needs a directory path to store its files) that its bootstrap can be hardcoded, thus giving us the initial egg. I also experimented with using back-ldap as the cn=config backing store; i.e., running a server with a configuration proxied from (and thus identical to) a separate server. But it's more reliable to just replicate the master's config to a local LDIF copy. There's no reason we couldn't provide hardcoded parameters for any other backend type as well, but back-ldif has the virtue of being human-readable/editable in case Something Went Wrong... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- La mélancolie c’est communiste Tout le monde y a droit de temps en temps La mélancolie n’est pas capitaliste C’est même gratuit pour les perdants La mélancolie c’est pacifiste On ne lui rentre jamais dedans La mélancolie oh tu sais ça existe Elle se prend même avec des gants La mélancolie c’est pour les syndicalistes Il faut juste sa carte de permanent Miossec (2006) http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.lastfm.fr/listen/user/jeffmaury/personal -- Ersin ER http://www.ersiner.net
Re: Toward 2.0...
Ersin ER wrote: I think this whole configuration stuff should go into its own thread. Towards 2.0 is more general view thread. Yeah. I just used it as Stefan mentioned the moving configuration problem... -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
Another thing to consider is using Spring xml namespace support. For example: http://directory.apache.org/schema/apache-directory-configuration-2.0.0 http://directory.apache.org/schema/apache-directory-configuration-2.0.0.xsd It can get hairy unit testing massive graphs without Spring or an IOC container. Seems to me the core of the issue is that the server is not broken down into simple contexts that build upon each other. With that it should only take a few minutes to isolate an issue. Cheers, - Ole
Re: Toward 2.0...
I'm down with CiDIT over xbean. I'd do this for 2.0. I hate this XML crap. Maybe this is something I can dedicate myself to as others work on replication. I think this XBean stuff is way to mysterious without any documentation. Alex On Tue, Mar 17, 2009 at 7:07 PM, Emmanuel Lecharny elecha...@apache.orgwrote: - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. Main problem is the configuration, which has changed with each minor version. In order to make any progress here, we need to finalize the configuration for the 2.0 first. There must be a decision at some point about the technologies used etc. (XML, xbeans, stored in DIT, ...). We had discussions about this topic every month. Ok. Today, I spent something like 5 hours trying to get some new classes injected into LdapService, and make them work nicely with xbeans. So far, it's a plain failure. I will be very clear : if we are to continue with xbeans+spring, I will -1 the release. This is absolutely not mature, cryptic, unusable, undocumented. In other words, it recalls me Maven 1. Unless xbeans reach another level of stability, I want it out of the configuration. I'm fed of this piece of garbage. I think this is something we have to discuss at ApacheCon, but the decision should be done on the ML. -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
- Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. Main problem is the configuration, which has changed with each minor version. In order to make any progress here, we need to finalize the configuration for the 2.0 first. There must be a decision at some point about the technologies used etc. (XML, xbeans, stored in DIT, ...). We had discussions about this topic every month. Ok. Today, I spent something like 5 hours trying to get some new classes injected into LdapService, and make them work nicely with xbeans. So far, it's a plain failure. I will be very clear : if we are to continue with xbeans+spring, I will -1 the release. This is absolutely not mature, cryptic, unusable, undocumented. In other words, it recalls me Maven 1. Unless xbeans reach another level of stability, I want it out of the configuration. I'm fed of this piece of garbage. I think this is something we have to discuss at ApacheCon, but the decision should be done on the ML. -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
On Mar 17, 2009, at 4:07 PM, Emmanuel Lecharny wrote: - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. Main problem is the configuration, which has changed with each minor version. In order to make any progress here, we need to finalize the configuration for the 2.0 first. There must be a decision at some point about the technologies used etc. (XML, xbeans, stored in DIT, ...). We had discussions about this topic every month. Ok. Today, I spent something like 5 hours trying to get some new classes injected into LdapService, and make them work nicely with xbeans. So far, it's a plain failure. And your requests for help and complaints about specific problems are where? It was a long time ago but 5 hours was the same order of magnitude to the time it took me to convert everything to using xbean-spring (and this was my first use of it). So I wonder if your problems are due to some small factor that I considered too obvious at the time to document. From the point of view of just having written something I often think its really obvious how it works, and then a week later spend a lot of time puzzling out what I did :-) I will be very clear : if we are to continue with xbeans+spring, I will -1 the release. This is absolutely not mature, cryptic, unusable, undocumented. In other words, it recalls me Maven 1. Unless xbeans reach another level of stability, I want it out of the configuration. I'm fed of this piece of garbage. Not sure what you mean by stability. Xbean-spring hasn't been updated in a long time, what it does it seems to do just fine. Undocumented I can't argue with, but activemq seems to be pretty happy with its current implementation. I'll happily agree xbean-spring is pretty terrible, but its better than anything else I've seen anyone use. One thing I would eventually like to investigate that I think could be a big improvement is a domain specific assembly language written in scala. However IIUC a configuration/program in this DSL would need to be compiled. Some people like groovy builders but AFAICT they have no syntax checking until you try to run them. I think this is something we have to discuss at ApacheCon, but the decision should be done on the ML. sure thanks david jencks -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
Ok. Today, I spent something like 5 hours trying to get some new classes injected into LdapService, and make them work nicely with xbeans. So far, it's a plain failure. And your requests for help and complaints about specific problems are where? It was a long time ago but 5 hours was the same order of magnitude to the time it took me to convert everything to using xbean-spring (and this was my first use of it). So I wonder if your problems are due to some small factor that I considered too obvious at the time to document. From the point of view of just having written something I often think its really obvious how it works, and then a week later spend a lot of time puzzling out what I did :-) As usual, I tried first to get it working by myself. It may sound quite arrogant, but if I can't make it work, then I see no reason why the average joe programmer can. If it were so damn obvious, then I must be a total arse... (which is still an option, at this point ;) What fool me totally is that each single time I tried to understand the way it works, it always a real pain in the back orifice. Typically the signal of bad implemented techno to me... I will be very clear : if we are to continue with xbeans+spring, I will -1 the release. This is absolutely not mature, cryptic, unusable, undocumented. In other words, it recalls me Maven 1. Unless xbeans reach another level of stability, I want it out of the configuration. I'm fed of this piece of garbage. Not sure what you mean by stability. Xbean-spring hasn't been updated in a long time, what it does it seems to do just fine. Undocumented I can't argue with, but activemq seems to be pretty happy with its current implementation. Ok, stability is a bit too much. I'll happily agree xbean-spring is pretty terrible, but its better than anything else I've seen anyone use. That's the main issue... Xbeans is trying to alleviate the pain it is to manipulate class names in a Spring file. As I already said, it creates a decoupling which is painfull, as you have no clue about what is what in the Spring file when you have decoupled those two parts. We have more than 2500 classes in Directory (not counting Studio), and even after 5 years working on the project, I can't easily remember where all of them are located. Even if the previous Spring conf with the 500 char longs lines was impossible to read, at least, you were able to knwo exactly in which package you can find a class. So xbeans seems to me like aspirin when you have a cancer (the cancer being Spring) : it does not cure, and your stomach hurts... To be frank, and we discussed that ad nauseam with Alex, using Spring was one of the biggest mistakes we made. We don't need Inversion of control/Depenency Injection/whatever Fowler call it. This is useless in our case. What we need is a configuration which can be loaded at startup, and that's it. what we had before (a property file) was just plain ok. OpenLDAP is now storing the configuration in the DIT, and it works perfectly well. And if we don't want to go back to a property file, or config in the DIT, then JSON can be an intersting option. Everything but this infamous brackets all over the configration !!! Die, XML, die ! PS : I'll be pleased to share a beer or two with you David next week ! -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
Emmanuel Lecharny wrote: That's the main issue... Xbeans is trying to alleviate the pain it is to manipulate class names in a Spring file. As I already said, it creates a decoupling which is painfull, as you have no clue about what is what in the Spring file when you have decoupled those two parts. We have more than 2500 classes in Directory (not counting Studio), and even after 5 years working on the project, I can't easily remember where all of them are located. The name of the principle you are currntly violating by using Spring is called the Don't repeat yourself principle: http://en.wikipedia.org/wiki/Don%27t_repeat_yourself By storing class names in two places (the Java class file, and the Spring bean definition XML file) you are adding 2500 different opportunities for the classes and spring configs to go out of sync. And there is nothing in the class to suggest that a spring bean is somehow relevant, making refactoring impossible, and that kills code. To be frank, and we discussed that ad nauseam with Alex, using Spring was one of the biggest mistakes we made. We don't need Inversion of control/Depenency Injection/whatever Fowler call it. This is useless in our case. What we need is a configuration which can be loaded at startup, and that's it. what we had before (a property file) was just plain ok. OpenLDAP is now storing the configuration in the DIT, and it works perfectly well. And if we don't want to go back to a property file, or config in the DIT, then JSON can be an intersting option. Everything but this infamous brackets all over the configration !!! Die, XML, die ! When I use XML as a config, I define that config as an XSD, throw the XSD through xmlbeans to write all the parsing code for me, and then I make sure the config file is properly validated on load. If the end user does something wrong, the end user gets proper error messages telling them what they did wrong, and what line they did it wrong on. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Toward 2.0...
Graham Leggett wrote: Emmanuel Lecharny wrote: That's the main issue... Xbeans is trying to alleviate the pain it is to manipulate class names in a Spring file. As I already said, it creates a decoupling which is painfull, as you have no clue about what is what in the Spring file when you have decoupled those two parts. We have more than 2500 classes in Directory (not counting Studio), and even after 5 years working on the project, I can't easily remember where all of them are located. The name of the principle you are currntly violating by using Spring is called the Don't repeat yourself principle: http://en.wikipedia.org/wiki/Don%27t_repeat_yourself By storing class names in two places (the Java class file, and the Spring bean definition XML file) you are adding 2500 different opportunities for the classes and spring configs to go out of sync. And there is nothing in the class to suggest that a spring bean is somehow relevant, making refactoring impossible, and that kills code. = why using Spring ? :) To be frank, and we discussed that ad nauseam with Alex, using Spring was one of the biggest mistakes we made. We don't need Inversion of control/Depenency Injection/whatever Fowler call it. This is useless in our case. What we need is a configuration which can be loaded at startup, and that's it. what we had before (a property file) was just plain ok. OpenLDAP is now storing the configuration in the DIT, and it works perfectly well. And if we don't want to go back to a property file, or config in the DIT, then JSON can be an intersting option. Everything but this infamous brackets all over the configration !!! Die, XML, die ! When I use XML as a config, I define that config as an XSD, throw the XSD through xmlbeans to write all the parsing code for me, and then I make sure the config file is properly validated on load. The only little problem with this approach is that writing a XSD is an order of magnitude more painful than dealing with all the Spring crap. = Why using XML when not absolutely necessary ? ;) Die, XML, die ! (well, I may die before XML ... :/ ) -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
Emmanuel Lecharny wrote: The name of the principle you are currntly violating by using Spring is called the Don't repeat yourself principle: http://en.wikipedia.org/wiki/Don%27t_repeat_yourself By storing class names in two places (the Java class file, and the Spring bean definition XML file) you are adding 2500 different opportunities for the classes and spring configs to go out of sync. And there is nothing in the class to suggest that a spring bean is somehow relevant, making refactoring impossible, and that kills code. = why using Spring ? :) Exactly :) The only little problem with this approach is that writing a XSD is an order of magnitude more painful than dealing with all the Spring crap. = Why using XML when not absolutely necessary ? ;) Die, XML, die ! (well, I may die before XML ... :/ ) The trouble with properties is that they are not always that good at expressing hierarchies of concepts (something XML does well), and if you accidently spell a property wrong, that property silently vanishes, leaving the end user staring at the config for hours trying to figure out why it isn't working as expected. A config that is wrong should fail early and fail often (to misuse the expression), and the end user should ideally be told exactly what they did wrong. That said, both Fedora and OpenLDAP use the DIT for config, so that may be an XML free option :) Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Toward 2.0...
Graham Leggett wrote: That said, both Fedora and OpenLDAP use the DIT for config, so that may be an XML free option :) FDS doesn't exactly use the DIT. We had the same argument that you're having, in 1997 or so, and decided to write an 'ldif back end' for the server, specifically for config (there had been an earlier flat-file back end that provided some inspiration and possibly some code). This replaced the original UMich text config file (which I believe OpenLDAP still used until recently). So while the config manifests as LDAP entries, and can be read/written over-the-wire, it is stored in a text file as LDIF. If it were in the primary database, you'd have a chicken/egg problem.
Re: Toward 2.0...
The trouble with properties is that they are not always that good at expressing hierarchies of concepts (something XML does well), and if you accidently spell a property wrong, that property silently vanishes, leaving the end user staring at the config for hours trying to figure out why it isn't working as expected. JSon is hierarchical, even if it does not have nameSpace (which is _bad_). So it's a pretty good alternative. Also one thing that make me crazy s that with Spring, you have no clue about which bean is going to be instanciated and we=hen. I mean, you have to put breakpoints all over the code, when you want to debug the initialization. Something you don't have to do when using a simple file you load. That is typically what make me want to through my computer through my office or my appartement. For instance, having such error messages is self-explanatory. Two bullets in the head, and you send the bills to the parents... (more to read at the end of the damn long Stack trace) org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ldapService' defined in URL [file:/home/elecharny/apacheds/replication/apacheds/server-xml/target/classes/serverReplication.xml]: Cannot create inner bean '(inner bean)' of type [org.apache.directory.server.ldap.replication.ReplicaPeerConfiguration] while setting bean property 'replicationSystem'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#5' defined in URL [file:/home/elecharny/apacheds/replication/apacheds/server-xml/target/classes/serverReplication.xml]: Initialization of bean failed; nested exception is org.springframework.beans.TypeMismatchException: Failed to convert property value of type [java.lang.String] to required type [org.apache.directory.shared.ldap.name.LdapDN] for property 'principalDN'; nested exception is java.lang.IllegalArgumentException: Original must not be null at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:230) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1245) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1010) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:472) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409) at java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:221) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381) at org.apache.xbean.spring.context.FileSystemXmlApplicationContext.init(FileSystemXmlApplicationContext.java:149) at org.apache.xbean.spring.context.FileSystemXmlApplicationContext.init(FileSystemXmlApplicationContext.java:48) at org.apache.directory.server.SpringServerTest.testSpringServerReplication(SpringServerTest.java:125) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59) at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98) at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79) at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87) at
Re: Toward 2.0...
David Boreham wrote: Graham Leggett wrote: That said, both Fedora and OpenLDAP use the DIT for config, so that may be an XML free option :) FDS doesn't exactly use the DIT. We had the same argument that you're having, in 1997 or so, and decided to write an 'ldif back end' for the server, specifically for config (there had been an earlier flat-file back end that provided some inspiration and possibly some code). This replaced the original UMich text config file (which I believe OpenLDAP still used until recently). So while the config manifests as LDAP entries, and can be read/written over-the-wire, it is stored in a text file as LDIF. If it were in the primary database, you'd have a chicken/egg problem. OpenLDAP does the same thing. The idea is to have a LDIF based backend. This is also something we are considering. -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
I've got to say it : JUST SAY NO TO SPRING. (I can't remember ever using all-caps before in a post, during my 25 years using e-mail..)
Re: Toward 2.0...
Emmanuel Lecharny wrote: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ldapService' defined in URL [file:/home/elecharny/apacheds/replication/apacheds/server-xml/target/classes/serverReplication.xml]: Cannot create inner bean '(inner bean)' of type [org.apache.directory.server.ldap.replication.ReplicaPeerConfiguration] while setting bean property 'replicationSystem'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#5' defined in URL [file:/home/elecharny/apacheds/replication/apacheds/server-xml/target/classes/serverReplication.xml]: Initialization of bean failed; nested exception is org.springframework.beans.TypeMismatchException: Failed to convert property value of type [java.lang.String] to required type [org.apache.directory.shared.ldap.name.LdapDN] for property 'principalDN'; nested exception is java.lang.IllegalArgumentException: Original must not be null Insane, isn't it ? Absolutely. Insane is spending six weeks hunting down a problem where a set of clustered calculations would intermittently return different answers for the same input data, and it turned out that someone, somewhere, had cut and pasted 'singleton=true' into some bean, somewhere. Disaster. Spring is just an attempt to turn a strongly typed language (Java) into a weak typed language, which has the effect of moving problems that would have been picked up by your compiler, and moving them to be picked up by your end user instead, in the form of incomprehensible exceptions like the one above. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Toward 2.0...
David Boreham wrote: Graham Leggett wrote: That said, both Fedora and OpenLDAP use the DIT for config, so that may be an XML free option :) FDS doesn't exactly use the DIT. We had the same argument that you're having, in 1997 or so, and decided to write an 'ldif back end' for the server, specifically for config (there had been an earlier flat-file back end that provided some inspiration and possibly some code). Heh. Sounds like the same discussion we had in 2000, it took us a while to get out of the weeds of LDAPv3 before we could see it. ;) This replaced the original UMich text config file (which I believe OpenLDAP still used until recently). So while the config manifests as LDAP entries, and can be read/written over-the-wire, it is stored in a text file as LDIF. If it were in the primary database, you'd have a chicken/egg problem. Right. We wound up with an LDIF backend as well; it can still be configured as a primary database (but that would be stupid) but its main point is that it is so simple to configure (just needs a directory path to store its files) that its bootstrap can be hardcoded, thus giving us the initial egg. I also experimented with using back-ldap as the cn=config backing store; i.e., running a server with a configuration proxied from (and thus identical to) a separate server. But it's more reliable to just replicate the master's config to a local LDIF copy. There's no reason we couldn't provide hardcoded parameters for any other backend type as well, but back-ldif has the virtue of being human-readable/editable in case Something Went Wrong... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Toward 2.0...
Tony Stevenson wrote: The DRS system is of interest to me. What happens at the moment if the system crashes? Can we copy the flat files, daily, hourly? Would they be re-usable. i.e. Could I copy the data files, and could they be used again if copied over the other files? I am thinking about a backup strategy for the LDAP data. I use subversion to back up my LDAP servers - the ldif snapshot is dumped on a regular basis, and the ldif is then automatically checked into a repository dedicated for this purpose. This technique gives you backup depth, so that corruption of the LDAP server doesn't result in accidental corruption of the backups. Regards, Graham -- smime.p7s Description: S/MIME Cryptographic Signature
Re: Toward 2.0...
Tony Stevenson wrote: Emmanuel, Hi Tony, It's good to see some of these things on the list. However one or two of these will be required should we decide to use ADS for the ASF. I know we will be able to get a good basic LDAP service in place without some of these, *but* some are core requirements to the implementation of ADS within the ASF. I'm perfectly aware of that :) A LDAP server without replication is pretty useless, from a production POV. For example, the replication system will be one such core fundamental requirement. In fact we cannot really proceed a lot further with the current demo/testing environment without it. What state is it currently in? It's running. The concern we have about it is that we didn't used it _yet_ in a production environment (here the 'we' is Alex and me). We have to review it. However, two of our committers have worked on it, and one of them (Martin Alderson) has use the replication for its own project, after a few fixes, one year ago. We also have integration tests, still running after thousands of commits, so I guess the replication system is still robust, as we didn't modified the tests. The fact that replication has been mentioned into the 'Toward 2.0' list is because we want to be 100% sure that it works well, is solid, and covers all the corner cases. We also want to move the current implementation to something more versatile, based on the since added Change Log system. This is what we want to work on in the next 4 months. But the next 2 weeks will be dedicated to check that Mitoses (the current replication code name) is working well. Also before LDAP goes live we will need some good basic documentation, which I can certainly help produce. These will mainly focus on system runbooks, and who/what/where/how to make changes, stop/start services etc. We can work on that, for sure. As ADS is running as a daemon, it should be pretty easy to write the basic documentation. The DRS system is of interest to me. You bet a lot of people are interested in it ! What happens at the moment if the system crashes? ... Can we copy the flat files, daily, hourly? ... Would they be re-usable. i.e. Could I copy the data files, and could they be used again if copied over the other files? I am thinking about a backup strategy for the LDAP data. We do too. The think is that currently, we are relying on JDBM, and if the server crash severely well, you may lost some data. This is certainly not something we can accept. Now, this is the bad. About the good : a LDAP server is known to be read most of the time, and written a few. If we backup the JDBM files, hourly, we will be able to restart the server and only lost a few modification. The ChangeLog system will be extended in the next few days to write a journal (a LDIF file) in order to be able to restore the server state. As we have a unique revision number associated with each modification, a bit like subversion, we should be able to apply this journal starting at some specific point in the past we know to be good. Those two features are for us the _most_ important things we want to work on, and we want to start working on it right now, in order to get a production ready server. -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
On 04/01/2009 14:54, Emmanuel Lecharny wrote: [SNIP ...] ... Would they be re-usable. i.e. Could I copy the data files, and could they be used again if copied over the other files? I am thinking about a backup strategy for the LDAP data. We do too. The think is that currently, we are relying on JDBM, and if the server crash severely well, you may lost some data. This is certainly not something we can accept. Now, this is the bad. About the good : a LDAP server is known to be read most of the time, and written a few. If we backup the JDBM files, hourly, we will be able to restart the server and only lost a few modification. The ChangeLog system will be extended in the next few days to write a journal (a LDIF file) in order to be able to restore the server state. As we have a unique revision number associated with each modification, a bit like subversion, we should be able to apply this journal starting at some specific point in the past we know to be good. Copying files hourly should be ok, this should be less of an issue, as we would have a replica server on the other side of the world. Now that sounds good. Especially if it can be emailed to an ML after each change. Those two features are for us the _most_ important things we want to work on, and we want to start working on it right now, in order to get a production ready server. As our American friends might say, Shwet Cheers, Tony -- - Tony Stevenson t...@pc-tony.com // pct...@apache.org http://www.pc-tony.com/ 1024D/51047D66 ECAF DC55 C608 5E82 0B5E 3359 C9C7 924E 5104 7D66 -
Re: Toward 2.0...
Hi Emmanuel! Emmanuel Lecharny wrote: I will gather all the items present in the previous roadmap into three different lists : 1) Mandatory : what we absolutely need for a 2.0 2) Good to have : what we want to have in a 2.0 if we have time to do it 3) Not urgent : what we can push in a 2.5 release So here is the list : 1) Mandatory ... - Interface cleaning : we have to check all the common interfaces and clean them. It's not necessarily a costly task, but as we will freeze the API, we have to do that before 2.0-RC1 This one is really important if someone plans to extend the server. The task includes updating the javadoc of the interfaces as well. - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. Main problem is the configuration, which has changed with each minor version. In order to make any progress here, we need to finalize the configuration for the 2.0 first. There must be a decision at some point about the technologies used etc. (XML, xbeans, stored in DIT, ...). We had discussions about this topic every month. - VSLDAP : We would like to pass the STANDARD test this year. That should not be too complicated. In order to reduce workload, I would declare VSLDAP at STANDARD level as Good to have. Only BASE level is Mandatory from my point of view. Even for BASE, stabilizing the configuration soon is a must. Configuring the test suite is tricky (especially the SSL part), and it costs time to modify it if configuration changes. Greetings from Hamburg, Stefan
Re: Toward 2.0...
Graham Leggett wrote: Tony Stevenson wrote: The DRS system is of interest to me. What happens at the moment if the system crashes? Can we copy the flat files, daily, hourly? Would they be re-usable. i.e. Could I copy the data files, and could they be used again if copied over the other files? I am thinking about a backup strategy for the LDAP data. I use subversion to back up my LDAP servers - the ldif snapshot is dumped on a regular basis, and the ldif is then automatically checked into a repository dedicated for this purpose. This technique gives you backup depth, so that corruption of the LDAP server doesn't result in accidental corruption of the backups. That's a pretty smart solution, assuming the server does not contains too much data ! Extracting all the ASF entries from ADS will only take one or two seconds... Thanks Graham ! -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
Stefan Zoerner wrote: Hi Emmanuel! Hi Stefan ! - Interface cleaning : we have to check all the common interfaces and clean them. It's not necessarily a costly task, but as we will freeze the API, we have to do that before 2.0-RC1 This one is really important if someone plans to extend the server. The task includes updating the javadoc of the interfaces as well. Very true. I think we won't add a lot more features to the server, so the API cleaning could start as soon as we have a new base line (ie, when 1.5.5 will be closed). So I think we need to release 1.5.5 soon, even if it means we have to postpone some issues. All in all, 1.5.5 is manly about integrating MINA 2.0. - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. Main problem is the configuration, which has changed with each minor version. In order to make any progress here, we need to finalize the configuration for the 2.0 first. There must be a decision at some point about the technologies used etc. (XML, xbeans, stored in DIT, ...). We had discussions about this topic every month. Yeah, I can't agree more. I have spent a large part of the last 2 weeks working on the configuration and its documentation, and I must admit it's messy. It's time to stop changing it version after version. Frankly, the perfect configuration is a myth. Here is what I suggest : let's keep Spring+xbeans from now on, just because changing that will be disruptive. It's not perfect, but at least, it works. I also suggest that we freeze the configuration for 1.5.5. - VSLDAP : We would like to pass the STANDARD test this year. That should not be too complicated. In order to reduce workload, I would declare VSLDAP at STANDARD level as Good to have. Only BASE level is Mandatory from my point of view. +1. Thanks Stefan ! -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org
Re: Toward 2.0...
On Sun, Jan 4, 2009 at 3:39 PM, Emmanuel Lecharny elecha...@gmail.comwrote: Stefan Zoerner wrote: Hi Emmanuel! Hi Stefan ! - Interface cleaning : we have to check all the common interfaces and clean them. It's not necessarily a costly task, but as we will freeze the API, we have to do that before 2.0-RC1 This one is really important if someone plans to extend the server. The task includes updating the javadoc of the interfaces as well. Very true. I think we won't add a lot more features to the server, so the API cleaning could start as soon as we have a new base line (ie, when 1.5.5 will be closed). So I think we need to release 1.5.5 soon, even if it means we have to postpone some issues. All in all, 1.5.5 is manly about integrating MINA 2.0. - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. Main problem is the configuration, which has changed with each minor version. In order to make any progress here, we need to finalize the configuration for the 2.0 first. There must be a decision at some point about the technologies used etc. (XML, xbeans, stored in DIT, ...). We had discussions about this topic every month. Yeah, I can't agree more. I have spent a large part of the last 2 weeks working on the configuration and its documentation, and I must admit it's messy. It's time to stop changing it version after version. Frankly, the perfect configuration is a myth. Here is what I suggest : let's keep Spring+xbeans from now on, just because changing that will be disruptive. It's not perfect, but at least, it works. I also suggest that we freeze the configuration for 1.5.5. +1 I don't think we can get off spring+xbean soon with a configuration in DIT implementation. However you cannot ask for a freeze on anything until we decide to go to 2.0 RC1. Alex
Re: Toward 2.0...
Emmanuel, It's good to see some of these things on the list. However one or two of these will be required should we decide to use ADS for the ASF. I know we will be able to get a good basic LDAP service in place without some of these, *but* some are core requirements to the implementation of ADS within the ASF. For example, the replication system will be one such core fundamental requirement. In fact we cannot really proceed a lot further with the current demo/testing environment without it. What state is it currently in? Also before LDAP goes live we will need some good basic documentation, which I can certainly help produce. These will mainly focus on system runbooks, and who/what/where/how to make changes, stop/start services etc. The DRS system is of interest to me. What happens at the moment if the system crashes? Can we copy the flat files, daily, hourly? Would they be re-usable. i.e. Could I copy the data files, and could they be used again if copied over the other files? I am thinking about a backup strategy for the LDAP data. I am happy tp help when and where I can, but I am not a developer. Cheers, Tony On 02/01/2009 10:37, Emmanuel Lecharny wrote: Hi guys, as this is a new year, it's a good timing to define a realistic roadmap for ADS 2.0. We did that last year back in august, trying to define a set of small releases (from 1.5.1 to 1.5.9, targetting a 2.0-RC1), but we now need to get more focused on what is essential. So here is a proposal, for discussion ! I will gather all the items present in the previous roadmap into three different lists : 1) Mandatory : what we absolutely need for a 2.0 2) Good to have : what we want to have in a 2.0 if we have time to do it 3) Not urgent : what we can push in a 2.5 release So here is the list : 1) Mandatory - Replication : we currently have something working (Mitosis), but we have many aspects we want to check before releasing 2.0. So to speak : defining something else than Derby as the pending operation storage, integrate Quartz to manage scheduled operations, inject en entryUUID in each entry. Plus we have to define some test scenarios. - Disaster Recovery System (DSR) : this is absolutely a must have. If the server crash for any reason, we must be able to recover the base. We have the ChangeLog system written, which is a major part of the DSR, but we still need to cover the other parts (how to recover, configuration). - Interface cleaning : we have to check all the common interfaces and clean them. It's not necessarily a costly task, but as we will freeze the API, we have to do that before 2.0-RC1 - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. - VSLDAP : We would like to pass the STANDARD test this year. That should not be too complicated. - JIRA cleaning : we have many open issues, some of them are 3 years old. We have to check all the issues (208 as of today !), and decide which are dead (sometime, an issue is opened, and is not anymore valid a few months after, not because we have fixed it, but because it's not relevant anymore) 2) Good to have - Jetty integration : we need it for two different things at least : DSML gateway and an Http based UI for monitoring the server - Trace/Logging interceptor : should not be too complex, as it will be based on the changeLog interceptor (except that we also want to log every search and bind requests) - ChangeLog extended operations : we need to add some Extended Operation to allow users to drive the Change Log system (like reverting to a specific revision, etc) - Authz Control : this is important if we want to allow the replication system to deal with users right when replicating, or for the DSML gateway - Password Policy : it's a cool thing to have 3) Not urgent - Encrypted user Password : the code is almost ready, but I'm not sure it's urgent - CommandLine tools : well, who uses them ? - SP, Triggers : They are already working, if we want to improve the code (for instance, allowing Groovy to be used), I think it should be done for 2.5 - Group and Role cache : not really urgent. When we will work on TripleSec, then it will be time ! - Attribute tags : we would like to have this, but it's a pretty big change in the server. - AD authentication : same thing, we would like to have it, but it does not worth postponing 2.0 for ever - Schema entities : Important, but in LDAP, currently, nobody uses it ... - Kerberos : I think that we can spawn a new project for kerberos, as soon as ADS 2.0 is out. Kerberos can be a standalone server, based on ADS, at some point. - DHCP, DNS : Does it need to be an ADS project? I was thinking about moving them to MINA... As we already have a FTP server, plus a SSH server in MINA, it seems to make sense. That's pretty much it. Let's start the
Re: Toward 2.0...
Hi Emmanuel, On Fri, Jan 2, 2009 at 5:37 AM, Emmanuel Lecharny elecha...@gmail.comwrote: Hi guys, as this is a new year, it's a good timing to define a realistic roadmap for ADS 2.0. We did that last year back in august, trying to define a set of small releases (from 1.5.1 to 1.5.9, targetting a 2.0-RC1), but we now need to get more focused on what is essential. So here is a proposal, for discussion ! I will gather all the items present in the previous roadmap into three different lists : 1) Mandatory : what we absolutely need for a 2.0 2) Good to have : what we want to have in a 2.0 if we have time to do it 3) Not urgent : what we can push in a 2.5 release Sounds good. So here is the list : 1) Mandatory - Replication : we currently have something working (Mitosis), but we have many aspects we want to check before releasing 2.0. So to speak : defining something else than Derby as the pending operation storage, integrate Quartz to manage scheduled operations, inject en entryUUID in each entry. Plus we have to define some test scenarios. - Disaster Recovery System (DSR) : this is absolutely a must have. If the server crash for any reason, we must be able to recover the base. We have the ChangeLog system written, which is a major part of the DSR, but we still need to cover the other parts (how to recover, configuration). - Interface cleaning : we have to check all the common interfaces and clean them. It's not necessarily a costly task, but as we will freeze the API, we have to do that before 2.0-RC1 - Documentation : This is also an something we must deliver. Documentation is not only good for our users, it's also good for us, as developpers, because writtng documentation helps to see where the API is not consistent. - VSLDAP : We would like to pass the STANDARD test this year. That should not be too complicated. - JIRA cleaning : we have many open issues, some of them are 3 years old. We have to check all the issues (208 as of today !), and decide which are dead (sometime, an issue is opened, and is not anymore valid a few months after, not because we have fixed it, but because it's not relevant anymore) 2) Good to have - Jetty integration : we need it for two different things at least : DSML gateway and an Http based UI for monitoring the server - Trace/Logging interceptor : should not be too complex, as it will be based on the changeLog interceptor (except that we also want to log every search and bind requests) - ChangeLog extended operations : we need to add some Extended Operation to allow users to drive the Change Log system (like reverting to a specific revision, etc) - Authz Control : this is important if we want to allow the replication system to deal with users right when replicating, or for the DSML gateway - Password Policy : it's a cool thing to have 3) Not urgent - Encrypted user Password : the code is almost ready, but I'm not sure it's urgent - CommandLine tools : well, who uses them ? - SP, Triggers : They are already working, if we want to improve the code (for instance, allowing Groovy to be used), I think it should be done for 2.5 - Group and Role cache : not really urgent. When we will work on TripleSec, then it will be time ! - Attribute tags : we would like to have this, but it's a pretty big change in the server. - AD authentication : same thing, we would like to have it, but it does not worth postponing 2.0 for ever - Schema entities : Important, but in LDAP, currently, nobody uses it ... - Kerberos : I think that we can spawn a new project for kerberos, as soon as ADS 2.0 is out. Kerberos can be a standalone server, based on ADS, at some point. - DHCP, DNS : Does it need to be an ADS project? I was thinking about moving them to MINA... As we already have a FTP server, plus a SSH server in MINA, it seems to make sense. That's pretty much it. Let's start the discusiion ! I need to get back up to speed here after my absence. I'm a newbie now. The list is just perfect IMHO. I'm going to try to keep being involved in these discussions while attempting to chew off small tasks at first. I think revamping the UUID system so the UUID operational attribute is always created and maintained for entries regardless of replication being enabled is a good start along with fixing some bugs. I guess I need something creative along with bug fixing. Regards, Alex
