Forgot to mention the link. [1] - https://issues.apache.org/jira/browse/SYNAPSE-563
On Wed, Jul 22, 2009 at 2:09 PM, Hiranya Jayathilaka <[email protected]>wrote: > Hi Folks, > > I just finished a second implementation of this feature which uses a custom > IOEventDispatch and a single IOReactor instance. I have uploaded the patch > at [1]. > > However with this we are no longer able to configure SSL requirements at > endpoint level. We need to configure everything at transport level. Under > the HTTPS sender configuration we can define zero or more SSL profiles and > associate each profile with one or more servers. See the sample > configuration below. > > <transportSender name="https" > class="org.apache.synapse.transport.nhttp.HttpCoreNIOSSLSender"> > <parameter name="non-blocking" locked="false">true</parameter> > <parameter name="keystore" locked="false"> > <KeyStore> > <Location>lib/identity.jks</Location> > <Type>JKS</Type> > <Password>password</Password> > <KeyPassword>password</KeyPassword> > </KeyStore> > </parameter> > <parameter name="truststore" locked="false"> > <TrustStore> > <Location>lib/trust.jks</Location> > <Type>JKS</Type> > <Password>password</Password> > </TrustStore> > </parameter> > > *<parameter name="customSSLProfiles"> > <profile> > <servers>localhost:19002, www.testserver.com:80</servers> > <KeyStore> > <Location>/home/hiranya/cert/B/service.jks</Location> > <Type>JKS</Type> > <Password>123456</Password> > <KeyPassword>123456</KeyPassword> > </KeyStore> > <TrustStore> > <Location>/home/hiranya/cert/B/client.jks</Location> > <Type>JKS</Type> > <Password>123456</Password> > </TrustStore> > </profile> > </parameter>* > </transportSender> > > The above configuration creates one SSL profile and associates it with two > destination servers. The transport sender will lookup the available SSL > contexts and pick the correct one when creating a SSL IO session. > > I believe this implementation achives our goals and gives a fair amount of > control over how certificates are used to connect to different endpoints. > Please provide your feedback. > > Thanks, > Hiranya > > > > On Wed, Jul 22, 2009 at 10:56 AM, Supun Kamburugamuva > <[email protected]>wrote: > >> Hi, >> >> On Wed, Jul 22, 2009 at 8:20 AM, indika kumara <[email protected]>wrote: >> >>> Ruwan , It is not reactors that proportional to the users , but all >>> resources consumes by reactors. Did you see java doc from >>> ‘AbstractMultiworkerIOReactor’. >>> >>> <javadoc> >>> *Usually it is recommended to have one worker I/O reactor per physical >>> CPU core.* >>> <javadoc> >>> >>> I believe because It is few SSL profiles this may be acceptable. But , It >>> is better to get feedback from some one using Synapse in a real production >>> environment. >>> >> >> I think this comment is about server side use of reactors. But here it is >> a client side use. Anyway these are not things that can be decided that >> easily. Better to ask some one in production. >> >> Thanks, >> Supun.. >> >> >>> >>> Indika >>> >>> On Wed, Jul 22, 2009 at 6:12 AM, Ruwan Linton <[email protected]>wrote: >>> >>>> Indika, >>>> >>>> This might not be the approach that we need to do but I just wanted to >>>> get concrete in this discussion and see how we can implement this. I agree >>>> with you, this might not be a scalable solution but on the other hand we >>>> will have only a few SSL profiles, but not many in a given server.. so we >>>> are not talking about hundreds of reactors it will be like 3-5, also it is >>>> not proportional to the users in the system. >>>> >>>> Thanks for the review and I think this is very valuable. Hiranya, I >>>> suggest we wait for few other reviews and do the modifications to the code >>>> so that we gets to the sub optimal solution. You may investigate on the >>>> IOEventDispatcher approach in the mean time as well. >>>> >>>> Thanks, >>>> Ruwan >>>> >>>> >>>> On Tue, Jul 21, 2009 at 11:34 PM, indika kumara >>>> <[email protected]>wrote: >>>> >>>>> I didn’t look details at code level of patch. Patch is an >>>>> implementation of approach that uses MultiworkerIOReactor per SSLContext . >>>>> >>>>> Reactor is the heart of the Reactor pattern. It is all about >>>>> scalability. Generally, multiple reactors use for load balance between >>>>> reactors to match CPU and IO rates. By default, we use a >>>>> MultiworkerIOReactor which is a multi reactor implementation. In above >>>>> patch >>>>> we use collections of MultiworkerIOReactors. I don’t know how much >>>>> resources >>>>> it consumes. A system with n users to be scalable, the quality of physical >>>>> resources required to support them should be at most O(n). Therefore, >>>>> without increasing in loads, if physical resources consumption is >>>>> increased, >>>>> it is not a scalable system. >>>>> >>>>> >>>>> >>>>> BTW, this will be only needed by some set of users. General users may >>>>> not be affected. Therefore, usually there will be a one >>>>> MultiworkerIOReactor. So, this approach may be an option. But, it depends >>>>> on >>>>> actual users that need this feature. Would they like to have many >>>>> MultiworkerIOReactors in a Single Server? If they like, this solution may >>>>> be >>>>> OK. >>>>> >>>>> >>>>> >>>>> Please look at following doc which is from Javadoc of >>>>> ‘AbstractMultiworkerIOReactor’. >>>>> >>>>> >>>>> <javadoc> >>>>> >>>>> *Generic implementation of that can run multiple instances in separate >>>>> worker threads and distribute newly created I/O session equally across >>>>> those >>>>> I/O reactors for more optimal resource utilization and a better I/O >>>>> performance. Usually it is recommended to have one worker I/O reactor per >>>>> physical CPU core.* >>>>> >>>>> <javadoc> >>>>> >>>>> >>>>> >>>>> It is better to *ask from Oleg*?. He has written all these core codes. >>>>> If he thinks this is OK, then I believe, this approach may be OK. >>>>> >>>>> >>>>> >>>>> Indika >>>>> >>>>> >>>>> On Tue, Jul 21, 2009 at 8:56 PM, Ruwan Linton >>>>> <[email protected]>wrote: >>>>> >>>>>> If it is going to require IOEventDispatcher changes and a KeyManger >>>>>> impl that is too complex... >>>>>> >>>>>> Why don't we be concrete and have a look at the patch provided and try >>>>>> to overcome the drawbacks of the implementation (if there are any)? >>>>>> >>>>>> Asankha/Oleg, can you please have a look at the patch? >>>>>> >>>>>> Thanks, >>>>>> Ruwan >>>>>> >>>>>> >>>>>> On Tue, Jul 21, 2009 at 6:03 PM, indika kumara <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> I didn’t investigate the details of any suggested solution. Just said >>>>>>> what comes in to my mind. Possible approaches for complete solution may >>>>>>> be >>>>>>> based on OEventDispatch and Keymanger. >>>>>>> You can go deep and suggest what the easy yet complete solution is. >>>>>>> Then, everyone will agree with that. >>>>>>> >>>>>>> Indika >>>>>>> >>>>>>> On Tue, Jul 21, 2009 at 5:56 PM, Hiranya Jayathilaka < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 21, 2009 at 5:42 PM, indika kumara < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Even when using IOEventDispatch (What I suggested as my first >>>>>>>>> solution), we have to get information from IOSession to pick correct >>>>>>>>> Cert- >>>>>>>>> possibly remote domain name and ip. These things can get from >>>>>>>>> SSLSession too >>>>>>>>> - may be more. >>>>>>>>> >>>>>>>>> BTW, simple approach is best if it solves the problem. >>>>>>>>> >>>>>>>>> Any way, X509ExtendedKeyManager is in com.sun.net.ssl.internal.ssl >>>>>>>>> , it may be an issue to wrap this class. So , IOEventDispatch >>>>>>>>> solution may >>>>>>>>> be only feasible one. >>>>>>>> >>>>>>>> >>>>>>>> The X509ExtendedKeyManager abstract class comes from the >>>>>>>> javax.net.ssl package. It's implementations are JDK specific. But >>>>>>>> technically that shouldn't prevent us from wrapping it. Need to dig >>>>>>>> deeper >>>>>>>> into the API to get a clear idea on this. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Hiranya >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Indika >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jul 21, 2009 at 5:23 PM, Ruwan Linton < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Do we have the required control of figuring out the right identity >>>>>>>>>> certificates over the endpoint in this approach? >>>>>>>>>> >>>>>>>>>> For example can we provide the certificate to be used for a >>>>>>>>>> particular endpoint (may be as an alias) in the endpoint >>>>>>>>>> configuration? I >>>>>>>>>> guess that is the initial requirement. >>>>>>>>>> >>>>>>>>>> Ruwan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 21, 2009 at 4:31 PM, indika kumara < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> +1 for this approach if possible. It seems possible (cannot tell >>>>>>>>>>> exactly). Within SSLIOSessionHandler (httpcore), in the following >>>>>>>>>>> method >>>>>>>>>>> >>>>>>>>>>> *public void initialize(SSLEngine sslengine, HttpParams params) >>>>>>>>>>> , * >>>>>>>>>>> >>>>>>>>>>> we can put application specific thing into SSLSession, then >>>>>>>>>>> access within KeyManager implementation to pick the correct alias . >>>>>>>>>>> >>>>>>>>>>> BTW, the actual implementation of the X509ExtendedKeyManager is >>>>>>>>>>> in *com.sun.net.ssl.internal.ssl*. >>>>>>>>>>> >>>>>>>>>>> Indika >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 21, 2009 at 4:11 PM, Andreas Veithen< >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> > A quick look at the Javadoc shows that X509ExtendedKeyManager >>>>>>>>>>> has >>>>>>>>>>> > access to the SSLEngine from which the SSLSession can be >>>>>>>>>>> retrieved. On >>>>>>>>>>> > the other hand, SSLSession has methods >>>>>>>>>>> getValue/putValue/removeValue >>>>>>>>>>> > to store "application layer data". This could be a proper >>>>>>>>>>> solution. To >>>>>>>>>>> > be investigated. >>>>>>>>>>> > >>>>>>>>>>> > Andreas >>>>>>>>>>> > >>>>>>>>>>> > On Tue, Jul 21, 2009 at 12:32, Paul Fremantle<[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >> I agree. Maybe we could wrap the key manager and then be able >>>>>>>>>>> to >>>>>>>>>>> >> implement any kind of complexity (e..g multiple key stores, >>>>>>>>>>> etc) >>>>>>>>>>> >> behind that. The only question I have is whether you can pass >>>>>>>>>>> context >>>>>>>>>>> >> when you call across the key manager API. We need to pass some >>>>>>>>>>> context >>>>>>>>>>> >> so that the wrapper can do the right thing. Maybe thread local >>>>>>>>>>> context >>>>>>>>>>> >> would allow us to pass some context over that boundary. >>>>>>>>>>> >> >>>>>>>>>>> >> Paul >>>>>>>>>>> >> >>>>>>>>>>> >> On Tue, Jul 21, 2009 at 11:22 AM, Andreas >>>>>>>>>>> >> Veithen<[email protected]> wrote: >>>>>>>>>>> >>> On Tue, Jul 21, 2009 at 12:08, Hiranya Jayathilaka< >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>> Hi Folks, >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> I did some testing using two Axis2 instances each using its >>>>>>>>>>> own >>>>>>>>>>> >>>> certificates. I managed to get Synapse to connect to both >>>>>>>>>>> servers using a >>>>>>>>>>> >>>> single truststore. I had to put both client certs as well as >>>>>>>>>>> the CA cert in >>>>>>>>>>> >>>> the Synapse truststore for this scenario to work. So I guess >>>>>>>>>>> our current >>>>>>>>>>> >>>> implementation is good enough for a great extent. >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> However the problem I have is the way Synapse (or rather the >>>>>>>>>>> Java SSL API) >>>>>>>>>>> >>>> selects the client cert at runtime. As Andreas has mentioned >>>>>>>>>>> it seems to be >>>>>>>>>>> >>>> using some data sent by the server in selecting the correct >>>>>>>>>>> client cert. I >>>>>>>>>>> >>>> tried to find some documentation which properly explains >>>>>>>>>>> this procedure but >>>>>>>>>>> >>>> still couldn't find something satisfactory. However >>>>>>>>>>> according to most >>>>>>>>>>> >>>> resources and explanation provided by Oleg, it seems that >>>>>>>>>>> client can decide >>>>>>>>>>> >>>> which cert to present the way it wishes. In that case the >>>>>>>>>>> above scenario >>>>>>>>>>> >>>> might fail in certain situations. >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> So IMHO the best option (most scalable and robust) we got is >>>>>>>>>>> to implement a >>>>>>>>>>> >>>> custom IOEventDispatch. WDYT? >>>>>>>>>>> >>> >>>>>>>>>>> >>> Personally, I think that the HTTP(S) transport is already >>>>>>>>>>> complex >>>>>>>>>>> >>> enough and that we should only add to that complexity if >>>>>>>>>>> there is a >>>>>>>>>>> >>> good reason to do so. Assuming that the client certificate >>>>>>>>>>> selection >>>>>>>>>>> >>> algorithm doesn't give the expected results, I think that >>>>>>>>>>> implementing >>>>>>>>>>> >>> a custom IOEventDispatch is only the second best solution. We >>>>>>>>>>> should >>>>>>>>>>> >>> first investigate the possibility to change the behavior of >>>>>>>>>>> the key >>>>>>>>>>> >>> manager. Since this is the component that selects the client >>>>>>>>>>> >>> certificate, from a design perspective it would be the right >>>>>>>>>>> place do >>>>>>>>>>> >>> implement custom behavior. This approach would also have the >>>>>>>>>>> advantage >>>>>>>>>>> >>> of isolating the changes from the core of the transport. >>>>>>>>>>> >>> >>>>>>>>>>> >>> Andreas >>>>>>>>>>> >>> >>>>>>>>>>> >>>> Thanks, >>>>>>>>>>> >>>> Hiranya >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> On Tue, Jul 21, 2009 at 3:25 PM, Andreas Veithen < >>>>>>>>>>> [email protected]> >>>>>>>>>>> >>>> wrote: >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> On Tue, Jul 21, 2009 at 11:30, Oleg Kalnichevski< >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>> > On Tue, Jul 21, 2009 at 11:22:58AM +0200, Andreas Veithen >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>> >> > Well, if not through different stores, how can we let >>>>>>>>>>> the KeyManager >>>>>>>>>>> >>>>> >> > know >>>>>>>>>>> >>>>> >> > what cert to use for this particular endpoint? >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> If I remember well, this is how it works: during the >>>>>>>>>>> handshake, the >>>>>>>>>>> >>>>> >> server presents a list of trusted CAs to the client. The >>>>>>>>>>> client than >>>>>>>>>>> >>>>> >> selects the certificate that is signed (directly or >>>>>>>>>>> indirectly) by >>>>>>>>>>> >>>>> >> that CA and uses that to authenticate. I'm pretty sure >>>>>>>>>>> this is what >>>>>>>>>>> >>>>> >> happens when you create a java.net.URL with the https >>>>>>>>>>> scheme and call >>>>>>>>>>> >>>>> >> openConnection on it. Since behind the scene this uses >>>>>>>>>>> an SSLContext, >>>>>>>>>>> >>>>> >> chances are high that it also works with our HTTPS >>>>>>>>>>> transport (or that >>>>>>>>>>> >>>>> >> it would be pretty easy to make it work). >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> Of course this only satisfies the requirement if the two >>>>>>>>>>> endpoints use >>>>>>>>>>> >>>>> >> different CAs, which should be the usual case. >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> >> Andreas >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > Hi Andreas >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > I may be wrong about it but I believe the client can >>>>>>>>>>> present whatever >>>>>>>>>>> >>>>> > client >>>>>>>>>>> >>>>> > cert it pleases. That cert does not _have_ to be signed >>>>>>>>>>> by one of the >>>>>>>>>>> >>>>> > trusted >>>>>>>>>>> >>>>> > CA certs sent to client by the server. For instance, >>>>>>>>>>> common browsers >>>>>>>>>>> >>>>> > simply pop >>>>>>>>>>> >>>>> > up a UI dialog and let you pick any client certificate >>>>>>>>>>> available in the >>>>>>>>>>> >>>>> > certificate store, if the server requests client >>>>>>>>>>> authentication in the >>>>>>>>>>> >>>>> > course >>>>>>>>>>> >>>>> > of SSL context negotiation. >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > Oleg >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> That is possible, but it is only relevant for a scheme >>>>>>>>>>> where the >>>>>>>>>>> >>>>> consumer of the service creates a certificate himself >>>>>>>>>>> (typically a >>>>>>>>>>> >>>>> self-signed certificate) and somehow registers that with >>>>>>>>>>> the provider >>>>>>>>>>> >>>>> of the service. This implies that the provider has to >>>>>>>>>>> manage a list of >>>>>>>>>>> >>>>> recognized client certificates to authenticate the client. >>>>>>>>>>> I don't >>>>>>>>>>> >>>>> think that is a usual scheme for Web services (BTW, how >>>>>>>>>>> would you do >>>>>>>>>>> >>>>> that with Axis2?), but that it is more usual for the >>>>>>>>>>> provider to issue >>>>>>>>>>> >>>>> certificates to the consumer, so that authentication is >>>>>>>>>>> based on the >>>>>>>>>>> >>>>> signature on the client certificate. But again, this is a >>>>>>>>>>> question >>>>>>>>>>> >>>>> about the requirements. >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Andreas >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> >> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> >>>>> >> To unsubscribe, e-mail: >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>> >> For additional commands, e-mail: >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>> >> >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> >>>>> > To unsubscribe, e-mail: >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>> > For additional commands, e-mail: >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> > >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> >>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>> >>>>> For additional commands, e-mail: >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> -- >>>>>>>>>>> >>>> Hiranya Jayathilaka >>>>>>>>>>> >>>> Software Engineer; >>>>>>>>>>> >>>> WSO2 Inc.; http://wso2.org >>>>>>>>>>> >>>> E-mail: [email protected]; Mobile: +94 77 633 3491 >>>>>>>>>>> >>>> Blog: http://techfeast-hiranya.blogspot.com >>>>>>>>>>> >>>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> >>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>> >>> For additional commands, e-mail: [email protected] >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> -- >>>>>>>>>>> >> Paul Fremantle >>>>>>>>>>> >> Co-Founder and CTO, WSO2 >>>>>>>>>>> >> Apache Synapse PMC Chair >>>>>>>>>>> >> OASIS WS-RX TC Co-chair >>>>>>>>>>> >> >>>>>>>>>>> >> blog: http://pzf.fremantle.org >>>>>>>>>>> >> [email protected] >>>>>>>>>>> >> >>>>>>>>>>> >> "Oxygenating the Web Service Platform", www.wso2.com >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> >> To unsubscribe, e-mail: [email protected] >>>>>>>>>>> >> For additional commands, e-mail: [email protected] >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> > To unsubscribe, e-mail: [email protected] >>>>>>>>>>> > For additional commands, e-mail: [email protected] >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Ruwan Linton >>>>>>>>>> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb >>>>>>>>>> WSO2 Inc.; http://wso2.org >>>>>>>>>> email: [email protected]; cell: +94 77 341 3097 >>>>>>>>>> blog: http://ruwansblog.blogspot.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Hiranya Jayathilaka >>>>>>>> Software Engineer; >>>>>>>> WSO2 Inc.; http://wso2.org >>>>>>>> E-mail: [email protected]; Mobile: +94 77 633 3491 >>>>>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ruwan Linton >>>>>> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb >>>>>> WSO2 Inc.; http://wso2.org >>>>>> email: [email protected]; cell: +94 77 341 3097 >>>>>> blog: http://ruwansblog.blogspot.com >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Ruwan Linton >>>> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb >>>> WSO2 Inc.; http://wso2.org >>>> email: [email protected]; cell: +94 77 341 3097 >>>> blog: http://ruwansblog.blogspot.com >>>> >>> >>> >> >> >> -- >> Software Engineer, WSO2 Inc >> http://wso2.org >> supunk.blogspot.com >> >> >> > > > -- > Hiranya Jayathilaka > Software Engineer; > WSO2 Inc.; http://wso2.org > E-mail: [email protected]; Mobile: +94 77 633 3491 > Blog: http://techfeast-hiranya.blogspot.com > -- Hiranya Jayathilaka Software Engineer; WSO2 Inc.; http://wso2.org E-mail: [email protected]; Mobile: +94 77 633 3491 Blog: http://techfeast-hiranya.blogspot.com
