Hi Folks,

I just finished a second implementation of this feature which uses a custom
IOEventDispatch and a single IOReactor instance. I have uploaded the patch
at [1].

However with this we are no longer able to configure SSL requirements at
endpoint level. We need to configure everything at transport level. Under
the HTTPS sender configuration we can define zero or more SSL profiles and
associate each profile with one or more servers. See the sample
configuration below.

<transportSender name="https"
class="org.apache.synapse.transport.nhttp.HttpCoreNIOSSLSender">
        <parameter name="non-blocking" locked="false">true</parameter>
        <parameter name="keystore" locked="false">
            <KeyStore>
                <Location>lib/identity.jks</Location>
                <Type>JKS</Type>
                <Password>password</Password>
                <KeyPassword>password</KeyPassword>
            </KeyStore>
        </parameter>
        <parameter name="truststore" locked="false">
            <TrustStore>
                <Location>lib/trust.jks</Location>
                <Type>JKS</Type>
                <Password>password</Password>
            </TrustStore>
        </parameter>
    *<parameter name="customSSLProfiles">
        <profile>
            <servers>localhost:19002, www.testserver.com:80</servers>
            <KeyStore>
                <Location>/home/hiranya/cert/B/service.jks</Location>
                <Type>JKS</Type>
                <Password>123456</Password>
                <KeyPassword>123456</KeyPassword>
            </KeyStore>
            <TrustStore>
                <Location>/home/hiranya/cert/B/client.jks</Location>
                <Type>JKS</Type>
                <Password>123456</Password>
            </TrustStore>
        </profile>
    </parameter>*
</transportSender>

The above configuration creates one SSL profile and associates it with two
destination servers. The transport sender will lookup the available SSL
contexts and pick the correct one when creating a SSL IO session.

I believe this implementation achives our goals and gives a fair amount of
control over how certificates are used to connect to different endpoints.
Please provide your feedback.

Thanks,
Hiranya


On Wed, Jul 22, 2009 at 10:56 AM, Supun Kamburugamuva <[email protected]>wrote:

> Hi,
>
> On Wed, Jul 22, 2009 at 8:20 AM, indika kumara <[email protected]>wrote:
>
>> Ruwan , It is not reactors that proportional to the users , but all
>> resources consumes by reactors.  Did you see java doc from
>> ‘AbstractMultiworkerIOReactor’.
>>
>> <javadoc>
>> *Usually it is recommended to have one worker I/O reactor per physical
>> CPU core.*
>> <javadoc>
>>
>> I believe because It is few SSL profiles this may be acceptable. But , It
>> is better to get feedback from some one using Synapse in a real production
>> environment.
>>
>
> I think this comment is about server side use of reactors. But here it is a
> client side use. Anyway these are not things that can be decided that
> easily. Better to ask some one in production.
>
> Thanks,
> Supun..
>
>
>>
>> Indika
>>
>> On Wed, Jul 22, 2009 at 6:12 AM, Ruwan Linton <[email protected]>wrote:
>>
>>> Indika,
>>>
>>> This might not be the approach that we need to do but I just wanted to
>>> get concrete in this discussion and see how we can implement this. I agree
>>> with you, this might not be a scalable solution but on the other hand we
>>> will have only a few SSL profiles, but not many in a given server.. so we
>>> are not talking about hundreds of reactors it will be like 3-5, also it is
>>> not proportional to the users in the system.
>>>
>>> Thanks for the review and I think this is very valuable. Hiranya, I
>>> suggest we wait for few other reviews and do the modifications to the code
>>> so that we gets to the sub optimal solution. You may investigate on the
>>> IOEventDispatcher approach in the mean time as well.
>>>
>>> Thanks,
>>> Ruwan
>>>
>>>
>>> On Tue, Jul 21, 2009 at 11:34 PM, indika kumara 
>>> <[email protected]>wrote:
>>>
>>>> I didn’t look details at code level of patch. Patch is an implementation
>>>> of approach that uses MultiworkerIOReactor per SSLContext .
>>>>
>>>> Reactor is the heart of the Reactor pattern. It is all about
>>>> scalability. Generally, multiple reactors use for load balance between
>>>> reactors to match CPU and IO rates. By default, we use a
>>>> MultiworkerIOReactor which is a multi reactor implementation. In above 
>>>> patch
>>>> we use collections of MultiworkerIOReactors. I don’t know how much 
>>>> resources
>>>> it consumes. A system with n users to be scalable, the quality of physical
>>>> resources required to support them should be at most O(n). Therefore,
>>>> without increasing in loads, if physical resources consumption is 
>>>> increased,
>>>> it is not a scalable system.
>>>>
>>>>
>>>>
>>>> BTW, this will be only needed by some set of users. General users may
>>>> not be affected. Therefore, usually there will be a one
>>>> MultiworkerIOReactor. So, this approach may be an option. But, it depends 
>>>> on
>>>> actual users that need this feature. Would they like to have many
>>>> MultiworkerIOReactors in a Single Server? If they like, this solution may 
>>>> be
>>>> OK.
>>>>
>>>>
>>>>
>>>> Please look at following doc which is from Javadoc of
>>>> ‘AbstractMultiworkerIOReactor’.
>>>>
>>>>
>>>> <javadoc>
>>>>
>>>> *Generic implementation of that can run multiple instances in separate
>>>> worker threads and distribute newly created I/O session equally across 
>>>> those
>>>> I/O reactors for more optimal resource utilization and a better I/O
>>>> performance. Usually it is recommended to have one worker I/O reactor per
>>>> physical CPU core.*
>>>>
>>>> <javadoc>
>>>>
>>>>
>>>>
>>>> It is better to *ask from Oleg*?. He has written all these core codes.
>>>> If he thinks this is OK, then I believe, this approach may be OK.
>>>>
>>>>
>>>>
>>>> Indika
>>>>
>>>>
>>>> On Tue, Jul 21, 2009 at 8:56 PM, Ruwan Linton 
>>>> <[email protected]>wrote:
>>>>
>>>>> If it is going to require IOEventDispatcher changes and a KeyManger
>>>>> impl that is too complex...
>>>>>
>>>>> Why don't we be concrete and have a look at the patch provided and try
>>>>> to overcome the drawbacks of the implementation (if there are any)?
>>>>>
>>>>> Asankha/Oleg, can you please have a look at the patch?
>>>>>
>>>>> Thanks,
>>>>> Ruwan
>>>>>
>>>>>
>>>>> On Tue, Jul 21, 2009 at 6:03 PM, indika kumara 
>>>>> <[email protected]>wrote:
>>>>>
>>>>>> I didn’t investigate the details of any suggested solution. Just said
>>>>>> what comes in to my mind. Possible approaches for complete solution may 
>>>>>> be
>>>>>> based on OEventDispatch and Keymanger.
>>>>>> You can go deep and suggest what the easy yet complete solution is.
>>>>>> Then, everyone will agree with that.
>>>>>>
>>>>>> Indika
>>>>>>
>>>>>> On Tue, Jul 21, 2009 at 5:56 PM, Hiranya Jayathilaka <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 21, 2009 at 5:42 PM, indika kumara <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Even when using IOEventDispatch (What I suggested as my first
>>>>>>>> solution), we have to get information from IOSession to pick correct 
>>>>>>>> Cert-
>>>>>>>> possibly remote domain name and ip. These things can get from 
>>>>>>>> SSLSession too
>>>>>>>> - may be more.
>>>>>>>>
>>>>>>>> BTW, simple approach is best if it solves the problem.
>>>>>>>>
>>>>>>>> Any way, X509ExtendedKeyManager is in com.sun.net.ssl.internal.ssl ,
>>>>>>>> it may be an issue to wrap this class. So , IOEventDispatch solution 
>>>>>>>> may be
>>>>>>>> only feasible one.
>>>>>>>
>>>>>>>
>>>>>>> The X509ExtendedKeyManager abstract class comes from the
>>>>>>> javax.net.ssl package. It's implementations are JDK specific. But
>>>>>>> technically that shouldn't prevent us from wrapping it. Need to dig 
>>>>>>> deeper
>>>>>>> into the API to get a clear idea on this.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hiranya
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Indika
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jul 21, 2009 at 5:23 PM, Ruwan Linton <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Do we have the required control of figuring out the right identity
>>>>>>>>> certificates over the endpoint in this approach?
>>>>>>>>>
>>>>>>>>> For example can we provide the certificate to be used for a
>>>>>>>>> particular endpoint (may be as an alias) in the endpoint 
>>>>>>>>> configuration? I
>>>>>>>>> guess that is the initial requirement.
>>>>>>>>>
>>>>>>>>> Ruwan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jul 21, 2009 at 4:31 PM, indika kumara <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> +1 for this approach if possible. It seems possible (cannot tell
>>>>>>>>>> exactly). Within SSLIOSessionHandler (httpcore), in the following 
>>>>>>>>>> method
>>>>>>>>>>
>>>>>>>>>> *public void initialize(SSLEngine sslengine, HttpParams params) ,
>>>>>>>>>> *
>>>>>>>>>>
>>>>>>>>>> we can put application specific thing into SSLSession, then access
>>>>>>>>>> within KeyManager implementation to pick the correct alias .
>>>>>>>>>>
>>>>>>>>>> BTW, the actual implementation of the X509ExtendedKeyManager is in
>>>>>>>>>> *com.sun.net.ssl.internal.ssl*.
>>>>>>>>>>
>>>>>>>>>> Indika
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 21, 2009 at 4:11 PM, Andreas Veithen<
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>> > A quick look at the Javadoc shows that X509ExtendedKeyManager
>>>>>>>>>> has
>>>>>>>>>> > access to the SSLEngine from which the SSLSession can be
>>>>>>>>>> retrieved. On
>>>>>>>>>> > the other hand, SSLSession has methods
>>>>>>>>>> getValue/putValue/removeValue
>>>>>>>>>> > to store "application layer data". This could be a proper
>>>>>>>>>> solution. To
>>>>>>>>>> > be investigated.
>>>>>>>>>> >
>>>>>>>>>> > Andreas
>>>>>>>>>> >
>>>>>>>>>> > On Tue, Jul 21, 2009 at 12:32, Paul Fremantle<[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>> >> I agree. Maybe we could wrap the key manager and then be able
>>>>>>>>>> to
>>>>>>>>>> >> implement any kind of complexity (e..g multiple key stores,
>>>>>>>>>> etc)
>>>>>>>>>> >> behind that. The only question I have is whether you can pass
>>>>>>>>>> context
>>>>>>>>>> >> when you call across the key manager API. We need to pass some
>>>>>>>>>> context
>>>>>>>>>> >> so that the wrapper can do the right thing. Maybe thread local
>>>>>>>>>> context
>>>>>>>>>> >> would allow us to pass some context over that boundary.
>>>>>>>>>> >>
>>>>>>>>>> >> Paul
>>>>>>>>>> >>
>>>>>>>>>> >> On Tue, Jul 21, 2009 at 11:22 AM, Andreas
>>>>>>>>>> >> Veithen<[email protected]> wrote:
>>>>>>>>>> >>> On Tue, Jul 21, 2009 at 12:08, Hiranya Jayathilaka<
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>> >>>> Hi Folks,
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> I did some testing using two Axis2 instances each using its
>>>>>>>>>> own
>>>>>>>>>> >>>> certificates. I managed to get Synapse to connect to both
>>>>>>>>>> servers using a
>>>>>>>>>> >>>> single truststore. I had to put both client certs as well as
>>>>>>>>>> the CA cert in
>>>>>>>>>> >>>> the Synapse truststore for this scenario to work. So I guess
>>>>>>>>>> our current
>>>>>>>>>> >>>> implementation is good enough for a great extent.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> However the problem I have is the way Synapse (or rather the
>>>>>>>>>> Java SSL API)
>>>>>>>>>> >>>> selects the client cert at runtime. As Andreas has mentioned
>>>>>>>>>> it seems to be
>>>>>>>>>> >>>> using some data sent by the server in selecting the correct
>>>>>>>>>> client cert. I
>>>>>>>>>> >>>> tried to find some documentation which properly explains this
>>>>>>>>>> procedure but
>>>>>>>>>> >>>> still couldn't find something satisfactory. However according
>>>>>>>>>> to most
>>>>>>>>>> >>>> resources and explanation provided by Oleg, it seems that
>>>>>>>>>> client can decide
>>>>>>>>>> >>>> which cert to present the way it wishes. In that case the
>>>>>>>>>> above scenario
>>>>>>>>>> >>>> might fail in certain situations.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> So IMHO the best option (most scalable and robust) we got is
>>>>>>>>>> to implement a
>>>>>>>>>> >>>> custom IOEventDispatch. WDYT?
>>>>>>>>>> >>>
>>>>>>>>>> >>> Personally, I think that the HTTP(S) transport is already
>>>>>>>>>> complex
>>>>>>>>>> >>> enough and that we should only add to that complexity if there
>>>>>>>>>> is a
>>>>>>>>>> >>> good reason to do so. Assuming that the client certificate
>>>>>>>>>> selection
>>>>>>>>>> >>> algorithm doesn't give the expected results, I think that
>>>>>>>>>> implementing
>>>>>>>>>> >>> a custom IOEventDispatch is only the second best solution. We
>>>>>>>>>> should
>>>>>>>>>> >>> first investigate the possibility to change the behavior of
>>>>>>>>>> the key
>>>>>>>>>> >>> manager. Since this is the component that selects the client
>>>>>>>>>> >>> certificate, from a design perspective it would be the right
>>>>>>>>>> place do
>>>>>>>>>> >>> implement custom behavior. This approach would also have the
>>>>>>>>>> advantage
>>>>>>>>>> >>> of isolating the changes from the core of the transport.
>>>>>>>>>> >>>
>>>>>>>>>> >>> Andreas
>>>>>>>>>> >>>
>>>>>>>>>> >>>> Thanks,
>>>>>>>>>> >>>> Hiranya
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Tue, Jul 21, 2009 at 3:25 PM, Andreas Veithen <
>>>>>>>>>> [email protected]>
>>>>>>>>>> >>>> wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> On Tue, Jul 21, 2009 at 11:30, Oleg Kalnichevski<
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>> >>>>> > On Tue, Jul 21, 2009 at 11:22:58AM +0200, Andreas Veithen
>>>>>>>>>> wrote:
>>>>>>>>>> >>>>> >> > Well, if not through different stores, how can we let
>>>>>>>>>> the KeyManager
>>>>>>>>>> >>>>> >> > know
>>>>>>>>>> >>>>> >> > what cert to use for this particular endpoint?
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> If I remember well, this is how it works: during the
>>>>>>>>>> handshake, the
>>>>>>>>>> >>>>> >> server presents a list of trusted CAs to the client. The
>>>>>>>>>> client than
>>>>>>>>>> >>>>> >> selects the certificate that is signed (directly or
>>>>>>>>>> indirectly) by
>>>>>>>>>> >>>>> >> that CA and uses that to authenticate. I'm pretty sure
>>>>>>>>>> this is what
>>>>>>>>>> >>>>> >> happens when you create a java.net.URL with the https
>>>>>>>>>> scheme and call
>>>>>>>>>> >>>>> >> openConnection on it. Since behind the scene this uses an
>>>>>>>>>> SSLContext,
>>>>>>>>>> >>>>> >> chances are high that it also works with our HTTPS
>>>>>>>>>> transport (or that
>>>>>>>>>> >>>>> >> it would be pretty easy to make it work).
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> Of course this only satisfies the requirement if the two
>>>>>>>>>> endpoints use
>>>>>>>>>> >>>>> >> different CAs, which should be the usual case.
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >> Andreas
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> > Hi Andreas
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> > I may be wrong about it but I believe the client can
>>>>>>>>>> present whatever
>>>>>>>>>> >>>>> > client
>>>>>>>>>> >>>>> > cert it pleases. That cert does not _have_ to be signed by
>>>>>>>>>> one of the
>>>>>>>>>> >>>>> > trusted
>>>>>>>>>> >>>>> > CA certs sent to client by the server. For instance,
>>>>>>>>>> common browsers
>>>>>>>>>> >>>>> > simply pop
>>>>>>>>>> >>>>> > up a UI dialog and let you pick any client certificate
>>>>>>>>>> available in the
>>>>>>>>>> >>>>> > certificate store, if the server requests client
>>>>>>>>>> authentication in the
>>>>>>>>>> >>>>> > course
>>>>>>>>>> >>>>> > of SSL context negotiation.
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> > Oleg
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> That is possible, but it is only relevant for a scheme where
>>>>>>>>>> the
>>>>>>>>>> >>>>> consumer of the service creates a certificate himself
>>>>>>>>>> (typically a
>>>>>>>>>> >>>>> self-signed certificate) and somehow registers that with the
>>>>>>>>>> provider
>>>>>>>>>> >>>>> of the service. This implies that the provider has to manage
>>>>>>>>>> a list of
>>>>>>>>>> >>>>> recognized client certificates to authenticate the client. I
>>>>>>>>>> don't
>>>>>>>>>> >>>>> think that is a usual scheme for Web services (BTW, how
>>>>>>>>>> would you do
>>>>>>>>>> >>>>> that with Axis2?), but that it is more usual for the
>>>>>>>>>> provider to issue
>>>>>>>>>> >>>>> certificates to the consumer, so that authentication is
>>>>>>>>>> based on the
>>>>>>>>>> >>>>> signature on the client certificate. But again, this is a
>>>>>>>>>> question
>>>>>>>>>> >>>>> about the requirements.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Andreas
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> >>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> >>>>> >> To unsubscribe, e-mail:
>>>>>>>>>> [email protected]
>>>>>>>>>> >>>>> >> For additional commands, e-mail:
>>>>>>>>>> [email protected]
>>>>>>>>>> >>>>> >>
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> >
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> >>>>> > To unsubscribe, e-mail:
>>>>>>>>>> [email protected]
>>>>>>>>>> >>>>> > For additional commands, e-mail:
>>>>>>>>>> [email protected]
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>> >
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> >>>>> To unsubscribe, e-mail: [email protected]
>>>>>>>>>> >>>>> For additional commands, e-mail:
>>>>>>>>>> [email protected]
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> --
>>>>>>>>>> >>>> Hiranya Jayathilaka
>>>>>>>>>> >>>> Software Engineer;
>>>>>>>>>> >>>> WSO2 Inc.;  http://wso2.org
>>>>>>>>>> >>>> E-mail: [email protected];  Mobile: +94 77 633 3491
>>>>>>>>>> >>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>>>> >>>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> >>> To unsubscribe, e-mail: [email protected]
>>>>>>>>>> >>> For additional commands, e-mail: [email protected]
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> Paul Fremantle
>>>>>>>>>> >> Co-Founder and CTO, WSO2
>>>>>>>>>> >> Apache Synapse PMC Chair
>>>>>>>>>> >> OASIS WS-RX TC Co-chair
>>>>>>>>>> >>
>>>>>>>>>> >> blog: http://pzf.fremantle.org
>>>>>>>>>> >> [email protected]
>>>>>>>>>> >>
>>>>>>>>>> >> "Oxygenating the Web Service Platform", www.wso2.com
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> >> To unsubscribe, e-mail: [email protected]
>>>>>>>>>> >> For additional commands, e-mail: [email protected]
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> > To unsubscribe, e-mail: [email protected]
>>>>>>>>>> > For additional commands, e-mail: [email protected]
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ruwan Linton
>>>>>>>>> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
>>>>>>>>> WSO2 Inc.; http://wso2.org
>>>>>>>>> email: [email protected]; cell: +94 77 341 3097
>>>>>>>>> blog: http://ruwansblog.blogspot.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hiranya Jayathilaka
>>>>>>> Software Engineer;
>>>>>>> WSO2 Inc.;  http://wso2.org
>>>>>>> E-mail: [email protected];  Mobile: +94 77 633 3491
>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ruwan Linton
>>>>> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
>>>>> WSO2 Inc.; http://wso2.org
>>>>> email: [email protected]; cell: +94 77 341 3097
>>>>> blog: http://ruwansblog.blogspot.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Ruwan Linton
>>> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
>>> WSO2 Inc.; http://wso2.org
>>> email: [email protected]; cell: +94 77 341 3097
>>> blog: http://ruwansblog.blogspot.com
>>>
>>
>>
>
>
> --
> Software Engineer, WSO2 Inc
> http://wso2.org
> supunk.blogspot.com
>
>
>


-- 
Hiranya Jayathilaka
Software Engineer;
WSO2 Inc.;  http://wso2.org
E-mail: [email protected];  Mobile: +94 77 633 3491
Blog: http://techfeast-hiranya.blogspot.com

Reply via email to