On Fri, Feb 26, 2010 at 12:03 AM, Senaka Fernando <[email protected]> wrote:
> I wonder whether the changes made by Sumedha went into alpha2. Anyway, the
> results don't seem very informative to understand what actually happened.
> Can you do the same tests on H2 and paste any issues you see.
>
AFAIK, H2 doesn't support for higher concurrency. Hence I run with minimal
load. It was 100 iterations with 10 concurrent users.
[2010-02-26 00:31:35,494] ERROR - Failed to add resource to path
/test/thread_T29/resourceZ9. Deadlock detected. The current transaction was
rolled back. Details:
Session #312 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_CONTENT while
locking PUBLIC.REG_CONTENT_HISTORY (exclusive), PUBLIC.REG_RESOURCE_HISTORY
(exclusive), PUBLIC.REG_RESOURCE (exclusive).
Session #309 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_RESOURCE while
locking PUBLIC.REG_CONTENT (exclusive).; SQL statement:
INSERT INTO REG_RESOURCE (REG_PATH_ID, REG_NAME, REG_MEDIA_TYPE,
REG_CREATOR, REG_CREATED_TIME, REG_LAST_UPDATOR, REG_LAST_UPDATED_TIME,
REG_DESCRIPTION, REG_CONTENT_ID, REG_TENANT_ID) VALUES (?, ?, ?, ?, ?, ?, ?,
?, ?, ?) [40001-112]
org.h2.jdbc.JdbcSQLException: Deadlock detected. The current transaction was
rolled back. Details:
Session #312 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_CONTENT while
locking PUBLIC.REG_CONTENT_HISTORY (exclusive), PUBLIC.REG_RESOURCE_HISTORY
(exclusive), PUBLIC.REG_RESOURCE (exclusive).
Session #309 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_RESOURCE while
locking PUBLIC.REG_CONTENT (exclusive).; SQL statement:
INSERT INTO REG_RESOURCE (REG_PATH_ID, REG_NAME, REG_MEDIA_TYPE,
REG_CREATOR, REG_CREATED_TIME, REG_LAST_UPDATOR, REG_LAST_UPDATED_TIME,
REG_DESCRIPTION, REG_CONTENT_ID, REG_TENANT_ID) VALUES (?, ?, ?, ?, ?, ?, ?,
?, ?, ?) [40001-112]
at org.h2.message.Message.getSQLException(Message.java:107)
at org.h2.message.Message.getSQLException(Message.java:118)
at org.h2.message.Message.getSQLException(Message.java:77)
at org.h2.table.TableData.doLock(TableData.java:428)
at org.h2.table.TableData.lock(TableData.java:375)
at org.h2.command.dml.Insert.update(Insert.java:99)
at org.h2.command.CommandContainer.update(CommandContainer.java:71)
at org.h2.command.Command.executeUpdate(Command.java:207)
at
org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:139)
at
org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:128)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
at
org.wso2.carbon.registry.core.jdbc.dao.ResourceDAO.addResource(ResourceDAO.java:1267)
at
org.wso2.carbon.registry.core.jdbc.dao.ResourceDAO.add(ResourceDAO.java:573)
at
org.wso2.carbon.registry.core.jdbc.Repository.add(Repository.java:933)
at
org.wso2.carbon.registry.core.jdbc.Repository.put(Repository.java:287)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.put(EmbeddedRegistry.java:711)
at
org.wso2.carbon.registry.core.session.UserRegistry.put(UserRegistry.java:599)
at
org.wso2.carbon.registry.app.RegistryAdapter.postEntry(RegistryAdapter.java:1184)
at
org.wso2.carbon.registry.app.RegistryAdapter.postEntry(RegistryAdapter.java:52)
at
org.apache.abdera.protocol.server.impl.AbstractEntityCollectionAdapter.createNonMediaEntry(AbstractEntityCollectionAdapter.java:665)
at
org.apache.abdera.protocol.server.impl.AbstractEntityCollectionAdapter.postEntry(AbstractEntityCollectionAdapter.java:122)
at
org.wso2.carbon.registry.app.RegistryAdapter.postEntry(RegistryAdapter.java:1028)
at
org.apache.abdera.protocol.server.processors.CollectionRequestProcessor.processCollection(CollectionRequestProcessor.java:47)
at
org.apache.abdera.protocol.server.processors.CollectionRequestProcessor.process(CollectionRequestProcessor.java:38)
at
org.apache.abdera.protocol.server.impl.AbstractProvider.process(AbstractProvider.java:136)
at
org.apache.abdera.protocol.server.FilterChain.next(FilterChain.java:42)
at
org.apache.abdera.protocol.server.servlet.AbderaServlet.service(AbderaServlet.java:90)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.wso2.carbon.bridge.BridgeServlet.service(BridgeServlet.java:135)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:619)
[2010-02-26 00:31:49,743] ERROR - Failed to add resource to path /tes
Thanks,
Krishantha.
[1] http://pastebin.com/Ji6zTqMP
>
> Thanks,
> Senaka.
>
>
> On Thu, Feb 25, 2010 at 11:16 PM, Krishantha Samaraweera <
> [email protected]> wrote:
>
>> Since it will take time to setup perf tests with embedded registry,
>> APP based perf tests have been executed with alpha2 build using
>> Oracle. Following dead lock situation occurred just after starting the
>> worker1 class with 500 iterations and 20 concurrent users. Please
>> refer to [1] detail error log.
>>
>> [2010-02-25 22:39:25,171] ERROR - A SQLException error has occurred
>> when trying to close result set or prepared statement
>> java.sql.SQLException: ORA-00060: deadlock detected while waiting for
>> resource
>>
>> at
>> oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
>> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
>> at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
>> at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
>> at
>> oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
>> at
>> oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
>> at
>> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
>> at
>> oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285)
>> at
>> oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3368)
>> at
>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.ResourceDAO.deleteContentStream(ResourceDAO.java:658)
>> at
>> org.wso2.carbon.registry.core.jdbc.dao.ResourceVersionDAO.versionResource(ResourceVersionDAO.java:1052)
>> at
>> org.wso2.carbon.registry.core.jdbc.VersionRepository.createVersionForResource(VersionRepository.java:191)
>> at
>> org.wso2.carbon.registry.core.jdbc.VersionRepository.createSnapshot(VersionRepository.java:77)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.deleteSubTree(Repository.java:471)
>> at
>> org.wso2.carbon.registry.core.jdbc.Repository.delete(Repository.java:403)
>> at
>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.delete(EmbeddedRegistry.java:805)
>> at
>> org.wso2.carbon.registry.core.session.UserRegistry.delete(UserRegistry.java:625)
>> at
>> org.wso2.carbon.registry.app.RegistryAdapter.processDeleteRequest(RegistryAdapter.java:602)
>> at
>> org.wso2.carbon.registry.app.RegistryAdapter.extensionRequest(RegistryAdapter.java:124)
>> at
>> org.apache.abdera.protocol.server.impl.AbstractProvider.processExtensionRequest(AbstractProvider.java:196)
>> at
>> org.apache.abdera.protocol.server.impl.AbstractProvider.process(AbstractProvider.java:137)
>> at
>> org.apache.abdera.protocol.server.FilterChain.next(FilterChain.java:42)
>> at
>> org.apache.abdera.protocol.server.servlet.AbderaServlet.service(AbderaServlet.java:90)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>> at
>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
>> at
>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
>> at
>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>> at
>> org.wso2.carbon.bridge.BridgeServlet.service(BridgeServlet.java:135)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
>> at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>> at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>> at
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>> at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> Thanks,
>> Krishantha.
>>
>> [1] http://pastebin.com/Wi5RADvm
>>
>>
>> On Wed, Feb 24, 2010 at 1:08 PM, Krishantha Samaraweera
>> <[email protected]> wrote:
>> > On Wed, Feb 24, 2010 at 12:18 PM, Dimuthu Gamage <[email protected]>
>> wrote:
>> >> +1 for running perf tests. We can't rely on the atom based remote
>> registry
>> >> based perf tests that we used to run earlier, which doesn't have the
>> >> transactional database access across registry operations. We can use
>> the
>> >> same tests to run with embedded registry.
>> >
>> > Dimuthu, Can you help me to convert exiting APP based pref tests to
>> > run with embedded registry.
>> >
>> > We delayed perf test execution until we resolve oracle and MSSQL
>> > support with G-Reg. However, It is not good to delay this further
>> > anymore.
>> >
>> > So I will start perf tests with mysql within today.
>> >
>> > Thanks,
>> > Krishantha.
>> >
>> >>
>> >> With multiple registry operations are runing in one transaction, this
>> kind
>> >> of deadlock are possible. As senaka pointed out the solution can be
>> >> introducing a retry mechanism (handle exception and retry after a
>> random
>> >> interval).
>> >>
>> >> Thanks
>> >> Dimuthu
>> >>
>> >>
>> >> On Wed, Feb 24, 2010 at 12:08 PM, Senaka Fernando <[email protected]>
>> wrote:
>> >>>
>> >>> Hi it seems that you have persisted a resource + created an
>> association at
>> >>> the same time in two threads. How could this happen? Is it a part of
>> the
>> >>> same process? or two things? And +1 for running perf tests. But, this
>> time
>> >>> we need to eradicate the possibilities of such or at least introduce
>> some
>> >>> retry mechanism (or handle exceptions), instead of simply treating
>> this as a
>> >>> perf issue. Not everybody have the same hardware configurations that
>> we do,
>> >>> and there can be possibilities of such.
>> >>>
>> >>> Thanks,
>> >>> Senaka.
>> >>>
>> >>> On Wed, Feb 24, 2010 at 11:52 AM, Sumedha Rubasinghe <
>> [email protected]>
>> >>> wrote:
>> >>>>
>> >>>> Hi Hiranya,
>> >>>> Did you press save config button multiple times?
>> >>>> /sumedha
>> >>>>
>> >>>>
>> >>>> On Wed, Feb 24, 2010 at 11:28 AM, Hiranya Jayathilaka <
>> [email protected]>
>> >>>> wrote:
>> >>>>>
>> >>>>> Following exception encountered in the ESB while updating a proxy
>> >>>>> service:
>> >>>>>
>> >>>>> Exception in thread "Thread-23"
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.ServiceBusPersistenceException:
>> >>>>> Error while saving mediation configuration changes
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager.handleException(MediationPersistenceManager.java:278)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager.access$1200(MediationPersistenceManager.java:48)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager$MediationPersistenceWorker.run(MediationPersistenceManager.java:320)
>> >>>>> Caused by:
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.ServiceBusPersistenceException:
>> >>>>> Unable to persist the proxy service in the path :
>> >>>>> /repository/synapse/proxy-services/FooProxy
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.registry.AbstractRegistryStore.handleException(AbstractRegistryStore.java:101)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.registry.ProxyServiceRegistryStore.persistElement(ProxyServiceRegistryStore.java:72)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.AbstractStore.saveToRegistry(AbstractStore.java:154)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.AbstractStore.save(AbstractStore.java:82)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager.persistElement(MediationPersistenceManager.java:335)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager.access$1000(MediationPersistenceManager.java:48)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.MediationPersistenceManager$MediationPersistenceWorker.run(MediationPersistenceManager.java:314)
>> >>>>> Caused by:
>> org.wso2.carbon.registry.core.exceptions.RegistryException:
>> >>>>> Unable to persist element
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.registry.AbstractRegistryStore.persistElement(AbstractRegistryStore.java:95)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.registry.ProxyServiceRegistryStore.persistElement(ProxyServiceRegistryStore.java:69)
>> >>>>> ... 5 more
>> >>>>> Caused by:
>> org.wso2.carbon.registry.core.exceptions.RegistryException:
>> >>>>> Failed to add log entry for resource
>> >>>>> /_system/config/repository/synapse/proxy-services. Deadlock
>> detected. The
>> >>>>> current transaction was rolled back. Details:
>> >>>>> Session #9 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_RESOURCE
>> >>>>> while locking PUBLIC.REG_ASSOCIATION (exclusive), PUBLIC.REG_LOG
>> >>>>> (exclusive).
>> >>>>> Session #12 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_LOG
>> while
>> >>>>> locking PUBLIC.REG_CONTENT_HISTORY (exclusive),
>> PUBLIC.REG_RESOURCE_HISTORY
>> >>>>> (exclusive), PUBLIC.REG_RESOURCE (exclusive), PUBLIC.REG_CONTENT
>> >>>>> (exclusive), PUBLIC.REG_SNAPSHOT (exclusive).; SQL statement:
>> >>>>> INSERT INTO REG_LOG (REG_PATH, REG_USER_ID, REG_LOGGED_TIME,
>> REG_ACTION,
>> >>>>> REG_ACTION_DATA, REG_TENANT_ID) VALUES (?, ?, ?, ?, ?, ?)
>> [40001-112]
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.jdbc.dao.LogsDAO.addLog(LogsDAO.java:79)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.jdbc.Repository.update(Repository.java:966)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.jdbc.Repository.updateParent(Repository.java:947)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.jdbc.Repository.delete(Repository.java:403)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.delete(EmbeddedRegistry.java:807)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.session.UserRegistry.delete(UserRegistry.java:625)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.mediation.initializer.persistence.registry.AbstractRegistryStore.persistElement(AbstractRegistryStore.java:79)
>> >>>>> ... 6 more
>> >>>>> Caused by: org.h2.jdbc.JdbcSQLException: Deadlock detected. The
>> current
>> >>>>> transaction was rolled back. Details:
>> >>>>> Session #9 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_RESOURCE
>> >>>>> while locking PUBLIC.REG_ASSOCIATION (exclusive), PUBLIC.REG_LOG
>> >>>>> (exclusive).
>> >>>>> Session #12 (user: WSO2CARBON) is waiting to lock PUBLIC.REG_LOG
>> while
>> >>>>> locking PUBLIC.REG_CONTENT_HISTORY (exclusive),
>> PUBLIC.REG_RESOURCE_HISTORY
>> >>>>> (exclusive), PUBLIC.REG_RESOURCE (exclusive), PUBLIC.REG_CONTENT
>> >>>>> (exclusive), PUBLIC.REG_SNAPSHOT (exclusive).; SQL statement:
>> >>>>> INSERT INTO REG_LOG (REG_PATH, REG_USER_ID, REG_LOGGED_TIME,
>> REG_ACTION,
>> >>>>> REG_ACTION_DATA, REG_TENANT_ID) VALUES (?, ?, ?, ?, ?, ?)
>> [40001-112]
>> >>>>> at org.h2.message.Message.getSQLException(Message.java:107)
>> >>>>> at org.h2.message.Message.getSQLException(Message.java:118)
>> >>>>> at org.h2.message.Message.getSQLException(Message.java:77)
>> >>>>> at org.h2.table.TableData.doLock(TableData.java:428)
>> >>>>> at org.h2.table.TableData.lock(TableData.java:375)
>> >>>>> at org.h2.command.dml.Insert.update(Insert.java:99)
>> >>>>> at org.h2.command.CommandContainer.update(CommandContainer.java:71)
>> >>>>> at org.h2.command.Command.executeUpdate(Command.java:207)
>> >>>>> at
>> >>>>>
>> org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:139)
>> >>>>> at
>> >>>>>
>> org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:128)
>> >>>>> at
>> >>>>>
>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:102)
>> >>>>> at
>> >>>>>
>> org.wso2.carbon.registry.core.jdbc.dao.LogsDAO.addLog(LogsDAO.java:72)
>> >>>>> ... 12 more
>> >>>>> Thanks
>> >>>>> --
>> >>>>> Hiranya Jayathilaka
>> >>>>> Software Engineer;
>> >>>>> WSO2 Inc.; http://wso2.org
>> >>>>> E-mail: [email protected]; Mobile: +94 77 633 3491
>> >>>>> Blog: http://techfeast-hiranya.blogspot.com
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Carbon-dev mailing list
>> >>>>> [email protected]
>> >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Carbon-dev mailing list
>> >>>> [email protected]
>> >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Senaka Fernando
>> >>> Software Engineer
>> >>> WSO2 Inc.
>> >>> E-mail: senaka AT wso2.com; Mobile: +94 77 322 1818
>> >>>
>> >>> http://www.wso2.com/ - "Lean . Enterprise . Middleware"
>> >>>
>> >>> _______________________________________________
>> >>> Carbon-dev mailing list
>> >>> [email protected]
>> >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> Carbon-dev mailing list
>> >> [email protected]
>> >> https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>> >>
>> >>
>> >
>>
>> _______________________________________________
>> Carbon-dev mailing list
>> [email protected]
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>>
>
>
>
> --
> Senaka Fernando
> Software Engineer
> WSO2 Inc.
> E-mail: senaka AT wso2.com; Mobile: +94 77 322 1818
>
> http://www.wso2.com/ - "Lean . Enterprise . Middleware"
>
> _______________________________________________
> Carbon-dev mailing list
> [email protected]
> https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
>
>
_______________________________________________
Carbon-dev mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
