Hi Nirmal, My observation and maninda's observation is bit different. In his setup, the publisher never resumes, and if he starts a new publisher it works. So we are recreating the setup to find what is the actual behaviour seen there?
thanks, Sinthuja. On Mon, Aug 31, 2015 at 10:48 PM, Nirmal Fernando <[email protected]> wrote: > Hi Sinthuja, > > On Tue, Sep 1, 2015 at 11:12 AM, Sinthuja Ragendran <[email protected]> > wrote: > >> Hi Nirmal, >> >> Yes that is the expected behaviour, when the receiver queue is full. >> > > Exactly, thanks. So, do we still have an issue? Sorry, I couldn't locate > the real issue. > > >> >> There are different API's provided in the publisher, and in case if the >> user doesn't want the calling thread to block, then they can use >> tryPublish() which will publish only if it can, and so on. >> >> Thanks, >> Sinthuja. >> >> On Mon, Aug 31, 2015 at 10:37 PM, Nirmal Fernando <[email protected]> >> wrote: >> >>> All, >>> >>> Sorry for jumping in. But, shouldn't we expect this behaviour when we >>> are using a blocking executor? >>> >>> On Tue, Sep 1, 2015 at 11:00 AM, Sinthuja Ragendran <[email protected]> >>> wrote: >>> >>>> Hi Anjana, >>>> >>>> >>>> On Mon, Aug 31, 2015 at 10:23 PM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Sinthuja, >>>>> >>>>> For this, disable the indexing stuff and try the tests. Here, we are >>>>> just testing just the publishing of events to the server, indexing will >>>>> add >>>>> more load to the database, and can make it have timeouts etc.. For this, >>>>> we >>>>> can use a different record store for index staging part and so on. >>>>> >>>> >>>> Yeah, the latest test I ran was without indexing and with the CAPP >>>> maninda used for testing. Further, as mentioned i was able to see some >>>> times the publisher stops, and then resumes. At the time of publisher is >>>> stopped i got the threaddump which was attached in the last mail. There you >>>> can see the receiver queue is full, and worker threads are busy in >>>> inserting the records to DAL. I hope this is the same case in maninda's >>>> setup as well, but have to get a threaddump of DAS to confirm this. >>>> >>>> Thanks, >>>> Sinthuja. >>>> >>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> On Tue, Sep 1, 2015 at 10:47 AM, Sinthuja Ragendran <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi Maninda, >>>>>> >>>>>> I did a test with MySQL, and I was able to publish 10M events. There >>>>>> were some hickups as I mentioned before, and I could see the receiver >>>>>> queue >>>>>> is full, and the event sink worker threads are writing to database. >>>>>> Please >>>>>> refer the attached threaddump which was taken when the publisher is >>>>>> paused >>>>>> due to this. Please do run the test from your side, and share your >>>>>> observation. >>>>>> >>>>>> Thanks, >>>>>> Sinthuja. >>>>>> >>>>>> On Mon, Aug 31, 2015 at 8:50 PM, Sinthuja Ragendran < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Maninda, >>>>>>> >>>>>>> I'll also test with MySQL in local machine at the mean time, >>>>>>> apparently I observed really high CPU usage from DAS and publisher was >>>>>>> normal, as other way around of your observation. Please include the >>>>>>> sout to >>>>>>> the agent code as discussed offline and share the results. >>>>>>> >>>>>>> Thanks, >>>>>>> Sinthuja. >>>>>>> >>>>>>> On Mon, Aug 31, 2015 at 8:46 PM, Maninda Edirisooriya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Sinthuja, >>>>>>>> >>>>>>>> I have used MySQL in RDS. And I have used a indexing disabled >>>>>>>> version of smart home CApp to isolate issues. Here I have attached it. >>>>>>>> So I >>>>>>>> could not see any error in DAS side and that may be the low CPU usage >>>>>>>> in >>>>>>>> DAS than in publisher comparing to your setup as we discussed offline. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> *Maninda Edirisooriya* >>>>>>>> Senior Software Engineer >>>>>>>> >>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>> >>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>> *E-mail* : [email protected] >>>>>>>> *Skype* : @manindae >>>>>>>> *Twitter* : @maninda >>>>>>>> >>>>>>>> On Tue, Sep 1, 2015 at 8:06 AM, Sinthuja Ragendran < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Maninda, >>>>>>>>> >>>>>>>>> I tested this locally now, and I was able to see some hickups when >>>>>>>>> publishing. So at the point when the publisher is kind of paused >>>>>>>>> publishing, I started a new publisher, and that also succeeded only >>>>>>>>> upto >>>>>>>>> the publisher's event queue becomes full, and then that publisher also >>>>>>>>> stopped pushing. Can you confirm that same behaviour was observed in >>>>>>>>> publisher? I think this have made you to think the publisher has >>>>>>>>> become >>>>>>>>> hang state, but actually the receiver queue was full and it stops >>>>>>>>> accepting >>>>>>>>> the events further. >>>>>>>>> >>>>>>>>> And during that time, I was able to see multiple error logs in the >>>>>>>>> DAS side. Therefore I think the event persisting thread has become >>>>>>>>> very >>>>>>>>> slow, and hence the this behaviour was observed. I have attached the >>>>>>>>> DAS >>>>>>>>> threaddump, and I could see many threads are in blocked state on H2 >>>>>>>>> database. What is the database that you are using to test? I think >>>>>>>>> better >>>>>>>>> you try with MySQL, some other production recommended databases. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> >>>>>>>>> [2015-08-31 19:17:04,359] ERROR >>>>>>>>> {org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer} >>>>>>>>> - >>>>>>>>> Error in processing index batch operations: [-1000:__INDEX_DATA__] >>>>>>>>> does not >>>>>>>>> exist >>>>>>>>> org.wso2.carbon.analytics.datasource.commons.exception.AnalyticsTableNotAvailableException: >>>>>>>>> [-1000:__INDEX_DATA__] does not exist >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.get(RDBMSAnalyticsRecordStore.java:319) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.loadIndexOperationRecords(AnalyticsDataIndexer.java:588) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:391) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:381) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.access$100(AnalyticsDataIndexer.java:130) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer$IndexWorker.run(AnalyticsDataIndexer.java:1791) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> org.wso2.carbon.analytics.datasource.commons.exception.AnalyticsException: >>>>>>>>> Error in deleting records: Timeout trying to lock table >>>>>>>>> "ANX___8GIVT7RC_"; >>>>>>>>> SQL statement: >>>>>>>>> DELETE FROM ANX___8GIvT7Rc_ WHERE record_id IN >>>>>>>>> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) [50200-140] >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.delete(RDBMSAnalyticsRecordStore.java:519) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.delete(RDBMSAnalyticsRecordStore.java:491) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.deleteIndexRecords(AnalyticsDataIndexer.java:581) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:414) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:381) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.access$100(AnalyticsDataIndexer.java:130) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer$IndexWorker.run(AnalyticsDataIndexer.java:1791) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock >>>>>>>>> table "ANX___8GIVT7RC_"; SQL statement: >>>>>>>>> DELETE FROM ANX___8GIvT7Rc_ WHERE record_id IN >>>>>>>>> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) [50200-140] >>>>>>>>> at >>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327) >>>>>>>>> at org.h2.message.DbException.get(DbException.java:167) >>>>>>>>> at org.h2.message.DbException.get(DbException.java:144) >>>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466) >>>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404) >>>>>>>>> at org.h2.command.dml.Delete.update(Delete.java:50) >>>>>>>>> at >>>>>>>>> org.h2.command.CommandContainer.update(CommandContainer.java:70) >>>>>>>>> at org.h2.command.Command.executeUpdate(Command.java:199) >>>>>>>>> at >>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:141) >>>>>>>>> at >>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:127) >>>>>>>>> at >>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.delete(RDBMSAnalyticsRecordStore.java:514) >>>>>>>>> ... 9 more >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Aug 31, 2015 at 10:01 AM, Sinthuja Ragendran < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi maninda, >>>>>>>>>> >>>>>>>>>> Ok, thanks for information. I'll do the test locally and get back >>>>>>>>>> to you. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Sinthuja. >>>>>>>>>> >>>>>>>>>> On Mon, Aug 31, 2015 at 9:53 AM, Maninda Edirisooriya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Sinthuja, >>>>>>>>>>> >>>>>>>>>>> I tested with smart-home sample in latest DAS with [1] config >>>>>>>>>>> and DAS with the attached config directory. (There >>>>>>>>>>> data-bridge-config.xml >>>>>>>>>>> is as [2]) >>>>>>>>>>> I did the test on EC2 instances with MySQL RDS instance as DBs. >>>>>>>>>>> This issue was always reproducible when 10M events are published >>>>>>>>>>> with the sample. For some time events get published and then it will >>>>>>>>>>> suddenly stop receiving events. But you can see the client is busy >>>>>>>>>>> with the >>>>>>>>>>> CPU usage while DAS is almost idling. >>>>>>>>>>> No debug or logging was enabled. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> >>>>>>>>>>> <Agent> >>>>>>>>>>> <Name>Thrift</Name> >>>>>>>>>>> >>>>>>>>>>> <DataEndpointClass>org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint</DataEndpointClass> >>>>>>>>>>> >>>>>>>>>>> <TrustSore>src/main/resources/client-truststore.jks</TrustSore> >>>>>>>>>>> <TrustSorePassword>wso2carbon</TrustSorePassword> >>>>>>>>>>> <QueueSize>32768</QueueSize> >>>>>>>>>>> <BatchSize>200</BatchSize> >>>>>>>>>>> <CorePoolSize>5</CorePoolSize> >>>>>>>>>>> <MaxPoolSize>10</MaxPoolSize> >>>>>>>>>>> <KeepAliveTimeInPool>20</KeepAliveTimeInPool> >>>>>>>>>>> <ReconnectionInterval>30</ReconnectionInterval> >>>>>>>>>>> <MaxTransportPoolSize>250</MaxTransportPoolSize> >>>>>>>>>>> <MaxIdleConnections>250</MaxIdleConnections> >>>>>>>>>>> <EvictionTimePeriod>5500</EvictionTimePeriod> >>>>>>>>>>> <MinIdleTimeInPool>5000</MinIdleTimeInPool> >>>>>>>>>>> >>>>>>>>>>> <SecureMaxTransportPoolSize>250</SecureMaxTransportPoolSize> >>>>>>>>>>> <SecureMaxIdleConnections>250</SecureMaxIdleConnections> >>>>>>>>>>> <SecureEvictionTimePeriod>5500</SecureEvictionTimePeriod> >>>>>>>>>>> <SecureMinIdleTimeInPool>5000</SecureMinIdleTimeInPool> >>>>>>>>>>> </Agent> >>>>>>>>>>> >>>>>>>>>>> [2] >>>>>>>>>>> >>>>>>>>>>> <dataBridgeConfiguration> >>>>>>>>>>> >>>>>>>>>>> <workerThreads>10</workerThreads> >>>>>>>>>>> <eventBufferCapacity>1000</eventBufferCapacity> >>>>>>>>>>> <clientTimeoutMin>30</clientTimeoutMin> >>>>>>>>>>> >>>>>>>>>>> <dataReceiver name="Thrift"> >>>>>>>>>>> <config name="tcpPort">7611</config> >>>>>>>>>>> <config name="sslPort">7711</config> >>>>>>>>>>> </dataReceiver> >>>>>>>>>>> >>>>>>>>>>> <dataReceiver name="Binary"> >>>>>>>>>>> <config name="tcpPort">9611</config> >>>>>>>>>>> <config name="sslPort">9711</config> >>>>>>>>>>> <config name="sslReceiverThreadPoolSize">100</config> >>>>>>>>>>> <config name="tcpReceiverThreadPoolSize">100</config> >>>>>>>>>>> </dataReceiver> >>>>>>>>>>> >>>>>>>>>>> </dataBridgeConfiguration> >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>> Senior Software Engineer >>>>>>>>>>> >>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>> >>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 31, 2015 at 8:08 PM, Sinthuja Ragendran < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Are you running with debug mode in logging? And can you >>>>>>>>>>>> constantly reproduce this? Or it's intermittent? >>>>>>>>>>>> >>>>>>>>>>>> Please provide the publisher and receiver side configs to test >>>>>>>>>>>> this and see. As I have already tested more than 10M records, I'm >>>>>>>>>>>> not sure >>>>>>>>>>>> what is the case here. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Sinthuja. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Monday, August 31, 2015, Maninda Edirisooriya < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> When I started a 10M load test from Smart Home sample in DAS >>>>>>>>>>>>> it runs for some time and stops receiving events suddenly. >>>>>>>>>>>>> But publisher in client was running in higher CPU usage when >>>>>>>>>>>>> DAS was running with very low CPU. >>>>>>>>>>>>> When another data agent was spawned it started to publish >>>>>>>>>>>>> correctly which was confirming that the issue is with the client >>>>>>>>>>>>> side. >>>>>>>>>>>>> We analyzed the thread dump and found the highest using thread >>>>>>>>>>>>> is with the following stack traces when we analyzed it twice. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. >>>>>>>>>>>>> "main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable >>>>>>>>>>>>> [0x00007f85f250f000] >>>>>>>>>>>>> java.lang.Thread.State: RUNNABLE >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:148) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown >>>>>>>>>>>>> Source) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown >>>>>>>>>>>>> Source) >>>>>>>>>>>>> >>>>>>>>>>>>> 2. >>>>>>>>>>>>> "main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable >>>>>>>>>>>>> [0x00007f85f250f000] >>>>>>>>>>>>> java.lang.Thread.State: RUNNABLE >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.log4j.Category.callAppenders(Category.java:202) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.log4j.Category.forcedLog(Category.java:391) >>>>>>>>>>>>> at org.apache.log4j.Category.log(Category.java:856) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.isActiveDataEndpointExists(DataEndpointGroup.java:264) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.access$400(DataEndpointGroup.java:46) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:155) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown >>>>>>>>>>>>> Source) >>>>>>>>>>>>> at >>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown >>>>>>>>>>>>> Source) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> We suspect that *isActiveDataEndpointExists()* method is >>>>>>>>>>>>> called in >>>>>>>>>>>>> *org.wso2.carbon.analytics.eventsink.internal.queue.DataEndpointGroup* >>>>>>>>>>>>> class repeatedly because the disruptor ring buffer is filled in >>>>>>>>>>>>> client >>>>>>>>>>>>> side. Not sure why this happens. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>> >>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>> >>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Sent from iPhone >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Sinthuja Rajendran* >>>>>>>>>> Associate Technical Lead >>>>>>>>>> WSO2, Inc.:http://wso2.com >>>>>>>>>> >>>>>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>>>>> Mobile: +94774273955 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Sinthuja Rajendran* >>>>>>>>> Associate Technical Lead >>>>>>>>> WSO2, Inc.:http://wso2.com >>>>>>>>> >>>>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>>>> Mobile: +94774273955 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Sinthuja Rajendran* >>>>>>> Associate Technical Lead >>>>>>> WSO2, Inc.:http://wso2.com >>>>>>> >>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>> Mobile: +94774273955 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Sinthuja Rajendran* >>>>>> Associate Technical Lead >>>>>> WSO2, Inc.:http://wso2.com >>>>>> >>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>> Mobile: +94774273955 >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Senior Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> *Sinthuja Rajendran* >>>> Associate Technical Lead >>>> WSO2, Inc.:http://wso2.com >>>> >>>> Blog: http://sinthu-rajan.blogspot.com/ >>>> Mobile: +94774273955 >>>> >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> *Sinthuja Rajendran* >> Associate Technical Lead >> WSO2, Inc.:http://wso2.com >> >> Blog: http://sinthu-rajan.blogspot.com/ >> Mobile: +94774273955 >> >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Sinthuja Rajendran* Associate Technical Lead WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
