Thanks Sinthuja and Gokul. May be we could forget about DAS for a moment, and verify whether the blocking executor behaves properly, when the queue is available? Also, I hope after submitting a task, you guys didn't forget to call; afterExecute method of BlockingExecutor.
On Tue, Sep 1, 2015 at 11:21 AM, Gokul Balakrishnan <[email protected]> wrote: > Hi Nirmal, > > The issue we'd encountered is the datapublisher seemingly not publishing > any further data while consuming high CPU, while the persistence task in > DAS is also inactive. Since there was apparently no actual work being done, > we're investigating the issue. Further, a new publisher instance was able > to publish even when the first one was hanging. > > Thanks, > > On 1 September 2015 at 11:18, Nirmal Fernando <[email protected]> wrote: > >> Hi Sinthuja, >> >> On Tue, Sep 1, 2015 at 11:12 AM, Sinthuja Ragendran <[email protected]> >> wrote: >> >>> Hi Nirmal, >>> >>> Yes that is the expected behaviour, when the receiver queue is full. >>> >> >> Exactly, thanks. So, do we still have an issue? Sorry, I couldn't locate >> the real issue. >> >> >>> >>> There are different API's provided in the publisher, and in case if the >>> user doesn't want the calling thread to block, then they can use >>> tryPublish() which will publish only if it can, and so on. >>> >>> Thanks, >>> Sinthuja. >>> >>> On Mon, Aug 31, 2015 at 10:37 PM, Nirmal Fernando <[email protected]> >>> wrote: >>> >>>> All, >>>> >>>> Sorry for jumping in. But, shouldn't we expect this behaviour when we >>>> are using a blocking executor? >>>> >>>> On Tue, Sep 1, 2015 at 11:00 AM, Sinthuja Ragendran <[email protected]> >>>> wrote: >>>> >>>>> Hi Anjana, >>>>> >>>>> >>>>> On Mon, Aug 31, 2015 at 10:23 PM, Anjana Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Sinthuja, >>>>>> >>>>>> For this, disable the indexing stuff and try the tests. Here, we are >>>>>> just testing just the publishing of events to the server, indexing will >>>>>> add >>>>>> more load to the database, and can make it have timeouts etc.. For this, >>>>>> we >>>>>> can use a different record store for index staging part and so on. >>>>>> >>>>> >>>>> Yeah, the latest test I ran was without indexing and with the CAPP >>>>> maninda used for testing. Further, as mentioned i was able to see some >>>>> times the publisher stops, and then resumes. At the time of publisher is >>>>> stopped i got the threaddump which was attached in the last mail. There >>>>> you >>>>> can see the receiver queue is full, and worker threads are busy in >>>>> inserting the records to DAL. I hope this is the same case in maninda's >>>>> setup as well, but have to get a threaddump of DAS to confirm this. >>>>> >>>>> Thanks, >>>>> Sinthuja. >>>>> >>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> On Tue, Sep 1, 2015 at 10:47 AM, Sinthuja Ragendran < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Maninda, >>>>>>> >>>>>>> I did a test with MySQL, and I was able to publish 10M events. There >>>>>>> were some hickups as I mentioned before, and I could see the receiver >>>>>>> queue >>>>>>> is full, and the event sink worker threads are writing to database. >>>>>>> Please >>>>>>> refer the attached threaddump which was taken when the publisher is >>>>>>> paused >>>>>>> due to this. Please do run the test from your side, and share your >>>>>>> observation. >>>>>>> >>>>>>> Thanks, >>>>>>> Sinthuja. >>>>>>> >>>>>>> On Mon, Aug 31, 2015 at 8:50 PM, Sinthuja Ragendran < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Maninda, >>>>>>>> >>>>>>>> I'll also test with MySQL in local machine at the mean time, >>>>>>>> apparently I observed really high CPU usage from DAS and publisher was >>>>>>>> normal, as other way around of your observation. Please include the >>>>>>>> sout to >>>>>>>> the agent code as discussed offline and share the results. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sinthuja. >>>>>>>> >>>>>>>> On Mon, Aug 31, 2015 at 8:46 PM, Maninda Edirisooriya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Sinthuja, >>>>>>>>> >>>>>>>>> I have used MySQL in RDS. And I have used a indexing disabled >>>>>>>>> version of smart home CApp to isolate issues. Here I have attached >>>>>>>>> it. So I >>>>>>>>> could not see any error in DAS side and that may be the low CPU usage >>>>>>>>> in >>>>>>>>> DAS than in publisher comparing to your setup as we discussed offline. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> >>>>>>>>> *Maninda Edirisooriya* >>>>>>>>> Senior Software Engineer >>>>>>>>> >>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>> >>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>> *E-mail* : [email protected] >>>>>>>>> *Skype* : @manindae >>>>>>>>> *Twitter* : @maninda >>>>>>>>> >>>>>>>>> On Tue, Sep 1, 2015 at 8:06 AM, Sinthuja Ragendran < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Maninda, >>>>>>>>>> >>>>>>>>>> I tested this locally now, and I was able to see some hickups >>>>>>>>>> when publishing. So at the point when the publisher is kind of paused >>>>>>>>>> publishing, I started a new publisher, and that also succeeded only >>>>>>>>>> upto >>>>>>>>>> the publisher's event queue becomes full, and then that publisher >>>>>>>>>> also >>>>>>>>>> stopped pushing. Can you confirm that same behaviour was observed in >>>>>>>>>> publisher? I think this have made you to think the publisher has >>>>>>>>>> become >>>>>>>>>> hang state, but actually the receiver queue was full and it stops >>>>>>>>>> accepting >>>>>>>>>> the events further. >>>>>>>>>> >>>>>>>>>> And during that time, I was able to see multiple error logs in >>>>>>>>>> the DAS side. Therefore I think the event persisting thread has >>>>>>>>>> become >>>>>>>>>> very slow, and hence the this behaviour was observed. I have >>>>>>>>>> attached the >>>>>>>>>> DAS threaddump, and I could see many threads are in blocked state on >>>>>>>>>> H2 >>>>>>>>>> database. What is the database that you are using to test? I think >>>>>>>>>> better >>>>>>>>>> you try with MySQL, some other production recommended databases. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> >>>>>>>>>> [2015-08-31 19:17:04,359] ERROR >>>>>>>>>> {org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer} >>>>>>>>>> - >>>>>>>>>> Error in processing index batch operations: [-1000:__INDEX_DATA__] >>>>>>>>>> does not >>>>>>>>>> exist >>>>>>>>>> org.wso2.carbon.analytics.datasource.commons.exception.AnalyticsTableNotAvailableException: >>>>>>>>>> [-1000:__INDEX_DATA__] does not exist >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.get(RDBMSAnalyticsRecordStore.java:319) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.loadIndexOperationRecords(AnalyticsDataIndexer.java:588) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:391) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:381) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.access$100(AnalyticsDataIndexer.java:130) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer$IndexWorker.run(AnalyticsDataIndexer.java:1791) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> org.wso2.carbon.analytics.datasource.commons.exception.AnalyticsException: >>>>>>>>>> Error in deleting records: Timeout trying to lock table >>>>>>>>>> "ANX___8GIVT7RC_"; >>>>>>>>>> SQL statement: >>>>>>>>>> DELETE FROM ANX___8GIvT7Rc_ WHERE record_id IN >>>>>>>>>> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) [50200-140] >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.delete(RDBMSAnalyticsRecordStore.java:519) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.delete(RDBMSAnalyticsRecordStore.java:491) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.deleteIndexRecords(AnalyticsDataIndexer.java:581) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:414) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.processIndexOperations(AnalyticsDataIndexer.java:381) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer.access$100(AnalyticsDataIndexer.java:130) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.dataservice.indexing.AnalyticsDataIndexer$IndexWorker.run(AnalyticsDataIndexer.java:1791) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>> at >>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>> Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock >>>>>>>>>> table "ANX___8GIVT7RC_"; SQL statement: >>>>>>>>>> DELETE FROM ANX___8GIvT7Rc_ WHERE record_id IN >>>>>>>>>> (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) [50200-140] >>>>>>>>>> at >>>>>>>>>> org.h2.message.DbException.getJdbcSQLException(DbException.java:327) >>>>>>>>>> at org.h2.message.DbException.get(DbException.java:167) >>>>>>>>>> at org.h2.message.DbException.get(DbException.java:144) >>>>>>>>>> at org.h2.table.RegularTable.doLock(RegularTable.java:466) >>>>>>>>>> at org.h2.table.RegularTable.lock(RegularTable.java:404) >>>>>>>>>> at org.h2.command.dml.Delete.update(Delete.java:50) >>>>>>>>>> at >>>>>>>>>> org.h2.command.CommandContainer.update(CommandContainer.java:70) >>>>>>>>>> at org.h2.command.Command.executeUpdate(Command.java:199) >>>>>>>>>> at >>>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:141) >>>>>>>>>> at >>>>>>>>>> org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:127) >>>>>>>>>> at >>>>>>>>>> org.wso2.carbon.analytics.datasource.rdbms.RDBMSAnalyticsRecordStore.delete(RDBMSAnalyticsRecordStore.java:514) >>>>>>>>>> ... 9 more >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Aug 31, 2015 at 10:01 AM, Sinthuja Ragendran < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi maninda, >>>>>>>>>>> >>>>>>>>>>> Ok, thanks for information. I'll do the test locally and get >>>>>>>>>>> back to you. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Sinthuja. >>>>>>>>>>> >>>>>>>>>>> On Mon, Aug 31, 2015 at 9:53 AM, Maninda Edirisooriya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Sinthuja, >>>>>>>>>>>> >>>>>>>>>>>> I tested with smart-home sample in latest DAS with [1] config >>>>>>>>>>>> and DAS with the attached config directory. (There >>>>>>>>>>>> data-bridge-config.xml >>>>>>>>>>>> is as [2]) >>>>>>>>>>>> I did the test on EC2 instances with MySQL RDS instance as DBs. >>>>>>>>>>>> This issue was always reproducible when 10M events are >>>>>>>>>>>> published with the sample. For some time events get published and >>>>>>>>>>>> then it >>>>>>>>>>>> will suddenly stop receiving events. But you can see the client is >>>>>>>>>>>> busy >>>>>>>>>>>> with the CPU usage while DAS is almost idling. >>>>>>>>>>>> No debug or logging was enabled. >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> >>>>>>>>>>>> <Agent> >>>>>>>>>>>> <Name>Thrift</Name> >>>>>>>>>>>> >>>>>>>>>>>> <DataEndpointClass>org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint</DataEndpointClass> >>>>>>>>>>>> >>>>>>>>>>>> <TrustSore>src/main/resources/client-truststore.jks</TrustSore> >>>>>>>>>>>> <TrustSorePassword>wso2carbon</TrustSorePassword> >>>>>>>>>>>> <QueueSize>32768</QueueSize> >>>>>>>>>>>> <BatchSize>200</BatchSize> >>>>>>>>>>>> <CorePoolSize>5</CorePoolSize> >>>>>>>>>>>> <MaxPoolSize>10</MaxPoolSize> >>>>>>>>>>>> <KeepAliveTimeInPool>20</KeepAliveTimeInPool> >>>>>>>>>>>> <ReconnectionInterval>30</ReconnectionInterval> >>>>>>>>>>>> <MaxTransportPoolSize>250</MaxTransportPoolSize> >>>>>>>>>>>> <MaxIdleConnections>250</MaxIdleConnections> >>>>>>>>>>>> <EvictionTimePeriod>5500</EvictionTimePeriod> >>>>>>>>>>>> <MinIdleTimeInPool>5000</MinIdleTimeInPool> >>>>>>>>>>>> >>>>>>>>>>>> <SecureMaxTransportPoolSize>250</SecureMaxTransportPoolSize> >>>>>>>>>>>> <SecureMaxIdleConnections>250</SecureMaxIdleConnections> >>>>>>>>>>>> >>>>>>>>>>>> <SecureEvictionTimePeriod>5500</SecureEvictionTimePeriod> >>>>>>>>>>>> <SecureMinIdleTimeInPool>5000</SecureMinIdleTimeInPool> >>>>>>>>>>>> </Agent> >>>>>>>>>>>> >>>>>>>>>>>> [2] >>>>>>>>>>>> >>>>>>>>>>>> <dataBridgeConfiguration> >>>>>>>>>>>> >>>>>>>>>>>> <workerThreads>10</workerThreads> >>>>>>>>>>>> <eventBufferCapacity>1000</eventBufferCapacity> >>>>>>>>>>>> <clientTimeoutMin>30</clientTimeoutMin> >>>>>>>>>>>> >>>>>>>>>>>> <dataReceiver name="Thrift"> >>>>>>>>>>>> <config name="tcpPort">7611</config> >>>>>>>>>>>> <config name="sslPort">7711</config> >>>>>>>>>>>> </dataReceiver> >>>>>>>>>>>> >>>>>>>>>>>> <dataReceiver name="Binary"> >>>>>>>>>>>> <config name="tcpPort">9611</config> >>>>>>>>>>>> <config name="sslPort">9711</config> >>>>>>>>>>>> <config name="sslReceiverThreadPoolSize">100</config> >>>>>>>>>>>> <config name="tcpReceiverThreadPoolSize">100</config> >>>>>>>>>>>> </dataReceiver> >>>>>>>>>>>> >>>>>>>>>>>> </dataBridgeConfiguration> >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 31, 2015 at 8:08 PM, Sinthuja Ragendran < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Are you running with debug mode in logging? And can you >>>>>>>>>>>>> constantly reproduce this? Or it's intermittent? >>>>>>>>>>>>> >>>>>>>>>>>>> Please provide the publisher and receiver side configs to test >>>>>>>>>>>>> this and see. As I have already tested more than 10M records, I'm >>>>>>>>>>>>> not sure >>>>>>>>>>>>> what is the case here. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Sinthuja. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Monday, August 31, 2015, Maninda Edirisooriya < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> When I started a 10M load test from Smart Home sample in DAS >>>>>>>>>>>>>> it runs for some time and stops receiving events suddenly. >>>>>>>>>>>>>> But publisher in client was running in higher CPU usage when >>>>>>>>>>>>>> DAS was running with very low CPU. >>>>>>>>>>>>>> When another data agent was spawned it started to publish >>>>>>>>>>>>>> correctly which was confirming that the issue is with the client >>>>>>>>>>>>>> side. >>>>>>>>>>>>>> We analyzed the thread dump and found the highest using >>>>>>>>>>>>>> thread is with the following stack traces when we analyzed it >>>>>>>>>>>>>> twice. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. >>>>>>>>>>>>>> "main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable >>>>>>>>>>>>>> [0x00007f85f250f000] >>>>>>>>>>>>>> java.lang.Thread.State: RUNNABLE >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:148) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown >>>>>>>>>>>>>> Source) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown >>>>>>>>>>>>>> Source) >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. >>>>>>>>>>>>>> "main" prio=10 tid=0x00007f85ec00a800 nid=0x7843 runnable >>>>>>>>>>>>>> [0x00007f85f250f000] >>>>>>>>>>>>>> java.lang.Thread.State: RUNNABLE >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.log4j.Category.callAppenders(Category.java:202) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.log4j.Category.forcedLog(Category.java:391) >>>>>>>>>>>>>> at org.apache.log4j.Category.log(Category.java:856) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.isActiveDataEndpointExists(DataEndpointGroup.java:264) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.access$400(DataEndpointGroup.java:46) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.put(DataEndpointGroup.java:155) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup$EventQueue.access$300(DataEndpointGroup.java:97) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup.publish(DataEndpointGroup.java:94) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.databridge.agent.DataPublisher.publish(DataPublisher.java:183) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.publishLogEvents(Unknown >>>>>>>>>>>>>> Source) >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.wso2.carbon.das.smarthome.sample.SmartHomeAgent.main(Unknown >>>>>>>>>>>>>> Source) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We suspect that *isActiveDataEndpointExists()* method is >>>>>>>>>>>>>> called in >>>>>>>>>>>>>> *org.wso2.carbon.analytics.eventsink.internal.queue.DataEndpointGroup* >>>>>>>>>>>>>> class repeatedly because the disruptor ring buffer is filled in >>>>>>>>>>>>>> client >>>>>>>>>>>>>> side. Not sure why this happens. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>> >>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Sent from iPhone >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Sinthuja Rajendran* >>>>>>>>>>> Associate Technical Lead >>>>>>>>>>> WSO2, Inc.:http://wso2.com >>>>>>>>>>> >>>>>>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>>>>>> Mobile: +94774273955 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Sinthuja Rajendran* >>>>>>>>>> Associate Technical Lead >>>>>>>>>> WSO2, Inc.:http://wso2.com >>>>>>>>>> >>>>>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>>>>> Mobile: +94774273955 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Sinthuja Rajendran* >>>>>>>> Associate Technical Lead >>>>>>>> WSO2, Inc.:http://wso2.com >>>>>>>> >>>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>>> Mobile: +94774273955 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Sinthuja Rajendran* >>>>>>> Associate Technical Lead >>>>>>> WSO2, Inc.:http://wso2.com >>>>>>> >>>>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>>>> Mobile: +94774273955 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Sinthuja Rajendran* >>>>> Associate Technical Lead >>>>> WSO2, Inc.:http://wso2.com >>>>> >>>>> Blog: http://sinthu-rajan.blogspot.com/ >>>>> Mobile: +94774273955 >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> *Sinthuja Rajendran* >>> Associate Technical Lead >>> WSO2, Inc.:http://wso2.com >>> >>> Blog: http://sinthu-rajan.blogspot.com/ >>> Mobile: +94774273955 >>> >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Gokul Balakrishnan > Senior Software Engineer, > WSO2, Inc. http://wso2.com > Mob: +94 77 593 5789 | +1 650 272 9927 > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
