Hi Srinath, I was checking on this, and it seems, the issue is, if you've a large result set, and for the gadget, if you select this large result set as its source data, it will go and read everything. For example, if you use the smart home sample and just put all the raw records as the X axis of a bar chart, it will try to read all thousands of data records and try to render it. But this operation takes a bit of time, to do the JSON serialization and send it to the front end. And what makes this far more worse is, the dashboard automatically refreshes the data, and this happens in an asynchronous manner, and it doesn't wait for the earlier cycle to be finished. So these requests effectively builds up and eats up the CPU in the backend and can end up going OOM.
So I guess, first, we have to fix the refreshing of the gadgets somehow, so it will wait for the earlier request to finish, before making new ones. Then after that, we should also provide a sensible upper limit to the data records queries, so people won't mistakenly query millions of records from a raw data table. We have to do this carefully, because for a thing like a scatter plot, we may need to read thousands of records to render the image. So for things like that, it maybe easier for it to be rendered in the backend and sent to the front end. And also, Maninda also pointed out, since we have a natural refresh interval for a dashboard, we should also cache the data in the end backend, with a cache validity period similar to the dashboard interval, so we don't have to read data from the physical databases all the time. So as an immediate fix, shall we try to change the dashboard refresh logic and also put a upper limit to the backend data retrieval to about 1000 (by changing "LAST_INDEX" variable from -1 to 100 in analytics.jag at carbon-dashboards). @Dunith, will you be able to do these changes. Cheers, Anjana. On Fri, Aug 28, 2015 at 3:07 PM, Anjana Fernando <[email protected]> wrote: > Hi Srinath, > > Yeah, I also could reproduce the issue. Will check it out. > > Cheers, > Anjana. > > On Fri, Aug 28, 2015 at 2:38 PM, Srinath Perera <[email protected]> wrote: > >> This was a batch chart. >> >> On Fri, Aug 28, 2015 at 2:32 PM, Damith Wickramasinghe <[email protected]> >> wrote: >> >>> Hi, >>> >>> AFAIK That message "Data is not available for plotting. The chart will >>> be loaded once the dashboard receives events" normally comes for real time >>> flow gadgets. >>> >>> Regards, >>> Damith. >>> >>> On Fri, Aug 28, 2015 at 2:22 PM, Srinath Perera <[email protected]> >>> wrote: >>> >>>> Yes, I could see the preview ok. >>>> >>>> --Srinath >>>> >>>> On Fri, Aug 28, 2015 at 2:19 PM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Srinath, >>>>> >>>>> Did you have data in that table before you created the gadget? .. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> On Fri, Aug 28, 2015 at 2:16 PM, Srinath Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> It is fixed with new pack. >>>>>> >>>>>> However, when I create a Gadget ( based on beatch data) and add teh >>>>>> Gadget, it says "Data is not available for plotting. The chart will be >>>>>> loaded once the dashboard receives events". >>>>>> >>>>>> Thanks >>>>>> Srinath >>>>>> >>>>>> On Fri, Aug 28, 2015 at 8:58 AM, Srinath Perera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Yesterday packs had the same problem >>>>>>> >>>>>>> On Thu, Aug 27, 2015 at 4:20 PM, Maninda Edirisooriya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This was fixed in https://wso2.org/jira/browse/DAS-84 on yesterday >>>>>>>> by Dunith. Now the fix should be there in latest jenkins built packs. >>>>>>>> Thanks. >>>>>>>> >>>>>>>> >>>>>>>> *Maninda Edirisooriya* >>>>>>>> Senior Software Engineer >>>>>>>> >>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>> >>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>> *E-mail* : [email protected] >>>>>>>> *Skype* : @manindae >>>>>>>> *Twitter* : @maninda >>>>>>>> >>>>>>>> On Thu, Aug 27, 2015 at 4:16 PM, Srinath Perera <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> is this fixed? >>>>>>>>> >>>>>>>>> On Tue, Aug 25, 2015 at 4:41 PM, Maninda Edirisooriya < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Aug 25, 2015 at 3:28 PM, Srinath Perera <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Using yesterday's pack >>>>>>>>>>> >>>>>>>>>>> 1. Can we make creating a receiver part of the creating/ editing >>>>>>>>>>> new streams flow? e.g. by asking what transports to expose and >>>>>>>>>>> automatically creating it. >>>>>>>>>>> >>>>>>>>>>> 2. In the gadget generation wizard, x, y axis drop downs are now >>>>>>>>>>> not be populated. >>>>>>>>>>> >>>>>>>>>> Yes this issue was observed in latest packs. I am looking at it. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>> >>>>>>>>>>> 3. When you go into Gadget design view, there is no button to >>>>>>>>>>> come back. >>>>>>>>>>> >>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Srinath >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ============================ >>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>> Phone: 0772360902 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ============================ >>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>> Phone: 0772360902 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>> Site: http://people.apache.org/~hemapani/ >>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>> Phone: 0772360902 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Senior Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Software Engineer >>> WSO2 Inc.; http://wso2.com >>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>> lean.enterprise.middleware >>> >>> mobile: *+94728671315 <%2B94728671315>* >>> >>> >> >> >> -- >> ============================ >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
