Correct approch (for Viz) would be to sample it down rather than taking first 1000 ( but not for DAL data retrival case). Any chance we can do that?
On Fri, Aug 28, 2015 at 7:12 PM, Dunith Dhanushka <[email protected]> wrote: > Hi Anjana, > > I'll work on the above change. > > Regards, > Dunith > > On Fri, Aug 28, 2015 at 8:22 AM, Anjana Fernando <[email protected]> wrote: > >> Hi Srinath, >> >> I was checking on this, and it seems, the issue is, if you've a large >> result set, and for the gadget, if you select this large result set as its >> source data, it will go and read everything. For example, if you use the >> smart home sample and just put all the raw records as the X axis of a bar >> chart, it will try to read all thousands of data records and try to render >> it. But this operation takes a bit of time, to do the JSON serialization >> and send it to the front end. And what makes this far more worse is, the >> dashboard automatically refreshes the data, and this happens in an >> asynchronous manner, and it doesn't wait for the earlier cycle to be >> finished. So these requests effectively builds up and eats up the CPU in >> the backend and can end up going OOM. >> >> So I guess, first, we have to fix the refreshing of the gadgets somehow, >> so it will wait for the earlier request to finish, before making new ones. >> Then after that, we should also provide a sensible upper limit to the data >> records queries, so people won't mistakenly query millions of records from >> a raw data table. We have to do this carefully, because for a thing like a >> scatter plot, we may need to read thousands of records to render the image. >> So for things like that, it maybe easier for it to be rendered in the >> backend and sent to the front end. And also, Maninda also pointed out, >> since we have a natural refresh interval for a dashboard, we should also >> cache the data in the end backend, with a cache validity period similar to >> the dashboard interval, so we don't have to read data from the physical >> databases all the time. >> >> So as an immediate fix, shall we try to change the dashboard refresh >> logic and also put a upper limit to the backend data retrieval to about >> 1000 (by changing "LAST_INDEX" variable from -1 to 100 in analytics.jag at >> carbon-dashboards). @Dunith, will you be able to do these changes. >> >> Cheers, >> Anjana. >> >> >> On Fri, Aug 28, 2015 at 3:07 PM, Anjana Fernando <[email protected]> wrote: >> >>> Hi Srinath, >>> >>> Yeah, I also could reproduce the issue. Will check it out. >>> >>> Cheers, >>> Anjana. >>> >>> On Fri, Aug 28, 2015 at 2:38 PM, Srinath Perera <[email protected]> >>> wrote: >>> >>>> This was a batch chart. >>>> >>>> On Fri, Aug 28, 2015 at 2:32 PM, Damith Wickramasinghe < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> AFAIK That message "Data is not available for plotting. The chart will >>>>> be loaded once the dashboard receives events" normally comes for real time >>>>> flow gadgets. >>>>> >>>>> Regards, >>>>> Damith. >>>>> >>>>> On Fri, Aug 28, 2015 at 2:22 PM, Srinath Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> Yes, I could see the preview ok. >>>>>> >>>>>> --Srinath >>>>>> >>>>>> On Fri, Aug 28, 2015 at 2:19 PM, Anjana Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Srinath, >>>>>>> >>>>>>> Did you have data in that table before you created the gadget? .. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Fri, Aug 28, 2015 at 2:16 PM, Srinath Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> It is fixed with new pack. >>>>>>>> >>>>>>>> However, when I create a Gadget ( based on beatch data) and add >>>>>>>> teh Gadget, it says "Data is not available for plotting. The chart >>>>>>>> will be >>>>>>>> loaded once the dashboard receives events". >>>>>>>> >>>>>>>> Thanks >>>>>>>> Srinath >>>>>>>> >>>>>>>> On Fri, Aug 28, 2015 at 8:58 AM, Srinath Perera <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yesterday packs had the same problem >>>>>>>>> >>>>>>>>> On Thu, Aug 27, 2015 at 4:20 PM, Maninda Edirisooriya < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> This was fixed in https://wso2.org/jira/browse/DAS-84 on >>>>>>>>>> yesterday by Dunith. Now the fix should be there in latest jenkins >>>>>>>>>> built >>>>>>>>>> packs. >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>> Senior Software Engineer >>>>>>>>>> >>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>> >>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>> *Skype* : @manindae >>>>>>>>>> *Twitter* : @maninda >>>>>>>>>> >>>>>>>>>> On Thu, Aug 27, 2015 at 4:16 PM, Srinath Perera <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> is this fixed? >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 25, 2015 at 4:41 PM, Maninda Edirisooriya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 25, 2015 at 3:28 PM, Srinath Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Using yesterday's pack >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Can we make creating a receiver part of the creating/ >>>>>>>>>>>>> editing new streams flow? e.g. by asking what transports to >>>>>>>>>>>>> expose and >>>>>>>>>>>>> automatically creating it. >>>>>>>>>>>>> >>>>>>>>>>>>> 2. In the gadget generation wizard, x, y axis drop downs are >>>>>>>>>>>>> now not be populated. >>>>>>>>>>>>> >>>>>>>>>>>> Yes this issue was observed in latest packs. I am looking at >>>>>>>>>>>> it. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>> >>>>>>>>>>>>> 3. When you go into Gadget design view, there is no button to >>>>>>>>>>>>> come back. >>>>>>>>>>>>> >>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Srinath >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ============================ >>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ============================ >>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>> Phone: 0772360902 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>> Site: http://people.apache.org/~hemapani/ >>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>> Phone: 0772360902 >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Software Engineer >>>>> WSO2 Inc.; http://wso2.com >>>>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>>>> lean.enterprise.middleware >>>>> >>>>> mobile: *+94728671315 <%2B94728671315>* >>>>> >>>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> *Anjana Fernando* >> Senior Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > Regards, > > Dunith Dhanushka, > Senior Software Engineer > WSO2 Inc, > > Mobile - +94 71 8615744 > Blog - dunithd.wordpress.com <http://blog.dunith.com> > Twitter - @dunithd <http://twitter.com/dunithd> > -- ============================ Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
