Hi Anjana, Even for a bar graph, it is OK to randomly sample if it is bigger than some size (e.g. 10000) with a warning that data has been sampled. That is the best we can do, and it is much better than going OOM.
Thanks Srinath On Sat, Aug 29, 2015 at 10:31 AM, Anjana Fernando <[email protected]> wrote: > Hi, > > On Fri, Aug 28, 2015 at 9:18 PM, Srinath Perera <[email protected]> wrote: > >> Correct approch (for Viz) would be to sample it down rather than taking >> first 1000 ( but not for DAL data retrival case). Any chance we can do that? >> > > That also, should be done case by case for graph types I guess, for > example, it doesn't make sense to sample data for a bar graphs, like some > bars won't be visible! .. > > Cheers, > Anjana. > > >> >> On Fri, Aug 28, 2015 at 7:12 PM, Dunith Dhanushka <[email protected]> >> wrote: >> >>> Hi Anjana, >>> >>> I'll work on the above change. >>> >>> Regards, >>> Dunith >>> >>> On Fri, Aug 28, 2015 at 8:22 AM, Anjana Fernando <[email protected]> >>> wrote: >>> >>>> Hi Srinath, >>>> >>>> I was checking on this, and it seems, the issue is, if you've a large >>>> result set, and for the gadget, if you select this large result set as its >>>> source data, it will go and read everything. For example, if you use the >>>> smart home sample and just put all the raw records as the X axis of a bar >>>> chart, it will try to read all thousands of data records and try to render >>>> it. But this operation takes a bit of time, to do the JSON serialization >>>> and send it to the front end. And what makes this far more worse is, the >>>> dashboard automatically refreshes the data, and this happens in an >>>> asynchronous manner, and it doesn't wait for the earlier cycle to be >>>> finished. So these requests effectively builds up and eats up the CPU in >>>> the backend and can end up going OOM. >>>> >>>> So I guess, first, we have to fix the refreshing of the gadgets >>>> somehow, so it will wait for the earlier request to finish, before making >>>> new ones. Then after that, we should also provide a sensible upper limit to >>>> the data records queries, so people won't mistakenly query millions of >>>> records from a raw data table. We have to do this carefully, because for a >>>> thing like a scatter plot, we may need to read thousands of records to >>>> render the image. So for things like that, it maybe easier for it to be >>>> rendered in the backend and sent to the front end. And also, Maninda also >>>> pointed out, since we have a natural refresh interval for a dashboard, we >>>> should also cache the data in the end backend, with a cache validity period >>>> similar to the dashboard interval, so we don't have to read data from the >>>> physical databases all the time. >>>> >>>> So as an immediate fix, shall we try to change the dashboard refresh >>>> logic and also put a upper limit to the backend data retrieval to about >>>> 1000 (by changing "LAST_INDEX" variable from -1 to 100 in analytics.jag at >>>> carbon-dashboards). @Dunith, will you be able to do these changes. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>> On Fri, Aug 28, 2015 at 3:07 PM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Srinath, >>>>> >>>>> Yeah, I also could reproduce the issue. Will check it out. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> On Fri, Aug 28, 2015 at 2:38 PM, Srinath Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> This was a batch chart. >>>>>> >>>>>> On Fri, Aug 28, 2015 at 2:32 PM, Damith Wickramasinghe < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> AFAIK That message "Data is not available for plotting. The chart >>>>>>> will be loaded once the dashboard receives events" normally comes for >>>>>>> real >>>>>>> time flow gadgets. >>>>>>> >>>>>>> Regards, >>>>>>> Damith. >>>>>>> >>>>>>> On Fri, Aug 28, 2015 at 2:22 PM, Srinath Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, I could see the preview ok. >>>>>>>> >>>>>>>> --Srinath >>>>>>>> >>>>>>>> On Fri, Aug 28, 2015 at 2:19 PM, Anjana Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Srinath, >>>>>>>>> >>>>>>>>> Did you have data in that table before you created the gadget? .. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> On Fri, Aug 28, 2015 at 2:16 PM, Srinath Perera <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It is fixed with new pack. >>>>>>>>>> >>>>>>>>>> However, when I create a Gadget ( based on beatch data) and add >>>>>>>>>> teh Gadget, it says "Data is not available for plotting. The chart >>>>>>>>>> will be >>>>>>>>>> loaded once the dashboard receives events". >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Srinath >>>>>>>>>> >>>>>>>>>> On Fri, Aug 28, 2015 at 8:58 AM, Srinath Perera <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Yesterday packs had the same problem >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 27, 2015 at 4:20 PM, Maninda Edirisooriya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> This was fixed in https://wso2.org/jira/browse/DAS-84 on >>>>>>>>>>>> yesterday by Dunith. Now the fix should be there in latest jenkins >>>>>>>>>>>> built >>>>>>>>>>>> packs. >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Aug 27, 2015 at 4:16 PM, Srinath Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> is this fixed? >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Aug 25, 2015 at 4:41 PM, Maninda Edirisooriya < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Aug 25, 2015 at 3:28 PM, Srinath Perera < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Using yesterday's pack >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Can we make creating a receiver part of the creating/ >>>>>>>>>>>>>>> editing new streams flow? e.g. by asking what transports to >>>>>>>>>>>>>>> expose and >>>>>>>>>>>>>>> automatically creating it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. In the gadget generation wizard, x, y axis drop downs are >>>>>>>>>>>>>>> now not be populated. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes this issue was observed in latest packs. I am looking at >>>>>>>>>>>>>> it. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3. When you go into Gadget design view, there is no button >>>>>>>>>>>>>>> to come back. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Srinath >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ============================ >>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>> Phone: 0772360902 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Software Engineer >>>>>>> WSO2 Inc.; http://wso2.com >>>>>>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> mobile: *+94728671315 <%2B94728671315>* >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>> Site: http://people.apache.org/~hemapani/ >>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>> Phone: 0772360902 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Senior Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> Dunith Dhanushka, >>> Senior Software Engineer >>> WSO2 Inc, >>> >>> Mobile - +94 71 8615744 >>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>> Twitter - @dunithd <http://twitter.com/dunithd> >>> >> >> >> >> -- >> ============================ >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- ============================ Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
