Hi, As an immediate fix I set the LAST_INDEX value to 1000 so that a user won't accidentally load all available data in a table. I aslo modified the way we request data from backend by introducing a boolean flag called 'dataLoaded'. This flag will be set to true only when a data loading operation gets completed. Until then no data request will be sent to backend.
In the meantime I'll look forward to improve this from Viz side as Srinath suggested. Thanks, Dunith On Sat, Aug 29, 2015 at 12:01 AM, Anjana Fernando <[email protected]> wrote: > Hi, > > On Fri, Aug 28, 2015 at 9:18 PM, Srinath Perera <[email protected]> wrote: > >> Correct approch (for Viz) would be to sample it down rather than taking >> first 1000 ( but not for DAL data retrival case). Any chance we can do that? >> > > That also, should be done case by case for graph types I guess, for > example, it doesn't make sense to sample data for a bar graphs, like some > bars won't be visible! .. > > Cheers, > Anjana. > > >> >> On Fri, Aug 28, 2015 at 7:12 PM, Dunith Dhanushka <[email protected]> >> wrote: >> >>> Hi Anjana, >>> >>> I'll work on the above change. >>> >>> Regards, >>> Dunith >>> >>> On Fri, Aug 28, 2015 at 8:22 AM, Anjana Fernando <[email protected]> >>> wrote: >>> >>>> Hi Srinath, >>>> >>>> I was checking on this, and it seems, the issue is, if you've a large >>>> result set, and for the gadget, if you select this large result set as its >>>> source data, it will go and read everything. For example, if you use the >>>> smart home sample and just put all the raw records as the X axis of a bar >>>> chart, it will try to read all thousands of data records and try to render >>>> it. But this operation takes a bit of time, to do the JSON serialization >>>> and send it to the front end. And what makes this far more worse is, the >>>> dashboard automatically refreshes the data, and this happens in an >>>> asynchronous manner, and it doesn't wait for the earlier cycle to be >>>> finished. So these requests effectively builds up and eats up the CPU in >>>> the backend and can end up going OOM. >>>> >>>> So I guess, first, we have to fix the refreshing of the gadgets >>>> somehow, so it will wait for the earlier request to finish, before making >>>> new ones. Then after that, we should also provide a sensible upper limit to >>>> the data records queries, so people won't mistakenly query millions of >>>> records from a raw data table. We have to do this carefully, because for a >>>> thing like a scatter plot, we may need to read thousands of records to >>>> render the image. So for things like that, it maybe easier for it to be >>>> rendered in the backend and sent to the front end. And also, Maninda also >>>> pointed out, since we have a natural refresh interval for a dashboard, we >>>> should also cache the data in the end backend, with a cache validity period >>>> similar to the dashboard interval, so we don't have to read data from the >>>> physical databases all the time. >>>> >>>> So as an immediate fix, shall we try to change the dashboard refresh >>>> logic and also put a upper limit to the backend data retrieval to about >>>> 1000 (by changing "LAST_INDEX" variable from -1 to 100 in analytics.jag at >>>> carbon-dashboards). @Dunith, will you be able to do these changes. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>> On Fri, Aug 28, 2015 at 3:07 PM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi Srinath, >>>>> >>>>> Yeah, I also could reproduce the issue. Will check it out. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> On Fri, Aug 28, 2015 at 2:38 PM, Srinath Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> This was a batch chart. >>>>>> >>>>>> On Fri, Aug 28, 2015 at 2:32 PM, Damith Wickramasinghe < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> AFAIK That message "Data is not available for plotting. The chart >>>>>>> will be loaded once the dashboard receives events" normally comes for >>>>>>> real >>>>>>> time flow gadgets. >>>>>>> >>>>>>> Regards, >>>>>>> Damith. >>>>>>> >>>>>>> On Fri, Aug 28, 2015 at 2:22 PM, Srinath Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, I could see the preview ok. >>>>>>>> >>>>>>>> --Srinath >>>>>>>> >>>>>>>> On Fri, Aug 28, 2015 at 2:19 PM, Anjana Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Srinath, >>>>>>>>> >>>>>>>>> Did you have data in that table before you created the gadget? .. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> On Fri, Aug 28, 2015 at 2:16 PM, Srinath Perera <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It is fixed with new pack. >>>>>>>>>> >>>>>>>>>> However, when I create a Gadget ( based on beatch data) and add >>>>>>>>>> teh Gadget, it says "Data is not available for plotting. The chart >>>>>>>>>> will be >>>>>>>>>> loaded once the dashboard receives events". >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Srinath >>>>>>>>>> >>>>>>>>>> On Fri, Aug 28, 2015 at 8:58 AM, Srinath Perera <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Yesterday packs had the same problem >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 27, 2015 at 4:20 PM, Maninda Edirisooriya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> This was fixed in https://wso2.org/jira/browse/DAS-84 on >>>>>>>>>>>> yesterday by Dunith. Now the fix should be there in latest jenkins >>>>>>>>>>>> built >>>>>>>>>>>> packs. >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>> >>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>> >>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Aug 27, 2015 at 4:16 PM, Srinath Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> is this fixed? >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Aug 25, 2015 at 4:41 PM, Maninda Edirisooriya < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Aug 25, 2015 at 3:28 PM, Srinath Perera < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Using yesterday's pack >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Can we make creating a receiver part of the creating/ >>>>>>>>>>>>>>> editing new streams flow? e.g. by asking what transports to >>>>>>>>>>>>>>> expose and >>>>>>>>>>>>>>> automatically creating it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. In the gadget generation wizard, x, y axis drop downs are >>>>>>>>>>>>>>> now not be populated. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes this issue was observed in latest packs. I am looking at >>>>>>>>>>>>>> it. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3. When you go into Gadget design view, there is no button >>>>>>>>>>>>>>> to come back. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Srinath >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ============================ >>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>> Phone: 0772360902 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected] >>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Software Engineer >>>>>>> WSO2 Inc.; http://wso2.com >>>>>>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>>>>>> lean.enterprise.middleware >>>>>>> >>>>>>> mobile: *+94728671315 <%2B94728671315>* >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>> Site: http://people.apache.org/~hemapani/ >>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>> Phone: 0772360902 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Senior Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> Dunith Dhanushka, >>> Senior Software Engineer >>> WSO2 Inc, >>> >>> Mobile - +94 71 8615744 >>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>> Twitter - @dunithd <http://twitter.com/dunithd> >>> >> >> >> >> -- >> ============================ >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- Regards, Dunith Dhanushka, Senior Software Engineer WSO2 Inc, Mobile - +94 71 8615744 Blog - dunithd.wordpress.com <http://blog.dunith.com> Twitter - @dunithd <http://twitter.com/dunithd>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
