n Mon, Aug 31, 2015 at 9:47 AM, Srinath Perera <[email protected]> wrote:
> Let's chat about this f2f. I guess we need an upper limit for sampling as > well. > +1. Cheers, Anjana. > > Thanks > Srinath > > On Mon, Aug 31, 2015 at 9:35 AM, Anjana Fernando <[email protected]> wrote: > >> Hi Srinath, >> >> I see. Then we have to think how to do the sampling efficiently, even in >> the backend, we don't want millions of data to be read for the sampling >> process. >> >> Cheers, >> Anjana. >> >> On Mon, Aug 31, 2015 at 7:33 AM, Srinath Perera <[email protected]> wrote: >> >>> Hi Anjana, >>> >>> Even for a bar graph, it is OK to randomly sample if it is bigger than >>> some size (e.g. 10000) with a warning that data has been sampled. That is >>> the best we can do, and it is much better than going OOM. >>> >>> Thanks >>> Srinath >>> >>> On Sat, Aug 29, 2015 at 10:31 AM, Anjana Fernando <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> On Fri, Aug 28, 2015 at 9:18 PM, Srinath Perera <[email protected]> >>>> wrote: >>>> >>>>> Correct approch (for Viz) would be to sample it down rather than >>>>> taking first 1000 ( but not for DAL data retrival case). Any chance we can >>>>> do that? >>>>> >>>> >>>> That also, should be done case by case for graph types I guess, for >>>> example, it doesn't make sense to sample data for a bar graphs, like some >>>> bars won't be visible! .. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>>> >>>>> On Fri, Aug 28, 2015 at 7:12 PM, Dunith Dhanushka <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Anjana, >>>>>> >>>>>> I'll work on the above change. >>>>>> >>>>>> Regards, >>>>>> Dunith >>>>>> >>>>>> On Fri, Aug 28, 2015 at 8:22 AM, Anjana Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Srinath, >>>>>>> >>>>>>> I was checking on this, and it seems, the issue is, if you've a >>>>>>> large result set, and for the gadget, if you select this large result >>>>>>> set >>>>>>> as its source data, it will go and read everything. For example, if you >>>>>>> use >>>>>>> the smart home sample and just put all the raw records as the X axis of >>>>>>> a >>>>>>> bar chart, it will try to read all thousands of data records and try to >>>>>>> render it. But this operation takes a bit of time, to do the JSON >>>>>>> serialization and send it to the front end. And what makes this far more >>>>>>> worse is, the dashboard automatically refreshes the data, and this >>>>>>> happens >>>>>>> in an asynchronous manner, and it doesn't wait for the earlier cycle to >>>>>>> be >>>>>>> finished. So these requests effectively builds up and eats up the CPU in >>>>>>> the backend and can end up going OOM. >>>>>>> >>>>>>> So I guess, first, we have to fix the refreshing of the gadgets >>>>>>> somehow, so it will wait for the earlier request to finish, before >>>>>>> making >>>>>>> new ones. Then after that, we should also provide a sensible upper >>>>>>> limit to >>>>>>> the data records queries, so people won't mistakenly query millions of >>>>>>> records from a raw data table. We have to do this carefully, because >>>>>>> for a >>>>>>> thing like a scatter plot, we may need to read thousands of records to >>>>>>> render the image. So for things like that, it maybe easier for it to be >>>>>>> rendered in the backend and sent to the front end. And also, Maninda >>>>>>> also >>>>>>> pointed out, since we have a natural refresh interval for a dashboard, >>>>>>> we >>>>>>> should also cache the data in the end backend, with a cache validity >>>>>>> period >>>>>>> similar to the dashboard interval, so we don't have to read data from >>>>>>> the >>>>>>> physical databases all the time. >>>>>>> >>>>>>> So as an immediate fix, shall we try to change the dashboard refresh >>>>>>> logic and also put a upper limit to the backend data retrieval to about >>>>>>> 1000 (by changing "LAST_INDEX" variable from -1 to 100 in analytics.jag >>>>>>> at >>>>>>> carbon-dashboards). @Dunith, will you be able to do these changes. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> >>>>>>> On Fri, Aug 28, 2015 at 3:07 PM, Anjana Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Srinath, >>>>>>>> >>>>>>>> Yeah, I also could reproduce the issue. Will check it out. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> >>>>>>>> On Fri, Aug 28, 2015 at 2:38 PM, Srinath Perera <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> This was a batch chart. >>>>>>>>> >>>>>>>>> On Fri, Aug 28, 2015 at 2:32 PM, Damith Wickramasinghe < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> AFAIK That message "Data is not available for plotting. The chart >>>>>>>>>> will be loaded once the dashboard receives events" normally comes >>>>>>>>>> for real >>>>>>>>>> time flow gadgets. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Damith. >>>>>>>>>> >>>>>>>>>> On Fri, Aug 28, 2015 at 2:22 PM, Srinath Perera <[email protected] >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Yes, I could see the preview ok. >>>>>>>>>>> >>>>>>>>>>> --Srinath >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 28, 2015 at 2:19 PM, Anjana Fernando < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Srinath, >>>>>>>>>>>> >>>>>>>>>>>> Did you have data in that table before you created the gadget? >>>>>>>>>>>> .. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Anjana. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 28, 2015 at 2:16 PM, Srinath Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> It is fixed with new pack. >>>>>>>>>>>>> >>>>>>>>>>>>> However, when I create a Gadget ( based on beatch data) and >>>>>>>>>>>>> add teh Gadget, it says "Data is not available for plotting. The >>>>>>>>>>>>> chart will >>>>>>>>>>>>> be loaded once the dashboard receives events". >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Srinath >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Aug 28, 2015 at 8:58 AM, Srinath Perera < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yesterday packs had the same problem >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Aug 27, 2015 at 4:20 PM, Maninda Edirisooriya < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> This was fixed in https://wso2.org/jira/browse/DAS-84 on >>>>>>>>>>>>>>> yesterday by Dunith. Now the fix should be there in latest >>>>>>>>>>>>>>> jenkins built >>>>>>>>>>>>>>> packs. >>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>>> *E-mail* : [email protected] >>>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Aug 27, 2015 at 4:16 PM, Srinath Perera < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> is this fixed? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Aug 25, 2015 at 4:41 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Aug 25, 2015 at 3:28 PM, Srinath Perera < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Using yesterday's pack >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. Can we make creating a receiver part of the creating/ >>>>>>>>>>>>>>>>>> editing new streams flow? e.g. by asking what transports to >>>>>>>>>>>>>>>>>> expose and >>>>>>>>>>>>>>>>>> automatically creating it. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. In the gadget generation wizard, x, y axis drop downs >>>>>>>>>>>>>>>>>> are now not be populated. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes this issue was observed in latest packs. I am looking >>>>>>>>>>>>>>>>> at it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 3. When you go into Gadget design view, there is no >>>>>>>>>>>>>>>>>> button to come back. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>> Srinath >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ============================ >>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> >>>>>>>>>> mobile: *+94728671315 <%2B94728671315>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ============================ >>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>> Phone: 0772360902 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> Dunith Dhanushka, >>>>>> Senior Software Engineer >>>>>> WSO2 Inc, >>>>>> >>>>>> Mobile - +94 71 8615744 >>>>>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>>>>> Twitter - @dunithd <http://twitter.com/dunithd> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> ============================ >>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>> Site: http://people.apache.org/~hemapani/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> Phone: 0772360902 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> ============================ >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://people.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> *Anjana Fernando* >> Senior Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > ============================ > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
