Hi Alex - I agree that IDSR requires "immediate" response - but I we are usually talking hours, not minutes, right?
Knut On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye <atumwesi...@gmail.com> wrote: > Jason/Morten, > > I agree that real time analytics would not be easily implemented. May be > it is time to separate Aggregate and Tracker analytics as a start but it > also depends on what the tracker is being used for e.g surveys may require > no immediate analytics but IDSR/outbreaks may require instant update of > analytics. We are looking at IDSR features and as I see, real time > analytics will be a requirement. The IDSR requires real time analytics > since people need to respond to outbreaks and handle outbreak responses and > management in real time so that you can intervene and stop the outbreak. > > Here is what I propose (just a thought from my discussion with Calle) > We set a configurable (checked) variable/attribute that indicates that > if this variable is changed, the analytics process is started for example > changing population data has a very big impact on indicators that depend on > population,entering lab results or requests or patient updates for IDSR > requires immediate analytics, etc. If we had this attribute/variable, then > we would use it to identify the corresponding meta data that might need to > be / may have changed and update (through temp tables) the only affected > analytics tables. This way we can control the load required to run > analytics in realtime since the system will only be updating affected > changes. > Using the API to trigger analytics every after X minutes may be feasible > but not sustainable as we do not control how many threads may be running as > the API call through curl does not easily get feedback/update if the > previous analytics process has been completed for the new one to start > otherwise we can end up in a forever loop if the server resources are not > enough. > > Alex > > > > On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen <mor...@dhis2.org> > wrote: > >> 1 ns is fine... but i depends on what the ramifications are.. what if you >> start a new job every 1 ms? is that ok? what happens during table swap? is >> analytic blocked? >> >> Maybe this is already documented.. just curious :) >> >> -- >> Morten Olav Hansen >> Senior Engineer, DHIS 2 >> University of Oslo >> http://www.dhis2.org >> >> On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering < >> jason.p.picker...@gmail.com> wrote: >> >>> Why not ever 1 nanosecond? Its always going to take some amount of time, >>> question is what is reasonable. >>> >>> One could argue that real-time analytics in an aggregate data system is >>> not needed. This is not real time data. Its not even close to it. >>> Look at Google Analytics. They provide reports once a day, and you do >>> not see a whole lot of people complaning. Yes, you can get some limited >>> real time information from this as well, but its limited. The data must be >>> processed first, and that takes computational time. Same with DHIS2. >>> >>> From my experience, people think they need "real time analytics" when >>> they really are just in a rush. Data takes time to review and analayze and >>> whether its available now, 1 nanosecond from now, or 10 minutes from now, >>> makes no difference in the end, as the amount of time which is required to >>> digest that information is on a totally different time scale (hours days or >>> weeks). Once an hour is probably easily achievable depending on the scale >>> of the system however. >>> >>> As for the call to the API, just create a Bash script and call it as >>> frequently as you like with a cron task. >>> >>> This is a very simple one, but you should really check for things like >>> "Is analytics already running and should I trigger another run?" >>> >>> #!/bin/sh >>> >>> /usr/bin/curl >>> "localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2" >>> -X POST -u admin:district >/dev/null 2>&1 >>> >>> >>> >>> >>> On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen <mor...@dhis2.org> >>> wrote: >>> >>>> Well, it sounds like a bad solution for me ;) if every 10 min works >>>> fine.. why not every 5 min.. why not every 1 min.. >>>> >>>> -- >>>> Morten Olav Hansen >>>> Senior Engineer, DHIS 2 >>>> University of Oslo >>>> http://www.dhis2.org >>>> >>>> On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh < >>>> ibrahim.ba...@sl.ehealthafrica.org> wrote: >>>> >>>>> @Jason,Knut and Morten, Having analytics table run every 10 mins >>>>> sounds like a good place to start. but am kind of not sure how to >>>>> implement >>>>> this with the API calls and am sure this is not possible through the user >>>>> interface. If you guys can point me in the right direction with an example >>>>> of some sort that will greatly helpful and highly appreciated. >>>>> >>>>> Thanks. >>>>> >>>>> On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering < >>>>> jason.p.picker...@gmail.com> wrote: >>>>> >>>>>> No they are swapped now, so first they are built as temp tables, then >>>>>> swapped. So, this is the brief point in time in which things may not be >>>>>> available. >>>>>> >>>>>> Every 10 minutes might be OK, depending on your server, loading, and >>>>>> amount of data. It just requires some experimentation. >>>>>> >>>>>> Point is, "real time" analytics is not possible. Near-real time may >>>>>> be. >>>>>> >>>>>> On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen <mor...@dhis2.org >>>>>> > wrote: >>>>>> >>>>>>> But I thought all analytic tables was cleared out during >>>>>>> re-generation? is this not true? >>>>>>> >>>>>>> -- >>>>>>> Morten Olav Hansen >>>>>>> Senior Engineer, DHIS 2 >>>>>>> University of Oslo >>>>>>> http://www.dhis2.org >>>>>>> >>>>>>> On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring <knu...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Depends on the data. I trigger analytics every 10 min. >>>>>>>> Hi All, >>>>>>>> Initially i thought DHIS2 had fully or out-of-the-box realtime >>>>>>>> analytics and Dashboards. But after further inspection i realized that >>>>>>>> current data entry will not available for analysis or dashboards until >>>>>>>> the >>>>>>>> next day or by manually doing export tables. This is a real bottle >>>>>>>> neck to >>>>>>>> the intended implementation am working on. I have looked at >>>>>>>> Scheduling, but >>>>>>>> the options for analytics table is not ideal in my case. *Is there >>>>>>>> way this can be reduced or removed to gain real-time dashboards and >>>>>>>> analytics?* >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Bayoh. >>>>>>>> >>>>>>>> -- >>>>>>>> Ibrahim Rashid Bayoh >>>>>>>> Information Systems Coordinator, >>>>>>>> eHealth Africa(Sierra Leone) >>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone* >>>>>>>> Mobile: +232 88-765-638 >>>>>>>> ibrahim.ba...@sl.ehealthafrica.org >>>>>>>> http://ehealthafrica.org/ >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: https://launchpad.net/~dhis2-users >>>>>>>> Post to : dhis2-users@lists.launchpad.net >>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users >>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: https://launchpad.net/~dhis2-users >>>>>>>> Post to : dhis2-users@lists.launchpad.net >>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users >>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mailing list: https://launchpad.net/~dhis2-users >>>>>>> Post to : dhis2-users@lists.launchpad.net >>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users >>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jason P. Pickering >>>>>> email: jason.p.picker...@gmail.com >>>>>> tel:+46764147049 >>>>>> >>>>>> _______________________________________________ >>>>>> Mailing list: https://launchpad.net/~dhis2-users >>>>>> Post to : dhis2-users@lists.launchpad.net >>>>>> Unsubscribe : https://launchpad.net/~dhis2-users >>>>>> More help : https://help.launchpad.net/ListHelp >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Ibrahim Rashid Bayoh >>>>> Information Systems Coordinator, >>>>> eHealth Africa(Sierra Leone) >>>>> *117 Wilkinson Rd, Freetown, Sierra Leone* >>>>> Mobile: +232 88-765-638 >>>>> ibrahim.ba...@sl.ehealthafrica.org >>>>> http://ehealthafrica.org/ >>>>> >>>>> >>>> >>> >>> >>> -- >>> Jason P. Pickering >>> email: jason.p.picker...@gmail.com >>> tel:+46764147049 >>> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-users >> Post to : dhis2-users@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~dhis2-users >> More help : https://help.launchpad.net/ListHelp >> >> > > > -- > Alex Tumwesigye > > Technical Advisor - DHIS2 (Consultant), > Ministry of Health/AFENET > Kampala > Uganda > +256 774149 775, + 256 759 800161 > Skype ID: talexie > > IT Consultant (Servers, Networks and Security, Health Information Systems > - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant > > > "I don't want to be anything other than what I have been - one tree hill " > > _______________________________________________ > Mailing list: https://launchpad.net/~dhis2-users > Post to : dhis2-users@lists.launchpad.net > Unsubscribe : https://launchpad.net/~dhis2-users > More help : https://help.launchpad.net/ListHelp > > -- Knut Staring Dept. of Informatics, University of Oslo Norway: +4791880522 Skype: knutstar http://dhis2.org
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-users Post to : dhis2-users@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-users More help : https://help.launchpad.net/ListHelp
