Matt, That's true, but you do not need it at all: Take the world, splice it in polygons (according to the density of data availably and resolution needed); label you polygons, and get your data for each polygon with the label consisting of Where:What, with where being the label of the specific geo-area according to your above system, and what the label for what kind of data you push (like seismic etc.). And there you have your data format: Label to scalar! Now the htmengine outputs you anomaly scores for each Label Where:What and you take these to hierarchically (in a geo-hierarchie) build logistic regression models, trained by the anomaly output, and a binary value for whether a certain disaster happened there at a time X later time or not. (This needs some past data which is why the highest priority is getting the data polled and htmengine trained). You go for logistic regression because that is what literature finds to perform best. Now when that works, you have your 'live' data stream and get predictions in the form of probabilities for the disaster occurring X time in the future...
This was the basic idea.. of course you will need to test it and refine the architecture etc. But you got your work-around :) So htmengine is not supposed to do the entire job. its more for feature detection :) The problem researchers find when building log-reg models with real data (raw scalars of the sensors) is that they periodically make wrong predictions due to daily etc. patterns. This is what HTM should filter out ;) The point of using tuarus as a starter therefore is that you already have your basic infrastructure of companies (your geo-polygons) and different metrics (the different sensor data in that region).. Does it make more sense now? :) Of course a geoencoder and so would be nice in addition to capture more of the patterns, but this is what I would hope to achieve with the geo-hierarchy of log-reg models so they capture the spatial relationships in their input weights (of course only based on historical data)... I do not think the geoEncoder Would get this as well.. When running the demo_app, you find that the geoendoding with radius=Magnitude or any exponential function thereof makes HTM immune to regions where at least one strong quake happend... and you dont want that. but David, you may think about building a engine for java as well :) Just cause its faster ;D On Tue, Aug 4, 2015 at 2:09 AM, Matthew Taylor <[email protected]> wrote: > HTM Engine refers specifically to tools at > https://github.com/numenta/numenta-apps/tree/master/htmengine. It > allows you to run a bunch of models at once and scale upwards more > easily. It does not refer to the Network API, which is much further > down in the stack. > --------- > Matt Taylor > OS Community Flag-Bearer > Numenta > > > On Mon, Aug 3, 2015 at 5:01 PM, cogmission (David Ray) > <[email protected]> wrote: > > Pascal, Matt, > > > > I might have this wrong, but HTM Engine refers to the Network API no? > > HTM.java can process lat/long data in its Network API. If there is no > other > > criteria, perhaps HTM.java could be an alternative? > > > > On Mon, Aug 3, 2015 at 5:42 PM, Matthew Taylor <[email protected]> wrote: > >> > >> Hi Pascal, > >> > >> First of all, thanks for all your interest in NuPIC and HTM. I can > >> tell you're really excited about this stuff and I love that kind of > >> energy in our community! > >> > >> Taurus was established as a scalar anomaly detection system, not a > >> geospatial anomaly detection system. Because all of its data displays > >> are tuned towards scalar charts and graphs, I'm not sure how much help > >> it will be. > >> > >> This leads me to the next point.... HTM Engine was also build for > >> scalar anomaly detection, not geospatial :(. The pretty much means > >> that you will not be able to pass HTM Engine "timestamp / lat / long" > >> events. It currently only works for "timestamp / scalar value" input. > >> > >> But, this doesn't mean things are hopeless. IMO there are many great > >> opportunities to improve the interface of HTM Engine. There are > >> several things I wish that it did: > >> > >> - prediction > >> - multiple input fields > >> - geospatial encoding > >> > >> Just to name a few. If these are features that others want to see in > >> HTM engine, please file a ticket so we can prioritize our work: > >> https://github.com/numenta/numenta-apps/issues/new > >> > >> Regards, > >> > >> --------- > >> Matt Taylor > >> OS Community Flag-Bearer > >> Numenta > >> > >> > >> On Mon, Aug 3, 2015 at 2:14 PM, Pascal Weinberger > >> <[email protected]> wrote: > >> > Hey All! > >> > > >> > For the htmengine infrastructure for nostradamIQ, I was trying to > >> > understand > >> > Taurus and use it as a template to build on, as it already has > multiple > >> > metrics (stock and twitter) for multiple classes (companies). This is > >> > what > >> > would be appropriate for nostradamIQ as well... e.g. streaming > multiple > >> > metrics like air pressure etc. for multiple geofences. > >> > > >> > This is at least the idea... to get anomaly scores for each metric in > >> > each > >> > geolocation plus one HTM anomaly model on top of it all (to capture > >> > geophysical correlations) and then build a logistic-regression model > >> > combining the anomalies to evaluate the likelihood for different > natural > >> > disasters for each region ... > >> > > >> > Digging into Taurus, I learn that this is a bit too much for me > alone... > >> > I still think though that this is a really valuable project and feel > >> > like it > >> > could make a good contribution to our general safety and preparedness. > >> > > >> > My question now is two-fold: > >> > a) Do you think this is a valid approach? Do you have ideas? concerns? > >> > b) Would any more experienced programmer // engineer like to join and > >> > work > >> > on this? Anyone from Numenta? Take Grok to the next level? :) > >> > > >> > This would really help, as I am not experienced enough to build this > >> > complex > >> > architecture quickly alone and the earlier it stands and learns, the > >> > quicker > >> > we can all benefit from it. > >> > > >> > > >> > Thank you for any help! > >> > > >> > --Pascal > >> > > > > > > > > -- > > With kind regards, > > > > David Ray > > Java Solutions Architect > > > > Cortical.io > > Sponsor of: HTM.java > > > > [email protected] > > http://cortical.io > >
