Comments inline...
On Sun, May 25, 2014 at 6:21 AM, Stephen Crane <[email protected]> wrote:
> Hi,
> I have some time-series data from an environmental sensor that I'm
> interested in using to learn about NuPIC.
>
> It contains readings of light, humidity and temperature every 45-90s
> (depending on light level). There is a daily as well as a weekly period to
> this data. Here are the first few lines of this file:
>
> temperature,humidity,light,dttm
> float,float,int,datetime
> ,,,T
> 16.8,64.2,0,2014-05-10 00:00:16
> 16.8,64.3,0,2014-05-10 00:01:48
> 16.8,64.3,0,2014-05-10 00:03:21
> 16.8,64.2,0,2014-05-10 00:04:53
> 16.8,64.1,0,2014-05-10 00:06:26
> 16.8,64.4,0,2014-05-10 00:07:58
>
> I've attached a graph of the light level from this sensor from a typical 24
> hour period recently.
>
> Since there are about 10000 minutes in a week, what is the best way to
> approach this data?
I think you're going to have to experiment a bit. Looks like the data
doesn't get recorded at regular intervals. This should not matter, but
most of the experiments and tests I've seen with NuPIC have used
regular intervals, so I don't have direct experience with sending in
irregular intervals. If you set up aggregation, it should normalize
the input data before sending to NuPIC anyway.
> Aggregation will reduce the amount of data to be swarmed
> over but will lose some signal. How is aggregation performed?
The aggregator [1] has decent code docs, so I just created a PR to add
it to the automated documentation [2]. Hopefully this helps a bit.
>From the code docs:
"""
It supports aggregation of multiple records based on time.
Common use cases:
- Aggregate records by month
- Aggregate records every 3 months starting April 15th
- Aggregate records in 2.5 seconds intervals
Assumption: aggregated slices fit in memory. All the records that are aggregated
per period are stored in memory until the next slice starts and are only
aggregated then. If this assumption is too strong the script will need to write
slices to a temp storage or use incremental aggregation techniques.
"""
There is an example in the Running-Swarms wiki [3] of setting up
temporal aggregation that looks like this:
"""
"aggregation": {
"hours": 1,
"microseconds": 0,
"seconds": 0,
"fields": [
[
"timestamp",
"first"
]
],
"weeks": 0,
"months": 0,
"minutes": 0,
"days": 0,
"milliseconds": 0,
"years": 0
}
"""
This specifies that input will be chunked into 1-hour periods, each
aggregate row using the "first" timestamp value found within that
period. I assume that the values within the period are averaged across
it, but not positive.
> (I have just noticed that I haven't provided aggregation functions for any
> fields besides dttm. What does NuPIC do by default? What choices of
> aggregation function are available?)
I believe you can only aggregate over time. By default, NuPIC does no
aggregation. After looking at the code, it looks like the aggregation
functions are "first", "last", "mean" "weighted_mean", and "mode" [4].
> I have run a swarm over 5000 records and 10 minute aggregation using the
> attached search_def.json file.
Looks right to me.
> It produced the attached model_params.py
> file. Can I deduce from the sensorParams section that the swarm detected no
> time-of-day-related component to the light level but it thinks day-of-week
> is significant (even though it's only seen 2.5 days worth)?
Does seem a bit curious. But might be true if it sees that there is a
difference between the patterns within the days you gave it, more so
than the patterns of time-of-day. Do you have more data you could
swarm over? It might be hard for it to pick up the time-of-day
patterns when it has only seen 2.5 days.
>
> At this point, should I just edit the model_params.py file to encode my
> prior knowledge of the periodicity of the data, or is this pointless?
First, I would see if you can swarm over more data to get better model
params. If you think you have daily patterns, it needs to see more
than just a couple of days to pick up the pattern. It must have
noticed that there was a difference between the days within the data
you sent to use a day-of-week encoder.
[1] https://github.com/numenta/nupic/blob/master/py/nupic/data/aggregator.py
[2] https://github.com/numenta/nupic/pull/960
[3] https://github.com/numenta/nupic/wiki/Running-Swarms#a-simple-example
[4]
https://github.com/numenta/nupic/blob/master/py/nupic/data/aggregator.py#L136..L232
Regards,
---------
Matt Taylor
OS Community Flag-Bearer
Numenta
_______________________________________________
nupic mailing list
[email protected]
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
