Hey Matt, The folder I sent you contains data from user 1 only. Didn’t wanna swamp you with all the data files I have. Please change the 2 references of User_2_dir_*.txt to User_1_dir_*.txt (FILE_NAME_TRAIN and FILE_NAME_TEST).
Hopefully, that gets it working. I’m here if you need anything else. Thanks. Nick > On Dec 29, 2014, at 10:47 PM, Matthew Taylor <[email protected]> wrote: > > Nick, > > No such file or directory: > '/Users/mtaylor/dev/mitri/data/User_2_dir_train.txt' > > I'm missing the user "2" data files. > --------- > Matt Taylor > OS Community Flag-Bearer > Numenta > > > On Mon, Dec 29, 2014 at 12:01 PM, Nicholas Mitri <[email protected]> wrote: >> Hey Matt, >> >> Please find attached a folder containing the main py file and a data folder >> containing a sample of data for one of the users. >> Also attached is the swarm file, the data log (concatenation from all users) >> that was used to create it, and the description file containing the model >> parameters (I’ve modified this to search for better parameter choices than >> those produced by swarming, in the current version I’ve disabled the SP to >> feed a large encoder output directly to the TP). >> >> It should run with no issues. Please feel free to email me if it doesn’t run >> out of the box. >> >> Note: the swarm chooses the adaptive encoder. In my testing, the better >> choice is a scalar encoder with periodic bool set to True since the >> direction values wrap around (min = 0, max = 16). >> >> Thanks, >> Nick >> >> >> >>> On Dec 29, 2014, at 9:24 PM, Matthew Taylor <[email protected]> wrote: >>> >>> Nicholas, >>> >>> I ran your code with the same results you had. If you have updated >>> code, please post it. >>> >>> After talking to Subutai about this, we think the input data you're >>> using to swarm over doesn't seem to represent the data in your >>> labelled matrix file. How are you generating the input CSV data file? >>> It could be that the swarming data is not close enough to the >>> real-world data within the matrix file. Swarming tries to give you the >>> model that is best for predicting the next field in the input data, so >>> it's important that the data the swarm uses closely represents the >>> data the model actually sees. I don't think this is the case here. >>> >>> One thing I'm trying right now is to pass the labeled data into each >>> model during the training phase 10 times instead of just once to >>> reinforce the patterns in each model. The program is running now, not >>> sure if it will help or not but I might as well try. >>> >>> --------- >>> Matt Taylor >>> OS Community Flag-Bearer >>> Numenta >>> >>> >>> On Mon, Dec 29, 2014 at 7:25 AM, Nicholas Mitri <[email protected]> wrote: >>>> Hey Matt, everyone, >>>> >>>> I debugged the code and managed to get some sensible results. HTM is doing >>>> a >>>> great job of learning sequences but performing very poorly at >>>> generalization. So while it can recognize a sequence it had learned with >>>> high accuracy, when it’s fed a test sequence that it’s never seen, its >>>> classification accuracy plummets. To be clear, classification here is >>>> performed by assigning an HTM region to each class and observing which >>>> region outputs the least anomaly score averaged along a test sequence. >>>> >>>> I’ve tried tweaking the encoder parameters to quantize the input with a >>>> lower resolution in the hope that similar inputs will be better pooled. >>>> That >>>> didn’t pan out. Also, changing encoder output length or number of columns >>>> is >>>> causing the HTM to output no predictions at times even with a non-empty >>>> active column list. I have little idea why that keeps happening. >>>> >>>> Any hints as to how to get HTM to better perform here? I’ve included HMM >>>> results for comparison. SVM results are all 95+%. >>>> >>>> Thank you, >>>> Nick >>>> >>>> >>>> HTM Results: >>>> >>>> Data = sequence of directions (8 discrete direction) >>>> Note on accuracy: M1/M2 is shown here to represent 2 performance metrics. >>>> M1 >>>> is average anomaly, M2 is the sum of average anomaly normalized and >>>> prediction error normalized. >>>> >>>> Base training accuracy: 100 % at 2 training passes >>>> >>>> User Dependent: 56.25%/56.25% >>>> >>>> User Independent: N/A >>>> >>>> Mixed: 65.00 %/ 71.25% >>>> >>>> >>>> HMM (22-states) Results: >>>> >>>> Data = sequence of directions (16 discrete direction) >>>> >>>> Base training accuracy: 97.5% >>>> >>>> User Dependent: 76.25 % >>>> >>>> User Independent: 88.75 % >>>> >>>> Mixed: 88.75 % >>>> >>>> >>>> On Dec 11, 2014, at 7:16 PM, Matthew Taylor <[email protected]> wrote: >>>> >>>> Nicholas, can you paste a sample of the input data file? >>>> >>>> --------- >>>> Matt Taylor >>>> OS Community Flag-Bearer >>>> Numenta >>>> >>>> On Thu, Dec 11, 2014 at 7:50 AM, Nicholas Mitri <[email protected]> >>>> wrote: >>>>> >>>>> Hey all, >>>>> >>>>> I’m running into some trouble with using HTM for a gesture recognition >>>>> application and would appreciate some help. >>>>> First, the data is collected from 17 users performing 5 gestures of each >>>>> of 16 different gesture classes using motion sensors. The feature vector >>>>> for >>>>> each sample is a sequence of discretized directions calculated using >>>>> bezier >>>>> control points after curve fitting the gesture trace. >>>>> >>>>> For a baseline, I fed the data to 16 10-state HMMs for training and again >>>>> for testing. The classification accuracy achieved is 95.7%. >>>>> >>>>> For HTM, I created 16 CLA models using parameters from a medium swarm. I >>>>> ran the data through the models for training where each model is trained >>>>> on >>>>> only 1 gesture class. For testing, I fed the same data again with learning >>>>> turned off and recorded the anomaly score (averaged across each sequence) >>>>> for each model. Classification was done by seeking the model with the >>>>> minimum anomaly score. Accuracy turned out to be a puzzling 0.0%!! >>>>> >>>>> Below is the relevant section of the code. I would appreciate any hints. >>>>> Thanks, >>>>> Nick >>>>> >>>>> def run_experiment(): >>>>> print "Running experiment..." >>>>> >>>>> model = [0]*16 >>>>> for i in range(0, 16): >>>>> model[i] = ModelFactory.create(model_params, logLevel=0) >>>>> model[i].enableInference({"predictedField": FIELD_NAME}) >>>>> >>>>> with open(FILE_PATH, "rb") as f: >>>>> csv_reader = csv.reader(f) >>>>> data = [] >>>>> labels = [] >>>>> for row in csv_reader: >>>>> r = [int(item) for item in row[:-1]] >>>>> data.append(r) >>>>> labels.append(int(row[-1])) >>>>> >>>>> # data_train, data_test, labels_train, labels_test = >>>>> cross_validation.train_test_split(data, labels, test_size=0.4, >>>>> random_state=0) >>>>> data_train = data >>>>> data_test = data >>>>> labels_train = labels >>>>> labels_test = labels >>>>> >>>>> for passes in range(0, TRAINING_PASSES): >>>>> sample = 0 >>>>> for (ind, row) in enumerate(data_train): >>>>> for r in row: >>>>> value = int(r) >>>>> result = model[labels_train[ind]].run({FIELD_NAME: value, >>>>> '_learning': True}) >>>>> prediction = >>>>> result.inferences["multiStepBestPredictions"][1] >>>>> anomalyScore = result.inferences["anomalyScore"] >>>>> model[labels[ind]].resetSequenceStates() >>>>> sample += 1 >>>>> print "Processing training sample %i" % sample >>>>> if sample == 100: >>>>> break >>>>> >>>>> sample = 0 >>>>> labels_predicted = [] >>>>> for row in data_test: >>>>> anomaly = [0]*16 >>>>> for i in range(0, 16): >>>>> model[i].resetSequenceStates() >>>>> for r in row: >>>>> value = int(r) >>>>> result = model[i].run({FIELD_NAME: value, '_learning': >>>>> False}) >>>>> prediction = >>>>> result.inferences["multiStepBestPredictions"][1] >>>>> anomalyScore = result.inferences["anomalyScore"] >>>>> # print value, prediction, anomalyScore >>>>> if value == int(prediction) and anomalyScore == 0: >>>>> # print "No prediction made" >>>>> anomalyScore = 1 >>>>> anomaly[i] += anomalyScore >>>>> anomaly[i] /= len(row) >>>>> sample += 1 >>>>> print "Processing testing sample %i" % sample >>>>> labels_predicted.append(np.min(np.array(anomaly))) >>>>> print anomaly, row[-1] >>>>> if sample == 100: >>>>> break >>>>> >>>>> accuracy = np.sum(np.array(labels_predicted) == >>>>> np.array(labels_test))*100.0/len(labels_test) >>>>> print "Testing accuracy is %0.2f" % accuracy >>>>> >>>>> >>>> >>>> >>> >> >> >
