Hey Matt, You’re right, I did send you the wrong matrix-style file. I’ve converted it since which has solved the zero accuracy rate. I’ve also modified the code to support different training/testing scenarios. I will add some comments to the new code and post it along with the some data logs to run on. Thank you for taking the time to trouble shoot this!
Nick > On Dec 29, 2014, at 9:24 PM, Matthew Taylor <[email protected]> wrote: > > Nicholas, > > I ran your code with the same results you had. If you have updated > code, please post it. > > After talking to Subutai about this, we think the input data you're > using to swarm over doesn't seem to represent the data in your > labelled matrix file. How are you generating the input CSV data file? > It could be that the swarming data is not close enough to the > real-world data within the matrix file. Swarming tries to give you the > model that is best for predicting the next field in the input data, so > it's important that the data the swarm uses closely represents the > data the model actually sees. I don't think this is the case here. > > One thing I'm trying right now is to pass the labeled data into each > model during the training phase 10 times instead of just once to > reinforce the patterns in each model. The program is running now, not > sure if it will help or not but I might as well try. > > --------- > Matt Taylor > OS Community Flag-Bearer > Numenta > > > On Mon, Dec 29, 2014 at 7:25 AM, Nicholas Mitri <[email protected]> wrote: >> Hey Matt, everyone, >> >> I debugged the code and managed to get some sensible results. HTM is doing a >> great job of learning sequences but performing very poorly at >> generalization. So while it can recognize a sequence it had learned with >> high accuracy, when it’s fed a test sequence that it’s never seen, its >> classification accuracy plummets. To be clear, classification here is >> performed by assigning an HTM region to each class and observing which >> region outputs the least anomaly score averaged along a test sequence. >> >> I’ve tried tweaking the encoder parameters to quantize the input with a >> lower resolution in the hope that similar inputs will be better pooled. That >> didn’t pan out. Also, changing encoder output length or number of columns is >> causing the HTM to output no predictions at times even with a non-empty >> active column list. I have little idea why that keeps happening. >> >> Any hints as to how to get HTM to better perform here? I’ve included HMM >> results for comparison. SVM results are all 95+%. >> >> Thank you, >> Nick >> >> >> HTM Results: >> >> Data = sequence of directions (8 discrete direction) >> Note on accuracy: M1/M2 is shown here to represent 2 performance metrics. M1 >> is average anomaly, M2 is the sum of average anomaly normalized and >> prediction error normalized. >> >> Base training accuracy: 100 % at 2 training passes >> >> User Dependent: 56.25%/56.25% >> >> User Independent: N/A >> >> Mixed: 65.00 %/ 71.25% >> >> >> HMM (22-states) Results: >> >> Data = sequence of directions (16 discrete direction) >> >> Base training accuracy: 97.5% >> >> User Dependent: 76.25 % >> >> User Independent: 88.75 % >> >> Mixed: 88.75 % >> >> >> On Dec 11, 2014, at 7:16 PM, Matthew Taylor <[email protected]> wrote: >> >> Nicholas, can you paste a sample of the input data file? >> >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> On Thu, Dec 11, 2014 at 7:50 AM, Nicholas Mitri <[email protected]> wrote: >>> >>> Hey all, >>> >>> I’m running into some trouble with using HTM for a gesture recognition >>> application and would appreciate some help. >>> First, the data is collected from 17 users performing 5 gestures of each >>> of 16 different gesture classes using motion sensors. The feature vector for >>> each sample is a sequence of discretized directions calculated using bezier >>> control points after curve fitting the gesture trace. >>> >>> For a baseline, I fed the data to 16 10-state HMMs for training and again >>> for testing. The classification accuracy achieved is 95.7%. >>> >>> For HTM, I created 16 CLA models using parameters from a medium swarm. I >>> ran the data through the models for training where each model is trained on >>> only 1 gesture class. For testing, I fed the same data again with learning >>> turned off and recorded the anomaly score (averaged across each sequence) >>> for each model. Classification was done by seeking the model with the >>> minimum anomaly score. Accuracy turned out to be a puzzling 0.0%!! >>> >>> Below is the relevant section of the code. I would appreciate any hints. >>> Thanks, >>> Nick >>> >>> def run_experiment(): >>> print "Running experiment..." >>> >>> model = [0]*16 >>> for i in range(0, 16): >>> model[i] = ModelFactory.create(model_params, logLevel=0) >>> model[i].enableInference({"predictedField": FIELD_NAME}) >>> >>> with open(FILE_PATH, "rb") as f: >>> csv_reader = csv.reader(f) >>> data = [] >>> labels = [] >>> for row in csv_reader: >>> r = [int(item) for item in row[:-1]] >>> data.append(r) >>> labels.append(int(row[-1])) >>> >>> # data_train, data_test, labels_train, labels_test = >>> cross_validation.train_test_split(data, labels, test_size=0.4, >>> random_state=0) >>> data_train = data >>> data_test = data >>> labels_train = labels >>> labels_test = labels >>> >>> for passes in range(0, TRAINING_PASSES): >>> sample = 0 >>> for (ind, row) in enumerate(data_train): >>> for r in row: >>> value = int(r) >>> result = model[labels_train[ind]].run({FIELD_NAME: value, >>> '_learning': True}) >>> prediction = >>> result.inferences["multiStepBestPredictions"][1] >>> anomalyScore = result.inferences["anomalyScore"] >>> model[labels[ind]].resetSequenceStates() >>> sample += 1 >>> print "Processing training sample %i" % sample >>> if sample == 100: >>> break >>> >>> sample = 0 >>> labels_predicted = [] >>> for row in data_test: >>> anomaly = [0]*16 >>> for i in range(0, 16): >>> model[i].resetSequenceStates() >>> for r in row: >>> value = int(r) >>> result = model[i].run({FIELD_NAME: value, '_learning': >>> False}) >>> prediction = >>> result.inferences["multiStepBestPredictions"][1] >>> anomalyScore = result.inferences["anomalyScore"] >>> # print value, prediction, anomalyScore >>> if value == int(prediction) and anomalyScore == 0: >>> # print "No prediction made" >>> anomalyScore = 1 >>> anomaly[i] += anomalyScore >>> anomaly[i] /= len(row) >>> sample += 1 >>> print "Processing testing sample %i" % sample >>> labels_predicted.append(np.min(np.array(anomaly))) >>> print anomaly, row[-1] >>> if sample == 100: >>> break >>> >>> accuracy = np.sum(np.array(labels_predicted) == >>> np.array(labels_test))*100.0/len(labels_test) >>> print "Testing accuracy is %0.2f" % accuracy >>> >>> >> >> >
