I once thought about being a quant long before I started my current company :) .
I don't want to ruin the surprise for you, but because of volatility in the market and the fact that you are looking at daily data (unless you spend a lot of time writing a custom clustering implementation) you are more likely to see clusters like 'defensive stocks', 'bell weather stocks' and the like... whose performance is less about their sector and more about the overall market's "like" of the company. Since some sectors have a higher affinity to these categorizations, you are likely to see some patterns as you describe and will also see some sectors being clustered apart. One example is the 'tech' sector which has many companies that perform differently in any particular market climate. I found the much more interesting problem to be in the area of which non-sector companies influence other companies and to what degree. For instance, could I create a model that is "subconsciously aware" that stock1 is a supplier to stock2 who is a major vendor of stock3. If stock3 takes a dip, to what extent does that influence stock2's share price and could I short stock3. To build this model, I bought "tickdata" from a company called "Fitch" ( http://www.fitchgroup.com/fitchdata/) and built a stock simulator and neural net as part of my Master's project. I had about 1 year of data and could get pretty good at making some predictions, but my performance never came close to beating the major indexes. I bet if you had more data and a better algorithm, that you could get much better performance. Whether you could do better than the indexes is another question. Joe On Thu, Jul 15, 2010 at 7:51 AM, Florent Empis <[email protected]>wrote: > Hi, > > I want to learn more on clustering techniques. I have skimmed through > Programming Collective Intelligence and Mahout in Action in the past but I > don't have them on hand at the moment... :( > I've seen Isabel Drost mail about test data on http://mldata.org/about/ > I've had an idea of using http://mldata.org/repository/view/stockvalues/for > a pet project. > My idea is as follow: can we see a common behaviour between companies' > stock > value? > I would expect ending up with cluster of banking sector shares, utilities > share, media etc... and maybe some more unexpected cluster, who knows? > > My idea is basically: > 1°)Transform the dataset from values to daily variation as percentage > drop/raise (data is then normalized) > 2°)Apply clustering technique(s) > > The issue may seem silly but as I understand it, clustering happens in a 2 > (or more) dimension space. > I know I have 2 dimensions: variation and time, but I can't wrap my head on > the problem... > > I *think* that the K-Means example does exactly what I intend to do my > second step, is this correct? > However, I can grasp what the 2 dimensional display represent exactly: what > are the x and y axis ? > > Added question: I am fairly new to the M/R paradigm, but let's say I would > like to do step 1 (data normalization) in a M/R fashion. Would the > following > be a good idea: > My data is a matrix of k stock values S in n intervals of time. > I call the first stock in the file, first and second period: > S1,t & S1,t+1 ... > > Map Step: input: ((S1,t ... S1,t+n),... ,(Sk,t ... Sk,t+n) ) > output (( (S1,t;S1,t+1),...,(S1,t+n-1;S1,t+n)), ... ,( > (Sk,t;Sk,t+1),...,(Sk,t+n-1;Sk,t+n)) ) > Reduce Step: > ( (%S1,t+1.....%S1,t+n), ...,(%S1,t+1.....%S1,t+n)) > > I apologize for my beginner's questions but.... everyone has to start > somewhere :-) > > BR, > > Florent Empis >
