Nice function ! Thanks Paul. I don't have any C compilor here (i will try to make translate it in VB maybe). I try to anderstand... you want to compute noise varaince for each order and for each length... and from this find the better fit by minimising error variance. I agree this is really a good technics i think to find optimum length. And like you and Fred, yes i find (gaphicaly, even on simple sinus theorical) that include one peak more of a cycle (just one data before the peak, so the cross with 0 from the derivation is include) significantly improve frequency matching (just one data difference ! but this data include the reversal of the derivation sign).
But this technic can be very long to compute. You say you code a way so lenght is recursuive and take back previous calculation in account. I read fastly the code (i don't konw well C, but there is just FOR loop and arythmetic so i hope i have well anderstand your code ; ) ), and fin that you keep same Nom and Denom value during length variation... Keep "end" to "maxlen", but just modifiing "start" by shifting it from "stepwidth" at each iteration ? Problem is end stay always the same and so there are overlapping... example : 10 step, 100 stepwidth n=0 -> start = 1000+i, end = maxlen n=1 -> start = 900+i, end = maxlen .... n=step-1 -> start = 100+i, end = maxlen n=step -> start = 0+i, end = maxlen And Nom and Denom are not rest to 0 between each loop (to conserve previous data) : for n = 0 all is ok you compute Den/Nom between [1000+i;maxlen] for n = 1 ... hum you compute Den/Nom between [900+i;maxlen] + [1000+i;maxlen], so [1000+i;maxlen] is counted 2 times, ....etc... So "end" should change so no overlapping act ? Problem maybe, if end is modified is that new update of f/b are not taken in account (f/b should be different depending the lentgt and the order)... Seems difficult to make recursion on the length... Maybe you can explain me better the code if i am wrong (C is not my "dada" i don't know the word for this expression in english... "C is not my horse" for direct translation héhé) if we need to make computation start back from 0 for each length, will be heavy yes. For AR length it can be shortcut during recursion. In my code i test all AR order between 1 and MaxOrder, but it is recursive so you can stop it when you thing erreur is enough small (but maybe there is a higher order where error is more small... but we would never know it because we stop recursion earlier...). This can be a compromise. Cheers, Mich. ----- Original Message ----- From: Paul Ho To: [email protected] Sent: Friday, December 01, 2006 4:04 PM Subject: RE: [amibroker-ts] FW: Sample Data Range, Sampling Rate, and Window Effect on Au I have finally decided on automating length selection through minimising the forward prediction errors, just like auto selections of orders in linear prediction coeffs. The results oftens comes out at reasonably close to select a length to include a point just before a move takes place as Fred has suggested previously. There are few computing penalties as each iterations for one length forms the bases for the next length. Most of the work is done through memcof which is mostly rewritten. Feel free to use it for your own purposes. As to my point about using d(s(f)/df - yes, I Know both mich and myself are using deviation or differencing to find the peak. But I was referring to doing it mathmatically by solving the equations d(s(f)/df = 0 to obtain an equation for f which can be calculated directly to reduce the processing time. Then we might be able to talk about backtesting and so on. It is fast enough for charts right now but thats about all. Dll certain has speed advantage and I agree its about 3 - 5 times, but also allows flexibility in DSP work such as setting dimensions of vectors and using double precision. Combining both AFL and dll is a powerful development environment. Scripts have their place too, especially in dealing with com objects, but I am hopeless in debugging scripts. // **************************************************************************** *********************************** // * This inline function calculates AR coeffs and delivers them in a 3D array (len, degree, iterations) and // * noise variances in a 2D array (len, degree). It is expected that the optimium values lie within. // * The orginal source code from Numeric Recipes is heavily modified. // * Input : a vector of data[0...maxlen -1]; maxlen - max length of the vector being considered. // * Input : steps - No of steps between the minimum length and max length stepwidth - No of bars in each step. // * Input : degree - maximum order begin considered. // * Output : through pointers to two vectors ar and noisevariance. // * // **************************************************************************** *********************************** inline void memcof(float *data, int maxlen, int steps, int stepwidth, int degree, double ***ar, double **noisevariance) { int i,j, n, start = 0; double p=0.0; double *ef= new double[maxlen]; double *eb= new double[maxlen]; double *bshift = new double[maxlen]; double *refl = new double[degree + 1]; memset(refl, 0, (degree+1) * sizeof(double)); memset(ef, 0, maxlen* sizeof(double)); memset(eb, 0, maxlen* sizeof(double)); memset(bshift, 0, maxlen* sizeof(double)); int end = maxlen; // // When iterating through the steps, the first loop starts with // processing data of min length // in the 2nd loop it moves back one step and process upto the beginning of // first loop. 3rd loop moves back another step and so on. // for(n = 0; n < steps+1; n++) { start = (steps - n) * stepwidth; for (j = start;j < end ;j++) { p += SQR(data[j]); ef[j] = data[j]; eb[j] = data[j]; } noisevariance[n][0] = p/(maxlen - start); end = start; } for (i = 1;i< degree + 1;i++) { double num=0.0,denom=0.0; end = maxlen; for(n = 0; n <= steps; n++) { start = (steps - n) * stepwidth + i; // advance start by i for (j = start;j < end ;j++) { bshift[j] = eb[j-1]; num += ef[j]*bshift[j]; denom += SQR(ef[j])+SQR(bshift[j]); } refl[i]=2.0*num/denom; noisevariance[n][i] = noisevariance[n][i -1] * (1.0-SQR(refl[i])); for (j= 1;j < i ;j++) { ar[n][i][j] = ar[n][i -1][j] - ar[n][i-1][i - j] * refl[i]; } ar[n][i][i] = refl[i]; for (j=start;j < end ;j++) { eb[j] = bshift[j] - refl[i] *ef[j]; ef[j] -= refl[i] * bshift[j]; } end = start; } } delete []eb; delete []ef; delete []refl; delete []bshift; return; } ________________________________ From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of Tom Tom Sent: Monday, 27 November 2006 5:48 PM To: [email protected] Subject: RE: [amibroker-ts] FW: Sample Data Range, Sampling Rate, and Window Effect on Au Hello ! 1- Yes, I agree MEM dominant cycle (elhers) and MEM classic don't give same result for main cycle. But elhers for dominant cycle use Homondym (something like that... i don't remember me well : ) ) discriminator. So it seems to be different. It doens't use modeled data... but extract directly dominant cycle. Dominant cycle can be used for every bar, but yes MEM with AR modeling is more computer intensive, you are right ! I am working on code it in DLL, seems 2 or 3 times faster than AFL, not so much. For back testing it should take long time. But for computing it only for one bar (i.e: trading on bar close for exmple) it is ok i think (maybe 1/10 sec to compute forecast...). And problem for data length = n * yesterdays cycle, is that yesterday cycle depend from data length yesterday too... so it is recursive and the problem is always here. Maybe compute the length = one dominant cycle from elher ? (but dominante cycle need peridos parameter too ?). >>> I think practical way is maybe to know how long we keep each trade line in our account. For exemple for intraday trading if we keep position around 50 min in average, so if chart is 5 min bar, we should at least use 10 datas. Maybe no more so that way we only are interesting in cycle which have impact on our trading time horizon of 50 min. -> For d(S(f))/df it is already in the AFL i posted to extract dominant cycles. It keep only the cycles which have more power than the noise (i take the noise variance as refence to compute the spectrum in decibels). After i use RegLinSlope to find when it is zero so it is a maximum like you suggest it. This way there is often between 1 and 6 cycles. Fred, you said were saying you find around same number of dominant cycle too in one of your previous mail if i remember well... This way is A LOT MORE POWERFULL than just elhers dominant cycle. Just try 3 theorical added sinus... dominant cycle is lost... but by taking the 1 to 6 six dominant cycle from AR spectrum you find back the 3 main frequency without any pbl. 2- Good links thanks. I have to read a little about it... 3- For downsampling, i didn't tryed for now. But i want to experiment non linear (teking volatility and volume like i explained it before). I will post about it after i done it, but for now many work. Cheers, Mich. ----- Original Message ----- From: Paul Ho To: [email protected] ts%40yahoogroups.com> Sent: Wednesday, November 22, 2006 1:25 PM Subject: RE: [amibroker-ts] FW: Sample Data Range, Sampling Rate, and Window Effect on Au Mich, I guess my last post really discuss two separate but somehow related issues 1. How to programmatically determine the data length used for MEM a. John Elhers discussed in his book Mesa and Trading Market cycles p.88 what he used to determine data length. In the beginning he maximised the SNR to establish to optimum data length. Later on he set todays data length to yesterday cycle length, assuming that it requires one cycles worth of data to make a good measurement. Mich, you have said in your previous post that perhaps ½ cycle will suffice according to your experience. We of course can set today data length = n * yesterdays cycle period; There is also a practical problem in calculating dominant cycles. It is computing intensive for every bar. I wonder if it can be calculated more directly, for example using d(S(f))/df = 0. any views. b. When compared the dominant cycle plot using MEM and that using Elhers formula , the result are quite different. MEM doesnt always produced a dominant cycle for every bar. 2. How to deal with non-stationary signals a) My knowledge about Hilbert is fairly limited (left University more than a quarter of a century ago) is mainly from reading Elhers Rocket Science for traders. (pg 65) it is only about 10 pages I am willing to photocopy it send it to if its relevant. Another site Ive reading is http://wwwhitech.technion.ac.il/~feldman/intro.html <http://wwwhitech.technion.ac.il/~feldman/intro.html> and http://lucid.igpp.ucla.edu/lessons/ess265/2005/hilbert_huang_transformation. <http://lucid.igpp.ucla.edu/lessons/ess265/2005/hilbert_huang_transformation > pdf which provides some sort of introduction. EMD may be interesting. 3. The effect of downsampling on Spectral density a) It seems that down sampling often provides interesting result and clearer spectral peaks which are otherwise masked. Have you tried it 3. Ill have a look at your suggestions later on. Have you used any of it so far. Cheers Paul. __________________________________________________________ Live.com: le nouveau moteur de recherche par Messenger! http://www.windowslive.fr/livecom/ <http://www.windowslive.fr/livecom/> Send instant messages to your online friends http://au.messenger.yahoo.com _________________________________________________________________ Live.com : le moteur de recherche aux réponses bien carrossées! http://www.windowslive.fr/livecom/
