Hello everybody, I'm getting a problem that so far I could not manage to solve, I'm guessing it's a precision problem based on a past thread I found in the mailing-list archive, but I'm not completely sure…
Basically I'm writing a neural network language model using gsl. In the main training function several matrices and vectors are involved. In order to keep track and check everything I'm doing is correct, I wrote a first prototype in octave and then I rewrote the same things in C with gsl. When I run the two programs, at first results are the same, but after some iterations to train the network, results start to diverge, with the C implementation having worst results and cross-entropy fluctuating values. After checking several things, and not understanding the source of the problem, I wrote the matrices involved in the computations to a file, for both programs, octave and C versions, thus I can compare intermediate computation for both programs. Then I compare the corresponding matrices to check if they are equal (with gsl_matrix_equal) or not. Only 2 out of 26 matrix pairs are different, and by very small values, 10e-15 on average over matrix elements, or less. Please note that the two matrix pairs that are different are computed from other matrices that are equal in the two program versions, thus that's even more strange to me. My question is: Is this a precision problem or there should be a mistake somewhere ? Please note also that, although difference is very small, since in a neural network matrix are, let's say, interdependent, small differences in 2 matrices are propagated, after some iterations of training, to all the other matrices. Thus at the end the model trained by the C version is different from the one trained by the octave version, in particular the former is worst. So, in case this is a precision problem, does anybody know how to overcome it ? Any help would be more than appreciated. Thanks in advance Best regards Marco