Hi, I'm writing a paper and I need to calculate tf-idf. Whit your help I managed to get results, I needed, but the problem is that I need to be able to explain how each number was gotten. So I tried to understand how idf was calculated and the numbers i get don't correspond to those I should get .
I have 3 documents (each line a document) a a b c m m e a c d e e d j k l m m c When I calculate tf, I get this (1048576,[99,100,106,107,108,109],[1.0,1.0,1.0,1.0,1.0,2.0]) (1048576,[97,98,99,109],[2.0,1.0,1.0,2.0]) (1048576,[97,99,100,101],[1.0,1.0,1.0,3.0] idf is supposedly calculated idf = log((m + 1) / (d(t) + 1)) m -number of documents (3 in my case). d(t) - in how many documents is term present a: log(4/3) =0.1249387366 b: log(4/2) =0.3010299957 c: log(4/4) =0 d: log(4/3) =0.1249387366 e: log(4/2) =0.3010299957 l: log(4/2) =0.3010299957 m: log(4/3) =0.1249387366 When I output idf vector ` idf.idf.toArray.filter(_.>(0)).distinct.foreach(println(_)) ` I get : 1.3862943611198906 0.28768207245178085 0.6931471805599453 I understand why there are only 3 numbers, because only 3 are unique : log(4/2), log(4/3), log(4/4), but I don't understand how numbers in idf where calculated Best regards, Andrejs
