Oh, sorry, never-mind my last mail.
> On May 22, 2015, at 5:15 PM, Sebastian Raschka <se.rasc...@gmail.com> wrote: > > Thanks, Lars, that's what I thought (natural log). I will try some more > combinations later and browse through the source code to see if I can somehow > manage to reproduce the results. Maybe it would be good to write it up as an > example then for the documentation -- in case someone else is wondering about > it since it is slightly different from the "classic" tf-idf approach. > > Btw. is there anything that speaks against those negative values in the > feature vectors? I mean for e.g., SGD classifiers it can maybe be beneficial > to have values that can be positive and negative. > > Best, > Sebastian > > >> On May 22, 2015, at 12:00 PM, Lars Buitinck <larsm...@gmail.com> wrote: >> >> 2015-05-22 8:29 GMT+02:00 Sebastian Raschka <se.rasc...@gmail.com>: >>> The default equation is: >>> # idf = log ( number_of_docs / number_of_docs_where_term_appears ) >>> >>> And in the online documentation at >>> http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html >>> I found the additional info: >>>> smooth_idf : boolean, default=True >>>> Smooth idf weights by adding one to document frequencies, as if an extra >>>> document was seen containing every term in the collection exactly once. >>>> Prevents zero divisions. >>> >>> >>> So that I assume that the smooth_idf is calculated as follows: >>> # smooth_idf = log ( number_of_docs / (1 + >>> number_of_docs_where_term_appears) ) >> >> I don't have a full answer ready, but note that number_of_docs must >> also be incremented by the smoothing term (which is actually a >> misnomer, IIRC). Otherwise the logs can come out negative. >> >> Logs are also always natural logs in scikit-learn. >> >> HTH >> >> ------------------------------------------------------------------------------ >> One dashboard for servers and applications across Physical-Virtual-Cloud >> Widest out-of-the-box monitoring support with 50+ applications >> Performance metrics, stats and reports that give you Actionable Insights >> Deep dive visibility with transaction tracing using APM Insight. >> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y >> _______________________________________________ >> Scikit-learn-general mailing list >> Scikit-learn-general@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > > ------------------------------------------------------------------------------ > One dashboard for servers and applications across Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------------------------------------------------------------------ One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
