I suspect this method is underreported by any particular name, as it's a
straightforward greedy search. It is also very close to what I think many
researchers do in system development or report in system analysis, albeit
with more automation.
In the case of KNN, I would think metric learning could subsume or
outperform this.
On 28 April 2015 at 08:50, Andreas Mueller <t3k...@gmail.com> wrote:
> Maybe we would want mrmr first?
>
> http://penglab.janelia.org/proj/mRMR/
>
>
> On 04/27/2015 06:46 PM, Sebastian Raschka wrote:
> >> I guess that could be done, but has a much higher complexity than RFE.
> > Oh yes, I agree, the sequential feature algorithms are definitely
> computationally more costly.
> >
> >> It seems interesting. Is that really used in practice and is there any
> >> literature evaluating it?
> >
> > I am not sure how often it is used in practice nowadays, but I think it
> is one of the classic approaches for feature selection -- I learned about
> it a couple of years ago in a pattern classification class, and there is a
> relatively detailed article in
> >
> > Ferri, F., et al. "Comparative study of techniques for large-scale
> feature selection." Pattern Recognition in Practice IV (1994): 403-413.
> >
> > The optimal solution to feature selection would be to evaluate the
> performance of all possible feature combination, which is a little bit too
> costly in practice. The sequential forward or backward selection (SFS and
> SBS) algorithms are just a suboptimal solution, and there are some minor
> improvements, e.g,. Sequential Floating Forward Selection (SFFS) which
> allows for the removal of added features in later stages etc.
> >
> > I have an implementation of SBS that uses k-fold cross_val_score, and it
> is actually not a bad idea to use it for KNN to reduce overfitting as
> alternative to dimensionality reduction, for example, KNN cross-val mean
> accuracy on the wine dataset where the features are selected by SBS:
> http://i.imgur.com/ywDTHom.png?1
> >
> > But for scikit-learn, it may be better to implement SBBS or SFFS which
> is slightly more sophisticated.
> >
> >
> >> On Apr 27, 2015, at 6:00 PM, Andreas Mueller <t3k...@gmail.com> wrote:
> >>
> >> That is like a one-step look-ahead feature selection?
> >> I guess that could be done, but has a much higher complexity than RFE.
> >> RFE works for anything that returns "importances", not just linear
> models.
> >> It doesn't really work for KNN, as you say. [I wouldn't say
> >> non-parametric models. Trees are pretty non-parametric].
> >>
> >> It seems interesting. Is that really used in practice and is there any
> >> literature evaluating it?
> >> There is some discussion here
> >> http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf in 4.2
> >> but there is no empirical comparison or theoretical analysis.
> >>
> >> To be added to sklearn, you'd need to show that it is widely used and /
> >> or widely useful.
> >>
> >>
> >> On 04/27/2015 02:47 PM, Sebastian Raschka wrote:
> >>> Hi, I was wondering if sequential feature selection algorithms are
> currently implemented in scikit-learn. The closest that I could find was
> recursive feature elimination (RFE);
> http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html.
> However, unless the application requires a fixed number of features, I am
> not sure if it is necessarily worthwhile using it over regularized models.
> When I understand correctly, it works like this:
> >>>
> >>> {x1, x2, x3} --> eliminate xi with smallest corresponding weight
> >>>
> >>> {x1, x3} --> eliminate xi with smallest corresponding weight
> >>>
> >>> {x1}
> >>>
> >>> However, this would only work with linear, discriminative models right?
> >>>
> >>> Wouldn't be a classic "sequential feature selection" algorithm useful
> for non-regularized, nonparametric models e.g,. K-nearest neighbors as an
> alternative to dimensionality reduction for applications where the original
> features may need to be maintained? The RFE, for example, wouldn't work
> with KNN, and maybe the data is non-linearly separable so that RFE with a
> linear model doesn't make sense.
> >>>
> >>> In a nutshell, SFS algorithms simply add or remove one feature at the
> time based on the classifier performance.
> >>>
> >>> e.g., Sequential backward selection:
> >>>
> >>> {x1, x2, x3} ---> estimate performance on {x1, x2}, {x2, x3} and {x1,
> x3}, and pick the subset with the best performance
> >>> {x1, x3} ---> estimate performance on {x1}, {x3} and pick the subset
> with the best performance
> >>> {x1}
> >>>
> >>> where performance could be e.g., cross-val accuracy.
> >>>
> >>> What do you think?
> >>>
> >>> Best,
> >>> Sebastian
> >>>
> ------------------------------------------------------------------------------
> >>> One dashboard for servers and applications across
> Physical-Virtual-Cloud
> >>> Widest out-of-the-box monitoring support with 50+ applications
> >>> Performance metrics, stats and reports that give you Actionable
> Insights
> >>> Deep dive visibility with transaction tracing using APM Insight.
> >>> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> >>> _______________________________________________
> >>> Scikit-learn-general mailing list
> >>> Scikit-learn-general@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
> >>
> >>
> ------------------------------------------------------------------------------
> >> One dashboard for servers and applications across Physical-Virtual-Cloud
> >> Widest out-of-the-box monitoring support with 50+ applications
> >> Performance metrics, stats and reports that give you Actionable Insights
> >> Deep dive visibility with transaction tracing using APM Insight.
> >> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> >> _______________________________________________
> >> Scikit-learn-general mailing list
> >> Scikit-learn-general@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
> >
> >
> ------------------------------------------------------------------------------
> > One dashboard for servers and applications across Physical-Virtual-Cloud
> > Widest out-of-the-box monitoring support with 50+ applications
> > Performance metrics, stats and reports that give you Actionable Insights
> > Deep dive visibility with transaction tracing using APM Insight.
> > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> > _______________________________________________
> > Scikit-learn-general mailing list
> > Scikit-learn-general@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
