Hi Arezou.
As far as I know, if you don't regularize at all, the global optimum of
a compressing autoencoder is given by PCA, even with a sigmoid.
The optimization is non-convex, though, so it will be rather unlikely
that you will end up with a PCA-Equivalent basis.
The proof for the global optimality of PCA uses a limit in which you
make the first layer weights small and the second layer weights large,
so that you end up in the linear regime.
With a linear activation function, the optimization is convex and you
will get a projection to the PCA space.
Hth,
Andy
On 11/06/2014 09:07 PM, Arezou Moussavi wrote:
Hi everybody,
Does any one know if a sparse autoencoder with sigmoidal function (as
its activation function ) and less hidden units than the inputs units,
works as a feature extractor or as PCA?
I know if the number of hidden units are less than the number of input
units and if the activation function is linear then the autoencoder
works as PCA unless we put restrictions on the autoencoder. (source
2) So, I wondered if making the autoencoder "sparse" and using
sigmoidal activation function helps to employ the autoencoder as a
feature extractor (deep learner) when it has less hidden units than
input units. My goal is to stack them to make a deep architecture, but
each autoencoder has less hidden units than inputs. Source 1 claimed
that even one sigmoidal layer will lead the autoencoder works as PCA.
These two resources have different opinions :
http://en.wikipedia.org/wiki/Autoencoder
http://deeplearning.net/tutorial/dA.html
Thanks,
Arezou
------------------------------------------------------------------------------
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
