I am happy to see people interested in messing around with Julia for ML.
The best way to wrap your head around the concepts is usually to try it
out and see what happens.
My 2 cents are that I doubt that you will get competitive results with
neural networks for your regression problems (even with the modern
trickeries such as ReLU-variant hidden layers). Two of the strengths of
current deep learning are convolutions and very large datasets. This
allows them to handle high dimensional data like no other approach. I
suspect that is why you usually observe NNs in image, text and speech
classification tasks.
I haven't read any recent papers on the NN regression front, but from my
limited personal experience I would say that neural networks are just
too flexible to be useful for datasets that are smaller than a few
thousand observations (I am sure there are some data size guidelines in
the upcoming DL book).
Concerning DL in Julia, there is also https://github.com/dmlc/MXNet.jl.
Although I haven't tried it for regression.
On 2016-01-30 18:13, Cedric St-Jean wrote:
AFAIK deep learning in general does not have any problem with
redundant inputs. If you have fewer nodes in your first layer than
input nodes, then the redundant (or nearly-redundant) input nodes will
be combined into one node (... more or less). And there are approaches
that favor using so-called overcomplete representations with more
hidden nodes / layer than input nodes.
Cédric
On Saturday, January 30, 2016 at 9:46:06 AM UTC-5, [email protected]
wrote:
Thanks, that's pretty much my understanding. Scaling the inputs
seems to be important, too, from what I read. I'm also interested
in a framework that will trim off redundant inputs.
I have run the mocha tutorial examples, and it looks very
promising because the structure is clear, and there are C++ and
cuda backends. The C++ backend, with openmp, gives me a good
performance boost over the pure Julia backend. However, I'm not so
sure that it will allow for trimming redundant inputs. Also, I
have some ideas on how to restrict the net to remove
observationally equivalent configurations, which should aid in
training, and I don't think I could implement those ideas with mocha.
From what I see, the focus of much recent work in neural nets
seems to be on classification and labeling of images, and
regression examples using the modern tools seem to be scarce. I'm
wondering if that's because other tools work better for
regression, or simply because it's an old problem that is
considered to be well studied. I would like to see some examples
of regression nets that work well, using the modern tools, though,
if there are any out there.
On Saturday, January 30, 2016 at 2:32:16 PM UTC+1, Jason Eckstein
wrote:
I've been using NN for regression and I've experimented with
Mocha. I ended up coding my own network for speed purposes
but in general you simply leave the final output of the neural
network as a linear combination without applying an activation
function. That way the output can represent a real number
rather than compress it into a 0 to 1 or -1 to 1 range for
classification. You can leave the rest of the network unchanged.
On Saturday, January 30, 2016 at 3:45:27 AM UTC-7,
[email protected] wrote:
I'm interested in using neural networks (deep learning)
for multivariate multiple regression, with multiple real
valued inputs and multiple real valued outputs. At the
moment, the mocha.jl package looks very promising, but the
examples seem to be all for classification problems. Does
anyone have examples of use of mocha (or other deep
learning packages for Julia) for regression problems? Or
any tips for deep learning and regression?
