On Jan 8, 2023, at 11:03 AM, Russ Abbott
<[email protected]> wrote:
As indicated in my original reply, my interest in this
project grows from my relative ignorance of Deep Learning.
My career has focussed exclusively on symbolic computing.
I've worked with and taught (a) functional programming,
logic programming, and related issues in advanced Python;
(b) complex systems, agent-based modeling, genetic
algorithms, and related evolutionary processes, (c) a bit of
constraint programming, especially in MiniZinc, and (d)
reinforcement learning as Q-learning, which is reinforcement
learning without neural nets. I've always avoided neural
nets--and more generally numerical programming of any sort.
Deep learning has produced so many impressive results that
I've decided to devote much of my retirement life to
learning about it. I retired at the end of Spring 2022 and
(after a break) am now devoting much of my time to learning
more about Deep Neural Nets. So far, I've dipped my brain
into it at various points. I think I've learned a fair
amount. For example,
* I now know how to build a neural net (NN) that adds two
numbers using a single layer with a single neuron. It's
really quite simple and is, I think, a beautiful example
of how NNs work. If I were to teach an intro to NNs I'd
start with this.
* I've gone through the Kaggle Deep Learning sequence
mentioned earlier.
* I found a paper that shows how you can approximate
any differentiable function to any degree of accuracy
with a single-layer NN. (This is a very nice result,
although I believe it's not used explicitly in building
serious Deep NN systems.)
* From what I've seen so far, most serious DNNs are built
using Keras rather than PyTorch.
* I've looked at Jeremy Howard's fast.ai <http://fast.ai>
material. I was going to go through the course but
stopped when I found that it uses PyTorch. Also, it
seems to be built on fast.ai <http://fast.ai> libraries
that do a lot of the work for you without explanation.
And it seems to focus almost exclusively on
Convolutional NNs.
* My impression of DNNs is that to a great extent they are
/ad hoc/. There is no good way to determine the best
architecture to use for a given problem. By
architecture, I mean the number of layers, the number of
neurons in each layer, the types of layers, the
activation functions to use, etc.
* All DNNs that I've seen use Python as code glue rather
than R or some other language. I like Python--so I'm
pleased with that.
* To build serious NNs one should learn the Python
libraries Numpy (array manipulation) and Pandas (data
processing). Numpy especially seems to be used for
virtually all DNNs that I've seen.
* Keras and probably PyTorch include a number of
special-purpose neurons and layers that can be
included in one's DNN. These include: a DropOut layer,
LSTM (short-long-term memory) neurons, convolutional
layers, recurrent neural net layers (RNN), and more
recently transformers, which get credit for ChatGPT and
related programs. My impression is that these
special-purpose layers are /ad hoc/ in the same sense
that functions or libraries that one finds useful in a
programming language are /ad hoc/. They have been very
important for the success of DNNs, but they came into
existence because people invented them in the same way
that people invented useful functions and libraries.
* NN libraries also include a menagerie of activation
functions. An activation function acts as the final
control on the output of a layer. Different
activation functions are used for different purposes. To
be successful in building a DNN, one must understand
what those activation functions do for you and which
ones to use.
* I'm especially interested in DNNs that use reinforcement
learning. That's because the first DNN work that
impressed me was DeepMind's DNNs that learned to play
Atari games--and then Go, etc. An important advantage of
Reinforcement Learning (RL) is that it doesn't depend on
mountains of labeled data.
* I find RL systems more interesting than image
recognition systems. One of the striking features of
many image recognition systems is that they can be
thrown off by changing a small number of pixels in an
image. The changed image would look to a human observer
just like the original, but it might fool a trained NN
into labeling the image as a banana rather than, say, an
automobile, which is what it really is. To address this
problem people have developed Generative Adversarial
Networks (GANs) which attempt to find such weaknesses in
a neural net during training and then to train the NN
not to have those weaknesses. This is a fascinating
result, but as far as I can tell, it mainly shows how
fragile some NNs are and doesn't add much conceptual
depth to one's understanding of how NNs work.
I'm impressed with this list of things I sort of know. If
you had asked me before I started writing this email I
wouldn't have thought I had learned as much as I have. Even
so, I feel like I don't understand much of it beyond a
superficial level.
So far I've done all my exploration using Google's Colab
(Google's Python notebook implementation) and Kaggle's
similar Python notebook implementation. (I prefer Colab to
Kaggle.) Using either one, it's super nice not to have to
download and install anything!
I'm continuing my journey to learn more about DNNs. I'd be
happy to have company and to help develop materials to teach
about DNNs. (Developing teaching materials always helps me
learn the subject being covered.)
-- Russ Abbott
Professor Emeritus, Computer Science
California State University, Los Angeles
On Sun, Jan 8, 2023 at 1:48 AM glen <[email protected]>
wrote:
Yes, the money/expertise bar is still pretty high. But
TANSTAAFL still applies. And the overwhelming evidence
is coming in that specific models do better than those
trained up on diverse data sets, "better" meaning less
prone to subtle bullsh¡t. What I find fascinating is
tools like OpenAI *facilitate* trespassing. We have a
wonderful bloom of non-experts claiming they understand
things like "deep learning". But do they? An old
internet meme is brought to mind: "Do you even Linear
Algebra, bro?" >8^D
On 1/8/23 01:06, Jochen Fromm wrote:
> I have finished a number of Coursera courses recently,
including "Deep Learning & Neural Networks with Keras"
which was ok but not great. The problems with deep
learning are
>
> * to achieve impressive results like chatGPT from
OpenAi or LaMDA from Goggle you need to spend millions
on hardware
> * only big organisations can afford to create such
expensive models
> * the resulting network is s black box and it is
unclear why it works the way it does
>
> In the end it is just the same old back propagation
that has been known for decades, just on more computers
and trained on more data. Peter Norvig calls it "The
unreasonable effectiveness of data"
> https://research.google.com/pubs/archive/35179.pdf
>
> -J.
>
>
> -------- Original message --------
> From: Russ Abbott <[email protected]>
> Date: 1/8/23 12:20 AM (GMT+01:00)
> To: The Friday Morning Applied Complexity Coffee Group
<[email protected]>
> Subject: Re: [FRIAM] Deep learning training material
>
> Hi Pieter,
>
> A few comments.
>
> * Much of the actual deep learning material looks
like it came from the Kaggle "Deep Learning
<https://www.kaggle.com/learn/intro-to-deep-learning>"
sequence.
> * In my opinion, R is an ugly and /ad hoc/ language.
I'd stick to Python.
> * More importantly, I would put the
How-to-use-Python stuff into a preliminary class. Assume
your audience knows how to use Python and focus on Deep
Learning. Given that, there is only a minimal amount of
information about Deep Learning in the write-up. If I
were to attend the workshop and thought I would be
learning about Deep Learning, I would be
disappointed--at least with what's covered in the write-up.
>
> I say this because I've been looking for a good
intro to Deep Learning. Even though I taught Computer
Science for many years, and am now retired, I avoided
Deep Learning because it was so non-symbolic. My focus
has always been on symbolic computing. But Deep Learning
has produced so many extraordinarily impressive results,
I decided I should learn more about it. I haven't found
any really good material. If you are interested, I'd be
more than happy to work with you on developing some
introductory Deep Learning material.
>
> -- Russ Abbott
> Professor Emeritus, Computer Science
> California State University, Los Angeles
>
>
> On Thu, Jan 5, 2023 at 11:31 AM Pieter Steenekamp
<[email protected]
<mailto:[email protected]>> wrote:
>
> Thanks to the kind support of OpenAI's chatGPT, I
am in the process of gathering materials for a
comprehensive and hands-on deep learning workshop.
Although it is still a work in progress, I welcome any
interested parties to take a look and provide their
valuable input. Thank you!
>
> You can get it from:
>
https://www.dropbox.com/s/eyx4iumb0439wlx/deep%20learning%20training%20rev%2005012023.zip?dl=0
<https://www.dropbox.com/s/eyx4iumb0439wlx/deep%20learning%20training%20rev%2005012023.zip?dl=0>
>
--
ꙮ Mɥǝu ǝlǝdɥɐuʇs ɟᴉƃɥʇ' ʇɥǝ ƃɹɐss snɟɟǝɹs˙ ꙮ
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-.
--- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe / Thursdays
9a-12p Zoom https://bit.ly/virtualfriam
to (un)subscribe
http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives: 5/2017 thru present
https://redfish.com/pipermail/friam_redfish.com/
1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. ---
-.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p
Zoom https://bit.ly/virtualfriam
to (un)subscribe
http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives: 5/2017 thru present
https://redfish.com/pipermail/friam_redfish.com/
1/2003 thru 6/2021 http://friam.383.s1.nabble.com/
