[sympy] Re: Symbolic integrator using a neural network

[Gagandeep Singh (B17CS021)]

When I skimmed through the paper, I had the following queries:

1. Is integration really a tree to tree translation? Because, neural 
network is predicting the resulting expression tree for the input equation. 
However, integration is not a predictive operation. Moreover, how can we 
define that whether the model is over-fitting or not on the training data? 
What was the size of the training data set?

2. Does the data set of equations contains noise? Is there any uncertainty 
in the data set? For example, while translating from one language to 
another there are chances that one word can be mapped to different ones 
having the same meaning. However, here, it is not the case, there may be 
multiple results but we can check whether they are correct or not with 100% 
surety. 

3. Is this model able to generalise over any mathematical expression. The 
way they generated data sets is algorithmic and deterministic. It is not 
random(random number generators are itself deterministic). So, how can we 
say that this model is the one that outperforms any CAS?

Neural Networks don't learn, the way human beings do. They just imitate the 
underlying distribution of the data. But, I don't think that mathematical 
expressions have any such underlying distribution. Well, we can take a 
subset of those expressions which can have a distribution and I think 
that's the case with their model.

PS - I can be wrong at any many places above as, I am just a beginner in 
ML/DL/AI. Please correct me wherever possible. Thanks.

On Saturday, September 28, 2019 at 12:18:21 AM UTC+5:30, Aaron Meurer wrote:
>
> There's a review paper for ICLR 2020 on training a neural network to 
> do symbolic integration. They claim that it outperforms Mathematica by 
> a large margin. Machine learning papers can sometimes make overzealous 
> claims, so scepticism is in order. 
>
> https://openreview.net/pdf?id=S1eZYeHFDS 
> 
>  
>
> The don't seem to post any code. The paper is in double blind review, 
> so maybe it will be available later. Or maybe it is available now and 
> I don't see it. If someone knows, please post a link here. 
>
> They do cite the SymPy paper, but it's not clear if they actually use 
> SymPy. 
>
> I think it's an interesting concept. They claim that they generate 
> random functions and differentiate them to train the network. But I 
> wonder if one could instead take a large pattern matching integration 
> table like RUBI and train it on that, and produce something that works 
> better than RUBI. The nice thing about indefinite integration is it's 
> trivial to check if an answer is correct (just check if 
> diff(integral(f)) - f == 0), so heuristic approaches that can 
> sometimes give nonsense are tenable, because you can just throw out 
> wrong answers. 
>
> I'm also curious (and sceptical) on just how well a neural network can 
> "learn" symbolic mathematics and specifically an integration 
> algorithm. Another interesting thing to do would be to try to train a 
> network to integrate rational functions, to see if it can effectively 
> recreate the algorithm (for those who don't know, there is a complete 
> algorithm which can integrate any rational function). My guess is that 
> this sort of thing is still beyond the capabilities of a neural 
> network. 
>
> Aaron Meurer 
>

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/fa4ee5ee-a112-425b-ba2e-61d299812855%40googlegroups.com.

Re: [sympy] Symbolic integrator using a neural network

[Oscar Benjamin]

Neural nets are trained for a particular statistical distribution of
inputs and in the paper they describe their method for generating a
particular ensemble of possibilities. There might be something
inherent about the examples they give that means they are all solved
using a particular approach. From their description I could imagine
writing a pattern-matching integrator that would explicitly try to
reverse the way the examples are generated.

Perhaps the examples from e.g. the SymPy test suite would in some ways
represent a more "natural" distribution since they are written by
humans and show the kinds of problems that humans wanted to solve. It
would be interesting to see how the accuracy of the net looks on that
distribution of inputs (although any comparison with SymPy on that
data would be unfair).

Oscar

On Sat, 28 Sep 2019 at 07:30, Aaron Meurer  wrote:
>
> On Fri, Sep 27, 2019 at 11:56 PM Ondřej Čertík  wrote:
> >
> > On Fri, Sep 27, 2019, at 12:48 PM, Aaron Meurer wrote:
> > > There's a review paper for ICLR 2020 on training a neural network to
> > > do symbolic integration. They claim that it outperforms Mathematica by
> > > a large margin. Machine learning papers can sometimes make overzealous
> > > claims, so scepticism is in order.
> > >
> > > https://openreview.net/pdf?id=S1eZYeHFDS
> > >
> > > The don't seem to post any code. The paper is in double blind review,
> > > so maybe it will be available later. Or maybe it is available now and
> > > I don't see it. If someone knows, please post a link here.
> > >
> > > They do cite the SymPy paper, but it's not clear if they actually use 
> > > SymPy.
> >
> > They wrote:
> >
> > "The validity of a solution itself is not provided by the model, but by an 
> > external symbolic framework (Meurer et al., 2017). "
> >
> > So that seems to suggest they used SymPy to check the results.
> >
> > >
> > > I think it's an interesting concept. They claim that they generate
> > > random functions and differentiate them to train the network. But I
> > > wonder if one could instead take a large pattern matching integration
> > > table like RUBI and train it on that, and produce something that works
> > > better than RUBI. The nice thing about indefinite integration is it's
> > > trivial to check if an answer is correct (just check if
> > > diff(integral(f)) - f == 0), so heuristic approaches that can
> > > sometimes give nonsense are tenable, because you can just throw out
> > > wrong answers.
> > >
> > > I'm also curious (and sceptical) on just how well a neural network can
> > > "learn" symbolic mathematics and specifically an integration
> > > algorithm. Another interesting thing to do would be to try to train a
> > > network to integrate rational functions, to see if it can effectively
> > > recreate the algorithm (for those who don't know, there is a complete
> > > algorithm which can integrate any rational function). My guess is that
> > > this sort of thing is still beyond the capabilities of a neural
> > > network.
> >
> > I saw this paper too today. My main question is whether their approach is 
> > better than Rubi (say in Mathematica, as it doesn't yet work 100% in SymPy 
> > yet). They show that their approach is much better than Mathematica, but so 
> > is Rubi.
>
> It actually isn't clear to me yet that they've shown it. I want to see
> what their test suite of functions looks like.
>
> Aaron Meurer
>
> >
> > The ML approach seems like a brute force. So is Rubi. So it's fair to 
> > compare ML with Rubi. On the other hand, I feel it's unfair to compare 
> > brute force with an actual algorithm, such as Risch.
> >
> > Ondrej
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "sympy" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to sympy+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/sympy/db41cf67-acc9-4a84-8267-2742b748de4d%40www.fastmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sympy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sympy/CAKgW%3D6JExav6cUscxJbv%2B9QhY_S2AP52L9ycNzA%2Be2jtgdGSQQ%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/CAHVvXxTMoxh6fG8Yb%2B3%3Dgmr1bqt9tQXf_rWMTkLAAThRnW-wyw%40mail.gmail.com.

Re: [sympy] Symbolic integrator using a neural network

[Francesco Bonazzi]

Their paper appears to be an attempt at using the transformer model for 
language translation to symbolic math.

There is a Jupyter notebook with an example on how to create a translator 
from Portuguese to English using the transformer model:
https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb

If someone has some spare time, it would be interesting to see how this 
model would perform with SymPy (just add a tokenizer to the output of 
*srepr* and replace the Portuguese-English dataset).

On Saturday, 28 September 2019 08:30:30 UTC+2, Aaron Meurer wrote:
>
> On Fri, Sep 27, 2019 at 11:56 PM Ondřej Čertík  > wrote: 
> > 
> > On Fri, Sep 27, 2019, at 12:48 PM, Aaron Meurer wrote: 
> > > There's a review paper for ICLR 2020 on training a neural network to 
> > > do symbolic integration. They claim that it outperforms Mathematica by 
> > > a large margin. Machine learning papers can sometimes make overzealous 
> > > claims, so scepticism is in order. 
> > > 
> > > https://openreview.net/pdf?id=S1eZYeHFDS 
> > > 
> > > The don't seem to post any code. The paper is in double blind review, 
> > > so maybe it will be available later. Or maybe it is available now and 
> > > I don't see it. If someone knows, please post a link here. 
> > > 
> > > They do cite the SymPy paper, but it's not clear if they actually use 
> SymPy. 
> > 
> > They wrote: 
> > 
> > "The validity of a solution itself is not provided by the model, but by 
> an external symbolic framework (Meurer et al., 2017). " 
> > 
> > So that seems to suggest they used SymPy to check the results. 
> > 
> > > 
> > > I think it's an interesting concept. They claim that they generate 
> > > random functions and differentiate them to train the network. But I 
> > > wonder if one could instead take a large pattern matching integration 
> > > table like RUBI and train it on that, and produce something that works 
> > > better than RUBI. The nice thing about indefinite integration is it's 
> > > trivial to check if an answer is correct (just check if 
> > > diff(integral(f)) - f == 0), so heuristic approaches that can 
> > > sometimes give nonsense are tenable, because you can just throw out 
> > > wrong answers. 
> > > 
> > > I'm also curious (and sceptical) on just how well a neural network can 
> > > "learn" symbolic mathematics and specifically an integration 
> > > algorithm. Another interesting thing to do would be to try to train a 
> > > network to integrate rational functions, to see if it can effectively 
> > > recreate the algorithm (for those who don't know, there is a complete 
> > > algorithm which can integrate any rational function). My guess is that 
> > > this sort of thing is still beyond the capabilities of a neural 
> > > network. 
> > 
> > I saw this paper too today. My main question is whether their approach 
> is better than Rubi (say in Mathematica, as it doesn't yet work 100% in 
> SymPy yet). They show that their approach is much better than Mathematica, 
> but so is Rubi. 
>
> It actually isn't clear to me yet that they've shown it. I want to see 
> what their test suite of functions looks like. 
>
> Aaron Meurer 
>
> > 
> > The ML approach seems like a brute force. So is Rubi. So it's fair to 
> compare ML with Rubi. On the other hand, I feel it's unfair to compare 
> brute force with an actual algorithm, such as Risch. 
> > 
> > Ondrej 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "sympy" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to sy...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sympy/db41cf67-acc9-4a84-8267-2742b748de4d%40www.fastmail.com.
>  
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/f42e6f5d-a386-476b-ad15-bb962cfdcdba%40googlegroups.com.

Re: [sympy] Symbolic integrator using a neural network

[David Bailey]

On 28/09/2019 14:27, Oscar Benjamin wrote:

Neural nets are trained for a particular statistical distribution of
inputs and in the paper they describe their method for generating a
particular ensemble of possibilities. There might be something
inherent about the examples they give that means they are all solved
using a particular approach. From their description I could imagine
writing a pattern-matching integrator that would explicitly try to
reverse the way the examples are generated.

Perhaps the examples from e.g. the SymPy test suite would in some ways
represent a more "natural" distribution since they are written by
humans and show the kinds of problems that humans wanted to solve. It
would be interesting to see how the accuracy of the net looks on that
distribution of inputs (although any comparison with SymPy on that
data would be unfair).

I suppose a fair test might be to take a set of integrals from 
Abramowitz and Stegun, but of course, for indefinite integrals the 
accuracy should be 100% because the output can be checked by 
differentiation.


Sadly the paper mainly focuses on the generation of the test sets, 
because the rest is inevitably completely opaque!


I noticed this sentence:

"Unfortunately, functions do not have an integral which can be expressed 
with usual functions (e.g.f(x) = exp(x2) or f(x) = log(log(x))), and 
solutions to arbitrary differential equations cannot always be expressed 
with usual functions."


This suggests that their integrator will not handle anything that 
resolves to a higher transcendental functions - e.g. elliptic integrals.


I guess this is a rather special case of a significant maths problem 
that is hard in one direction but easy to check, and where pattern 
matching can be used extensively. It would be sad if one day the whole 
of SymPy were replaced with an opaque program like this - but I don't 
think that is likely.


David


--
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/8ccad222-8435-17c5-cc93-c07409edda28%40dbailey.co.uk.

Re: [sympy] Symbolic integrator using a neural network

[Aaron Meurer]

On Fri, Sep 27, 2019 at 11:56 PM Ondřej Čertík  wrote:
>
> On Fri, Sep 27, 2019, at 12:48 PM, Aaron Meurer wrote:
> > There's a review paper for ICLR 2020 on training a neural network to
> > do symbolic integration. They claim that it outperforms Mathematica by
> > a large margin. Machine learning papers can sometimes make overzealous
> > claims, so scepticism is in order.
> >
> > https://openreview.net/pdf?id=S1eZYeHFDS
> >
> > The don't seem to post any code. The paper is in double blind review,
> > so maybe it will be available later. Or maybe it is available now and
> > I don't see it. If someone knows, please post a link here.
> >
> > They do cite the SymPy paper, but it's not clear if they actually use SymPy.
>
> They wrote:
>
> "The validity of a solution itself is not provided by the model, but by an 
> external symbolic framework (Meurer et al., 2017). "
>
> So that seems to suggest they used SymPy to check the results.
>
> >
> > I think it's an interesting concept. They claim that they generate
> > random functions and differentiate them to train the network. But I
> > wonder if one could instead take a large pattern matching integration
> > table like RUBI and train it on that, and produce something that works
> > better than RUBI. The nice thing about indefinite integration is it's
> > trivial to check if an answer is correct (just check if
> > diff(integral(f)) - f == 0), so heuristic approaches that can
> > sometimes give nonsense are tenable, because you can just throw out
> > wrong answers.
> >
> > I'm also curious (and sceptical) on just how well a neural network can
> > "learn" symbolic mathematics and specifically an integration
> > algorithm. Another interesting thing to do would be to try to train a
> > network to integrate rational functions, to see if it can effectively
> > recreate the algorithm (for those who don't know, there is a complete
> > algorithm which can integrate any rational function). My guess is that
> > this sort of thing is still beyond the capabilities of a neural
> > network.
>
> I saw this paper too today. My main question is whether their approach is 
> better than Rubi (say in Mathematica, as it doesn't yet work 100% in SymPy 
> yet). They show that their approach is much better than Mathematica, but so 
> is Rubi.

It actually isn't clear to me yet that they've shown it. I want to see
what their test suite of functions looks like.

Aaron Meurer

>
> The ML approach seems like a brute force. So is Rubi. So it's fair to compare 
> ML with Rubi. On the other hand, I feel it's unfair to compare brute force 
> with an actual algorithm, such as Risch.
>
> Ondrej
>
> --
> You received this message because you are subscribed to the Google Groups 
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to sympy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/sympy/db41cf67-acc9-4a84-8267-2742b748de4d%40www.fastmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/CAKgW%3D6JExav6cUscxJbv%2B9QhY_S2AP52L9ycNzA%2Be2jtgdGSQQ%40mail.gmail.com.