# Re: [theano-users] Split Op (OpFromGraph) to save intermediate results for grad

```"forward the precomputed output" means that Op1 already computed the final
output, therefore Op2 just has to behaves as identity in the forward pass ```
```
The intermediate value is already an output of Op1 as shown in the example
code, sorry if that wasn't clear.

Nicolas

Le mardi 8 août 2017 20:56:12 UTC+2, nouiz a écrit :
>
> I don't understand what you mean by "forward the precomputed output"
>
> What I would recommand is to make 1 op for the forward. The intermediate
> value that can be reused for the gradient, make them output. Don't use them
> in the forward, but you can reuse them your grad override.
>
> Frédéric
>
> On Mon, Jul 31, 2017 at 9:43 AM <nicolas....@gmail.com <javascript:>>
> wrote:
>
>> I am trying to build an Op with a custom/optimized gradient formula. To
>> override the automatic differenciation, I'm trying to use OpFromGraph.
>> The gradient formula can reuse intermediate results from the feed forward
>> pass, so I have tried to split the Op in two: Op1 computes the intermediate
>> and final result and gives all of it to Op2, Op2 forwards the final result
>> and takes care of the gradient computation given all the necessary values.
>>
>> Note that the gradient of the loss wrt the intermediate results is never
>> needed.
>>
>> Below is a what I believe to be a minimal working example of my problem,
>> it exhibits a strange conversion error related to the gradient computation
>> with the intermediate values. Please take note of the presence of an
>> integral variable.
>>
>> import numpy as np
>> import theano.tensor as T
>> import theano
>>
>>
>> def make_ops():
>>     x = T.vector()
>>     m = T.bvector()
>>
>>     r = m.sum().astype('floatX')  # intermediate value
>>     z = x * m / r  # final result
>>
>>
>>         return [
>>             T.DisconnectedType()()  # variable has integral type
>>             # T.zeros_like(inputs[1])
>>         ]
>>
>>
>>     op1 = theano.OpFromGraph(
>>         inputs=[x, m],
>>         outputs=[z, m, r],
>>         inline=True,
>>         name="op1")
>>
>>
>>     z = T.vector()
>>     r_forwarded = T.scalar()
>>
>>         _, m_, r_ = inputs
>>         # I think the error could be around here
>> <<<<<<<<<<------------------------------
>>         dr_ = T.zeros_like(r_)
>>         return [m_ / r_, dm_, dr_]
>>
>>     op2 = theano.OpFromGraph(
>>         inputs=[z, m, r_forwarded],
>>         outputs=[z],  # Op 2 forwards the precomputed output
>>         inline=True,
>>         name="op2")
>>
>>     return op1, op2
>>
>>
>> def main():
>>     op1, op2 = make_ops()
>>     x = T.vector(name="x")
>>     m = T.bvector(name="m")
>>     z_intermediate, m_forwarded, r = op1(x, m)
>>     z = op2(z_intermediate, m, r)
>>
>>     print(g.eval({x: np.array([1., .3, .0, .2], dtype=np.float32),
>>                   m: np.array([1, 0, 1, 1], dtype=np.int8)}))
>>
>>
>> if __name__ == "__main__":
>>     main()
>>
>> (Note: I had tried to hijack my previous question thread with this
>> problem but it went unnoticed, sorry for double posting)
>>
>> Thank you
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> For more options, visit https://groups.google.com/d/optout.
>>
>

--

---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email