Hi Chirag,

I think it would be better to also cc the mail list.

I assume you are trying to implement A3C or something likes this.
Actually this has almost been done. See my PR  
This is my work last summer. To compute the gradient, you can use 
And there is also an actor_critic worker to show how to use this.

The most annoying thing is it doesn’t work and I don’t know why. I and Marcus 
tried hard but didn’t find any obvious logic bug.
So if you want tor implement A3C I think the simplest way is to find the bug.
I have some hints for you:
1. Even we don’t have shared layers among actor and critic, A3C should work 
well on small task like CartPole. If you do want shared layers, you need to 
look into https://github.com/mlpack/mlpack/pull/1091 
<https://github.com/mlpack/mlpack/pull/1091> (I highly recommend you not to do 
this first, as this is not critical)
2. I believe the bug may lie in the async mechanism so it’s difficult to debug 
(It’s possible I’m wrong). A good practice I think is to implement A2C and 
corresponding PPO, which I believe is the state-of-the-art technique. You can 
implement the vectorized environment, i.e. the interaction with the environment 
is parallelized and synchronous, while the optimization occurs at a single 
thread. See OpenAI baselines (tensorflow, https://github.com/openai/baselines 
<https://github.com/openai/baselines>) or my A2C (pytorch, 
<https://github.com/ShangtongZhang/DeepRL/blob/master/agent/A2C_agent.py>) to 
see how this idea works. I believe it’s much easier to implement and debug. 
Once you implement the vectorized environment, it’s easy to plugin all the 
algorithms, e.g. one/n-step q learning, n-step salsa, actor-critic and PPO. 
From my experience, if tuned properly, the speed is comparable to fully async 
3. If you do want A3C and want to find that bug. I think you can implement 
actor-critic with experience replay first to verify if it works in single 
thread case (Note this is wrong theoretically as to do this you need to use 
off-policy actor-critic, while in practice you can just ignore the importance 
sampling ratio and treat the data in the buffer as on-policy, it should work 
and is enough to check the implementation in small task like CartPole)

BTW your understanding about how forward and backward in DQN is absolutely 

Hope this can help,

Best regards,

Shangtong Zhang,
Second year graduate student,
Department of Computing Science,
University of Alberta
Github <https://github.com/ShangtongZhang> | Stackoverflow 
> On Feb 19, 2018, at 00:58, Chirag Ramdas <chiragram...@gmail.com> wrote:
> I think I can probably write a custom compute_gradients() method for my 
> backprop here, but i wanted to know if mlpack's implementation provides me 
> with something similar to a convenient Forward() + Backword() pair which i 
> can use for my requirements here..
> Yours Sincerely,
> Chirag Pabbaraju,
> B.E.(Hons.) Computer Science Engineering,
> BITS Pilani K.K. Birla Goa Campus,
> Off NH17B, Zuarinagar,
> Goa, India
> chiragram...@gmail.com <mailto:chiragram...@gmail.com> | +91-9860632945
> On Mon, Feb 19, 2018 at 1:26 PM, Chirag Ramdas <chiragram...@gmail.com 
> <mailto:chiragram...@gmail.com>> wrote:
> Hello,
> I had an implementation question to ask.. So from the neural network 
> implementation i saw (ffn_impl), eg.  lines 146-156 
> <https://github.com/chogba/mlpack/blob/master/src/mlpack/methods/reinforcement_learning/q_learning_impl.hpp#L146-L156>
>  , you first forwarded the network on the states and saw what output (Q 
> value) it was giving for each action.. thereafter, you updated the targets 
> for the actions which you actually saw from your experience replay mechanism 
> and this updated target matrix now behaves like your labels which you wanted 
> the neural net to actually predict.. now, i saw from the q_learning_test.hpp 
> file that you are initialising the FFN with MeanSquaredError, so i am 
> assuming if you pass this target matrix to learningNetwork.Backward(), it 
> computes the gradients of the mean squared error with respect to all the 
> parameters. Thereafter, with these gradients and the optimizer which you have 
> specified eg.Adam,etc,  updater.Update() updates the parameters of the 
> network.
> Do correct me if i was wrong anywhere..
> So now my question is.. I am faced with a custom optimisation function, and i 
> am required to compute gradients of this function with respect to each of the 
> parameters of my neural net.. The Forward() + Backward() pair which was 
> called in the above implementation required me to compute 1) what my network 
> computes for an input 2) what i believe it should have computed, and 
> thereafter computes the gradients by itself. But I simply have an objective 
> function (no notion of what the network should have computed ie labels) and 
> correspondingly an update rule which i want to follow..
> Precisely, i have a policy function pi which is approximated by a neural net 
> parameterised by theta, and which outputs the probabilities of performing 
> each action given a state.. now, i want the following update rule for the 
> parameters..
> <Screen Shot 2018-02-19 at 1.10.32 PM.png>
> basically, i am asking if i can have my neural net optimise an objective 
> function which i myself specify, in some form.
> I looked at the implementation of ffn, but i couldn't figure out how i could 
> do this.. hope my question was clear.. 
> Thanks a lot!
> Yours Sincerely,
> Chirag Pabbaraju,
> B.E.(Hons.) Computer Science Engineering,
> BITS Pilani K.K. Birla Goa Campus,
> Off NH17B, Zuarinagar,
> Goa, India
> chiragram...@gmail.com <mailto:chiragram...@gmail.com> | +91-9860632945

mlpack mailing list

Reply via email to