[GitHub] [madlib] kaknikhil edited a comment on pull request #564: MLP: Add rmsprop and Adam optimization techniques

[GitBox]

kaknikhil edited a comment on pull request #564:
URL: https://github.com/apache/madlib/pull/564#issuecomment-829729170


   While running mlp with verbose set to true with either rmsprop or adam, we 
print the wrong loss value to the console i.e. it always prints the epsilon 
value instead of the loss value
   
   I had to check the output table to get the actual loss value
   
   ```
   SELECT madlib.mlp_classification(
       'iris_data_packed',
       'mlp_model',
       'independent_varname',
       'dependent_varname',
       ARRAY[5],
       'learning_rate_init=0.1, learning_rate_policy=constant, n_iterations=10, 
solver=rmsprop',
       'tanh',
       NULL,
       FALSE,
       TRUE
   );
   INFO:  Iteration: 1, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 2, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 3, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 4, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 5, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 6, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 7, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 8, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   INFO:  Iteration: 9, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
   
   madlib=# SELECT loss FROM mlp_model ;
          loss
   -------------------
    0.924061498034721
   (1 row)
   
   
   SELECT madlib.mlp_classification(
       'iris_data_packed',
       'mlp_model',
       'independent_varname',
       'dependent_varname',
       ARRAY[5],
       'learning_rate_init=0.1, learning_rate_policy=constant, n_iterations=10, 
solver=rmsprop, eps=0.5',
       'tanh',
       NULL,
       FALSE,
       TRUE
   );
   INFO:  Iteration: 1, Loss: <0.5>
   CONTEXT:  PL/Python function "mlp_classification"
    mlp_classification
   --------------------
   
   (1 row)
   ```
   
   I believe the wrong loss value also messes up the tolerance calculation. If 
I do not set tolerance, it only runs 1 iteration
   ```
   SELECT madlib.mlp_classification(
       'iris_data_packed',
       'mlp_model',
       'independent_varname',
       'dependent_varname',
       ARRAY[5],
       'learning_rate_init=0.1, learning_rate_policy=constant, n_iterations=10, 
solver=rmsprop',
       'tanh',
       NULL,
       FALSE,
       TRUE
   );
   INFO:  Iteration: 1, Loss: <1e-07>
   CONTEXT:  PL/Python function "mlp_classification"
    mlp_classification
   --------------------
   
   (1 row)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org