Baunsgaard commented on pull request #1154:
URL: https://github.com/apache/systemds/pull/1154#issuecomment-760756975
> There is now a second createPS function in the
ParamservBuiltinCPInstruction. If all the additional arguments are specified
the parameter server is able to validate after each epoch. It will do so if
LOG.info is enabled.
> This feature is implemented for the federated parameter server ONLY, but
is easily implemented for the other cases too.
I can understand why this is done with the logging. But i think this is a
slight misuse of the logging in the system, since ideally we (or at least me)
want no difference in execution (or as close to) when we set different logging
levels. This violates it since the validation in the parameter server can take
many seconds of execution time.
I would suggest adding a parameter some other way to enable / disable the
validation. While it is still possible that we output the validation scores
through LOG.info when that other "unrelated to logging level" setting is set.
But maybe I'm the only one with this opinion? @mboehm7 ?
furthermore designwise, you might want to consider doing this validation in
parallel with the distribution of model parameters. since we know that the
controller is doing nothing while the workers are training, you could do the
validation on a separate thread at no cost to execution time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
