Hi, Igniters!
Currently we don't have an ability to deploy automatically user-defined
preprocessors and vectorizers. Client's code should be deployed manually to
Ignite server nodes.
I have an idea how to fix it. If we pass user's classloader and one of
user-defined classes from fit-level to
ComputeUtils.affinityCallWithRetries() then we wiil be able to use
GridPeerDeployAware interface to send informtation about this classloader
to server nodes.
To support this ability we can define interfaces like these:
public interface DeployableObject {
public List<Object> getDependencies();
}
and
public interface DeployingContext {
public Class<?> userClass();
public ClassLoader clientClassLoader();
}
DeployableObject will be mark for our ignite-ml final classes like trainers
or concrete preprocessors and it can be able to return all dependencies
that should be deployed to server nodes if it's needed. If these
dependencies are DeployableObjects too then depenndencies will be unfolded
recursively. Classes that isn't defined as DeployableObject will be
recognized as user-defined (NOTE: all leaf classes in our hierarchy will be
DeployableObject).
This list of DeployableObjects will be user for define user class loader
and one of these objects will be used for passing to GridPeerDeployAware.
So, this logic allows to pass user-defined Preprocessors and Vectorizers to
training algorithms and pipelines.
What do you think?
Sincerely
Alexey Platonov
