I am looking to implement a custom executor, but I do not want to require users to distribute the JARs and other related resources associated with my executor onto every slave node.
I see from the code that I can addUris and even ".tgz" that get downloaded and extracted during the execution of the executor, but I am not clear if this will be practical if my JARs and other resources needed by the java executor will be in the 10s or 100s of megabytes. It is important that I don't require the user to install anything on the slave nodes other than the base Mesos package since my Framework application is very dynamic and code jars can vary by job in most cases. So I really want everything dynamically downloaded per job run and I would also like the resources to be cached if they have not changed...etc. Is there any plan to support something like what exist in YARN which deals with these of lifecycle resource manage application dependencies across the cluster? Here is theYARN overview of what I am kind of asking about: http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/ Thanks in advance. Thanks, Sam Taha http://www.grandlogic.com
