Re: [openstack-dev] [Sahara] improve oozie engine for common lib management
I think it could be useful and pretty easy to implement. Feel free to file blueprint+spec. Andrew. On Mon, May 11, 2015 at 9:49 AM, Sergey Lukjanov slukja...@mirantis.com wrote: Hi, do you think it could be implemented based on job binaries? It sounds like it's a type of job binary that should be always uploaded to Oozie for the tenant where it lives. (A bit crazy, but could useful). Thanks. On Mon, May 11, 2015 at 6:39 PM, lu jander juvenboy1...@gmail.com wrote: Hi, All Currently, oozie share lib is not well known and can be hardly used by the users, so I think we can make it less oozieness and more friendly for the users, it can be used for running jobs which are using third party libs. If many jobs use the same libs, oozie share lib can make it as a common share lib. here is the bp, https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib I will write a spec soon after scheduled edp jobs bp. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours, Sergey Lukjanov Sahara Technical Lead (OpenStack Data Processing) Principal Software Engineer Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Sahara] improve oozie engine for common lib management
Hi, All Currently, oozie share lib is not well known and can be hardly used by the users, so I think we can make it less oozieness and more friendly for the users, it can be used for running jobs which are using third party libs. If many jobs use the same libs, oozie share lib can make it as a common share lib. here is the bp, https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib I will write a spec soon after scheduled edp jobs bp. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Sahara] improve oozie engine for common lib management
Hi, I’m chen, working in the same team as lu jander. Actually , we’re planning to implement the common lib management base on bp expose-elementshttps://blueprints.launchpad.net/sahara/+spec/expose-elements. We would like to build common supported 3-rd part libs into sahara images, so user do not need to upload these libs themselves, and each lib would be an element which would be an attribute value for a running cluster. The improve for oozie engine is used for other “common libs” user want to use but could not be found in Sahara image. User do has the ability to upload them to sahara by themselves, and it would be implemented based on job binaries. With a user friendly interface to let user know what libs are available on currently running cluster which would combine these libs already in images and uploaded by users, user only need to pick up libs they want to use. Oozie.libpath would be set automately by sahara based on the libs user chose. Thanks. -chen From: lu jander [mailto:juvenboy1...@gmail.com] Sent: Tuesday, May 12, 2015 1:10 AM To: OpenStack Development Mailing List (not for usage questions) Cc: Lu, Huichun; Yu, Zhidong Subject: Re: [openstack-dev] [Sahara] improve oozie engine for common lib management Hi Sergey yes, it is based on job binaries. what we should do is provide a user friendly interface, to let user know how to set oozie.libpath( may be we can show it as share lib path in the job execution config horizon page) there are two scenarios as below: (1) if Job A and Job B all need lib A, we can upload lib A through job binary into sahara, then before we run Job A and B, user can set oozie.libpath to the path where lib A exists. (2) if Job A and Job B need system lib or other libs already exists in the OS, user needs not to upload additional job binary 2015-05-12 0:49 GMT+08:00 Sergey Lukjanov slukja...@mirantis.commailto:slukja...@mirantis.com: Hi, do you think it could be implemented based on job binaries? It sounds like it's a type of job binary that should be always uploaded to Oozie for the tenant where it lives. (A bit crazy, but could useful). Thanks. On Mon, May 11, 2015 at 6:39 PM, lu jander juvenboy1...@gmail.commailto:juvenboy1...@gmail.com wrote: Hi, All Currently, oozie share lib is not well known and can be hardly used by the users, so I think we can make it less oozieness and more friendly for the users, it can be used for running jobs which are using third party libs. If many jobs use the same libs, oozie share lib can make it as a common share lib. here is the bp, https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib I will write a spec soon after scheduled edp jobs bp. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours, Sergey Lukjanov Sahara Technical Lead (OpenStack Data Processing) Principal Software Engineer Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Sahara] improve oozie engine for common lib management
Hi, do you think it could be implemented based on job binaries? It sounds like it's a type of job binary that should be always uploaded to Oozie for the tenant where it lives. (A bit crazy, but could useful). Thanks. On Mon, May 11, 2015 at 6:39 PM, lu jander juvenboy1...@gmail.com wrote: Hi, All Currently, oozie share lib is not well known and can be hardly used by the users, so I think we can make it less oozieness and more friendly for the users, it can be used for running jobs which are using third party libs. If many jobs use the same libs, oozie share lib can make it as a common share lib. here is the bp, https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib I will write a spec soon after scheduled edp jobs bp. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours, Sergey Lukjanov Sahara Technical Lead (OpenStack Data Processing) Principal Software Engineer Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Sahara] improve oozie engine for common lib management
Hi Sergey yes, it is based on job binaries. what we should do is provide a user friendly interface, to let user know how to set oozie.libpath( may be we can show it as share lib path in the job execution config horizon page) there are two scenarios as below: (1) if Job A and Job B all need lib A, we can upload lib A through job binary into sahara, then before we run Job A and B, user can set oozie.libpath to the path where lib A exists. (2) if Job A and Job B need system lib or other libs already exists in the OS, user needs not to upload additional job binary 2015-05-12 0:49 GMT+08:00 Sergey Lukjanov slukja...@mirantis.com: Hi, do you think it could be implemented based on job binaries? It sounds like it's a type of job binary that should be always uploaded to Oozie for the tenant where it lives. (A bit crazy, but could useful). Thanks. On Mon, May 11, 2015 at 6:39 PM, lu jander juvenboy1...@gmail.com wrote: Hi, All Currently, oozie share lib is not well known and can be hardly used by the users, so I think we can make it less oozieness and more friendly for the users, it can be used for running jobs which are using third party libs. If many jobs use the same libs, oozie share lib can make it as a common share lib. here is the bp, https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib I will write a spec soon after scheduled edp jobs bp. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours, Sergey Lukjanov Sahara Technical Lead (OpenStack Data Processing) Principal Software Engineer Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
