Thanks Kuan for insight. Much appreciated. Mich
Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 19 January 2017 at 16:43, Kuan Feng <k...@ca.ibm.com> wrote: > Greetings, Dr Mich Talebzadeh, > > This is Kuan from IBM Platform team. Thank you for your interests in > Platform Symphony and Spark product. > > I'm writing this mail to clarify the "EGO-YARN" in that blog post you were > referring to. EGO is an enterprise > resource orchestration component used in all IBM Platform products. It > provides rich feature for better manage > various resource across huge cluster. EGO has its own interface, which is > different to YARN. EGO-YARN a Symphony component > to simulate YARN interface on top of EGO. In other words, it provides YARN > interface, but does not actually use YARN. > > In terms of improving Spark job management, we have product "IBM Spectrum > Conductor with Spark" > https://www.ibm.com/developerworks/community/ > blogs/281605c9-7369-46dc-ad03-70d9ad377480/entry/Conductor_ > with_Spark_2.2?lang=en > It provides rich enterprise spark job management features. Please feel > free to let me know if you have any questions > or something you want to have with spark. > > Then back to the blog post itself, since all Platform Symphony products > use the same resource orchestrator behind the scene, > this blog post is to help customers to onboard their existing Hadoop > applications (using YARN interface) to our Platform Computing cluster > for better application efficiency and cluster resource utilization. > > So, if you just want to start something with Spark, I'd like to suggest to > go to IBM Spectrum Conductor with Spark directly. If > you have a solution with Spark and other Hadoop frameworks together, and > you want to put them on the same enterprise cluster managed > by our product, you may try this blog to onboard your Hadoop applications. > Please feel free to let me know if you want to further discuss. > > btw, IBM renamed "Platform" product family to "Spectrum" product family, > you may see both in materials depends on when it is published, they are the > same. > > best regards! > > Kuan Feng (冯宽) > > Software Architect, IBM Spectrum Computing Development, IBM Systems > 3600 Steeles Ave. East Markham, ON Canada L3R 9Z7 > Tel: 1-905-316-0550 <(905)%20316-0550> > E-mail: k...@ca.ibm.com > > > From: Mich Talebzadeh <mich.talebza...@gmail.com> > To: "user @spark" <user@spark.apache.org> > Date: 19/01/2017 15:23 > Subject: Yarn resource management for Spark with IBM Platform Symphony > ------------------------------ > > > > > Hi, > > IBM stating that when *Yarn is integrated with IBM platform symphony,* > <https://www.ibm.com/developerworks/community/blogs/46ecec34-bd69-43f7-a627-7c469c1eddf8/entry/Enable_your_Hadoop_application_to_run_on_an_IBM_Platform_Symphony_EGO_YARN_environment4?lang=en> > you have more control on your Spark jobs including HA failover using > Platform Symphony ha. > > Has anyone had any experience of using Yarn with IBM Platform Symphony at > all including Proof of Concept? > > Thanks > > > Dr Mich Talebzadeh > > LinkedIn > *https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw* > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw> > > > *http://talebzadehmich.wordpress.com* > <http://talebzadehmich.wordpress.com/> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > >
