Re: [DISCUSS] Hadoop Accelerator as part of Apache Ignite
Hi Guys, Sorry about missing in action for couple weeks. Just came back from family vacation. Love to see this coming as part of ignite. As Branko and Cos had mentioned, I think GridGain needs to send SGA for this new component since it is not part of initial proposal. - Henry On Tue, Dec 30, 2014 at 10:56 AM, Dmitriy Setrakyan wrote: > Hi, > > I want to open up a discussion about adding GridGain Hadoop Accelerator as > a feature of Apache Ignite. > > As some of you may know, Hadoop Accelerator is now offered as a part of > GridGain open source edition. It is built on top of Ignite In-Memory Data > Fabric technology and provides plug-n-play acceleration for Hadoop. It also > recently has been integrated with Apache BigTop. > > The acceleration is achieved by providing the following Hadoop components > in memory: > >- IgniteFS, in-memory Hadoop-compliant file system, which natively plugs >into Hadoop, and is built on top of Ignite data grid. >- Ignite MapReduce, very fast Hadoop MapReduce implementation, which is >built on top of Ignite computation framework. > > I anticipate that some questions will arise around how Ignite Hadoop > Accelerator is different from Apache Spark. The reality is that they are > very different. > > One of the main differences is that Ignite Hadoop Accelerator will offer > acceleration of the existing Hadoop MapReduce computations that run > natively on Hadoop, while Spark essentially takes you off of Hadoop > MapReduce into its own DSL. Additionally, because of the fast MapReduce > implementation, Ignite Hadoop Accelerator will also accelerate native Hive > queries, while Spark provides its own SQL engine. > > More information about Hadoop Accelerator can be found here: > http://hadoop.gridgain.org/ > > Please let me know your thoughts.
Re: [DISCUSS] Hadoop Accelerator as part of Apache Ignite
Agree about SGA. I will try to get it executed next week. D. On Sat, Jan 3, 2015 at 3:02 AM, Branko Čibej wrote: > On 02.01.2015 19:49, Konstantin Boudnik wrote: > > +1 on what Brane said. Considering that the Accelerator code is already > in the > > open and licensed under ASL2.0 I don't see any issues with just adding > it to > > the existing Ignite code base. > > An SGA must still be executed since that's what allows us to change the > copyright, along with relicensing (which, as you say, is not needed in > this case). > > -- Brane > > > > As far as the PPMC is Ok with handling the > > additional dev. load. > > > > With my Apache Bigtop hat on, I'd say it'd be even better as we'll be > able to > > use the Apache Incubator project instead of the current github source. > > > > Cos > > > > On Tue, Dec 30, 2014 at 10:56AM, Dmitriy Setrakyan wrote: > >> Hi, > >> > >> I want to open up a discussion about adding GridGain Hadoop Accelerator > as > >> a feature of Apache Ignite. > >> > >> As some of you may know, Hadoop Accelerator is now offered as a part of > >> GridGain open source edition. It is built on top of Ignite In-Memory > Data > >> Fabric technology and provides plug-n-play acceleration for Hadoop. It > also > >> recently has been integrated with Apache BigTop. > >> > >> The acceleration is achieved by providing the following Hadoop > components > >> in memory: > >> > >>- IgniteFS, in-memory Hadoop-compliant file system, which natively > plugs > >>into Hadoop, and is built on top of Ignite data grid. > >>- Ignite MapReduce, very fast Hadoop MapReduce implementation, which > is > >>built on top of Ignite computation framework. > >> > >> I anticipate that some questions will arise around how Ignite Hadoop > >> Accelerator is different from Apache Spark. The reality is that they are > >> very different. > >> > >> One of the main differences is that Ignite Hadoop Accelerator will offer > >> acceleration of the existing Hadoop MapReduce computations that run > >> natively on Hadoop, while Spark essentially takes you off of Hadoop > >> MapReduce into its own DSL. Additionally, because of the fast MapReduce > >> implementation, Ignite Hadoop Accelerator will also accelerate native > Hive > >> queries, while Spark provides its own SQL engine. > >> > >> More information about Hadoop Accelerator can be found here: > >> http://hadoop.gridgain.org/ > >> > >> Please let me know your thoughts. > >
Re: [DISCUSS] Hadoop Accelerator as part of Apache Ignite
On 02.01.2015 19:49, Konstantin Boudnik wrote: > +1 on what Brane said. Considering that the Accelerator code is already in the > open and licensed under ASL2.0 I don't see any issues with just adding it to > the existing Ignite code base. An SGA must still be executed since that's what allows us to change the copyright, along with relicensing (which, as you say, is not needed in this case). -- Brane > As far as the PPMC is Ok with handling the > additional dev. load. > > With my Apache Bigtop hat on, I'd say it'd be even better as we'll be able to > use the Apache Incubator project instead of the current github source. > > Cos > > On Tue, Dec 30, 2014 at 10:56AM, Dmitriy Setrakyan wrote: >> Hi, >> >> I want to open up a discussion about adding GridGain Hadoop Accelerator as >> a feature of Apache Ignite. >> >> As some of you may know, Hadoop Accelerator is now offered as a part of >> GridGain open source edition. It is built on top of Ignite In-Memory Data >> Fabric technology and provides plug-n-play acceleration for Hadoop. It also >> recently has been integrated with Apache BigTop. >> >> The acceleration is achieved by providing the following Hadoop components >> in memory: >> >>- IgniteFS, in-memory Hadoop-compliant file system, which natively plugs >>into Hadoop, and is built on top of Ignite data grid. >>- Ignite MapReduce, very fast Hadoop MapReduce implementation, which is >>built on top of Ignite computation framework. >> >> I anticipate that some questions will arise around how Ignite Hadoop >> Accelerator is different from Apache Spark. The reality is that they are >> very different. >> >> One of the main differences is that Ignite Hadoop Accelerator will offer >> acceleration of the existing Hadoop MapReduce computations that run >> natively on Hadoop, while Spark essentially takes you off of Hadoop >> MapReduce into its own DSL. Additionally, because of the fast MapReduce >> implementation, Ignite Hadoop Accelerator will also accelerate native Hive >> queries, while Spark provides its own SQL engine. >> >> More information about Hadoop Accelerator can be found here: >> http://hadoop.gridgain.org/ >> >> Please let me know your thoughts.
Re: [DISCUSS] Hadoop Accelerator as part of Apache Ignite
+1 on what Brane said. Considering that the Accelerator code is already in the open and licensed under ASL2.0 I don't see any issues with just adding it to the existing Ignite code base. As far as the PPMC is Ok with handling the additional dev. load. With my Apache Bigtop hat on, I'd say it'd be even better as we'll be able to use the Apache Incubator project instead of the current github source. Cos On Tue, Dec 30, 2014 at 10:56AM, Dmitriy Setrakyan wrote: > Hi, > > I want to open up a discussion about adding GridGain Hadoop Accelerator as > a feature of Apache Ignite. > > As some of you may know, Hadoop Accelerator is now offered as a part of > GridGain open source edition. It is built on top of Ignite In-Memory Data > Fabric technology and provides plug-n-play acceleration for Hadoop. It also > recently has been integrated with Apache BigTop. > > The acceleration is achieved by providing the following Hadoop components > in memory: > >- IgniteFS, in-memory Hadoop-compliant file system, which natively plugs >into Hadoop, and is built on top of Ignite data grid. >- Ignite MapReduce, very fast Hadoop MapReduce implementation, which is >built on top of Ignite computation framework. > > I anticipate that some questions will arise around how Ignite Hadoop > Accelerator is different from Apache Spark. The reality is that they are > very different. > > One of the main differences is that Ignite Hadoop Accelerator will offer > acceleration of the existing Hadoop MapReduce computations that run > natively on Hadoop, while Spark essentially takes you off of Hadoop > MapReduce into its own DSL. Additionally, because of the fast MapReduce > implementation, Ignite Hadoop Accelerator will also accelerate native Hive > queries, while Spark provides its own SQL engine. > > More information about Hadoop Accelerator can be found here: > http://hadoop.gridgain.org/ > > Please let me know your thoughts. signature.asc Description: Digital signature
Re: [DISCUSS] Hadoop Accelerator as part of Apache Ignite
On 30.12.2014 19:56, Dmitriy Setrakyan wrote: > Hi, > > I want to open up a discussion about adding GridGain Hadoop Accelerator as > a feature of Apache Ignite. > > As some of you may know, Hadoop Accelerator is now offered as a part of > GridGain open source edition. It is built on top of Ignite In-Memory Data > Fabric technology and provides plug-n-play acceleration for Hadoop. It also > recently has been integrated with Apache BigTop. > > The acceleration is achieved by providing the following Hadoop components > in memory: > >- IgniteFS, in-memory Hadoop-compliant file system, which natively plugs >into Hadoop, and is built on top of Ignite data grid. >- Ignite MapReduce, very fast Hadoop MapReduce implementation, which is >built on top of Ignite computation framework. > > I anticipate that some questions will arise around how Ignite Hadoop > Accelerator is different from Apache Spark. The reality is that they are > very different. > > One of the main differences is that Ignite Hadoop Accelerator will offer > acceleration of the existing Hadoop MapReduce computations that run > natively on Hadoop, while Spark essentially takes you off of Hadoop > MapReduce into its own DSL. Additionally, because of the fast MapReduce > implementation, Ignite Hadoop Accelerator will also accelerate native Hive > queries, while Spark provides its own SQL engine. > > More information about Hadoop Accelerator can be found here: > http://hadoop.gridgain.org/ > > Please let me know your thoughts. Actually, I haven't a clue what the Incubator policy is on additional software grants to a podling. As far as I can tell, this should be fine, if the PPMC is willing to manage additional code, and GridGain is willing to extend the SGA (or create another SGA) for this new code, and the Champion and Mentors don't object ... a lot of "if"s but I'm guessing they're mostly irrelevant for this case. :) The IPMC (general@incubator) should know in advance that this is going to happen, and the new or extended SGA must by all means be submitted before the code import; I can't think of any other considerations. -- Brane
