Why are we creating work for ourselves? What is wrong with having 2
downloads?
Hadoop accelerator edition exists for the following 2 purposes only:
- accelerate HDFS with Ignite In-Memory File System (IGFS)
- accelerate Hadoop MapReduce with Ignite In-Memory MapReduce
I agree with the original email from Valentin that Spark libs should not be
included into hadoop-accelerator download. Spark integration is not part of
Ignite Hadoop Accelerator and should be included only into the Ignite
fabric download.
D.
On Tue, Dec 6, 2016 at 12:30 AM, Sergey Kozlov <skoz...@gridgain.com> wrote:
> Hi
>
> In general I agree with Vladimir but would suggest more technical details:
>
> Due the need to collect particular CLASS_PATHs for fabric and hadoop
> editions we can change the logic of processing of libs directory
>
> 1. Introduce libs/hadoop and libs/fabric directories. These directories are
> root directories for specific modules for hadoop and fabric
> editions respectively
> 2. Change collecting of directories for CLASS_PATH for ignite.sh:
> - collect everything for libs except libs/hadoop
> - collect everything from libs/fabric
> 3. Add ignite-hadoop-accelerator.{sh|bat} script (also it may make initial
> setup instead of setup-hadoop.sh) that constructs CLASS_PATH by following
> way:
> - collect everything for libs except libs/fabirc
> - collect everything from libs/hadoop
>
> This approach allows us following:
> - share common modules across both editions (just put in libs)
> - do not share edition-specific modules (either put in libs/hadoop or in
> libs/fabric)
>
>
>
>
> On Mon, Dec 5, 2016 at 11:56 PM, Vladimir Ozerov <voze...@gridgain.com>
> wrote:
>
> > Agree. I do not see any reasons to have two different products. Instead,
> > just add ignite-hadoop.jar to distribution, and add separate script to
> > start Accelerator. We can go the same way as we did for "platforms":
> create
> > separate top-level folder "hadoop" in Fabric distribution and put all
> > realted Hadoop Acceleratro stuff there.
> >
> > On Fri, Dec 2, 2016 at 10:46 PM, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > In general, I don't quite understand why we should move any component
> > > outside of Fabric. The concept of Fabric is to have everything, no? :)
> In
> > > other words, if a cluster was once setup for Hadoop Acceleration, why
> not
> > > allow to create a cache and/or run a task using native Ignite APIs
> > sometime
> > > later. We follow this approach with all our components and modules, but
> > not
> > > with ignite-hadoop for some reason.
> > >
> > > If we get rid of Hadoop Accelerator build, initial setup of Hadoop
> > > integration can potentially become a bit more complicated, but with
> > proper
> > > documentation I don't think this is going to be a problem, because it
> > > requires multiple steps now anyway. And frankly the same can be said
> > about
> > > any optional module we have - enabling it requires some additional
> steps
> > as
> > > it doesn't work out of the box.
> > >
> > > -Val
> > >
> > > On Fri, Dec 2, 2016 at 11:38 AM, Denis Magda <dma...@apache.org>
> wrote:
> > >
> > >> Dmitriy,
> > >>
> > >> > - the "lib/" folder has much fewer libraries that in fabric,
> simply
> > >> > becomes many dependencies don't make sense for hadoop environment
> > >>
> > >> This reason why the discussion moved to this direction is exactly in
> > that.
> > >>
> > >> How do we decide what should be a part of Hadoop Accelerator and what
> > >> should be excluded? If you read through Val and Cos comments below
> > you’ll
> > >> get more insights.
> > >>
> > >> In general, we need to have a clear understanding on what's Hadoop
> > >> Accelerator distribution use case. This will help us to come up with a
> > >> final decision.
> > >>
> > >> If the accelerator is supposed to be plugged-in into an existed Hadoop
> > >> environment by enabling MapReduce and/IGFS at the configuration level
> > then
> > >> we should simply remove ignite-indexing, ignite-spark modules and add
> > >> additional logging libs as well as AWS, GCE integrations’ packages.
> > >>
> > >> But, wait, what if a user wants to leverage from Ignite Spark
> > >> Integration, Ignite SQL or Geospatial queries, Ignite streaming
> > >> capabilities after he has already plugged-in the accelerator. What if
> > he is
> > >> ready to modify his existed code. He can’t simply switch to the fabric
> > on
> > >> an application side because the fabric doesn’t include accelerator’s
> > libs
> > >> that are still needed. He can’t solely rely on the accelerator
> > distribution
> > >> as well which misses some libs. And, obviously, the user starts
> > shuffling
> > >> libs in between the fabric and accelerator to get what is required.
> > >>
> > >> Vladimir, can you share your thoughts on this?
> > >>
> > >> —
> > >> Denis
> > >>
> > >>
> > >>
> > >> > On Nov 30, 2016, at 11:18 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > >> wrote:
> > >> >
> > >> > Guys,
> > >> >
> > >> > I just downloaded the hadoop accelerator and here are the
> differences
> > >> from
> > >> > the fabric edition that jump at me right away:
> > >> >
> > >> > - the "bin/" folder has "setup-hadoop" scripts
> > >> > - the "config/" folder has "hadoop" subfolder with necessary
> > >> > hadoop-related configuration
> > >> > - the "lib/" folder has much fewer libraries that in fabric,
> simply
> > >> > becomes many dependencies don't make sense for hadoop environment
> > >> >
> > >> > I currently don't see how we can merge the hadoop accelerator with
> > >> standard
> > >> > fabric edition.
> > >> >
> > >> > D.
> > >> >
> > >> > On Thu, Dec 1, 2016 at 9:54 AM, Denis Magda <dma...@apache.org>
> > wrote:
> > >> >
> > >> >> Vovan,
> > >> >>
> > >> >> As one of hadoop maintainers, please share your point of view on
> > this.
> > >> >>
> > >> >> —
> > >> >> Denis
> > >> >>
> > >> >>> On Nov 30, 2016, at 10:49 PM, Sergey Kozlov <skoz...@gridgain.com
> >
> > >> >> wrote:
> > >> >>>
> > >> >>> Denis
> > >> >>>
> > >> >>> I agree that at the moment there's no reason to split into fabric
> > and
> > >> >>> hadoop editions.
> > >> >>>
> > >> >>> On Thu, Dec 1, 2016 at 4:45 AM, Denis Magda <dma...@apache.org>
> > >> wrote:
> > >> >>>
> > >> >>>> Hadoop Accelerator doesn’t require any additional libraries in
> > >> compare
> > >> >> to
> > >> >>>> those we have in the fabric build. It only lacks some of them as
> > Val
> > >> >>>> mentioned below.
> > >> >>>>
> > >> >>>> Wouldn’t it better to discontinue Hadoop Accelerator edition and
> > >> simply
> > >> >>>> deliver hadoop jar and its configs as a part of the fabric?
> > >> >>>>
> > >> >>>> —
> > >> >>>> Denis
> > >> >>>>
> > >> >>>>> On Nov 27, 2016, at 3:12 PM, Dmitriy Setrakyan <
> > >> dsetrak...@apache.org>
> > >> >>>> wrote:
> > >> >>>>>
> > >> >>>>> Separate edition for the Hadoop Accelerator was primarily driven
> > by
> > >> the
> > >> >>>>> default libraries. Hadoop Accelerator requires many more
> libraries
> > >> as
> > >> >>>> well
> > >> >>>>> as configuration settings compared to the standard fabric
> > download.
> > >> >>>>>
> > >> >>>>> Now, as far as spark integration is concerned, I am not sure
> which
> > >> >>>> edition
> > >> >>>>> it belongs in, Hadoop Accelerator or standard fabric.
> > >> >>>>>
> > >> >>>>> D.
> > >> >>>>>
> > >> >>>>> On Sat, Nov 26, 2016 at 7:39 PM, Denis Magda <dma...@apache.org
> >
> > >> >> wrote:
> > >> >>>>>
> > >> >>>>>> *Dmitriy*,
> > >> >>>>>>
> > >> >>>>>> I do believe that you should know why the community decided to
> a
> > >> >>>> separate
> > >> >>>>>> edition for the Hadoop Accelerator. What was the reason for
> that?
> > >> >>>>>> Presently, as I see, it brings more confusion and difficulties
> > >> rather
> > >> >>>> then
> > >> >>>>>> benefit.
> > >> >>>>>>
> > >> >>>>>> —
> > >> >>>>>> Denis
> > >> >>>>>>
> > >> >>>>>> On Nov 26, 2016, at 2:14 PM, Konstantin Boudnik <
> c...@apache.org>
> > >> >> wrote:
> > >> >>>>>>
> > >> >>>>>> In fact I am very much agree with you. Right now, running the
> > >> >>>> "accelerator"
> > >> >>>>>> component in Bigtop disto gives one a pretty much complete
> fabric
> > >> >>>> anyway.
> > >> >>>>>> But
> > >> >>>>>> in order to make just an accelerator component we perform
> quite a
> > >> bit
> > >> >> of
> > >> >>>>>> woodoo magic during the packaging stage of the Bigtop build,
> > >> shuffling
> > >> >>>> jars
> > >> >>>>>> from here and there. And that's quite crazy, honestly ;)
> > >> >>>>>>
> > >> >>>>>> Cos
> > >> >>>>>>
> > >> >>>>>> On Mon, Nov 21, 2016 at 03:33PM, Valentin Kulichenko wrote:
> > >> >>>>>>
> > >> >>>>>> I tend to agree with Denis. I see only these differences
> between
> > >> >> Hadoop
> > >> >>>>>> Accelerator and Fabric builds (correct me if I miss something):
> > >> >>>>>>
> > >> >>>>>> - Limited set of available modules and no optional modules in
> > >> Hadoop
> > >> >>>>>> Accelerator.
> > >> >>>>>> - No ignite-hadoop module in Fabric.
> > >> >>>>>> - Additional scripts, configs and instructions included in
> Hadoop
> > >> >>>>>> Accelerator.
> > >> >>>>>>
> > >> >>>>>> And the list of included modules frankly looks very weird. Here
> > are
> > >> >> only
> > >> >>>>>> some of the issues I noticed:
> > >> >>>>>>
> > >> >>>>>> - ignite-indexing and ignite-spark are mandatory. Even if we
> need
> > >> them
> > >> >>>>>> for Hadoop Acceleration (which I doubt), are they really
> required
> > >> or
> > >> >>>> can
> > >> >>>>>> be
> > >> >>>>>> optional?
> > >> >>>>>> - We force to use ignite-log4j module without providing other
> > >> logger
> > >> >>>>>> options (e.g., SLF).
> > >> >>>>>> - We don't include ignite-aws module. How to use Hadoop
> > Accelerator
> > >> >>>> with
> > >> >>>>>> S3 discovery?
> > >> >>>>>> - Etc.
> > >> >>>>>>
> > >> >>>>>> It seems to me that if we try to fix all this issue, there will
> > be
> > >> >>>>>> virtually no difference between Fabric and Hadoop Accelerator
> > >> builds
> > >> >>>> except
> > >> >>>>>> couple of scripts and config files. If so, there is no reason
> to
> > >> have
> > >> >>>> two
> > >> >>>>>> builds.
> > >> >>>>>>
> > >> >>>>>> -Val
> > >> >>>>>>
> > >> >>>>>> On Mon, Nov 21, 2016 at 3:13 PM, Denis Magda <
> dma...@apache.org>
> > >> >> wrote:
> > >> >>>>>>
> > >> >>>>>> On the separate note, in the Bigtop, we start looking into
> > changing
> > >> >> the
> > >> >>>>>>
> > >> >>>>>> way we
> > >> >>>>>>
> > >> >>>>>> deliver Ignite and we'll likely to start offering the whole
> 'data
> > >> >>>> fabric'
> > >> >>>>>> experience instead of the mere "hadoop-acceleration”.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> And you still will be using hadoop-accelerator libs of Ignite,
> > >> right?
> > >> >>>>>>
> > >> >>>>>> I’m thinking of if there is a need to keep releasing Hadoop
> > >> >> Accelerator
> > >> >>>> as
> > >> >>>>>> a separate delivery.
> > >> >>>>>> What if we start releasing the accelerator as a part of the
> > >> standard
> > >> >>>>>> fabric binary putting hadoop-accelerator libs under ‘optional’
> > >> folder?
> > >> >>>>>>
> > >> >>>>>> —
> > >> >>>>>> Denis
> > >> >>>>>>
> > >> >>>>>> On Nov 21, 2016, at 12:19 PM, Konstantin Boudnik <
> c...@apache.org
> > >
> > >> >>>> wrote:
> > >> >>>>>>
> > >> >>>>>> What Denis said: spark has been added to the Hadoop accelerator
> > as
> > >> a
> > >> >> way
> > >> >>>>>>
> > >> >>>>>> to
> > >> >>>>>>
> > >> >>>>>> boost the performance of more than just MR compute of the
> Hadoop
> > >> >> stack,
> > >> >>>>>>
> > >> >>>>>> IIRC.
> > >> >>>>>>
> > >> >>>>>> For what it worth, Spark is considered a part of Hadoop at
> large.
> > >> >>>>>>
> > >> >>>>>> On the separate note, in the Bigtop, we start looking into
> > changing
> > >> >> the
> > >> >>>>>>
> > >> >>>>>> way we
> > >> >>>>>>
> > >> >>>>>> deliver Ignite and we'll likely to start offering the whole
> 'data
> > >> >>>> fabric'
> > >> >>>>>> experience instead of the mere "hadoop-acceleration".
> > >> >>>>>>
> > >> >>>>>> Cos
> > >> >>>>>>
> > >> >>>>>> On Mon, Nov 21, 2016 at 09:54AM, Denis Magda wrote:
> > >> >>>>>>
> > >> >>>>>> Val,
> > >> >>>>>>
> > >> >>>>>> Ignite Hadoop module includes not only the map-reduce
> accelerator
> > >> but
> > >> >>>>>>
> > >> >>>>>> Ignite
> > >> >>>>>>
> > >> >>>>>> Hadoop File System component as well. The latter can be used in
> > >> >>>>>>
> > >> >>>>>> deployments
> > >> >>>>>>
> > >> >>>>>> like HDFS+IGFS+Ignite Spark + Spark.
> > >> >>>>>>
> > >> >>>>>> Considering this I’m for the second solution proposed by you:
> put
> > >> both
> > >> >>>>>>
> > >> >>>>>> 2.10
> > >> >>>>>>
> > >> >>>>>> and 2.11 ignite-spark modules under ‘optional’ folder of Ignite
> > >> Hadoop
> > >> >>>>>> Accelerator distribution.
> > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-4254 <
> > >> >>>>>>
> > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-4254>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> BTW, this task may be affected or related to the following
> ones:
> > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-3596 <
> > >> >>>>>>
> > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-3596>
> > >> >>>>>>
> > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-3822
> > >> >>>>>>
> > >> >>>>>> —
> > >> >>>>>> Denis
> > >> >>>>>>
> > >> >>>>>> On Nov 19, 2016, at 1:26 PM, Valentin Kulichenko <
> > >> >>>>>>
> > >> >>>>>> valentin.kuliche...@gmail.com> wrote:
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> Hadoop Accelerator is a plugin to Ignite and this plugin is
> used
> > by
> > >> >>>>>>
> > >> >>>>>> Hadoop
> > >> >>>>>>
> > >> >>>>>> when running its jobs. ignite-spark module only provides
> > IgniteRDD
> > >> >>>>>>
> > >> >>>>>> which
> > >> >>>>>>
> > >> >>>>>> Hadoop obviously will never use.
> > >> >>>>>>
> > >> >>>>>> Is there another use case for Hadoop Accelerator which I'm
> > missing?
> > >> >>>>>>
> > >> >>>>>> -Val
> > >> >>>>>>
> > >> >>>>>> On Sat, Nov 19, 2016 at 3:12 AM, Dmitriy Setrakyan <
> > >> >>>>>>
> > >> >>>>>> dsetrak...@apache.org>
> > >> >>>>>>
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>> Why do you think that spark module is not needed in our hadoop
> > >> build?
> > >> >>>>>>
> > >> >>>>>> On Fri, Nov 18, 2016 at 5:44 PM, Valentin Kulichenko <
> > >> >>>>>> valentin.kuliche...@gmail.com> wrote:
> > >> >>>>>>
> > >> >>>>>> Folks,
> > >> >>>>>>
> > >> >>>>>> Is there anyone who understands the purpose of including
> > >> ignite-spark
> > >> >>>>>> module in the Hadoop Accelerator build? I can't figure out a
> use
> > >> >>>>>>
> > >> >>>>>> case for
> > >> >>>>>>
> > >> >>>>>> which it's needed.
> > >> >>>>>>
> > >> >>>>>> In case we actually need it there, there is an issue then. We
> > >> >>>>>>
> > >> >>>>>> actually
> > >> >>>>>>
> > >> >>>>>> have
> > >> >>>>>>
> > >> >>>>>> two ignite-spark modules, for 2.10 and 2.11. In Fabric build
> > >> >>>>>>
> > >> >>>>>> everything
> > >> >>>>>>
> > >> >>>>>> is
> > >> >>>>>>
> > >> >>>>>> good, we put both in 'optional' folder and user can enable
> either
> > >> >>>>>>
> > >> >>>>>> one.
> > >> >>>>>>
> > >> >>>>>> But
> > >> >>>>>>
> > >> >>>>>> in Hadoop Accelerator there is only 2.11 which means that the
> > build
> > >> >>>>>>
> > >> >>>>>> doesn't
> > >> >>>>>>
> > >> >>>>>> work with 2.10 out of the box.
> > >> >>>>>>
> > >> >>>>>> We should either remove the module from the build, or fix the
> > >> issue.
> > >> >>>>>>
> > >> >>>>>> -Val
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> >>> --
> > >> >>> Sergey Kozlov
> > >> >>> GridGain Systems
> > >> >>> www.gridgain.com
> > >> >>
> > >> >>
> > >>
> > >>
> > >
> >
> >
> > --
> > Vladimir Ozerov
> > Senior Software Architect
> > GridGain Systems
> > www.gridgain.com
> > *+7 (960) 283 98 40*
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>
