Work for ourselves - is to maintain two separate editions, while everything
can be easily merged into a single distribution.
On Wed, Dec 7, 2016 at 3:29 AM, Dmitriy Setrakyan <dsetrak...@apache.org>
wrote:
> Why are we creating work for ourselves? What is wrong with having 2
> downloads?
>
> Hadoop accelerator edition exists for the following 2 purposes only:
>
> - accelerate HDFS with Ignite In-Memory File System (IGFS)
> - accelerate Hadoop MapReduce with Ignite In-Memory MapReduce
>
> I agree with the original email from Valentin that Spark libs should not be
> included into hadoop-accelerator download. Spark integration is not part of
> Ignite Hadoop Accelerator and should be included only into the Ignite
> fabric download.
>
> D.
>
>
>
> On Tue, Dec 6, 2016 at 12:30 AM, Sergey Kozlov <skoz...@gridgain.com>
> wrote:
>
> > Hi
> >
> > In general I agree with Vladimir but would suggest more technical
> details:
> >
> > Due the need to collect particular CLASS_PATHs for fabric and hadoop
> > editions we can change the logic of processing of libs directory
> >
> > 1. Introduce libs/hadoop and libs/fabric directories. These directories
> are
> > root directories for specific modules for hadoop and fabric
> > editions respectively
> > 2. Change collecting of directories for CLASS_PATH for ignite.sh:
> > - collect everything for libs except libs/hadoop
> > - collect everything from libs/fabric
> > 3. Add ignite-hadoop-accelerator.{sh|bat} script (also it may make
> initial
> > setup instead of setup-hadoop.sh) that constructs CLASS_PATH by following
> > way:
> > - collect everything for libs except libs/fabirc
> > - collect everything from libs/hadoop
> >
> > This approach allows us following:
> > - share common modules across both editions (just put in libs)
> > - do not share edition-specific modules (either put in libs/hadoop or in
> > libs/fabric)
> >
> >
> >
> >
> > On Mon, Dec 5, 2016 at 11:56 PM, Vladimir Ozerov <voze...@gridgain.com>
> > wrote:
> >
> > > Agree. I do not see any reasons to have two different products.
> Instead,
> > > just add ignite-hadoop.jar to distribution, and add separate script to
> > > start Accelerator. We can go the same way as we did for "platforms":
> > create
> > > separate top-level folder "hadoop" in Fabric distribution and put all
> > > realted Hadoop Acceleratro stuff there.
> > >
> > > On Fri, Dec 2, 2016 at 10:46 PM, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > In general, I don't quite understand why we should move any component
> > > > outside of Fabric. The concept of Fabric is to have everything, no?
> :)
> > In
> > > > other words, if a cluster was once setup for Hadoop Acceleration, why
> > not
> > > > allow to create a cache and/or run a task using native Ignite APIs
> > > sometime
> > > > later. We follow this approach with all our components and modules,
> but
> > > not
> > > > with ignite-hadoop for some reason.
> > > >
> > > > If we get rid of Hadoop Accelerator build, initial setup of Hadoop
> > > > integration can potentially become a bit more complicated, but with
> > > proper
> > > > documentation I don't think this is going to be a problem, because it
> > > > requires multiple steps now anyway. And frankly the same can be said
> > > about
> > > > any optional module we have - enabling it requires some additional
> > steps
> > > as
> > > > it doesn't work out of the box.
> > > >
> > > > -Val
> > > >
> > > > On Fri, Dec 2, 2016 at 11:38 AM, Denis Magda <dma...@apache.org>
> > wrote:
> > > >
> > > >> Dmitriy,
> > > >>
> > > >> > - the "lib/" folder has much fewer libraries that in fabric,
> > simply
> > > >> > becomes many dependencies don't make sense for hadoop
> environment
> > > >>
> > > >> This reason why the discussion moved to this direction is exactly in
> > > that.
> > > >>
> > > >> How do we decide what should be a part of Hadoop Accelerator and
> what
> > > >> should be excluded? If you read through Val and Cos comments below
> > > you’ll
> > > >> get more insights.
> > > >>
> > > >> In general, we need to have a clear understanding on what's Hadoop
> > > >> Accelerator distribution use case. This will help us to come up
> with a
> > > >> final decision.
> > > >>
> > > >> If the accelerator is supposed to be plugged-in into an existed
> Hadoop
> > > >> environment by enabling MapReduce and/IGFS at the configuration
> level
> > > then
> > > >> we should simply remove ignite-indexing, ignite-spark modules and
> add
> > > >> additional logging libs as well as AWS, GCE integrations’ packages.
> > > >>
> > > >> But, wait, what if a user wants to leverage from Ignite Spark
> > > >> Integration, Ignite SQL or Geospatial queries, Ignite streaming
> > > >> capabilities after he has already plugged-in the accelerator. What
> if
> > > he is
> > > >> ready to modify his existed code. He can’t simply switch to the
> fabric
> > > on
> > > >> an application side because the fabric doesn’t include accelerator’s
> > > libs
> > > >> that are still needed. He can’t solely rely on the accelerator
> > > distribution
> > > >> as well which misses some libs. And, obviously, the user starts
> > > shuffling
> > > >> libs in between the fabric and accelerator to get what is required.
> > > >>
> > > >> Vladimir, can you share your thoughts on this?
> > > >>
> > > >> —
> > > >> Denis
> > > >>
> > > >>
> > > >>
> > > >> > On Nov 30, 2016, at 11:18 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > Guys,
> > > >> >
> > > >> > I just downloaded the hadoop accelerator and here are the
> > differences
> > > >> from
> > > >> > the fabric edition that jump at me right away:
> > > >> >
> > > >> > - the "bin/" folder has "setup-hadoop" scripts
> > > >> > - the "config/" folder has "hadoop" subfolder with necessary
> > > >> > hadoop-related configuration
> > > >> > - the "lib/" folder has much fewer libraries that in fabric,
> > simply
> > > >> > becomes many dependencies don't make sense for hadoop
> environment
> > > >> >
> > > >> > I currently don't see how we can merge the hadoop accelerator with
> > > >> standard
> > > >> > fabric edition.
> > > >> >
> > > >> > D.
> > > >> >
> > > >> > On Thu, Dec 1, 2016 at 9:54 AM, Denis Magda <dma...@apache.org>
> > > wrote:
> > > >> >
> > > >> >> Vovan,
> > > >> >>
> > > >> >> As one of hadoop maintainers, please share your point of view on
> > > this.
> > > >> >>
> > > >> >> —
> > > >> >> Denis
> > > >> >>
> > > >> >>> On Nov 30, 2016, at 10:49 PM, Sergey Kozlov <
> skoz...@gridgain.com
> > >
> > > >> >> wrote:
> > > >> >>>
> > > >> >>> Denis
> > > >> >>>
> > > >> >>> I agree that at the moment there's no reason to split into
> fabric
> > > and
> > > >> >>> hadoop editions.
> > > >> >>>
> > > >> >>> On Thu, Dec 1, 2016 at 4:45 AM, Denis Magda <dma...@apache.org>
> > > >> wrote:
> > > >> >>>
> > > >> >>>> Hadoop Accelerator doesn’t require any additional libraries in
> > > >> compare
> > > >> >> to
> > > >> >>>> those we have in the fabric build. It only lacks some of them
> as
> > > Val
> > > >> >>>> mentioned below.
> > > >> >>>>
> > > >> >>>> Wouldn’t it better to discontinue Hadoop Accelerator edition
> and
> > > >> simply
> > > >> >>>> deliver hadoop jar and its configs as a part of the fabric?
> > > >> >>>>
> > > >> >>>> —
> > > >> >>>> Denis
> > > >> >>>>
> > > >> >>>>> On Nov 27, 2016, at 3:12 PM, Dmitriy Setrakyan <
> > > >> dsetrak...@apache.org>
> > > >> >>>> wrote:
> > > >> >>>>>
> > > >> >>>>> Separate edition for the Hadoop Accelerator was primarily
> driven
> > > by
> > > >> the
> > > >> >>>>> default libraries. Hadoop Accelerator requires many more
> > libraries
> > > >> as
> > > >> >>>> well
> > > >> >>>>> as configuration settings compared to the standard fabric
> > > download.
> > > >> >>>>>
> > > >> >>>>> Now, as far as spark integration is concerned, I am not sure
> > which
> > > >> >>>> edition
> > > >> >>>>> it belongs in, Hadoop Accelerator or standard fabric.
> > > >> >>>>>
> > > >> >>>>> D.
> > > >> >>>>>
> > > >> >>>>> On Sat, Nov 26, 2016 at 7:39 PM, Denis Magda <
> dma...@apache.org
> > >
> > > >> >> wrote:
> > > >> >>>>>
> > > >> >>>>>> *Dmitriy*,
> > > >> >>>>>>
> > > >> >>>>>> I do believe that you should know why the community decided
> to
> > a
> > > >> >>>> separate
> > > >> >>>>>> edition for the Hadoop Accelerator. What was the reason for
> > that?
> > > >> >>>>>> Presently, as I see, it brings more confusion and
> difficulties
> > > >> rather
> > > >> >>>> then
> > > >> >>>>>> benefit.
> > > >> >>>>>>
> > > >> >>>>>> —
> > > >> >>>>>> Denis
> > > >> >>>>>>
> > > >> >>>>>> On Nov 26, 2016, at 2:14 PM, Konstantin Boudnik <
> > c...@apache.org>
> > > >> >> wrote:
> > > >> >>>>>>
> > > >> >>>>>> In fact I am very much agree with you. Right now, running the
> > > >> >>>> "accelerator"
> > > >> >>>>>> component in Bigtop disto gives one a pretty much complete
> > fabric
> > > >> >>>> anyway.
> > > >> >>>>>> But
> > > >> >>>>>> in order to make just an accelerator component we perform
> > quite a
> > > >> bit
> > > >> >> of
> > > >> >>>>>> woodoo magic during the packaging stage of the Bigtop build,
> > > >> shuffling
> > > >> >>>> jars
> > > >> >>>>>> from here and there. And that's quite crazy, honestly ;)
> > > >> >>>>>>
> > > >> >>>>>> Cos
> > > >> >>>>>>
> > > >> >>>>>> On Mon, Nov 21, 2016 at 03:33PM, Valentin Kulichenko wrote:
> > > >> >>>>>>
> > > >> >>>>>> I tend to agree with Denis. I see only these differences
> > between
> > > >> >> Hadoop
> > > >> >>>>>> Accelerator and Fabric builds (correct me if I miss
> something):
> > > >> >>>>>>
> > > >> >>>>>> - Limited set of available modules and no optional modules in
> > > >> Hadoop
> > > >> >>>>>> Accelerator.
> > > >> >>>>>> - No ignite-hadoop module in Fabric.
> > > >> >>>>>> - Additional scripts, configs and instructions included in
> > Hadoop
> > > >> >>>>>> Accelerator.
> > > >> >>>>>>
> > > >> >>>>>> And the list of included modules frankly looks very weird.
> Here
> > > are
> > > >> >> only
> > > >> >>>>>> some of the issues I noticed:
> > > >> >>>>>>
> > > >> >>>>>> - ignite-indexing and ignite-spark are mandatory. Even if we
> > need
> > > >> them
> > > >> >>>>>> for Hadoop Acceleration (which I doubt), are they really
> > required
> > > >> or
> > > >> >>>> can
> > > >> >>>>>> be
> > > >> >>>>>> optional?
> > > >> >>>>>> - We force to use ignite-log4j module without providing other
> > > >> logger
> > > >> >>>>>> options (e.g., SLF).
> > > >> >>>>>> - We don't include ignite-aws module. How to use Hadoop
> > > Accelerator
> > > >> >>>> with
> > > >> >>>>>> S3 discovery?
> > > >> >>>>>> - Etc.
> > > >> >>>>>>
> > > >> >>>>>> It seems to me that if we try to fix all this issue, there
> will
> > > be
> > > >> >>>>>> virtually no difference between Fabric and Hadoop Accelerator
> > > >> builds
> > > >> >>>> except
> > > >> >>>>>> couple of scripts and config files. If so, there is no reason
> > to
> > > >> have
> > > >> >>>> two
> > > >> >>>>>> builds.
> > > >> >>>>>>
> > > >> >>>>>> -Val
> > > >> >>>>>>
> > > >> >>>>>> On Mon, Nov 21, 2016 at 3:13 PM, Denis Magda <
> > dma...@apache.org>
> > > >> >> wrote:
> > > >> >>>>>>
> > > >> >>>>>> On the separate note, in the Bigtop, we start looking into
> > > changing
> > > >> >> the
> > > >> >>>>>>
> > > >> >>>>>> way we
> > > >> >>>>>>
> > > >> >>>>>> deliver Ignite and we'll likely to start offering the whole
> > 'data
> > > >> >>>> fabric'
> > > >> >>>>>> experience instead of the mere "hadoop-acceleration”.
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>> And you still will be using hadoop-accelerator libs of
> Ignite,
> > > >> right?
> > > >> >>>>>>
> > > >> >>>>>> I’m thinking of if there is a need to keep releasing Hadoop
> > > >> >> Accelerator
> > > >> >>>> as
> > > >> >>>>>> a separate delivery.
> > > >> >>>>>> What if we start releasing the accelerator as a part of the
> > > >> standard
> > > >> >>>>>> fabric binary putting hadoop-accelerator libs under
> ‘optional’
> > > >> folder?
> > > >> >>>>>>
> > > >> >>>>>> —
> > > >> >>>>>> Denis
> > > >> >>>>>>
> > > >> >>>>>> On Nov 21, 2016, at 12:19 PM, Konstantin Boudnik <
> > c...@apache.org
> > > >
> > > >> >>>> wrote:
> > > >> >>>>>>
> > > >> >>>>>> What Denis said: spark has been added to the Hadoop
> accelerator
> > > as
> > > >> a
> > > >> >> way
> > > >> >>>>>>
> > > >> >>>>>> to
> > > >> >>>>>>
> > > >> >>>>>> boost the performance of more than just MR compute of the
> > Hadoop
> > > >> >> stack,
> > > >> >>>>>>
> > > >> >>>>>> IIRC.
> > > >> >>>>>>
> > > >> >>>>>> For what it worth, Spark is considered a part of Hadoop at
> > large.
> > > >> >>>>>>
> > > >> >>>>>> On the separate note, in the Bigtop, we start looking into
> > > changing
> > > >> >> the
> > > >> >>>>>>
> > > >> >>>>>> way we
> > > >> >>>>>>
> > > >> >>>>>> deliver Ignite and we'll likely to start offering the whole
> > 'data
> > > >> >>>> fabric'
> > > >> >>>>>> experience instead of the mere "hadoop-acceleration".
> > > >> >>>>>>
> > > >> >>>>>> Cos
> > > >> >>>>>>
> > > >> >>>>>> On Mon, Nov 21, 2016 at 09:54AM, Denis Magda wrote:
> > > >> >>>>>>
> > > >> >>>>>> Val,
> > > >> >>>>>>
> > > >> >>>>>> Ignite Hadoop module includes not only the map-reduce
> > accelerator
> > > >> but
> > > >> >>>>>>
> > > >> >>>>>> Ignite
> > > >> >>>>>>
> > > >> >>>>>> Hadoop File System component as well. The latter can be used
> in
> > > >> >>>>>>
> > > >> >>>>>> deployments
> > > >> >>>>>>
> > > >> >>>>>> like HDFS+IGFS+Ignite Spark + Spark.
> > > >> >>>>>>
> > > >> >>>>>> Considering this I’m for the second solution proposed by you:
> > put
> > > >> both
> > > >> >>>>>>
> > > >> >>>>>> 2.10
> > > >> >>>>>>
> > > >> >>>>>> and 2.11 ignite-spark modules under ‘optional’ folder of
> Ignite
> > > >> Hadoop
> > > >> >>>>>> Accelerator distribution.
> > > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-4254 <
> > > >> >>>>>>
> > > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-4254>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>> BTW, this task may be affected or related to the following
> > ones:
> > > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-3596 <
> > > >> >>>>>>
> > > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-3596>
> > > >> >>>>>>
> > > >> >>>>>> https://issues.apache.org/jira/browse/IGNITE-3822
> > > >> >>>>>>
> > > >> >>>>>> —
> > > >> >>>>>> Denis
> > > >> >>>>>>
> > > >> >>>>>> On Nov 19, 2016, at 1:26 PM, Valentin Kulichenko <
> > > >> >>>>>>
> > > >> >>>>>> valentin.kuliche...@gmail.com> wrote:
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>> Hadoop Accelerator is a plugin to Ignite and this plugin is
> > used
> > > by
> > > >> >>>>>>
> > > >> >>>>>> Hadoop
> > > >> >>>>>>
> > > >> >>>>>> when running its jobs. ignite-spark module only provides
> > > IgniteRDD
> > > >> >>>>>>
> > > >> >>>>>> which
> > > >> >>>>>>
> > > >> >>>>>> Hadoop obviously will never use.
> > > >> >>>>>>
> > > >> >>>>>> Is there another use case for Hadoop Accelerator which I'm
> > > missing?
> > > >> >>>>>>
> > > >> >>>>>> -Val
> > > >> >>>>>>
> > > >> >>>>>> On Sat, Nov 19, 2016 at 3:12 AM, Dmitriy Setrakyan <
> > > >> >>>>>>
> > > >> >>>>>> dsetrak...@apache.org>
> > > >> >>>>>>
> > > >> >>>>>> wrote:
> > > >> >>>>>>
> > > >> >>>>>> Why do you think that spark module is not needed in our
> hadoop
> > > >> build?
> > > >> >>>>>>
> > > >> >>>>>> On Fri, Nov 18, 2016 at 5:44 PM, Valentin Kulichenko <
> > > >> >>>>>> valentin.kuliche...@gmail.com> wrote:
> > > >> >>>>>>
> > > >> >>>>>> Folks,
> > > >> >>>>>>
> > > >> >>>>>> Is there anyone who understands the purpose of including
> > > >> ignite-spark
> > > >> >>>>>> module in the Hadoop Accelerator build? I can't figure out a
> > use
> > > >> >>>>>>
> > > >> >>>>>> case for
> > > >> >>>>>>
> > > >> >>>>>> which it's needed.
> > > >> >>>>>>
> > > >> >>>>>> In case we actually need it there, there is an issue then. We
> > > >> >>>>>>
> > > >> >>>>>> actually
> > > >> >>>>>>
> > > >> >>>>>> have
> > > >> >>>>>>
> > > >> >>>>>> two ignite-spark modules, for 2.10 and 2.11. In Fabric build
> > > >> >>>>>>
> > > >> >>>>>> everything
> > > >> >>>>>>
> > > >> >>>>>> is
> > > >> >>>>>>
> > > >> >>>>>> good, we put both in 'optional' folder and user can enable
> > either
> > > >> >>>>>>
> > > >> >>>>>> one.
> > > >> >>>>>>
> > > >> >>>>>> But
> > > >> >>>>>>
> > > >> >>>>>> in Hadoop Accelerator there is only 2.11 which means that the
> > > build
> > > >> >>>>>>
> > > >> >>>>>> doesn't
> > > >> >>>>>>
> > > >> >>>>>> work with 2.10 out of the box.
> > > >> >>>>>>
> > > >> >>>>>> We should either remove the module from the build, or fix the
> > > >> issue.
> > > >> >>>>>>
> > > >> >>>>>> -Val
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>>
> > > >> >>> --
> > > >> >>> Sergey Kozlov
> > > >> >>> GridGain Systems
> > > >> >>> www.gridgain.com
> > > >> >>
> > > >> >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > Vladimir Ozerov
> > > Senior Software Architect
> > > GridGain Systems
> > > www.gridgain.com
> > > *+7 (960) 283 98 40*
> > >
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
> >
>
--
Vladimir Ozerov
Senior Software Architect
GridGain Systems
www.gridgain.com
*+7 (960) 283 98 40*
