Val, Ignite Hadoop module includes not only the map-reduce accelerator but Ignite Hadoop File System component as well. The latter can be used in deployments like HDFS+IGFS+Ignite Spark + Spark.
Considering this I’m for the second solution proposed by you: put both 2.10 and 2.11 ignite-spark modules under ‘optional’ folder of Ignite Hadoop Accelerator distribution. https://issues.apache.org/jira/browse/IGNITE-4254 <https://issues.apache.org/jira/browse/IGNITE-4254> BTW, this task may be affected or related to the following ones: https://issues.apache.org/jira/browse/IGNITE-3596 <https://issues.apache.org/jira/browse/IGNITE-3596> https://issues.apache.org/jira/browse/IGNITE-3822 — Denis > On Nov 19, 2016, at 1:26 PM, Valentin Kulichenko > <valentin.kuliche...@gmail.com> wrote: > > Hadoop Accelerator is a plugin to Ignite and this plugin is used by Hadoop > when running its jobs. ignite-spark module only provides IgniteRDD which > Hadoop obviously will never use. > > Is there another use case for Hadoop Accelerator which I'm missing? > > -Val > > On Sat, Nov 19, 2016 at 3:12 AM, Dmitriy Setrakyan <dsetrak...@apache.org> > wrote: > >> Why do you think that spark module is not needed in our hadoop build? >> >> On Fri, Nov 18, 2016 at 5:44 PM, Valentin Kulichenko < >> valentin.kuliche...@gmail.com> wrote: >> >>> Folks, >>> >>> Is there anyone who understands the purpose of including ignite-spark >>> module in the Hadoop Accelerator build? I can't figure out a use case for >>> which it's needed. >>> >>> In case we actually need it there, there is an issue then. We actually >> have >>> two ignite-spark modules, for 2.10 and 2.11. In Fabric build everything >> is >>> good, we put both in 'optional' folder and user can enable either one. >> But >>> in Hadoop Accelerator there is only 2.11 which means that the build >> doesn't >>> work with 2.10 out of the box. >>> >>> We should either remove the module from the build, or fix the issue. >>> >>> -Val >>> >>
