Revin, I doubt IgniteRDD#getPrefferredLocations has any affect on data frames, but this is an interesting point. Nikolay, as a developer of this functionality, can you please comment on this?
-Val On Wed, Jan 3, 2018 at 1:22 PM, Revin Chalil <rcha...@expedia.com> wrote: > Thanks Val for the info on indexes with DF. Do you know if adding index / > affinitykeys on the cache help with the join, when the IgniteRDD is joined > with a spark DF? The below from docs say that > > “IgniteRDD also provides affinity information to Spark via > getPrefferredLocations method so that RDD computations use data locality.” > > I was wondering, if the affinitykey on the cache can be utilized in the > spark join? > > > On 1/3/18, 12:27 PM, "vkulichenko" <valentin.kuliche...@gmail.com> wrote: > > Indexes would not be used during joins, at least in current > implementation. > Current integration is implemented as a regular Spark data source which > provides each relation separately. Spark then performs join by itself, > so > Ignite indexes do not help. > > The easiest way to get binaries would be to use a nightly build [1] , > but it > seems to be broken for some reason (latest is from May 31). I guess > the only > option at the moment is to build from source. > > [1] > https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/ > lastSuccessfulBuild/ > > -Val > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > >
