Re: Integration of Spark and Ignite. Prototype.

Val, Thank you. I fixed issues and answered questions from comments. Please, take a look. 2017-12-13 3:28 GMT+03:00 Valentin Kulichenko : > Hi Nikolay, > > I reviewed the code and left several comments in the ticket [1]. Please > take a look. > > [1]

Re: Integration of Spark and Ignite. Prototype.

Hi Nikolay, I reviewed the code and left several comments in the ticket [1]. Please take a look. [1] https://issues.apache.org/jira/browse/IGNITE-3084 -Val On Mon, Dec 4, 2017 at 3:03 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Denis, > > Nikolay was doing final changes

Re: Integration of Spark and Ignite. Prototype.

Denis, Nikolay was doing final changes and TC stabilization. I'm planning to do final review this week, so hopefully we will merge the code soon. -Val On Mon, Dec 4, 2017 at 1:31 PM, Denis Magda wrote: > Nikolay, Val, > > Since we agreed to release the feature without the

Re: Integration of Spark and Ignite. Prototype.

Nikolay, Val, Since we agreed to release the feature without the strategy support, can the current integration meet the world in 2.4 release? Please chime in this conversation: http://apache-ignite-developers.2346864.n4.nabble.com/Time-and-scope-for-Apache-Ignite-2-4-td24987.html — Denis > On

Re: Integration of Spark and Ignite. Prototype.

Guys, Looking into the parallel discussion about the strategy support I would change my initial stance and support the idea of releasing the integration in its current state. Is the code ready to be merged into the master? Let’s concentrate on this first and handle the strategy support as a

Re: Integration of Spark and Ignite. Prototype.

Nikolay, Let's estimate the strategy implementation work, and then decide weather to merge the code in current state or not. If anything is unclear, please start a separate discussion. -Val On Fri, Nov 24, 2017 at 5:42 AM, Николай Ижиков wrote: > Hello, Val, Denis. > >

Re: Integration of Spark and Ignite. Prototype.

Hello, Val, Denis. > Personally, I think that we should release the integration only after the strategy is fully supported. I see two major reason to propose merge of DataFrame API implementation without custom strategy: 1. My PR is relatively huge, already. From my experience of interaction

Re: Integration of Spark and Ignite. Prototype.

Val, Nikolay, Personally, I think that we should release the integration only after the strategy is fully supported. Without the strategy we don’t really leverage from Ignite’s SQL engine and introduce redundant data movement between Ignite and Spark nodes. How big is the effort to support

Re: Integration of Spark and Ignite. Prototype.

Nikolay, Custom strategy implementation is actually very important for this integration. Basically, it will allow to create a SQL query for Ignite and execute it directly on the cluster. Your current implementation only adds a new DataSource which means that Spark will fetch data in its own

Re: Integration of Spark and Ignite. Prototype.

Hello. > 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two Catalog implementations and what is the difference? IgniteCatalog removed. > 5. I don't like that IgniteStrategy and IgniteOptimization have to be set manually on SQLContext each time it's createdIs there any way to

Re: Integration of Spark and Ignite. Prototype.

Hello, Valentin. My answers is below. Dmitry, do we need to move discussion to Jira? > 1. Why do we have org.apache.spark.sql.ignite package in our codebase? As I mentioned earlier, to implement and override Spark Catalog one have to use internal(private) Spark API. So I have to use package

Re: Integration of Spark and Ignite. Prototype.

Val, thanks for the review. Can I ask you to add the same comments to the ticket? On Tue, Oct 17, 2017 at 3:20 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Nikolay, Anton, > > I did a high level review of the code. First of all, impressive results! > However, I have some

Re: Integration of Spark and Ignite. Prototype.

Nikolay, Anton, I did a high level review of the code. First of all, impressive results! However, I have some questions/comments. 1. Why do we have org.apache.spark.sql.ignite package in our codebase? Can these classes reside under org.apache.ignite.spark instead? 2. IgniteRelationProvider

Re: Integration of Spark and Ignite. Prototype.

Sounds awesome. I'll try to review API & tests this week. Val, Your review still required :) On Tue, Oct 17, 2017 at 2:36 PM, Николай Ижиков wrote: > Yes > > 17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" < > avinogra...@gridgain.com> написал: > > > Nikolay, >

Re: Integration of Spark and Ignite. Prototype.

Yes 17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" < avinogra...@gridgain.com> написал: > Nikolay, > > So, it will be able to start regular spark and ignite clusters and, using > peer classloading via spark-context, perform any DataFrame request, > correct? > > On Tue, Oct 17, 2017 at

Re: Integration of Spark and Ignite. Prototype.

Nikolay, So, it will be able to start regular spark and ignite clusters and, using peer classloading via spark-context, perform any DataFrame request, correct? On Tue, Oct 17, 2017 at 2:25 PM, Николай Ижиков wrote: > Hello, Anton. > > An example you provide is a path to

Re: Integration of Spark and Ignite. Prototype.

Hello, Anton. An example you provide is a path to a master *local* file. These libraries are added to the classpath for each remote node running submitted job. Please, see documentation: http://spark.apache.org/docs/latest/api/java/org/apache/ spark/SparkContext.html#addJar(java.lang.String)

Re: Integration of Spark and Ignite. Prototype.

Nikolay, > With Data Frame API implementation there are no requirements to have any > Ignite files on spark worker nodes. What do you mean? I see code like: spark.sparkContext.addJar(MAVEN_HOME + "/org/apache/ignite/ignite-core/2.3.0-SNAPSHOT/ignite-core-2.3.0-SNAPSHOT.jar") On Mon, Oct 16,

Re: Integration of Spark and Ignite. Prototype.

Hello, guys. I have created example application to run Ignite Data Frame on standalone Spark cluster. With Data Frame API implementation there are no requirements to have any Ignite files on spark worker nodes. I ran this application on the free dataset: ATP tennis match statistics. data -

Re: Integration of Spark and Ignite. Prototype.

Hi Nikolay, Sorry for delay on this, got a little swamped lately. I will do my best to review the code this week. -Val On Mon, Oct 9, 2017 at 11:48 AM, Николай Ижиков wrote: > Hello, Valentin. > > Did you have a chance to look at my changes? > > Now I think I have done

Re: Integration of Spark and Ignite. Prototype.

Hello, Valentin. Did you have a chance to look at my changes? Now I think I have done almost all required features. I want to make some performance test to ensure my implementation work properly with a significant amount of data. And I definitely need some feedback for my changes. 2017-10-09

Re: Integration of Spark and Ignite. Prototype.

Hello, guys. Which version of Spark do we want to use? 1. Currently, Ignite depends on Spark 2.1.0. * Can be run on JDK 7. * Still supported: 2.1.2 will be released soon. 2. Latest Spark version is 2.2.0. * Can be run only on JDK 8+ * Released Jul 11, 2017. * Already

Re: Integration of Spark and Ignite. Prototype.

I will review in the next few days. -Val On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda wrote: > Hello Nikolay, > > This is good news. Finally this capability is coming to Ignite. > > Val, Vladimir, could you do a preliminary review? > > Answering on your questions. > > 1.

Re: Integration of Spark and Ignite. Prototype.

Hello Nikolay, This is good news. Finally this capability is coming to Ignite. Val, Vladimir, could you do a preliminary review? Answering on your questions. 1. Yardstick should be enough for performance measurements. As a Spark user, I will be curious to know what’s the point of this

Integration of Spark and Ignite. Prototype.

Hello, guys. Currently, I’m working on integration between Spark and Ignite [1]. For now, I implement following: * Ignite DataSource implementation(IgniteRelationProvider) * DataFrame support for Ignite SQL table. * IgniteCatalog implementation for a transparent resolving of ignites