[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-05 Thread Denis Magda (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386619#comment-16386619
 ] 

Denis Magda commented on IGNITE-7655:
-

[~NIzhikov], [~abchaudhri] , thanks guys! The doc looks good to me, closing the 
ticket.

 

[~abchaudhri] , please create a new Jira ticket for the examples and set its 
version to 2.5. Place all the examples in respective Ignite examples folders, 
add tests and issue a pull-request. Nikolay and I will be happy to review the 
examples and accept them into the main code base.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteCatalogExample.java, 
> JavaIgniteCatalogExample.java, JavaIgniteCatalogExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java, 
> JavaIgniteDataFrameWriteExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-05 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386076#comment-16386076
 ] 

Akmal Chaudhri commented on IGNITE-7655:


[~NIzhikov], i think everything is done. I have converted the Scala programs to 
Java. I have added the code snippets to the documentation. I have added a very 
small section about SparkSession to the documentation. It needs [~dmagda] to 
review all this now.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteCatalogExample.java, 
> JavaIgniteCatalogExample.java, JavaIgniteCatalogExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java, 
> JavaIgniteDataFrameWriteExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-04 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385678#comment-16385678
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

Hello, [~abchaudhri]

Do you still need help?
If yes, please, write your questions and I will try to help with examples.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteCatalogExample.java, 
> JavaIgniteCatalogExample.java, JavaIgniteCatalogExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java, 
> JavaIgniteDataFrameWriteExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-04 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385185#comment-16385185
 ] 

Akmal Chaudhri commented on IGNITE-7655:


[~dmagda], some Java code snippets added to the documentation and a small 
introductory section on SparkSession, which could be expanded.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteCatalogExample.java, 
> JavaIgniteCatalogExample.java, JavaIgniteCatalogExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java, 
> JavaIgniteDataFrameWriteExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-03 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384689#comment-16384689
 ] 

Akmal Chaudhri commented on IGNITE-7655:


One small line of code missing from the Java Catalog file:
{code:java}
df2.show();
{code}
It is fixed now.

[^JavaIgniteCatalogExample.java]

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteCatalogExample.java, 
> JavaIgniteCatalogExample.java, JavaIgniteCatalogExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java, 
> JavaIgniteDataFrameWriteExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-03 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384595#comment-16384595
 ] 

Akmal Chaudhri commented on IGNITE-7655:


Attached are the three Scala programs converted to Java.

[^JavaIgniteCatalogExample.java]

[^JavaIgniteDataFrameExample.java]

[^JavaIgniteDataFrameWriteExample.java]

Again, probably some improvements, but they work.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteCatalogExample.java, 
> JavaIgniteCatalogExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, 
> JavaIgniteDataFrameWriteExample.java, JavaIgniteDataFrameWriteExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-02 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384260#comment-16384260
 ] 

Akmal Chaudhri commented on IGNITE-7655:


Attached is an example using Java.


[^JavaIgniteDataFrameExample.java]

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
> Attachments: JavaIgniteDataFrameExample.java
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-02 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383947#comment-16383947
 ] 

Akmal Chaudhri commented on IGNITE-7655:


[~vkulichenko], [~NIzhikov], thank you.

[~dmagda] assisted me today and we were able to get some simple Java code 
working with Ignite DataFrames. I need to look further at the Spark 
documentation and I think I will have the equivalent Java code ready quite soon.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-02 Thread Valentin Kulichenko (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383925#comment-16383925
 ] 

Valentin Kulichenko commented on IGNITE-7655:
-

[~abchaudhri], this is actually Spark API not Ignite API, so I think you should 
look for Java examples in Spark and them adapt them to Ignite.

[~NIzhikov], probably makes sense to create a Java example for data frames 
integration and include it into the product. Can you please create a ticket?

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-02 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383379#comment-16383379
 ] 

Akmal Chaudhri commented on IGNITE-7655:


[~vkulichenko], I am trying to take the existing Scala code example:

[https://github.com/apache/ignite/blob/master/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameExample.scala]

and rewrite it for Java. I am blocked here for the moment:
{code:java}
val igniteDF = spark.read
.format(FORMAT_IGNITE) //Data source type.
.option(OPTION_TABLE, "person") //Table to read.
.option(OPTION_CONFIG_FILE, CONFIG) //Ignite config.
.load()
.filter(col("id") >= 2) //Filter clause.
.filter(col("name") like "%M%") //Another filter clause.
{code}
The Java code to create the tables works ok. However, I don't know Scala, so 
trying to do a conversion from one language to another is difficult.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-01 Thread Valentin Kulichenko (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382830#comment-16382830
 ] 

Valentin Kulichenko commented on IGNITE-7655:
-

[~abchaudhri], do you have {{ignite-spark}} module on the classpath? This code 
compiles for me in Java:
{code:java}
String sss = IgniteDataFrameSettings.OPTION_CONFIG_FILE();{code}
Can you show the example that you tried to create and describe what is not 
working?

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-03-01 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382547#comment-16382547
 ] 

Akmal Chaudhri commented on IGNITE-7655:


[~NIzhikov], I have been trying to build some Java code examples, equivalent to 
your Scala code. However, I notice that:

*org.apache.ignite.spark.IgniteDataFrameSettings*

is only implemented in Scala. Is that correct please?

Thank you.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-28 Thread Denis Magda (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381195#comment-16381195
 ] 

Denis Magda commented on IGNITE-7655:
-

[~abchaudhri] , as discussed, please add Java code snippets similar to the ones 
we have for scala.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-28 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380491#comment-16380491
 ] 

Akmal Chaudhri commented on IGNITE-7655:


I have gone through the document and improved the readability and formatting.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-27 Thread Denis Magda (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379766#comment-16379766
 ] 

Denis Magda commented on IGNITE-7655:
-

[~NIzhikov] , thanks a lot for that!

[~abchaudhri] , please do a primarily review and editing making sure the doc is 
perceived as a getting started guide. Once you're done assign this task to me, 
I'll check it up as an Ignite user who have never used this new feature before 
and willing to start.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Akmal Chaudhri
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378016#comment-16378016
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

> Also, we need to have examples of Java API usage. If you don't have anything 
> ready, then Akmal can create Java versions of the Scala example for 2.5.

I don't have ready java examples. It would be great, ff Akmal can help with it.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378014#comment-16378014
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

[~dmagda] [~abchaudhri]

> Could you have a look and update the doc one more time?

I tried to merge my edits with current Akmal version of the page.
Please, check it.

> Please make sure this option below is documented: {{SaveMode.Append + 
> OPTION_STREAMER_ALLOW_OVERWRITE=true}}

Done.

> The problem with the existing Scala examples is that they are added to the 
> folder of Java examples:

It's interesting, actually.
I've checked it locally and it works with scala examples in spark folder.
Anyway, I will move examples to the right folder and make sure it tested on TC.

> If it's not late to merge the changes to 2.4, please fix it.

OK. I will fix it in master and mail to dev-list.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Denis Magda (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377875#comment-16377875
 ] 

Denis Magda commented on IGNITE-7655:
-

[~NIzhikov] , thanks for the explanation! It helped. Unfortunately, I don't see 
the changes (plus there are some TODOs left on the page). Akmal told me he had 
troubles with the same page earlier today. Probably, that's because you both 
might have been editing it at the same time. 

Could you have a look and update the doc one more time? Please make sure this 
option below is documented:

{{SaveMode.Append}} + {{OPTION_STREAMER_ALLOW_OVERWRITE=true}}

The problem with the existing Scala examples is that they are added to the 
folder of Java examples:

[https://github.com/apache/ignite/tree/master/examples/src/main/spark/org/apache/ignite/examples/spark]

while they need to be here:

[https://github.com/apache/ignite/tree/master/examples/src/main/scala/org/apache/ignite/scalar/examples/spark]

If it's not late to merge the changes to 2.4, please fix it. Ask the community 
for permission. Otherwise, create a ticket for 2.5.

Also, we need to have examples of Java API usage. If you don't have anything 
ready, then Akmal can create Java versions of the Scala example for 2.5.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377334#comment-16377334
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

Hello, [~abchaudhri].

> The GitHub examples are in Scala, but I will try and build the project and 
> run the code to test that it works.

Actually, all examples should be tested in Ignite Example plan.
I checked it and found that {{IgniteExamplesSparkSelfTestSuite}} doesn't 
included in "Ignite Example" run.
I will fix it, so all Spark examples will be tested.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Akmal Chaudhri (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377306#comment-16377306
 ] 

Akmal Chaudhri commented on IGNITE-7655:


[~NIzhikov]

I have done some minor edits throughout the document to improve readability. 
The GitHub examples are in Scala, but I will try and build the project and run 
the code to test that it works.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376840#comment-16376840
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

[~dmagda]

I fixed all issues, please, take a look.

https://dash.readme.io/project/apacheignite-fs/v2.3/docs/ignite-data-frame

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-26 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376785#comment-16376785
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

[~dmagda] 

As far as I can see you edited my explanation of {{Overwrite}} mode. I think it 
incorrect now.
{{Overwrite}} mode means we will do next steps:

1. Drop table from Ignite if it already exists.

2. Create table. We will use schema from Data frame we need to save.

3. Inserts data from data frame to new table.

I wrote this changes in page. Does it look right for you?

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-25 Thread Nikolay Izhikov (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376422#comment-16376422
 ] 

Nikolay Izhikov commented on IGNITE-7655:
-

[~dmagda]

> What does appending mean here? As far as I understand, you're just going to 
> insert new data ignoring the data which keys are already in the table.

Appends mean we will not execute `DROP TABLE`/`CREATE TABLE`.
Since all inserts are executed through IgniteDataStreamer behaviour will be 
like this:

* `OPTION_STREAMER_ALLOW_OVERWRITE=false` - existing keys will be ignored.
* `OPTION_STREAMER_ALLOW_OVERWRITE=true` - existing key/value will overwritten 
with DataFrame data.

> which mode is a default? 

ErrorIfExsists

> Is there a mode that updates existing rows, inserts new from the data frame 
> and does NOT delete the rows that were in the table initially?

`SaveMode.Append` + `OPTION_STREAMER_ALLOW_OVERWRITE=true`

> How do I set a mode? You start talking about the data frames right away but 
> do not explain how to use them.

One can use `dataFrame.write.mode(SaveMode.Append)`. It's a part of Spark API. 
OK, I will add an example to the page.

> I would start "Saving Data Frame" section with the explanation of how all 
> this works

OK. Will do.

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

2018-02-23 Thread Denis Magda (JIRA)

[ 
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375007#comment-16375007
 ] 

Denis Magda commented on IGNITE-7655:
-

[~NIzhikov] , please clarify the following questions.
 # You're saying "the data frame will be appended to the existing table. 
Existing table data will not be changed.". What does appending mean here? As 
far as I understand, you're just going to *insert* new data ignoring the data 
which keys are already in the table.
 # which mode is a default? 
 # Is there a mode that updates existing rows, inserts new from the data frame 
and does NOT delete the rows that were in the table initially?
 # How do I set a mode? You start talking about the data frames right away but 
do not explain how to use them.
 # You say "Note, internally all inserts are done through IgniteDataStreamer. 
Several optional parameters exists to configure internal streamer. Please, see 
full list of available options.". I would start "Saving Data Frame" section 
with the explanation of how all this works. From what I see, we're going to use 
CREATE TABLE to create a table and then store data in Ignite with the streamer, 
right? Please elaborate more on this in the doc.

[~abchaudhri], could you please finalize the review after [~NIzhikov] addresses 
the notes above? Please feel free to work together to make this doc exceptional 
and easy to follow!

> Spark Data Frames: saving data frames into Ignite needs to be documented
> 
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)