[jira] [Comment Edited] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented
[ https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386076#comment-16386076 ] Akmal Chaudhri edited comment on IGNITE-7655 at 3/5/18 4:05 PM: [~NIzhikov], I think everything is done. I have converted the Scala programs to Java. I have added the code snippets to the documentation. I have added a very small section about SparkSession to the documentation. It needs [~dmagda] to review all this now. was (Author: abchaudhri): [~NIzhikov], i think everything is done. I have converted the Scala programs to Java. I have added the code snippets to the documentation. I have added a very small section about SparkSession to the documentation. It needs [~dmagda] to review all this now. > Spark Data Frames: saving data frames into Ignite needs to be documented > > > Key: IGNITE-7655 > URL: https://issues.apache.org/jira/browse/IGNITE-7655 > Project: Ignite > Issue Type: Bug > Components: documentation, spark >Reporter: Nikolay Izhikov >Assignee: Denis Magda >Priority: Major > Fix For: 2.4 > > Attachments: JavaIgniteCatalogExample.java, > JavaIgniteCatalogExample.java, JavaIgniteCatalogExample.java, > JavaIgniteDataFrameExample.java, JavaIgniteDataFrameExample.java, > JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java, > JavaIgniteDataFrameWriteExample.java > > > Once IGNITE-7337 is ready for merge. > This new feature of Ignite needs to be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented
[ https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384260#comment-16384260 ] Akmal Chaudhri edited comment on IGNITE-7655 at 3/2/18 10:49 PM: - Attached is an example using Java. [^JavaIgniteDataFrameExample.java] Probably needs improvement, but it works. was (Author: abchaudhri): Attached is an example using Java. [^JavaIgniteDataFrameExample.java] > Spark Data Frames: saving data frames into Ignite needs to be documented > > > Key: IGNITE-7655 > URL: https://issues.apache.org/jira/browse/IGNITE-7655 > Project: Ignite > Issue Type: Bug > Components: documentation, spark >Reporter: Nikolay Izhikov >Assignee: Akmal Chaudhri >Priority: Major > Fix For: 2.4 > > Attachments: JavaIgniteDataFrameExample.java > > > Once IGNITE-7337 is ready for merge. > This new feature of Ignite needs to be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented
[ https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384260#comment-16384260 ] Akmal Chaudhri edited comment on IGNITE-7655 at 3/2/18 10:41 PM: - Attached is an example using Java. [^JavaIgniteDataFrameExample.java] was (Author: abchaudhri): Attached is an example using Java. [^JavaIgniteDataFrameExample.java] > Spark Data Frames: saving data frames into Ignite needs to be documented > > > Key: IGNITE-7655 > URL: https://issues.apache.org/jira/browse/IGNITE-7655 > Project: Ignite > Issue Type: Bug > Components: documentation, spark >Reporter: Nikolay Izhikov >Assignee: Akmal Chaudhri >Priority: Major > Fix For: 2.4 > > Attachments: JavaIgniteDataFrameExample.java > > > Once IGNITE-7337 is ready for merge. > This new feature of Ignite needs to be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented
[ https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379766#comment-16379766 ] Denis Magda edited comment on IGNITE-7655 at 2/28/18 4:31 AM: -- [~NIzhikov] , thanks a lot for that! [~abchaudhri] , please do a primarily review and editing making sure the doc is perceived as a getting started guide. Once you're done assign this task to me, I'll check it up as an Ignite user who has never used this new feature before and willing to start. was (Author: dmagda): [~NIzhikov] , thanks a lot for that! [~abchaudhri] , please do a primarily review and editing making sure the doc is perceived as a getting started guide. Once you're done assign this task to me, I'll check it up as an Ignite user who have never used this new feature before and willing to start. > Spark Data Frames: saving data frames into Ignite needs to be documented > > > Key: IGNITE-7655 > URL: https://issues.apache.org/jira/browse/IGNITE-7655 > Project: Ignite > Issue Type: Bug > Components: documentation, spark >Reporter: Nikolay Izhikov >Assignee: Akmal Chaudhri >Priority: Major > Fix For: 2.4 > > > Once IGNITE-7337 is ready for merge. > This new feature of Ignite needs to be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented
[
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376422#comment-16376422
]
Nikolay Izhikov edited comment on IGNITE-7655 at 2/26/18 12:20 PM:
---
[~dmagda]
> What does appending mean here? As far as I understand, you're just going to
> insert new data ignoring the data which keys are already in the table.
Appends mean we will not execute {{DROP TABLE}}/{{CREATE TABLE}}.
Since all inserts are executed through IgniteDataStreamer behaviour will be
like this:
* {{OPTION_STREAMER_ALLOW_OVERWRITE=false}} - if key already exists in Ignite
cache then no insert will be execute.
* {{OPTION_STREAMER_ALLOW_OVERWRITE=true}} - if a key already exists in Ignite
cache then it will be overwritten with data frame content.
> which mode is a default?
ErrorIfExsists
> Is there a mode that updates existing rows, inserts new from the data frame
> and does NOT delete the rows that were in the table initially?
{{SaveMode.Append}} + {{OPTION_STREAMER_ALLOW_OVERWRITE=true}}
> How do I set a mode? You start talking about the data frames right away but
> do not explain how to use them.
One can use {{dataFrame.write.mode(SaveMode.Append)}}. It's a part of Spark
API.
OK, I will add an example to the page.
> I would start "Saving Data Frame" section with the explanation of how all
> this works
OK. Will do.
was (Author: nizhikov):
[~dmagda]
> What does appending mean here? As far as I understand, you're just going to
> insert new data ignoring the data which keys are already in the table.
Appends mean we will not execute {{DROP TABLE}}/{{CREATE TABLE}}.
Since all inserts are executed through IgniteDataStreamer behaviour will be
like this:
* {{OPTION_STREAMER_ALLOW_OVERWRITE=false}} - existing keys will be ignored.
* {{OPTION_STREAMER_ALLOW_OVERWRITE=true}} - existing key/value will
overwritten with DataFrame data.
> which mode is a default?
ErrorIfExsists
> Is there a mode that updates existing rows, inserts new from the data frame
> and does NOT delete the rows that were in the table initially?
{{SaveMode.Append}} + {{OPTION_STREAMER_ALLOW_OVERWRITE=true}}
> How do I set a mode? You start talking about the data frames right away but
> do not explain how to use them.
One can use {{dataFrame.write.mode(SaveMode.Append)}}. It's a part of Spark
API.
OK, I will add an example to the page.
> I would start "Saving Data Frame" section with the explanation of how all
> this works
OK. Will do.
> Spark Data Frames: saving data frames into Ignite needs to be documented
>
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
> Issue Type: Bug
> Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented
[
https://issues.apache.org/jira/browse/IGNITE-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376422#comment-16376422
]
Nikolay Izhikov edited comment on IGNITE-7655 at 2/26/18 6:01 AM:
--
[~dmagda]
> What does appending mean here? As far as I understand, you're just going to
> insert new data ignoring the data which keys are already in the table.
Appends mean we will not execute {{DROP TABLE}}/{{CREATE TABLE}}.
Since all inserts are executed through IgniteDataStreamer behaviour will be
like this:
* {{OPTION_STREAMER_ALLOW_OVERWRITE=false}} - existing keys will be ignored.
* {{OPTION_STREAMER_ALLOW_OVERWRITE=true}} - existing key/value will
overwritten with DataFrame data.
> which mode is a default?
ErrorIfExsists
> Is there a mode that updates existing rows, inserts new from the data frame
> and does NOT delete the rows that were in the table initially?
{{SaveMode.Append}} + {{OPTION_STREAMER_ALLOW_OVERWRITE=true}}
> How do I set a mode? You start talking about the data frames right away but
> do not explain how to use them.
One can use {{dataFrame.write.mode(SaveMode.Append)}}. It's a part of Spark
API.
OK, I will add an example to the page.
> I would start "Saving Data Frame" section with the explanation of how all
> this works
OK. Will do.
was (Author: nizhikov):
[~dmagda]
> What does appending mean here? As far as I understand, you're just going to
> insert new data ignoring the data which keys are already in the table.
Appends mean we will not execute `DROP TABLE`/`CREATE TABLE`.
Since all inserts are executed through IgniteDataStreamer behaviour will be
like this:
* `OPTION_STREAMER_ALLOW_OVERWRITE=false` - existing keys will be ignored.
* `OPTION_STREAMER_ALLOW_OVERWRITE=true` - existing key/value will overwritten
with DataFrame data.
> which mode is a default?
ErrorIfExsists
> Is there a mode that updates existing rows, inserts new from the data frame
> and does NOT delete the rows that were in the table initially?
`SaveMode.Append` + `OPTION_STREAMER_ALLOW_OVERWRITE=true`
> How do I set a mode? You start talking about the data frames right away but
> do not explain how to use them.
One can use `dataFrame.write.mode(SaveMode.Append)`. It's a part of Spark API.
OK, I will add an example to the page.
> I would start "Saving Data Frame" section with the explanation of how all
> this works
OK. Will do.
> Spark Data Frames: saving data frames into Ignite needs to be documented
>
>
> Key: IGNITE-7655
> URL: https://issues.apache.org/jira/browse/IGNITE-7655
> Project: Ignite
> Issue Type: Bug
> Components: documentation, spark
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.4
>
>
> Once IGNITE-7337 is ready for merge.
> This new feature of Ignite needs to be documented.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
