Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Denis Magda]

Wonderful, thanks Nikolay!

--
Denis

On Fri, Apr 27, 2018 at 11:50 PM, Nikolay Izhikov 
wrote:

> Hell, Denis.
>
> I've done review and merge yersteday.
>
> master commit - https://github.com/apache/ignite/commit/
> fcc4d4a357413eb4856ca5c72d301738568060e2
> 2.5 commit - https://github.com/apache/ignite/commit/
> 2679cc5a18eb00599c9f00b0f46dbcaa6352c0cb
>
> The task is in resolved state.
>
> В Пт, 27/04/2018 в 23:32 -0700, Denis Magda пишет:
> > Akmal, Nikolay,
> >
> > Do you have any progress on this? Let's finish with this ticket within
> 2.5
> > scope and merge it there.
> >
> > --
> > Denis
> >
> > On Thu, Apr 19, 2018 at 10:07 PM, Akmal Chaudhri <
> > akmal.chaud...@gridgain.com> wrote:
> >
> > > Nikolay,
> > >
> > > Ok. I _think_ it should be better now, so we can try again.
> > >
> > > 1. I removed the 3 pull requests.
> > > 2. I renamed the Java code files with a "Java" prefix.
> > > 3. I created a test file for the Java code, using the existing test
> file
> > > for the Scala code.
> > > 4. I updated the TestSuite with bullet #3 above.
> > > 5. I combined all changes into a single pull request.
> > >
> > > Note that one of the tests still fails locally due to the path issue. I
> > > have no idea how to fix this, although the code works fine.
> > >
> > > Thank you.
> > >
> > >
> > >
> > > On 19 April 2018 at 22:41, Nikolay Izhikov 
> wrote:
> > >
> > > > Hello, Akmal.
> > > >
> > > > 1. As a first step. Let's combine all changes you want to merge into
> a
> > > > single pull request.
> > > >
> > > > I suggest that your changes relates only to "examples" module.
> > > >
> > > > 2. So, when you will have one pull request, please, run "Examples"
> test
> > > > suite on Team City for your branch.
> > > > After that, attach link to execution into issue.
> > > >
> > > > After completing step 1 I will be able to review your changes.
> > > > After completing step 2 and review we will be to merge your changes
> to
> > > > master.
> > > >
> > > > В Чт, 19/04/2018 в 15:30 +0100, Akmal Chaudhri пишет:
> > > > > Nikolay,
> > > > >
> > > > > The code is the same for the attached version to the ticket, as
> well as
> > > >
> > > > the
> > > > > pull version. The tests are also the same as those provided for the
> > >
> > > Scala
> > > > > DF examples. I have checked and they work correctly with the
> exception
> > >
> > > of
> > > > > the path issue which I previously mentioned on the dev list.
> > > > >
> > > > > I'm afraid I am new to this whole process, so need someone in the
> > > >
> > > > community
> > > > > to assist.
> > > > >
> > > > > To summarise,
> > > > >
> > > > > 1. The Java Spark DF code is equivalent in functionality to the
> Scala
> > > >
> > > > Spark
> > > > > DF code. These Java examples were requested by Denis Magda, since
> only
> > > > > Scala examples were previously available.
> > > > > 2. The tests scripts provided with the Scala DF code work just fine
> > >
> > > with
> > > > > the Java code.
> > > > > 3. There is a problem with a path issue in test #2, where the code
> > >
> > > reads
> > > > a
> > > > > JSON file. Since the Java code is equivalent to the Scala code, I
> don't
> > > > > know how the Scala code passed this test.
> > > > >
> > > > >
> > > > >
> > > > > On 18 April 2018 at 22:11, Nikolay Izhikov 
> > >
> > > wrote:
> > > > >
> > > > > > Hello, Akmal.
> > > > > >
> > > > > > I see 3 pull requests attached to the ticket [1], [2], [3].
> > > > > > I see 3 java files attached to the ticket, also.
> > > > > >
> > > > > > Which changes you want to be reviewed and merge? Please, clarify.
> > > > > >
> > > > > > Delete all unnecessary pull requests link from the ticket.
> > > > > > Add new examples to the tests so we can test it on the Team City.
> > > > > > You can take IgniteDataFrameSelfTest as an example.
> > > > > >
> > > > > > I also suggest to rename java examples with "Java" prefix.
> > > > > >
> > > > > > IgniteDataFrameWriteExample.java ->
> JavaIgniteDataFrameWriteExampl
> > > >
> > > > e.java
> > > > > >
> > > > > > [1] https://github.com/apache/ignite/pull/3857
> > > > > > [2] https://github.com/apache/ignite/pull/3858I
> > > > > > [3] https://github.com/apache/ignite/pull/3859
> > > > > > [4] https://github.com/apache/ignite/blob/master/examples/
> > > > > > src/test/spark/org/apache/ignite/spark/examples/
> > > > > > IgniteDataFrameSelfTest.java
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> > > > > > > If any community members have time, please review the code.
> Three
> > > >
> > > > Java
> > > > > > > files are attached to the Jira ticket:
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/IGNITE-7909
> > > > > > >
> > > > > > > The code should be functionally equivalent to the Scala Data
> Frames
> > > >
> > > > code
> > > > > > > that was shipped in 2.4.
> > > > > > >
> > > > > > > Data Frame documentation is 

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Nikolay Izhikov]

Hell, Denis.

I've done review and merge yersteday.

master commit - 
https://github.com/apache/ignite/commit/fcc4d4a357413eb4856ca5c72d301738568060e2
2.5 commit - 
https://github.com/apache/ignite/commit/2679cc5a18eb00599c9f00b0f46dbcaa6352c0cb

The task is in resolved state.

В Пт, 27/04/2018 в 23:32 -0700, Denis Magda пишет:
> Akmal, Nikolay,
> 
> Do you have any progress on this? Let's finish with this ticket within 2.5
> scope and merge it there.
> 
> --
> Denis
> 
> On Thu, Apr 19, 2018 at 10:07 PM, Akmal Chaudhri <
> akmal.chaud...@gridgain.com> wrote:
> 
> > Nikolay,
> > 
> > Ok. I _think_ it should be better now, so we can try again.
> > 
> > 1. I removed the 3 pull requests.
> > 2. I renamed the Java code files with a "Java" prefix.
> > 3. I created a test file for the Java code, using the existing test file
> > for the Scala code.
> > 4. I updated the TestSuite with bullet #3 above.
> > 5. I combined all changes into a single pull request.
> > 
> > Note that one of the tests still fails locally due to the path issue. I
> > have no idea how to fix this, although the code works fine.
> > 
> > Thank you.
> > 
> > 
> > 
> > On 19 April 2018 at 22:41, Nikolay Izhikov  wrote:
> > 
> > > Hello, Akmal.
> > > 
> > > 1. As a first step. Let's combine all changes you want to merge into a
> > > single pull request.
> > > 
> > > I suggest that your changes relates only to "examples" module.
> > > 
> > > 2. So, when you will have one pull request, please, run "Examples" test
> > > suite on Team City for your branch.
> > > After that, attach link to execution into issue.
> > > 
> > > After completing step 1 I will be able to review your changes.
> > > After completing step 2 and review we will be to merge your changes to
> > > master.
> > > 
> > > В Чт, 19/04/2018 в 15:30 +0100, Akmal Chaudhri пишет:
> > > > Nikolay,
> > > > 
> > > > The code is the same for the attached version to the ticket, as well as
> > > 
> > > the
> > > > pull version. The tests are also the same as those provided for the
> > 
> > Scala
> > > > DF examples. I have checked and they work correctly with the exception
> > 
> > of
> > > > the path issue which I previously mentioned on the dev list.
> > > > 
> > > > I'm afraid I am new to this whole process, so need someone in the
> > > 
> > > community
> > > > to assist.
> > > > 
> > > > To summarise,
> > > > 
> > > > 1. The Java Spark DF code is equivalent in functionality to the Scala
> > > 
> > > Spark
> > > > DF code. These Java examples were requested by Denis Magda, since only
> > > > Scala examples were previously available.
> > > > 2. The tests scripts provided with the Scala DF code work just fine
> > 
> > with
> > > > the Java code.
> > > > 3. There is a problem with a path issue in test #2, where the code
> > 
> > reads
> > > a
> > > > JSON file. Since the Java code is equivalent to the Scala code, I don't
> > > > know how the Scala code passed this test.
> > > > 
> > > > 
> > > > 
> > > > On 18 April 2018 at 22:11, Nikolay Izhikov 
> > 
> > wrote:
> > > > 
> > > > > Hello, Akmal.
> > > > > 
> > > > > I see 3 pull requests attached to the ticket [1], [2], [3].
> > > > > I see 3 java files attached to the ticket, also.
> > > > > 
> > > > > Which changes you want to be reviewed and merge? Please, clarify.
> > > > > 
> > > > > Delete all unnecessary pull requests link from the ticket.
> > > > > Add new examples to the tests so we can test it on the Team City.
> > > > > You can take IgniteDataFrameSelfTest as an example.
> > > > > 
> > > > > I also suggest to rename java examples with "Java" prefix.
> > > > > 
> > > > > IgniteDataFrameWriteExample.java -> JavaIgniteDataFrameWriteExampl
> > > 
> > > e.java
> > > > > 
> > > > > [1] https://github.com/apache/ignite/pull/3857
> > > > > [2] https://github.com/apache/ignite/pull/3858I
> > > > > [3] https://github.com/apache/ignite/pull/3859
> > > > > [4] https://github.com/apache/ignite/blob/master/examples/
> > > > > src/test/spark/org/apache/ignite/spark/examples/
> > > > > IgniteDataFrameSelfTest.java
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> > > > > > If any community members have time, please review the code. Three
> > > 
> > > Java
> > > > > > files are attached to the Jira ticket:
> > > > > > 
> > > > > > https://issues.apache.org/jira/browse/IGNITE-7909
> > > > > > 
> > > > > > The code should be functionally equivalent to the Scala Data Frames
> > > 
> > > code
> > > > > > that was shipped in 2.4.
> > > > > > 
> > > > > > Data Frame documentation is here:
> > > > > > 
> > > > > > https://apacheignite-fs.readme.io/docs/ignite-data-frame
> > > > > > 
> > > > > > Thank you!

signature.asc
Description: This is a digitally signed message part

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Denis Magda]

Akmal, Nikolay,

Do you have any progress on this? Let's finish with this ticket within 2.5
scope and merge it there.

--
Denis

On Thu, Apr 19, 2018 at 10:07 PM, Akmal Chaudhri <
akmal.chaud...@gridgain.com> wrote:

> Nikolay,
>
> Ok. I _think_ it should be better now, so we can try again.
>
> 1. I removed the 3 pull requests.
> 2. I renamed the Java code files with a "Java" prefix.
> 3. I created a test file for the Java code, using the existing test file
> for the Scala code.
> 4. I updated the TestSuite with bullet #3 above.
> 5. I combined all changes into a single pull request.
>
> Note that one of the tests still fails locally due to the path issue. I
> have no idea how to fix this, although the code works fine.
>
> Thank you.
>
>
>
> On 19 April 2018 at 22:41, Nikolay Izhikov  wrote:
>
> > Hello, Akmal.
> >
> > 1. As a first step. Let's combine all changes you want to merge into a
> > single pull request.
> >
> > I suggest that your changes relates only to "examples" module.
> >
> > 2. So, when you will have one pull request, please, run "Examples" test
> > suite on Team City for your branch.
> > After that, attach link to execution into issue.
> >
> > After completing step 1 I will be able to review your changes.
> > After completing step 2 and review we will be to merge your changes to
> > master.
> >
> > В Чт, 19/04/2018 в 15:30 +0100, Akmal Chaudhri пишет:
> > > Nikolay,
> > >
> > > The code is the same for the attached version to the ticket, as well as
> > the
> > > pull version. The tests are also the same as those provided for the
> Scala
> > > DF examples. I have checked and they work correctly with the exception
> of
> > > the path issue which I previously mentioned on the dev list.
> > >
> > > I'm afraid I am new to this whole process, so need someone in the
> > community
> > > to assist.
> > >
> > > To summarise,
> > >
> > > 1. The Java Spark DF code is equivalent in functionality to the Scala
> > Spark
> > > DF code. These Java examples were requested by Denis Magda, since only
> > > Scala examples were previously available.
> > > 2. The tests scripts provided with the Scala DF code work just fine
> with
> > > the Java code.
> > > 3. There is a problem with a path issue in test #2, where the code
> reads
> > a
> > > JSON file. Since the Java code is equivalent to the Scala code, I don't
> > > know how the Scala code passed this test.
> > >
> > >
> > >
> > > On 18 April 2018 at 22:11, Nikolay Izhikov 
> wrote:
> > >
> > > > Hello, Akmal.
> > > >
> > > > I see 3 pull requests attached to the ticket [1], [2], [3].
> > > > I see 3 java files attached to the ticket, also.
> > > >
> > > > Which changes you want to be reviewed and merge? Please, clarify.
> > > >
> > > > Delete all unnecessary pull requests link from the ticket.
> > > > Add new examples to the tests so we can test it on the Team City.
> > > > You can take IgniteDataFrameSelfTest as an example.
> > > >
> > > > I also suggest to rename java examples with "Java" prefix.
> > > >
> > > > IgniteDataFrameWriteExample.java -> JavaIgniteDataFrameWriteExampl
> > e.java
> > > >
> > > > [1] https://github.com/apache/ignite/pull/3857
> > > > [2] https://github.com/apache/ignite/pull/3858I
> > > > [3] https://github.com/apache/ignite/pull/3859
> > > > [4] https://github.com/apache/ignite/blob/master/examples/
> > > > src/test/spark/org/apache/ignite/spark/examples/
> > > > IgniteDataFrameSelfTest.java
> > > >
> > > >
> > > >
> > > >
> > > > В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> > > > > If any community members have time, please review the code. Three
> > Java
> > > > > files are attached to the Jira ticket:
> > > > >
> > > > > https://issues.apache.org/jira/browse/IGNITE-7909
> > > > >
> > > > > The code should be functionally equivalent to the Scala Data Frames
> > code
> > > > > that was shipped in 2.4.
> > > > >
> > > > > Data Frame documentation is here:
> > > > >
> > > > > https://apacheignite-fs.readme.io/docs/ignite-data-frame
> > > > >
> > > > > Thank you!
> >
>

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Akmal Chaudhri]

Nikolay,

Ok. I _think_ it should be better now, so we can try again.

1. I removed the 3 pull requests.
2. I renamed the Java code files with a "Java" prefix.
3. I created a test file for the Java code, using the existing test file
for the Scala code.
4. I updated the TestSuite with bullet #3 above.
5. I combined all changes into a single pull request.

Note that one of the tests still fails locally due to the path issue. I
have no idea how to fix this, although the code works fine.

Thank you.



On 19 April 2018 at 22:41, Nikolay Izhikov  wrote:

> Hello, Akmal.
>
> 1. As a first step. Let's combine all changes you want to merge into a
> single pull request.
>
> I suggest that your changes relates only to "examples" module.
>
> 2. So, when you will have one pull request, please, run "Examples" test
> suite on Team City for your branch.
> After that, attach link to execution into issue.
>
> After completing step 1 I will be able to review your changes.
> After completing step 2 and review we will be to merge your changes to
> master.
>
> В Чт, 19/04/2018 в 15:30 +0100, Akmal Chaudhri пишет:
> > Nikolay,
> >
> > The code is the same for the attached version to the ticket, as well as
> the
> > pull version. The tests are also the same as those provided for the Scala
> > DF examples. I have checked and they work correctly with the exception of
> > the path issue which I previously mentioned on the dev list.
> >
> > I'm afraid I am new to this whole process, so need someone in the
> community
> > to assist.
> >
> > To summarise,
> >
> > 1. The Java Spark DF code is equivalent in functionality to the Scala
> Spark
> > DF code. These Java examples were requested by Denis Magda, since only
> > Scala examples were previously available.
> > 2. The tests scripts provided with the Scala DF code work just fine with
> > the Java code.
> > 3. There is a problem with a path issue in test #2, where the code reads
> a
> > JSON file. Since the Java code is equivalent to the Scala code, I don't
> > know how the Scala code passed this test.
> >
> >
> >
> > On 18 April 2018 at 22:11, Nikolay Izhikov  wrote:
> >
> > > Hello, Akmal.
> > >
> > > I see 3 pull requests attached to the ticket [1], [2], [3].
> > > I see 3 java files attached to the ticket, also.
> > >
> > > Which changes you want to be reviewed and merge? Please, clarify.
> > >
> > > Delete all unnecessary pull requests link from the ticket.
> > > Add new examples to the tests so we can test it on the Team City.
> > > You can take IgniteDataFrameSelfTest as an example.
> > >
> > > I also suggest to rename java examples with "Java" prefix.
> > >
> > > IgniteDataFrameWriteExample.java -> JavaIgniteDataFrameWriteExampl
> e.java
> > >
> > > [1] https://github.com/apache/ignite/pull/3857
> > > [2] https://github.com/apache/ignite/pull/3858I
> > > [3] https://github.com/apache/ignite/pull/3859
> > > [4] https://github.com/apache/ignite/blob/master/examples/
> > > src/test/spark/org/apache/ignite/spark/examples/
> > > IgniteDataFrameSelfTest.java
> > >
> > >
> > >
> > >
> > > В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> > > > If any community members have time, please review the code. Three
> Java
> > > > files are attached to the Jira ticket:
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-7909
> > > >
> > > > The code should be functionally equivalent to the Scala Data Frames
> code
> > > > that was shipped in 2.4.
> > > >
> > > > Data Frame documentation is here:
> > > >
> > > > https://apacheignite-fs.readme.io/docs/ignite-data-frame
> > > >
> > > > Thank you!
>

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Nikolay Izhikov]

Hello, Akmal.

1. As a first step. Let's combine all changes you want to merge into a single 
pull request.

I suggest that your changes relates only to "examples" module.

2. So, when you will have one pull request, please, run "Examples" test suite 
on Team City for your branch.
After that, attach link to execution into issue.

After completing step 1 I will be able to review your changes.
After completing step 2 and review we will be to merge your changes to master.

В Чт, 19/04/2018 в 15:30 +0100, Akmal Chaudhri пишет:
> Nikolay,
> 
> The code is the same for the attached version to the ticket, as well as the
> pull version. The tests are also the same as those provided for the Scala
> DF examples. I have checked and they work correctly with the exception of
> the path issue which I previously mentioned on the dev list.
> 
> I'm afraid I am new to this whole process, so need someone in the community
> to assist.
> 
> To summarise,
> 
> 1. The Java Spark DF code is equivalent in functionality to the Scala Spark
> DF code. These Java examples were requested by Denis Magda, since only
> Scala examples were previously available.
> 2. The tests scripts provided with the Scala DF code work just fine with
> the Java code.
> 3. There is a problem with a path issue in test #2, where the code reads a
> JSON file. Since the Java code is equivalent to the Scala code, I don't
> know how the Scala code passed this test.
> 
> 
> 
> On 18 April 2018 at 22:11, Nikolay Izhikov  wrote:
> 
> > Hello, Akmal.
> > 
> > I see 3 pull requests attached to the ticket [1], [2], [3].
> > I see 3 java files attached to the ticket, also.
> > 
> > Which changes you want to be reviewed and merge? Please, clarify.
> > 
> > Delete all unnecessary pull requests link from the ticket.
> > Add new examples to the tests so we can test it on the Team City.
> > You can take IgniteDataFrameSelfTest as an example.
> > 
> > I also suggest to rename java examples with "Java" prefix.
> > 
> > IgniteDataFrameWriteExample.java -> JavaIgniteDataFrameWriteExample.java
> > 
> > [1] https://github.com/apache/ignite/pull/3857
> > [2] https://github.com/apache/ignite/pull/3858I
> > [3] https://github.com/apache/ignite/pull/3859
> > [4] https://github.com/apache/ignite/blob/master/examples/
> > src/test/spark/org/apache/ignite/spark/examples/
> > IgniteDataFrameSelfTest.java
> > 
> > 
> > 
> > 
> > В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> > > If any community members have time, please review the code. Three Java
> > > files are attached to the Jira ticket:
> > > 
> > > https://issues.apache.org/jira/browse/IGNITE-7909
> > > 
> > > The code should be functionally equivalent to the Scala Data Frames code
> > > that was shipped in 2.4.
> > > 
> > > Data Frame documentation is here:
> > > 
> > > https://apacheignite-fs.readme.io/docs/ignite-data-frame
> > > 
> > > Thank you!

signature.asc
Description: This is a digitally signed message part

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Akmal Chaudhri]

Nikolay,

The code is the same for the attached version to the ticket, as well as the
pull version. The tests are also the same as those provided for the Scala
DF examples. I have checked and they work correctly with the exception of
the path issue which I previously mentioned on the dev list.

I'm afraid I am new to this whole process, so need someone in the community
to assist.

To summarise,

1. The Java Spark DF code is equivalent in functionality to the Scala Spark
DF code. These Java examples were requested by Denis Magda, since only
Scala examples were previously available.
2. The tests scripts provided with the Scala DF code work just fine with
the Java code.
3. There is a problem with a path issue in test #2, where the code reads a
JSON file. Since the Java code is equivalent to the Scala code, I don't
know how the Scala code passed this test.



On 18 April 2018 at 22:11, Nikolay Izhikov  wrote:

> Hello, Akmal.
>
> I see 3 pull requests attached to the ticket [1], [2], [3].
> I see 3 java files attached to the ticket, also.
>
> Which changes you want to be reviewed and merge? Please, clarify.
>
> Delete all unnecessary pull requests link from the ticket.
> Add new examples to the tests so we can test it on the Team City.
> You can take IgniteDataFrameSelfTest as an example.
>
> I also suggest to rename java examples with "Java" prefix.
>
> IgniteDataFrameWriteExample.java -> JavaIgniteDataFrameWriteExample.java
>
> [1] https://github.com/apache/ignite/pull/3857
> [2] https://github.com/apache/ignite/pull/3858I
> [3] https://github.com/apache/ignite/pull/3859
> [4] https://github.com/apache/ignite/blob/master/examples/
> src/test/spark/org/apache/ignite/spark/examples/
> IgniteDataFrameSelfTest.java
>
>
>
>
> В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> > If any community members have time, please review the code. Three Java
> > files are attached to the Jira ticket:
> >
> > https://issues.apache.org/jira/browse/IGNITE-7909
> >
> > The code should be functionally equivalent to the Scala Data Frames code
> > that was shipped in 2.4.
> >
> > Data Frame documentation is here:
> >
> > https://apacheignite-fs.readme.io/docs/ignite-data-frame
> >
> > Thank you!
>

Re: [IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Nikolay Izhikov]

Hello, Akmal.

I see 3 pull requests attached to the ticket [1], [2], [3].
I see 3 java files attached to the ticket, also.

Which changes you want to be reviewed and merge? Please, clarify.

Delete all unnecessary pull requests link from the ticket.
Add new examples to the tests so we can test it on the Team City.
You can take IgniteDataFrameSelfTest as an example.

I also suggest to rename java examples with "Java" prefix.

IgniteDataFrameWriteExample.java -> JavaIgniteDataFrameWriteExample.java

[1] https://github.com/apache/ignite/pull/3857
[2] https://github.com/apache/ignite/pull/3858I
[3] https://github.com/apache/ignite/pull/3859
[4] 
https://github.com/apache/ignite/blob/master/examples/src/test/spark/org/apache/ignite/spark/examples/IgniteDataFrameSelfTest.java




В Ср, 18/04/2018 в 17:20 +0100, Akmal Chaudhri пишет:
> If any community members have time, please review the code. Three Java
> files are attached to the Jira ticket:
> 
> https://issues.apache.org/jira/browse/IGNITE-7909
> 
> The code should be functionally equivalent to the Scala Data Frames code
> that was shipped in 2.4.
> 
> Data Frame documentation is here:
> 
> https://apacheignite-fs.readme.io/docs/ignite-data-frame
> 
> Thank you!

signature.asc
Description: This is a digitally signed message part

[IGNITE-7909]: Java code examples are needed for Spark Data Frames.

[Akmal Chaudhri]

If any community members have time, please review the code. Three Java
files are attached to the Jira ticket:

https://issues.apache.org/jira/browse/IGNITE-7909

The code should be functionally equivalent to the Scala Data Frames code
that was shipped in 2.4.

Data Frame documentation is here:

https://apacheignite-fs.readme.io/docs/ignite-data-frame

Thank you!

Re: Fwd: [jira] [Created] (IGNITE-7909) Java code examples are needed for Spark Data Frames.

[Dmitry Pavlov]

I see now. Thank you for clarification.

пн, 12 мар. 2018 г., 13:12 Nikolay Izhikov <nizhi...@apache.org>:

> Hello, Dmitry.
>
> Examples I create written in scala [1].
>
> The goal of task is to rewrite these examples in java.
>
> [1]
> https://github.com/apache/ignite/blob/master/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameExample.scala
>
> В Пн, 12/03/2018 в 10:07 +, Dmitry Pavlov пишет:
> > Hi Nikolay,
> >
> > Сan your examples, that you have done recently, come up as a solution to
> > this ticket?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > -- Forwarded message -
> > From: Akmal Chaudhri (JIRA) <j...@apache.org>
> > Date: вс, 11 мар. 2018 г. в 2:24
> > Subject: [jira] [Created] (IGNITE-7909) Java code examples are needed for
> > Spark Data Frames.
> > To: <dev@ignite.apache.org>
> >
> >
> > Akmal Chaudhri created IGNITE-7909:
> > --
> >
> >  Summary: Java code examples are needed for Spark Data
> Frames.
> >  Key: IGNITE-7909
> >  URL: https://issues.apache.org/jira/browse/IGNITE-7909
> >  Project: Ignite
> >   Issue Type: Improvement
> >   Components: spark
> > Affects Versions: 2.5
> > Reporter: Akmal Chaudhri
> > Assignee: Akmal Chaudhri
> >      Attachments: JavaIgniteCatalogExample.java,
> > JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java
> >
> > Existing Scala code examples have been developed to illustrate Ignite
> > support for Spark Data Frames. But Java code examples are also required.
> > Some Java code has already been developed but requires further testing.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v7.6.3#76005)

Re: Fwd: [jira] [Created] (IGNITE-7909) Java code examples are needed for Spark Data Frames.

[Nikolay Izhikov]

Hello, Dmitry.

Examples I create written in scala [1].

The goal of task is to rewrite these examples in java.

[1] 
https://github.com/apache/ignite/blob/master/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameExample.scala

В Пн, 12/03/2018 в 10:07 +, Dmitry Pavlov пишет:
> Hi Nikolay,
> 
> Сan your examples, that you have done recently, come up as a solution to
> this ticket?
> 
> Sincerely,
> Dmitriy Pavlov
> 
> -- Forwarded message -
> From: Akmal Chaudhri (JIRA) <j...@apache.org>
> Date: вс, 11 мар. 2018 г. в 2:24
> Subject: [jira] [Created] (IGNITE-7909) Java code examples are needed for
> Spark Data Frames.
> To: <dev@ignite.apache.org>
> 
> 
> Akmal Chaudhri created IGNITE-7909:
> --
> 
>  Summary: Java code examples are needed for Spark Data Frames.
>  Key: IGNITE-7909
>  URL: https://issues.apache.org/jira/browse/IGNITE-7909
>  Project: Ignite
>   Issue Type: Improvement
>   Components: spark
> Affects Versions: 2.5
> Reporter: Akmal Chaudhri
> Assignee: Akmal Chaudhri
>  Attachments: JavaIgniteCatalogExample.java,
> JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java
> 
> Existing Scala code examples have been developed to illustrate Ignite
> support for Spark Data Frames. But Java code examples are also required.
> Some Java code has already been developed but requires further testing.
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

signature.asc
Description: This is a digitally signed message part

Fwd: [jira] [Created] (IGNITE-7909) Java code examples are needed for Spark Data Frames.

[Dmitry Pavlov]

Hi Nikolay,

Сan your examples, that you have done recently, come up as a solution to
this ticket?

Sincerely,
Dmitriy Pavlov

-- Forwarded message -
From: Akmal Chaudhri (JIRA) <j...@apache.org>
Date: вс, 11 мар. 2018 г. в 2:24
Subject: [jira] [Created] (IGNITE-7909) Java code examples are needed for
Spark Data Frames.
To: <dev@ignite.apache.org>


Akmal Chaudhri created IGNITE-7909:
--

 Summary: Java code examples are needed for Spark Data Frames.
 Key: IGNITE-7909
 URL: https://issues.apache.org/jira/browse/IGNITE-7909
 Project: Ignite
  Issue Type: Improvement
  Components: spark
Affects Versions: 2.5
Reporter: Akmal Chaudhri
Assignee: Akmal Chaudhri
 Attachments: JavaIgniteCatalogExample.java,
JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java

Existing Scala code examples have been developed to illustrate Ignite
support for Spark Data Frames. But Java code examples are also required.
Some Java code has already been developed but requires further testing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7909) Java code examples are needed for Spark Data Frames.

[Akmal Chaudhri (JIRA)]

Akmal Chaudhri created IGNITE-7909:
--

 Summary: Java code examples are needed for Spark Data Frames.
 Key: IGNITE-7909
 URL: https://issues.apache.org/jira/browse/IGNITE-7909
 Project: Ignite
  Issue Type: Improvement
  Components: spark
Affects Versions: 2.5
Reporter: Akmal Chaudhri
Assignee: Akmal Chaudhri
 Attachments: JavaIgniteCatalogExample.java, 
JavaIgniteDataFrameExample.java, JavaIgniteDataFrameWriteExample.java

Existing Scala code examples have been developed to illustrate Ignite support 
for Spark Data Frames. But Java code examples are also required. Some Java code 
has already been developed but requires further testing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Spark data frames

[Valentin Kulichenko]

That's a great idea! Nikolay, let me know if you need any help with the
presentation, I will be happy to help.

-Val

On Fri, Feb 16, 2018 at 12:19 AM, Nikolay Izhikov <nizhi...@apache.org>
wrote:

> Ok, Igniters.
>
> I will do it in a few weeks.
> I need time to prepare to the talk.
>
> В Пт, 16/02/2018 в 07:16 +, Dmitry Pavlov пишет:
> > +1 for starting new topic from Nikolay when ' Community Meeting to
> > Introduce Ignite Spark Data Frames'  is ready to be announced.
> >
> > пт, 16 февр. 2018 г. в 9:27, Denis Magda <dma...@apache.org>:
> >
> > > I'm in :)
> > >
> > > Nikolay, we can use my GoToMeeting account to host the webinar. To draw
> > > more attention I would suggest starting a more specific thread titled
> like
> > > "[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
> > > discussion sounds too generic, folks could simply pass by.
> > >
> > > Negotiated?
> > >
> > > --
> > > Denis
> > >
> > > On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > >
> > > > Dmitry, it's a great idea!
> > > >
> > > > Nikolay, I also have the interest to get familiar with Spark Data
> > > > Frames integration.
> > > >
> > > > I'd prefer webinar of the format similar to "Ignite Persistent Store
> > > > Walkthrough" by Denis Magda, which has been presented some times ago.
> > > >
> > > > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> > > > <dsetrak...@apache.org> wrote:
> > > > > I am definitely interested. Great idea!
> > > > >
> > > > > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov <
> nizhi...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hello, Dmitry.
> > > > > >
> > > > > > If other community member are also iterested in that kind of
> > > >
> > > > information I
> > > > > > can try to do the talk.
> > > > > >
> > > > > > В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > > > > > > Hi Nikolay,
> > > > > > >
> > > > > > > I've notices there are a number of very lively discussions on
> dev
> > >
> > > list
> > > > > > about SparkDataFrames. But I, for example, can't fully
> understand it
> > > > > > because it is not well-known code for me.
> > > > > > >
> > > > > > > I suppose Ignite community has other members, which are not
> aware of
> > > > > >
> > > > > > recent feature SparkDataFrame and its pros.
> > > > > > >
> > > > > > > What do you think about short talk arrangement for community
> to tell
> > > > > >
> > > > > > about this module, e.g. for 30 minutes? Could you please do
> this? I
> > > >
> > > > think
> > > > > > Denis M. can help with infrastructure.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards, Vyacheslav D.
> > > >
>

Re: Spark data frames

[Nikolay Izhikov]

Ok, Igniters.

I will do it in a few weeks.
I need time to prepare to the talk.

В Пт, 16/02/2018 в 07:16 +, Dmitry Pavlov пишет:
> +1 for starting new topic from Nikolay when ' Community Meeting to
> Introduce Ignite Spark Data Frames'  is ready to be announced.
> 
> пт, 16 февр. 2018 г. в 9:27, Denis Magda <dma...@apache.org>:
> 
> > I'm in :)
> > 
> > Nikolay, we can use my GoToMeeting account to host the webinar. To draw
> > more attention I would suggest starting a more specific thread titled like
> > "[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
> > discussion sounds too generic, folks could simply pass by.
> > 
> > Negotiated?
> > 
> > --
> > Denis
> > 
> > On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur <daradu...@gmail.com>
> > wrote:
> > 
> > > Dmitry, it's a great idea!
> > > 
> > > Nikolay, I also have the interest to get familiar with Spark Data
> > > Frames integration.
> > > 
> > > I'd prefer webinar of the format similar to "Ignite Persistent Store
> > > Walkthrough" by Denis Magda, which has been presented some times ago.
> > > 
> > > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> > > <dsetrak...@apache.org> wrote:
> > > > I am definitely interested. Great idea!
> > > > 
> > > > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov <nizhi...@apache.org>
> > > > wrote:
> > > > 
> > > > > Hello, Dmitry.
> > > > > 
> > > > > If other community member are also iterested in that kind of
> > > 
> > > information I
> > > > > can try to do the talk.
> > > > > 
> > > > > В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > > > > > Hi Nikolay,
> > > > > > 
> > > > > > I've notices there are a number of very lively discussions on dev
> > 
> > list
> > > > > about SparkDataFrames. But I, for example, can't fully understand it
> > > > > because it is not well-known code for me.
> > > > > > 
> > > > > > I suppose Ignite community has other members, which are not aware of
> > > > > 
> > > > > recent feature SparkDataFrame and its pros.
> > > > > > 
> > > > > > What do you think about short talk arrangement for community to tell
> > > > > 
> > > > > about this module, e.g. for 30 minutes? Could you please do this? I
> > > 
> > > think
> > > > > Denis M. can help with infrastructure.
> > > > > > 
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > 
> > > 
> > > 
> > > --
> > > Best Regards, Vyacheslav D.
> > > 

signature.asc
Description: This is a digitally signed message part

Re: Spark data frames

[Dmitry Pavlov]

+1 for starting new topic from Nikolay when ' Community Meeting to
Introduce Ignite Spark Data Frames'  is ready to be announced.

пт, 16 февр. 2018 г. в 9:27, Denis Magda <dma...@apache.org>:

> I'm in :)
>
> Nikolay, we can use my GoToMeeting account to host the webinar. To draw
> more attention I would suggest starting a more specific thread titled like
> "[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
> discussion sounds too generic, folks could simply pass by.
>
> Negotiated?
>
> --
> Denis
>
> On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur <daradu...@gmail.com>
> wrote:
>
> > Dmitry, it's a great idea!
> >
> > Nikolay, I also have the interest to get familiar with Spark Data
> > Frames integration.
> >
> > I'd prefer webinar of the format similar to "Ignite Persistent Store
> > Walkthrough" by Denis Magda, which has been presented some times ago.
> >
> > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> > <dsetrak...@apache.org> wrote:
> > > I am definitely interested. Great idea!
> > >
> > > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov <nizhi...@apache.org>
> > > wrote:
> > >
> > >> Hello, Dmitry.
> > >>
> > >> If other community member are also iterested in that kind of
> > information I
> > >> can try to do the talk.
> > >>
> > >> В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > >> > Hi Nikolay,
> > >> >
> > >> > I've notices there are a number of very lively discussions on dev
> list
> > >> about SparkDataFrames. But I, for example, can't fully understand it
> > >> because it is not well-known code for me.
> > >> >
> > >> > I suppose Ignite community has other members, which are not aware of
> > >> recent feature SparkDataFrame and its pros.
> > >> >
> > >> > What do you think about short talk arrangement for community to tell
> > >> about this module, e.g. for 30 minutes? Could you please do this? I
> > think
> > >> Denis M. can help with infrastructure.
> > >> >
> > >> > Sincerely,
> > >> > Dmitriy Pavlov
> > >>
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>

Re: Spark data frames

[Denis Magda]

I'm in :)

Nikolay, we can use my GoToMeeting account to host the webinar. To draw
more attention I would suggest starting a more specific thread titled like
"[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
discussion sounds too generic, folks could simply pass by.

Negotiated?

--
Denis

On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur <daradu...@gmail.com>
wrote:

> Dmitry, it's a great idea!
>
> Nikolay, I also have the interest to get familiar with Spark Data
> Frames integration.
>
> I'd prefer webinar of the format similar to "Ignite Persistent Store
> Walkthrough" by Denis Magda, which has been presented some times ago.
>
> On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> <dsetrak...@apache.org> wrote:
> > I am definitely interested. Great idea!
> >
> > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov <nizhi...@apache.org>
> > wrote:
> >
> >> Hello, Dmitry.
> >>
> >> If other community member are also iterested in that kind of
> information I
> >> can try to do the talk.
> >>
> >> В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> >> > Hi Nikolay,
> >> >
> >> > I've notices there are a number of very lively discussions on dev list
> >> about SparkDataFrames. But I, for example, can't fully understand it
> >> because it is not well-known code for me.
> >> >
> >> > I suppose Ignite community has other members, which are not aware of
> >> recent feature SparkDataFrame and its pros.
> >> >
> >> > What do you think about short talk arrangement for community to tell
> >> about this module, e.g. for 30 minutes? Could you please do this? I
> think
> >> Denis M. can help with infrastructure.
> >> >
> >> > Sincerely,
> >> > Dmitriy Pavlov
> >>
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Spark data frames

[Vyacheslav Daradur]

Dmitry, it's a great idea!

Nikolay, I also have the interest to get familiar with Spark Data
Frames integration.

I'd prefer webinar of the format similar to "Ignite Persistent Store
Walkthrough" by Denis Magda, which has been presented some times ago.

On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
<dsetrak...@apache.org> wrote:
> I am definitely interested. Great idea!
>
> On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov <nizhi...@apache.org>
> wrote:
>
>> Hello, Dmitry.
>>
>> If other community member are also iterested in that kind of information I
>> can try to do the talk.
>>
>> В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
>> > Hi Nikolay,
>> >
>> > I've notices there are a number of very lively discussions on dev list
>> about SparkDataFrames. But I, for example, can't fully understand it
>> because it is not well-known code for me.
>> >
>> > I suppose Ignite community has other members, which are not aware of
>> recent feature SparkDataFrame and its pros.
>> >
>> > What do you think about short talk arrangement for community to tell
>> about this module, e.g. for 30 minutes? Could you please do this? I think
>> Denis M. can help with infrastructure.
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>>



-- 
Best Regards, Vyacheslav D.

Re: Spark data frames

[Dmitriy Setrakyan]

I am definitely interested. Great idea!

On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov 
wrote:

> Hello, Dmitry.
>
> If other community member are also iterested in that kind of information I
> can try to do the talk.
>
> В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > Hi Nikolay,
> >
> > I've notices there are a number of very lively discussions on dev list
> about SparkDataFrames. But I, for example, can't fully understand it
> because it is not well-known code for me.
> >
> > I suppose Ignite community has other members, which are not aware of
> recent feature SparkDataFrame and its pros.
> >
> > What do you think about short talk arrangement for community to tell
> about this module, e.g. for 30 minutes? Could you please do this? I think
> Denis M. can help with infrastructure.
> >
> > Sincerely,
> > Dmitriy Pavlov
>

Re: Spark data frames

[Nikolay Izhikov]

Hello, Dmitry.

If other community member are also iterested in that kind of information I can 
try to do the talk.

В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> Hi Nikolay,
> 
> I've notices there are a number of very lively discussions on dev list about 
> SparkDataFrames. But I, for example, can't fully understand it because it is 
> not well-known code for me.
> 
> I suppose Ignite community has other members, which are not aware of recent 
> feature SparkDataFrame and its pros.
> 
> What do you think about short talk arrangement for community to tell about 
> this module, e.g. for 30 minutes? Could you please do this? I think Denis M. 
> can help with infrastructure.
> 
> Sincerely,
> Dmitriy Pavlov

signature.asc
Description: This is a digitally signed message part

Spark data frames

[Dmitry Pavlov]

Hi Nikolay,

I've notices there are a number of very lively discussions on dev list
about SparkDataFrames. But I, for example, can't fully understand it
because it is not well-known code for me.

I suppose Ignite community has other members, which are not aware of recent
feature SparkDataFrame and its pros.

What do you think about short talk arrangement for community to tell about
this module, e.g. for 30 minutes? Could you please do this? I think Denis
M. can help with infrastructure.

Sincerely,
Dmitriy Pavlov

Re: Saving Spark Data Frames merged to master

[Dmitriy Setrakyan]

Agree, it is ok to merge, since we are waiting for the page replacement
(eviction) performance fix anyway.

Spark data frames is a long-awaited feature by our users, so it does make
sense to provide a complete support in 2.4.

D.

On Fri, Feb 9, 2018 at 2:55 PM, Denis Magda <dma...@apache.org> wrote:

> +1
>
> It wasn’t an undiscussed merge. The question was raised here before [1].
>
> Anyway, Anton thanks for being on guard all the times! :)
>
> [1] http://apache-ignite-developers.2346864.n4.nabble.
> com/Apache-Ignite-2-4-release-td26031i20.html#a26807 <
> http://apache-ignite-developers.2346864.n4.nabble.
> com/Apache-Ignite-2-4-release-td26031i20.html#a26807>
>
> —
> Denis
>
> > On Feb 9, 2018, at 11:41 AM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >
> > I think it's OK to merge it to 2.4, especially since the release is
> > delayed. This is a fairly small feature which is fully isolated from
> > everything else, so there are no risks. At the same time, it makes data
> > frames integration much more valuable.
> >
> > -Val
> >
> > On Fri, Feb 9, 2018 at 5:20 AM, Nikolay Izhikov <nizhi...@apache.org>
> wrote:
> >
> >> Hello, Anton.
> >>
> >> I have no any objections.
> >>
> >> Seems like some kind of misunderstanding from my side.
> >>
> >> As far as I can understand mail from Dmitriy Setrakyan [1] He agreed to
> >> include IGNITE-7337 to 2.4.
> >> If the Ccommuntiy decides to postpone this feature in 2.5 release I'm
> fully
> >> OK with it.
> >>
> >> [1]
> >> http://apache-ignite-developers.2346864.n4.nabble.
> >> com/Apache-Ignite-2-4-release-tp26031p26807.html
> >>
> >>
> >> 2018-02-09 14:58 GMT+03:00 Anton Vinogradov <a...@apache.org>:
> >>
> >>> Nikolay,
> >>>
> >>> 2.4 is almost ready to be released.
> >>> We're fixing final issues to provide stable and fast release.
> >>> Merging something to 2.4 except blockers is not possible at this phase
> of
> >>> release process.
> >>>
> >>> Hope, 2.5, with your changes, will be released soon :)
> >>>
> >>> On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org>
> >>> wrote:
> >>>
> >>>> Hello, Igniters.
> >>>>
> >>>> Good news.
> >>>>
> >>>> IGNITE-7337 [1](Spark Data Frames: support saving a data frame in
> >> Ignite)
> >>>> are merged to master.
> >>>>
> >>>> For now we can both - read from and write to Ignite SQL table with
> Data
> >>>> Frame API.
> >>>> Big thanks to Valentin Kulichenko for a quick review.
> >>>> So it seems we can include this feature to 2.4 release.
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-7337
> >>>
> >>
>
>

Re: Saving Spark Data Frames merged to master

[Denis Magda]

+1 

It wasn’t an undiscussed merge. The question was raised here before [1].

Anyway, Anton thanks for being on guard all the times! :)

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-4-release-td26031i20.html#a26807
 
<http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-4-release-td26031i20.html#a26807>

—
Denis

> On Feb 9, 2018, at 11:41 AM, Valentin Kulichenko 
> <valentin.kuliche...@gmail.com> wrote:
> 
> I think it's OK to merge it to 2.4, especially since the release is
> delayed. This is a fairly small feature which is fully isolated from
> everything else, so there are no risks. At the same time, it makes data
> frames integration much more valuable.
> 
> -Val
> 
> On Fri, Feb 9, 2018 at 5:20 AM, Nikolay Izhikov <nizhi...@apache.org> wrote:
> 
>> Hello, Anton.
>> 
>> I have no any objections.
>> 
>> Seems like some kind of misunderstanding from my side.
>> 
>> As far as I can understand mail from Dmitriy Setrakyan [1] He agreed to
>> include IGNITE-7337 to 2.4.
>> If the Ccommuntiy decides to postpone this feature in 2.5 release I'm fully
>> OK with it.
>> 
>> [1]
>> http://apache-ignite-developers.2346864.n4.nabble.
>> com/Apache-Ignite-2-4-release-tp26031p26807.html
>> 
>> 
>> 2018-02-09 14:58 GMT+03:00 Anton Vinogradov <a...@apache.org>:
>> 
>>> Nikolay,
>>> 
>>> 2.4 is almost ready to be released.
>>> We're fixing final issues to provide stable and fast release.
>>> Merging something to 2.4 except blockers is not possible at this phase of
>>> release process.
>>> 
>>> Hope, 2.5, with your changes, will be released soon :)
>>> 
>>> On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org>
>>> wrote:
>>> 
>>>> Hello, Igniters.
>>>> 
>>>> Good news.
>>>> 
>>>> IGNITE-7337 [1](Spark Data Frames: support saving a data frame in
>> Ignite)
>>>> are merged to master.
>>>> 
>>>> For now we can both - read from and write to Ignite SQL table with Data
>>>> Frame API.
>>>> Big thanks to Valentin Kulichenko for a quick review.
>>>> So it seems we can include this feature to 2.4 release.
>>>> 
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-7337
>>> 
>> 

Re: Saving Spark Data Frames merged to master

[Valentin Kulichenko]

I think it's OK to merge it to 2.4, especially since the release is
delayed. This is a fairly small feature which is fully isolated from
everything else, so there are no risks. At the same time, it makes data
frames integration much more valuable.

-Val

On Fri, Feb 9, 2018 at 5:20 AM, Nikolay Izhikov <nizhi...@apache.org> wrote:

> Hello, Anton.
>
> I have no any objections.
>
> Seems like some kind of misunderstanding from my side.
>
> As far as I can understand mail from Dmitriy Setrakyan [1] He agreed to
> include IGNITE-7337 to 2.4.
> If the Ccommuntiy decides to postpone this feature in 2.5 release I'm fully
> OK with it.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.
> com/Apache-Ignite-2-4-release-tp26031p26807.html
>
>
> 2018-02-09 14:58 GMT+03:00 Anton Vinogradov <a...@apache.org>:
>
> > Nikolay,
> >
> > 2.4 is almost ready to be released.
> > We're fixing final issues to provide stable and fast release.
> > Merging something to 2.4 except blockers is not possible at this phase of
> > release process.
> >
> > Hope, 2.5, with your changes, will be released soon :)
> >
> > On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org>
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > Good news.
> > >
> > > IGNITE-7337 [1](Spark Data Frames: support saving a data frame in
> Ignite)
> > > are merged to master.
> > >
> > > For now we can both - read from and write to Ignite SQL table with Data
> > > Frame API.
> > > Big thanks to Valentin Kulichenko for a quick review.
> > > So it seems we can include this feature to 2.4 release.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-7337
> >
>

Re: Saving Spark Data Frames merged to master

[Nikolay Izhikov]

Hello, Anton.

I have no any objections.

Seems like some kind of misunderstanding from my side.

As far as I can understand mail from Dmitriy Setrakyan [1] He agreed to
include IGNITE-7337 to 2.4.
If the Ccommuntiy decides to postpone this feature in 2.5 release I'm fully
OK with it.

[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-4-release-tp26031p26807.html


2018-02-09 14:58 GMT+03:00 Anton Vinogradov <a...@apache.org>:

> Nikolay,
>
> 2.4 is almost ready to be released.
> We're fixing final issues to provide stable and fast release.
> Merging something to 2.4 except blockers is not possible at this phase of
> release process.
>
> Hope, 2.5, with your changes, will be released soon :)
>
> On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org>
> wrote:
>
> > Hello, Igniters.
> >
> > Good news.
> >
> > IGNITE-7337 [1](Spark Data Frames: support saving a data frame in Ignite)
> > are merged to master.
> >
> > For now we can both - read from and write to Ignite SQL table with Data
> > Frame API.
> > Big thanks to Valentin Kulichenko for a quick review.
> > So it seems we can include this feature to 2.4 release.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7337
>

Re: Saving Spark Data Frames merged to master

[Anton Vinogradov]

Nikolay,

2.4 is almost ready to be released.
We're fixing final issues to provide stable and fast release.
Merging something to 2.4 except blockers is not possible at this phase of
release process.

Hope, 2.5, with your changes, will be released soon :)

On Fri, Feb 9, 2018 at 7:19 AM, Nikolay Izhikov <nizhi...@apache.org> wrote:

> Hello, Igniters.
>
> Good news.
>
> IGNITE-7337 [1](Spark Data Frames: support saving a data frame in Ignite)
> are merged to master.
>
> For now we can both - read from and write to Ignite SQL table with Data
> Frame API.
> Big thanks to Valentin Kulichenko for a quick review.
> So it seems we can include this feature to 2.4 release.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7337

Saving Spark Data Frames merged to master

[Nikolay Izhikov]

Hello, Igniters.

Good news.

IGNITE-7337 [1](Spark Data Frames: support saving a data frame in Ignite) are 
merged to master.

For now we can both - read from and write to Ignite SQL table with Data Frame 
API.
Big thanks to Valentin Kulichenko for a quick review.
So it seems we can include this feature to 2.4 release.

[1] https://issues.apache.org/jira/browse/IGNITE-7337

signature.asc
Description: This is a digitally signed message part

[jira] [Created] (IGNITE-7655) Spark Data Frames: saving data frames into Ignite needs to be documented

[Nikolay Izhikov (JIRA)]

Nikolay Izhikov created IGNITE-7655:
---

 Summary: Spark Data Frames: saving data frames into Ignite needs 
to be documented
 Key: IGNITE-7655
 URL: https://issues.apache.org/jira/browse/IGNITE-7655
 Project: Ignite
  Issue Type: Bug
  Components: spark
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
 Fix For: 2.4


Once IGNITE-7337 is ready for merge.
This new feature of Ignite needs to be documented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Spark data frames integration merged

[Nikolay Izhikov]

Hello, Denis.

> Nikolay, could you document the feature before the release [1]?

Yes, I can. 
I will document these feature in a next few days.


В Пт, 29/12/2017 в 13:37 -0800, Denis Magda пишет:
> Great news,
> 
> Thanks Nikolay and Val!
> 
> Nikolay, could you document the feature before the release [1]? I’ve
> granted you required permission.
> 
> More on the doc process can be found here [2].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-7345
> [2] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Documen
> t
> 
> —
> Denis
> 
> > On Dec 29, 2017, at 1:22 PM, Valentin Kulichenko  > k...@gmail.com> wrote:
> > 
> > Igniters,
> > 
> > Great news! We completed and merged first part of integration with
> > Spark data frames [1]. It contains implementation of Spark data
> > source which allows to use DataFrame API to query Ignite data, as
> > well as join it with other data frames originated from different
> > sources.
> > 
> > Next planned steps are the following:
> > - Implement custom execution strategy to avoid transferring data
> > from Ignite to Spark when possible [2]. This should give serious
> > performance improvement in cases when only Ignite tables
> > participate in a query.
> > - Implement ability to save a data frame into Ignite via
> > DataFrameWrite API [3].
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-3084
> > [2] https://issues.apache.org/jira/browse/IGNITE-7077
> > [3] https://issues.apache.org/jira/browse/IGNITE-7337
> > 
> > Nikolay Izhikov, thanks for the contribution and for all the hard
> > work!
> > 
> > -Val
> 
> 

Re: Spark data frames integration merged

[Denis Magda]

Revin,

Excellent, please keep me in the loop and let me know once you achieve the next 
milestone being ready for the production. This type of use cases help to spread 
a word about Ignite which is really-really helpful!

—
Denis

> On Jan 5, 2018, at 12:27 AM, Revin Chalil <rcha...@expedia.com> wrote:
> 
> Thanks Denis. I watched your recent 2 webinars and they were very helpful.
>  
> I can definitely create a page explaining how (currently three) ignite 
> shared-rdd caches are shared across multiple spark streaming apps for data 
> enrichment here at expedia, once the solution is stabilized. We are not in 
> production yet. I have enabled native persistence and had some hiccups during 
> our testing but is looking better today.
>  
> We are currently working to optimize the join between incremental data and 
> shared-rdd dataframe in spark as there are several spark Apps and the total 
> memory is limited. This part does not have much to do with Ignite but mostly 
> spark optimization, I believe. We do load the entire ignite-cache (~50GB 
> each) into spark executors and the cache is trimmed based on the business 
> rules, daily.
>  
> We will keep in touch and thanks again for all the great work and help 
> everyone.
>  
> Revin
>  
> From: Denis Magda <dma...@apache.org>
> Date: Thursday, January 4, 2018 at 12:34 PM
> To: Revin Chalil <rcha...@expedia.com>
> Cc: "dev@ignite.apache.org" <dev@ignite.apache.org>
> Subject: Re: Spark data frames integration merged
>  
> Revin, 
>  
> As as side note, do you have a public article published or any other relevant 
> material that explains how Ignite is used at Expedia?
>  
> You would help the community out a lot if such information is referenced from 
> this page:
> https://ignite.apache.org/provenusecases.html 
> <https://ignite.apache.org/provenusecases.html>
>  
> —
> Denis
>  
> On Jan 3, 2018, at 11:24 AM, Revin Chalil <rcha...@expedia.com 
> <mailto:rcha...@expedia.com>> wrote:
>  
> Thank you and this is great news. 
> 
> We currently use the Ignite cache as a Reference dataset RDD in Spark, 
> convert it into a spark DataFrame and then join this DF with the 
> incoming-data DF. I hope we can change this 3 step process to a single step 
> with the Spark DF integration. If so, would index / affinitykeys on the join 
> columns help with performance? We currently do not have them defined on the 
> Reference dataset. Are there examples available joining ignite DF with Spark 
> DF? Also, what is the best way to get the latest executables with the 
> IGNITE-3084 included? Thanks again. 
> 
> 
> On 12/29/17, 10:34 PM, "Nikolay Izhikov" <nizhikov@gmail.com 
> <mailto:nizhikov@gmail.com>> wrote:
> 
>Thank you, guys.
> 
>Val, thanks for all reviews, advices and patience.
> 
>Anton, thanks for ignite wisdom you share with me.
> 
>Looking forward for next issues :)
> 
>P.S Happy New Year for all Ignite community!
> 
>В Пт, 29/12/2017 в 13:22 -0800, Valentin Kulichenko пишет:
> 
> Igniters,
> 
> Great news! We completed and merged first part of integration with
> Spark data frames [1]. It contains implementation of Spark data
> source which allows to use DataFrame API to query Ignite data, as
> well as join it with other data frames originated from different
> sources.
> 
> Next planned steps are the following:
> - Implement custom execution strategy to avoid transferring data from
> Ignite to Spark when possible [2]. This should give serious
> performance improvement in cases when only Ignite tables participate
> in a query.
> - Implement ability to save a data frame into Ignite via
> DataFrameWrite API [3].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084 
> <https://issues.apache.org/jira/browse/IGNITE-3084>
> [2] https://issues.apache.org/jira/browse/IGNITE-7077 
> <https://issues.apache.org/jira/browse/IGNITE-7077>
> [3] https://issues.apache.org/jira/browse/IGNITE-7337 
> <https://issues.apache.org/jira/browse/IGNITE-7337>
> 
> Nikolay Izhikov, thanks for the contribution and for all the hard
> work!
> 
> -Val
>  
> 
>  

Re: Spark data frames integration merged

[Nikolay Izhikov]

Hello, guys.

Currently `getPreferredLocations` implemented in 
`IgniteRDD -> IgniteAbstractRDD`.

But DataFrame implementation uses 
`IgniteSQLDataFrameRDD -> IgniteSqlRDD -> IgniteAbstractRDD`

Where `->` is extension.

So, for now, getPreferredLocation doesn't implemented for a
IgniteDataFrame.

Please, take a look [1], [2].

I think it a very good idea to implement `getPreferredLocation` inside
`IgniteSQLDataFrameRDD` or event inside `IgniteAbstractRDD`

Can someone file a ticket? Or I can do it by myself.


[1] - https://github.com/apache/ignite/blob/master/modules/spark/src/ma
in/scala/org/apache/ignite/spark/IgniteRDD.scala#L50

[2] - https://github.com/apache/ignite/blob/master/modules/spark/src/ma
in/scala/org/apache/ignite/spark/impl/IgniteSQLDataFrameRDD.scala#L40


В Ср, 03/01/2018 в 15:35 -0800, Valentin Kulichenko пишет:
> Revin,
> 
> I doubt IgniteRDD#getPrefferredLocations has any affect on data
> frames, but this is an interesting point. Nikolay, as a developer of
> this functionality, can you please comment on this?
> 
> -Val
> 
> On Wed, Jan 3, 2018 at 1:22 PM, Revin Chalil 
> wrote:
> > Thanks Val for the info on indexes with DF. Do you know if adding
> > index / affinitykeys on the cache help with the join, when the
> > IgniteRDD is joined with a spark DF? The below from docs say that
> > 
> > “IgniteRDD also provides affinity information to Spark via
> > getPrefferredLocations method so that RDD computations use data
> > locality.”
> > 
> > I was wondering, if the affinitykey on the cache can be utilized in
> > the spark join?
> > 
> > 
> > On 1/3/18, 12:27 PM, "vkulichenko" 
> > wrote:
> > 
> > Indexes would not be used during joins, at least in current
> > implementation.
> > Current integration is implemented as a regular Spark data
> > source which
> > provides each relation separately. Spark then performs join by
> > itself, so
> > Ignite indexes do not help.
> > 
> > The easiest way to get binaries would be to use a nightly build
> > [1] , but it
> > seems to be broken for some reason (latest is from May 31). I
> > guess the only
> > option at the moment is to build from source.
> > 
> > [1]
> > https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-night
> > ly/lastSuccessfulBuild/
> > 
> > -Val
> > 
> > 
> > 
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > 
> > 
> 
> 

Re: Spark data frames integration merged

[Revin Chalil]

Thanks Denis. I watched your recent 2 webinars and they were very helpful.

I can definitely create a page explaining how (currently three) ignite 
shared-rdd caches are shared across multiple spark streaming apps for data 
enrichment here at expedia, once the solution is stabilized. We are not in 
production yet. I have enabled native persistence and had some hiccups during 
our testing but is looking better today.

We are currently working to optimize the join between incremental data and 
shared-rdd dataframe in spark as there are several spark Apps and the total 
memory is limited. This part does not have much to do with Ignite but mostly 
spark optimization, I believe. We do load the entire ignite-cache (~50GB each) 
into spark executors and the cache is trimmed based on the business rules, 
daily.

We will keep in touch and thanks again for all the great work and help everyone.

Revin

From: Denis Magda <dma...@apache.org>
Date: Thursday, January 4, 2018 at 12:34 PM
To: Revin Chalil <rcha...@expedia.com>
Cc: "dev@ignite.apache.org" <dev@ignite.apache.org>
Subject: Re: Spark data frames integration merged

Revin,

As as side note, do you have a public article published or any other relevant 
material that explains how Ignite is used at Expedia?

You would help the community out a lot if such information is referenced from 
this page:
https://ignite.apache.org/provenusecases.html

—
Denis

On Jan 3, 2018, at 11:24 AM, Revin Chalil 
<rcha...@expedia.com<mailto:rcha...@expedia.com>> wrote:

Thank you and this is great news.

We currently use the Ignite cache as a Reference dataset RDD in Spark, convert 
it into a spark DataFrame and then join this DF with the incoming-data DF. I 
hope we can change this 3 step process to a single step with the Spark DF 
integration. If so, would index / affinitykeys on the join columns help with 
performance? We currently do not have them defined on the Reference dataset. 
Are there examples available joining ignite DF with Spark DF? Also, what is the 
best way to get the latest executables with the IGNITE-3084 included? Thanks 
again.


On 12/29/17, 10:34 PM, "Nikolay Izhikov" 
<nizhikov@gmail.com<mailto:nizhikov@gmail.com>> wrote:

   Thank you, guys.

   Val, thanks for all reviews, advices and patience.

   Anton, thanks for ignite wisdom you share with me.

   Looking forward for next issues :)

   P.S Happy New Year for all Ignite community!

   В Пт, 29/12/2017 в 13:22 -0800, Valentin Kulichenko пишет:

Igniters,

Great news! We completed and merged first part of integration with
Spark data frames [1]. It contains implementation of Spark data
source which allows to use DataFrame API to query Ignite data, as
well as join it with other data frames originated from different
sources.

Next planned steps are the following:
- Implement custom execution strategy to avoid transferring data from
Ignite to Spark when possible [2]. This should give serious
performance improvement in cases when only Ignite tables participate
in a query.
- Implement ability to save a data frame into Ignite via
DataFrameWrite API [3].

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://issues.apache.org/jira/browse/IGNITE-7077
[3] https://issues.apache.org/jira/browse/IGNITE-7337

Nikolay Izhikov, thanks for the contribution and for all the hard
work!

-Val

Re: Spark data frames integration merged

[Denis Magda]

Revin,

As as side note, do you have a public article published or any other relevant 
material that explains how Ignite is used at Expedia?

You would help the community out a lot if such information is referenced from 
this page:
https://ignite.apache.org/provenusecases.html 
<https://ignite.apache.org/provenusecases.html>

—
Denis

> On Jan 3, 2018, at 11:24 AM, Revin Chalil <rcha...@expedia.com> wrote:
> 
> Thank you and this is great news. 
> 
> We currently use the Ignite cache as a Reference dataset RDD in Spark, 
> convert it into a spark DataFrame and then join this DF with the 
> incoming-data DF. I hope we can change this 3 step process to a single step 
> with the Spark DF integration. If so, would index / affinitykeys on the join 
> columns help with performance? We currently do not have them defined on the 
> Reference dataset. Are there examples available joining ignite DF with Spark 
> DF? Also, what is the best way to get the latest executables with the 
> IGNITE-3084 included? Thanks again. 
> 
> 
> On 12/29/17, 10:34 PM, "Nikolay Izhikov" <nizhikov@gmail.com> wrote:
> 
>Thank you, guys.
> 
>Val, thanks for all reviews, advices and patience.
> 
>Anton, thanks for ignite wisdom you share with me.
> 
>Looking forward for next issues :)
> 
>P.S Happy New Year for all Ignite community!
> 
>В Пт, 29/12/2017 в 13:22 -0800, Valentin Kulichenko пишет:
>> Igniters,
>> 
>> Great news! We completed and merged first part of integration with
>> Spark data frames [1]. It contains implementation of Spark data
>> source which allows to use DataFrame API to query Ignite data, as
>> well as join it with other data frames originated from different
>> sources.
>> 
>> Next planned steps are the following:
>> - Implement custom execution strategy to avoid transferring data from
>> Ignite to Spark when possible [2]. This should give serious
>> performance improvement in cases when only Ignite tables participate
>> in a query.
>> - Implement ability to save a data frame into Ignite via
>> DataFrameWrite API [3].
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-3084
>> [2] https://issues.apache.org/jira/browse/IGNITE-7077
>> [3] https://issues.apache.org/jira/browse/IGNITE-7337
>> 
>> Nikolay Izhikov, thanks for the contribution and for all the hard
>> work!
>> 
>> -Val
> 
> 

Re: Spark data frames integration merged

[Valentin Kulichenko]

Revin,

I doubt IgniteRDD#getPrefferredLocations has any affect on data frames, but
this is an interesting point. Nikolay, as a developer of this
functionality, can you please comment on this?

-Val

On Wed, Jan 3, 2018 at 1:22 PM, Revin Chalil  wrote:

> Thanks Val for the info on indexes with DF. Do you know if adding index /
> affinitykeys on the cache help with the join, when the IgniteRDD is joined
> with a spark DF? The below from docs say that
>
> “IgniteRDD also provides affinity information to Spark via
> getPrefferredLocations method so that RDD computations use data locality.”
>
> I was wondering, if the affinitykey on the cache can be utilized in the
> spark join?
>
>
> On 1/3/18, 12:27 PM, "vkulichenko"  wrote:
>
> Indexes would not be used during joins, at least in current
> implementation.
> Current integration is implemented as a regular Spark data source which
> provides each relation separately. Spark then performs join by itself,
> so
> Ignite indexes do not help.
>
> The easiest way to get binaries would be to use a nightly build [1] ,
> but it
> seems to be broken for some reason (latest is from May 31). I guess
> the only
> option at the moment is to build from source.
>
> [1]
> https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/
> lastSuccessfulBuild/
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>

Re: Spark data frames integration merged

[Revin Chalil]

Thank you and this is great news. 

We currently use the Ignite cache as a Reference dataset RDD in Spark, convert 
it into a spark DataFrame and then join this DF with the incoming-data DF. I 
hope we can change this 3 step process to a single step with the Spark DF 
integration. If so, would index / affinitykeys on the join columns help with 
performance? We currently do not have them defined on the Reference dataset. 
Are there examples available joining ignite DF with Spark DF? Also, what is the 
best way to get the latest executables with the IGNITE-3084 included? Thanks 
again. 


On 12/29/17, 10:34 PM, "Nikolay Izhikov" <nizhikov@gmail.com> wrote:

Thank you, guys.

Val, thanks for all reviews, advices and patience.

Anton, thanks for ignite wisdom you share with me.

Looking forward for next issues :)

P.S Happy New Year for all Ignite community!

В Пт, 29/12/2017 в 13:22 -0800, Valentin Kulichenko пишет:
> Igniters,
> 
> Great news! We completed and merged first part of integration with
> Spark data frames [1]. It contains implementation of Spark data
> source which allows to use DataFrame API to query Ignite data, as
> well as join it with other data frames originated from different
> sources.
> 
> Next planned steps are the following:
> - Implement custom execution strategy to avoid transferring data from
> Ignite to Spark when possible [2]. This should give serious
> performance improvement in cases when only Ignite tables participate
> in a query.
> - Implement ability to save a data frame into Ignite via
> DataFrameWrite API [3].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
> [2] https://issues.apache.org/jira/browse/IGNITE-7077
> [3] https://issues.apache.org/jira/browse/IGNITE-7337
> 
> Nikolay Izhikov, thanks for the contribution and for all the hard
> work!
> 
> -Val

Re: Spark data frames integration merged

[Nikolay Izhikov]

Thank you, guys.

Val, thanks for all reviews, advices and patience.

Anton, thanks for ignite wisdom you share with me.

Looking forward for next issues :)

P.S Happy New Year for all Ignite community!

В Пт, 29/12/2017 в 13:22 -0800, Valentin Kulichenko пишет:
> Igniters,
> 
> Great news! We completed and merged first part of integration with
> Spark data frames [1]. It contains implementation of Spark data
> source which allows to use DataFrame API to query Ignite data, as
> well as join it with other data frames originated from different
> sources.
> 
> Next planned steps are the following:
> - Implement custom execution strategy to avoid transferring data from
> Ignite to Spark when possible [2]. This should give serious
> performance improvement in cases when only Ignite tables participate
> in a query.
> - Implement ability to save a data frame into Ignite via
> DataFrameWrite API [3].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
> [2] https://issues.apache.org/jira/browse/IGNITE-7077
> [3] https://issues.apache.org/jira/browse/IGNITE-7337
> 
> Nikolay Izhikov, thanks for the contribution and for all the hard
> work!
> 
> -Val

Re: Spark data frames integration merged

[Denis Magda]

Great news,

Thanks Nikolay and Val!

Nikolay, could you document the feature before the release [1]? I’ve granted 
you required permission.

More on the doc process can be found here [2].

[1] https://issues.apache.org/jira/browse/IGNITE-7345 
<https://issues.apache.org/jira/browse/IGNITE-7345>
[2] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document 
<https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document>

—
Denis

> On Dec 29, 2017, at 1:22 PM, Valentin Kulichenko 
> <valentin.kuliche...@gmail.com> wrote:
> 
> Igniters,
> 
> Great news! We completed and merged first part of integration with Spark data 
> frames [1]. It contains implementation of Spark data source which allows to 
> use DataFrame API to query Ignite data, as well as join it with other data 
> frames originated from different sources.
> 
> Next planned steps are the following:
> - Implement custom execution strategy to avoid transferring data from Ignite 
> to Spark when possible [2]. This should give serious performance improvement 
> in cases when only Ignite tables participate in a query.
> - Implement ability to save a data frame into Ignite via DataFrameWrite API 
> [3].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084 
> <https://issues.apache.org/jira/browse/IGNITE-3084>
> [2] https://issues.apache.org/jira/browse/IGNITE-7077 
> <https://issues.apache.org/jira/browse/IGNITE-7077>
> [3] https://issues.apache.org/jira/browse/IGNITE-7337 
> <https://issues.apache.org/jira/browse/IGNITE-7337>
> 
> Nikolay Izhikov, thanks for the contribution and for all the hard work!
> 
> -Val

[jira] [Created] (IGNITE-7345) Spark Data Frames and Ignite Documentation

[Denis Magda (JIRA)]

Denis Magda created IGNITE-7345:
---

 Summary: Spark Data Frames and Ignite Documentation
 Key: IGNITE-7345
 URL: https://issues.apache.org/jira/browse/IGNITE-7345
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Denis Magda
Assignee: Nikolay Izhikov
 Fix For: 2.4


Spark Data frames integration [1] needs to be documented in this [2] domain.

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Spark-data-frames-integration-merged-td25817.html
[2] https://apacheignite-fs.readme.io/docs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Spark data frames integration merged

[Valentin Kulichenko]

Igniters,

Great news! We completed and merged first part of integration with Spark
data frames [1]. It contains implementation of Spark data source which
allows to use DataFrame API to query Ignite data, as well as join it with
other data frames originated from different sources.

Next planned steps are the following:
- Implement custom execution strategy to avoid transferring data from
Ignite to Spark when possible [2]. This should give serious performance
improvement in cases when only Ignite tables participate in a query.
- Implement ability to save a data frame into Ignite via DataFrameWrite API
[3].

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://issues.apache.org/jira/browse/IGNITE-7077
[3] https://issues.apache.org/jira/browse/IGNITE-7337

Nikolay Izhikov, thanks for the contribution and for all the hard work!

-Val

[jira] [Created] (IGNITE-7337) Spark Data Frames: support saving a data frame in Ignite

[Valentin Kulichenko (JIRA)]

Valentin Kulichenko created IGNITE-7337:
---

 Summary: Spark Data Frames: support saving a data frame in Ignite
 Key: IGNITE-7337
 URL: https://issues.apache.org/jira/browse/IGNITE-7337
 Project: Ignite
  Issue Type: New Feature
  Components: spark
Affects Versions: 2.3
Reporter: Valentin Kulichenko
Assignee: Nikolay Izhikov
Priority: Critical
 Fix For: 2.4


Once Ignite data source for Spark is implemented, we need to add an ability to 
store a data frame in Ignite. Most likely if should be enough to provide 
implementation for the following traits:
* {{InsertableRelation}}
* {{CreatableRelationProvider}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)