Re: Feature request: Java-specific transform method in Dataset

[kant kodali]

I am not affiliated with Flink or Spark but I do think some of the thoughts
here

makes sense

On Sun, Jul 1, 2018 at 4:12 PM, Sean Owen  wrote:

> It's true, that is one of the issues to be solved by the 2.12-compatible
> build, because it otherwise introduces an overload ambiguity for Java 8
> lambdas. But for that reason I think the current transform() method would
> start working with lambdas. That would only help 2.12 builds; maybe that's
> an OK solution?
>
>
> On Sun, Jul 1, 2018, 2:36 PM Reynold Xin  wrote:
>
>> This wouldn’t be a problem with Scala 2.12 right?
>>
>> On Sun, Jul 1, 2018 at 12:23 PM Sean Owen  wrote:
>>
>>> I see, transform() doesn't have the same overload that other methods do
>>> in order to support Java 8 lambdas as you'd expect. One option is to
>>> introduce something like MapFunction for transform and introduce an
>>> overload.
>>>
>>> I think transform() isn't used much at all, so maybe why it wasn't
>>> Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
>>> might. I think it could be OK to add the overload to match how map works.
>>>
>>> On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales 
>>> wrote:
>>>
 No, because Function1 from Scala is not a functional interface.
 You can see a simple example of what I'm trying to accomplish In the
 unit test here:
 https://github.com/void/spark/blob/java-transform/sql/core/
 src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73


 On Sun, Jul 1, 2018 at 2:48 PM Sean Owen  wrote:

> Don't Java 8 lambdas let you do this pretty immediately? Can you give
> an example here of what you want to do and how you are trying to do it?
>
> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales 
> wrote:
>
>> Hi,
>>  it would be nice to have an easier way to use the Dataset transform
>> method from Java than implementing a Function1 from Scala.
>>
>> I've made a simple implentation here:
>>
>> https://github.com/void/spark/tree/java-transform
>>
>> Should I open a JIRA?
>>
>> Ismael Carnales
>>
>

Re: Feature request: Java-specific transform method in Dataset

[Sean Owen]

It's true, that is one of the issues to be solved by the 2.12-compatible
build, because it otherwise introduces an overload ambiguity for Java 8
lambdas. But for that reason I think the current transform() method would
start working with lambdas. That would only help 2.12 builds; maybe that's
an OK solution?

On Sun, Jul 1, 2018, 2:36 PM Reynold Xin  wrote:

> This wouldn’t be a problem with Scala 2.12 right?
>
> On Sun, Jul 1, 2018 at 12:23 PM Sean Owen  wrote:
>
>> I see, transform() doesn't have the same overload that other methods do
>> in order to support Java 8 lambdas as you'd expect. One option is to
>> introduce something like MapFunction for transform and introduce an
>> overload.
>>
>> I think transform() isn't used much at all, so maybe why it wasn't
>> Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
>> might. I think it could be OK to add the overload to match how map works.
>>
>> On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales 
>> wrote:
>>
>>> No, because Function1 from Scala is not a functional interface.
>>> You can see a simple example of what I'm trying to accomplish In the
>>> unit test here:
>>>
>>> https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>>>
>>>
>>> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen  wrote:
>>>
 Don't Java 8 lambdas let you do this pretty immediately? Can you give
 an example here of what you want to do and how you are trying to do it?

 On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales 
 wrote:

> Hi,
>  it would be nice to have an easier way to use the Dataset transform
> method from Java than implementing a Function1 from Scala.
>
> I've made a simple implentation here:
>
> https://github.com/void/spark/tree/java-transform
>
> Should I open a JIRA?
>
> Ismael Carnales
>

Re: Feature request: Java-specific transform method in Dataset

[Reynold Xin]

This wouldn’t be a problem with Scala 2.12 right?

On Sun, Jul 1, 2018 at 12:23 PM Sean Owen  wrote:

> I see, transform() doesn't have the same overload that other methods do in
> order to support Java 8 lambdas as you'd expect. One option is to introduce
> something like MapFunction for transform and introduce an overload.
>
> I think transform() isn't used much at all, so maybe why it wasn't
> Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
> might. I think it could be OK to add the overload to match how map works.
>
> On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales 
> wrote:
>
>> No, because Function1 from Scala is not a functional interface.
>> You can see a simple example of what I'm trying to accomplish In the unit
>> test here:
>>
>> https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>>
>>
>> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen  wrote:
>>
>>> Don't Java 8 lambdas let you do this pretty immediately? Can you give an
>>> example here of what you want to do and how you are trying to do it?
>>>
>>> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales 
>>> wrote:
>>>
 Hi,
  it would be nice to have an easier way to use the Dataset transform
 method from Java than implementing a Function1 from Scala.

 I've made a simple implentation here:

 https://github.com/void/spark/tree/java-transform

 Should I open a JIRA?

 Ismael Carnales

>>>

Re: Feature request: Java-specific transform method in Dataset

[Sean Owen]

I see, transform() doesn't have the same overload that other methods do in
order to support Java 8 lambdas as you'd expect. One option is to introduce
something like MapFunction for transform and introduce an overload.

I think transform() isn't used much at all, so maybe why it wasn't
Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it
might. I think it could be OK to add the overload to match how map works.

On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales  wrote:

> No, because Function1 from Scala is not a functional interface.
> You can see a simple example of what I'm trying to accomplish In the unit
> test here:
>
> https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73
>
>
> On Sun, Jul 1, 2018 at 2:48 PM Sean Owen  wrote:
>
>> Don't Java 8 lambdas let you do this pretty immediately? Can you give an
>> example here of what you want to do and how you are trying to do it?
>>
>> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales 
>> wrote:
>>
>>> Hi,
>>>  it would be nice to have an easier way to use the Dataset transform
>>> method from Java than implementing a Function1 from Scala.
>>>
>>> I've made a simple implentation here:
>>>
>>> https://github.com/void/spark/tree/java-transform
>>>
>>> Should I open a JIRA?
>>>
>>> Ismael Carnales
>>>
>>

Re: Feature request: Java-specific transform method in Dataset

[Ismael Carnales]

No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit
test here:
https://github.com/void/spark/blob/java-transform/sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java#L73


On Sun, Jul 1, 2018 at 2:48 PM Sean Owen  wrote:

> Don't Java 8 lambdas let you do this pretty immediately? Can you give an
> example here of what you want to do and how you are trying to do it?
>
> On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales  wrote:
>
>> Hi,
>>  it would be nice to have an easier way to use the Dataset transform
>> method from Java than implementing a Function1 from Scala.
>>
>> I've made a simple implentation here:
>>
>> https://github.com/void/spark/tree/java-transform
>>
>> Should I open a JIRA?
>>
>> Ismael Carnales
>>
>

Re: Feature request: Java-specific transform method in Dataset

[Sean Owen]

Don't Java 8 lambdas let you do this pretty immediately? Can you give an
example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales  wrote:

> Hi,
>  it would be nice to have an easier way to use the Dataset transform
> method from Java than implementing a Function1 from Scala.
>
> I've made a simple implentation here:
>
> https://github.com/void/spark/tree/java-transform
>
> Should I open a JIRA?
>
> Ismael Carnales
>

Feature request: Java-specific transform method in Dataset

[Ismael Carnales]

Hi,
 it would be nice to have an easier way to use the Dataset transform method
from Java than implementing a Function1 from Scala.

I've made a simple implentation here:

https://github.com/void/spark/tree/java-transform

Should I open a JIRA?

Ismael Carnales