Re: OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

[Bryan Cutler]

The example works for me, please check your environment and ensure you are
using Spark 2.3.0 where OneHotEncoderEstimator was introduced.

On Fri, May 18, 2018 at 12:57 AM, Matteo Cossu  wrote:

> Hi,
>
> are you sure Dataset has a method withColumns?
>
> On 15 May 2018 at 16:58, Mina Aslani  wrote:
>
>> Hi,
>>
>> I get below error when I try to run oneHotEncoderEstimator example.
>> https://github.com/apache/spark/blob/b74366481cc87490adf4e69
>> d26389ec737548c15/examples/src/main/java/org/apache/
>> spark/examples/ml/JavaOneHotEncoderEstimatorExample.java#L67
>>
>> Which is this line of the code:
>> https://github.com/apache/spark/blob/master/mllib/src/main/
>> scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348
>>
>> Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: 
>> org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
>>  at 
>> org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)
>>
>>
>> Can you please let me know, what is the cause? Any workaround?
>>
>> Seeing the example in the repo, looks like that at some point it used be
>> running fine. And, now it's not working. Also, oneHotEncoder is deprecated.
>>
>> I really appreciate your quick response.
>>
>> Regards,
>> Mina
>>
>>
>

Re: OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

[Matteo Cossu]

Hi,

are you sure Dataset has a method withColumns?

On 15 May 2018 at 16:58, Mina Aslani  wrote:

> Hi,
>
> I get below error when I try to run oneHotEncoderEstimator example.
> https://github.com/apache/spark/blob/b74366481cc87490adf4e69d26389e
> c737548c15/examples/src/main/java/org/apache/spark/examples/ml/
> JavaOneHotEncoderEstimatorExample.java#L67
>
> Which is this line of the code:
> https://github.com/apache/spark/blob/master/mllib/src/
> main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348
>
> Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: 
> org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
>   at 
> org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)
>
>
> Can you please let me know, what is the cause? Any workaround?
>
> Seeing the example in the repo, looks like that at some point it used be
> running fine. And, now it's not working. Also, oneHotEncoder is deprecated.
>
> I really appreciate your quick response.
>
> Regards,
> Mina
>
>

OneHotEncoderEstimator - java.lang.NoSuchMethodError: org.apache.spark.sql.Dataset.withColumns

[Mina Aslani]

Hi,

I get below error when I try to run oneHotEncoderEstimator example.
https://github.com/apache/spark/blob/b74366481cc87490adf4e69d26389ec737548c15/examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderEstimatorExample.java#L67

Which is this line of the code:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala#L348

Exception in thread "streaming-job-executor-0"
java.lang.NoSuchMethodError:
org.apache.spark.sql.Dataset.withColumns(Lscala/collection/Seq;Lscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
at 
org.apache.spark.ml.feature.OneHotEncoderModel.transform(OneHotEncoderEstimator.scala:348)


Can you please let me know, what is the cause? Any workaround?

Seeing the example in the repo, looks like that at some point it used be
running fine. And, now it's not working. Also, oneHotEncoder is deprecated.

I really appreciate your quick response.

Regards,
Mina