Re: Spark ML - Java implementation of custom Transformer

[Yanbo Liang]

Hi Mehdi,

Could you share your code and then we can help you to figure out the
problem?
Actually JavaTestParams can work well but there is some compatibility issue
for JavaDeveloperApiExample.
We have removed JavaDeveloperApiExample temporary at Spark 2.0 in order to
not confuse users. Since the solution for the compatibility issue has been
figured out, we will add it back at 2.1.

Thanks
Yanbo

2016-06-27 11:57 GMT-07:00 Mehdi Meziane :

> Hi all,
>
> We have some problems while implementing custom Transformers in JAVA
> (SPARK 1.6.1).
> We do override the method copy, but it crashes with an AbstractMethodError.
>
> If we extends the UnaryTransformer, and do not override the copy method,
> it works without any error.
>
> We tried to write the copy like in these examples :
>
> https://github.com/apache/spark/blob/branch-2.0/mllib/src/test/java/org/apache/spark/ml/param/JavaTestParams.java
>
> https://github.com/eBay/Spark/blob/branch-1.6/examples/src/main/java/org/apache/spark/examples/ml/JavaDeveloperApiExample.java
>
> None of it worked.
>
> The copy is defined in the Params class as :
>
>   /**
>* Creates a copy of this instance with the same UID and some extra
> params.
>* Subclasses should implement this method and set the return type
> properly.
>*
>* @see [[defaultCopy()]]
>*/
>   def copy(extra: ParamMap): Params
>
> Any idea?
> Thanks,
>
> Mehdi
>

Spark ML - Java implementation of custom Transformer

[Mehdi Meziane]

Hi all, 


We have some problems while implementing custom Transformers in JAVA (SPARK 
1.6.1). 
We do override the method copy, but it crashes with an AbstractMethodError. 


If we extends the UnaryTransformer, and do not override the copy method, it 
works without any error. 


We tried to write the copy like in these examples : 
https://github.com/apache/spark/blob/branch-2.0/mllib/src/test/java/org/apache/spark/ml/param/JavaTestParams.java
 
https://github.com/eBay/Spark/blob/branch-1.6/examples/src/main/java/org/apache/spark/examples/ml/JavaDeveloperApiExample.java
 


None of it worked. 


The copy is defined in the Params class as : 


/** 
* Creates a copy of this instance with the same UID and some extra params. 
* Subclasses should implement this method and set the return type properly. 
* 
* @see [[defaultCopy()]] 
*/ 
def copy(extra: ParamMap): Params 


Any idea? 
Thanks, 


Mehdi