[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990892#comment-16990892
]
Sabarish Sasidharan commented on SPARK-12606:
-
For folks who run into this, I was able to solve this issue by using a static
uid generated like below
{code:java}
private static final String uid =
Identifiable$.MODULE$.randomUID("sentence_embed_finisher_2");
@Override
public String uid() {
return uid;
}
{code}
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
>Priority: Major
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190904#comment-16190904
]
Akos Tomasits commented on SPARK-12606:
---
We have run into the same issue. We cannot create proper Java transformers
derived from UnaryTransformer.
We would like to use these custom transformers through CrossValidator, that in
the end requires a constructor with a string (uid) parameter. I guess the
custom transformer is supposed to set the provided uid in this constructor,
however, the object's uid() method is called before the constructor finishes.
This leads to the above mentioned "null__inputCol" error.
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-un
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086961#comment-16086961
]
Davis commented on SPARK-12606:
---
This happens on spark 2.2.0. In essence we can not a write a Transformer in
Java extending UnaryTransformer.
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051726#comment-16051726
]
Ilyes Hachani commented on SPARK-12606:
---
[~srowen] Can I rewrite it ?
What does setting the UID in the constructor means?
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044508#comment-16044508
]
Ilyes Hachani commented on SPARK-12606:
---
Any update on this?
I am using Java and extending the class but I get problem when I use
"setInputCol"
{code}
"java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
does not belong to TextCleaner_40f8c0be7bc7."
{code}
I tried setting the uid inside the constructor to no result.
{code:title=TextCleaner.java|borderStyle=solid}
private final String uid ;
public TextCleaner(){
uid = Identifiable$.MODULE$.randomUID("TextCleaner");
}
@Override
public String uid() {
return uid;
}
{code}
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15955191#comment-15955191
]
Guillaume Dardelet commented on SPARK-12606:
I had the same issue in Scala and I solved it by overloading the constructor so
that it initialises the UID.
The error comes from the initialisation of the parameter "inputCol".
You get "null__inputCol" because when the parameter was initialised, your class
didn't have a uid.
Therefore, instead of
class Lemmatizer extends UnaryTransformer[String, String, Lemmatizer] {
override val uid: String = Identifiable.randomUID("lemmatizer")
protected def createTransformFunc: String) => String = ???
protected def outputDataType: DataType = StringType
}
Do this:
class Lemmatizer(override val uid: String) extends UnaryTransformer[String,
String, Lemmatizer] {
def this() = this( Identifiable.randomUID("lemmatizer") )
protected def createTransformFunc: String) => String = ???
protected def outputDataType: DataType = StringType
}
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return wor
[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?
[
https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842630#comment-15842630
]
Hyukjin Kwon commented on SPARK-12606:
--
Do you mind if I ask to fix the title and contents in more details and clear as
a descriptive JIRA?
I thought this is a question when I read this first. I hope the JIRA focuses
only on the issue without irrelevant information (as it seems there have been
no interests on this issue more than a year).
> Scala/Java compatibility issue Re: how to extend java transformer from Scala
> UnaryTransformer ?
> ---
>
> Key: SPARK-12606
> URL: https://issues.apache.org/jira/browse/SPARK-12606
> Project: Spark
> Issue Type: Bug
> Components: ML
>Affects Versions: 1.5.2
> Environment: Java 8, Mac OS, Spark-1.5.2
>Reporter: Andrew Davidson
> Labels: transformers
>
> Hi Andy,
> I suspect that you hit the Scala/Java compatibility issue, I can also
> reproduce this issue, so could you file a JIRA to track this issue?
> Yanbo
> 2016-01-02 3:38 GMT+08:00 Andy Davidson :
> I am trying to write a trivial transformer I use use in my pipeline. I am
> using java and spark 1.5.2. It was suggested that I use the Tokenize.scala
> class as an example. This should be very easy how ever I do not understand
> Scala, I am having trouble debugging the following exception.
> Any help would be greatly appreciated.
> Happy New Year
> Andy
> java.lang.IllegalArgumentException: requirement failed: Param null__inputCol
> does not belong to Stemmer_2f3aa96d-7919-4eaa-ad54-f7c620b92d1c.
> at scala.Predef$.require(Predef.scala:233)
> at org.apache.spark.ml.param.Params$class.shouldOwn(params.scala:557)
> at org.apache.spark.ml.param.Params$class.set(params.scala:436)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at org.apache.spark.ml.param.Params$class.set(params.scala:422)
> at org.apache.spark.ml.PipelineStage.set(Pipeline.scala:37)
> at
> org.apache.spark.ml.UnaryTransformer.setInputCol(Transformer.scala:83)
> at com.pws.xxx.ml.StemmerTest.test(StemmerTest.java:30)
> public class StemmerTest extends AbstractSparkTest {
> @Test
> public void test() {
> Stemmer stemmer = new Stemmer()
> .setInputCol("raw”) //line 30
> .setOutputCol("filtered");
> }
> }
> /**
> * @ see
> spark-1.5.1/mllib/src/main/scala/org/apache/spark/ml/feature/Tokenizer.scala
> * @ see
> https://chimpler.wordpress.com/2014/06/11/classifiying-documents-using-naive-bayes-on-apache-spark-mllib/
> * @ see
> http://www.tonytruong.net/movie-rating-prediction-with-apache-spark-and-hortonworks/
> *
> * @author andrewdavidson
> *
> */
> public class Stemmer extends UnaryTransformer, List,
> Stemmer> implements Serializable{
> static Logger logger = LoggerFactory.getLogger(Stemmer.class);
> private static final long serialVersionUID = 1L;
> private static final ArrayType inputType =
> DataTypes.createArrayType(DataTypes.StringType, true);
> private final String uid = Stemmer.class.getSimpleName() + "_" +
> UUID.randomUUID().toString();
> @Override
> public String uid() {
> return uid;
> }
> /*
>override protected def validateInputType(inputType: DataType): Unit = {
> require(inputType == StringType, s"Input type must be string type but got
> $inputType.")
> }
> */
> @Override
> public void validateInputType(DataType inputTypeArg) {
> String msg = "inputType must be " + inputType.simpleString() + " but
> got " + inputTypeArg.simpleString();
> assert (inputType.equals(inputTypeArg)) : msg;
> }
>
> @Override
> public Function1, List> createTransformFunc() {
> //
> http://stackoverflow.com/questions/6545066/using-scala-from-java-passing-functions-as-parameters
> Function1, List> f = new
> AbstractFunction1, List>() {
> public List apply(List words) {
> for(String word : words) {
> logger.error("AEDWIP input word: {}", word);
> }
> return words;
> }
> };
>
> return f;
> }
> @Override
> public DataType outputDataType() {
> return DataTypes.createArrayType(DataTypes.StringType, true);
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
