[
https://issues.apache.org/jira/browse/SPARK-24666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhongYu updated SPARK-24666:
----------------------------
Description:
We found that Word2Vec generate large absolute value vectors when numIterations
are large, and if numIterations are large enough (>20), the vector's value many
be *infinity(or -**infinity)***, resulting in useless vectors.
In normal situations, vectors values are mainly around -1.0~1.0 when
numIterations = 1.
The bug is shown on spark 2.0.X, 2.1.X, 2.2.X, 2.3.X, 2.4.X
There are already issues report this bug:
https://issues.apache.org/jira/browse/SPARK-5261 , but the bug fix works seems
missing.
Other people's reports:
[https://stackoverflow.com/questions/49741956/infinity-vectors-in-spark-mllib-word2vec]
[http://apache-spark-user-list.1001560.n3.nabble.com/word2vec-outputs-Infinity-Infinity-vectors-with-increasing-iterations-td29020.html]
=======================================================
Here are the code to reproduce the issue. You can download title.akas.tsv from
[https://datasets.imdbws.com/] and upload to hdfs.
{code:java}
import org.apache.spark.sql.SparkSession
import org.apache.spark.ml.feature.Word2Vec
case class Sentences(name: String, words: Array[String])
// IMDB raw data title.akas.tsv download from https://datasets.imdbws.com/
val dataset = spark.read
.option("header", "true").option("sep", "\t")
.option("quote", "").option("nullValue", "\\N")
.csv("/tmp/word2vec/title.akas.tsv")
.filter("region = 'US' or language = 'en'")
.select("title")
.as[String]
.map(s => Sentences(s, s.split(' ')))
.persist()
println("Training model...")
val word2Vec = new Word2Vec()
.setInputCol("words")
.setOutputCol("vector")
.setVectorSize(64)
.setWindowSize(4)
.setNumPartitions(50)
.setMinCount(5)
.setMaxIter(30)
val model = word2Vec.fit(dataset)
model.getVectors.show()
{code}
When set maxIter to 30, you will get the result
{code:java}
scala> model.getVectors.show()
+-------------+--------------------+
| word| vector|
+-------------+--------------------+
| Unspoken|[-Infinity,-Infin...|
| Talent|[Infinity,-Infini...|
| Hourglass|[1.09657520526310...|
|Nickelodeon's|[2.20436549446219...|
| Priests|[-1.9625896848389...|
| Religion:|[-3.8815759928213...|
| Bu|[-7.9722236466752...|
| Totoro:|[-4.1829056206528...|
| Trouble,|[2.51985378203136...|
| Hatter|[8.49108115961009...|
| '79|[-5.4560309784650...|
| Vile|[-1.2059769646379...|
| 9/11|[Infinity,-Infini...|
| Santino|[6.30405421282099...|
| Motives|[1.96207712570869...|
| '13|[-1.7641987324084...|
| Fierce|[-Infinity,Infini...|
| Stover|[5.10057474120744...|
| 'It|[1.08629989605664...|
| Butts|[Infinity,Infinit...|
+-------------+--------------------+
only showing top 20 rows
{code}
was:
We found that Word2Vec generate large absolute value vectors when numIterations
are large, and if numIterations are large enough (>20), the vector's value many
be *infinity(or -**infinity)***, resulting in useless vectors.
In normal situations, vectors values are mainly around -1.0~1.0 when
numIterations = 1.
The bug is shown on spark 2.0.X, 2.1.X, 2.2.X, 2.3.X.
There are already issues report this bug:
https://issues.apache.org/jira/browse/SPARK-5261 , but the bug fix works seems
missing.
Other people's reports:
[https://stackoverflow.com/questions/49741956/infinity-vectors-in-spark-mllib-word2vec]
[http://apache-spark-user-list.1001560.n3.nabble.com/word2vec-outputs-Infinity-Infinity-vectors-with-increasing-iterations-td29020.html]
> Word2Vec generate infinity vectors when numIterations are large
> ---------------------------------------------------------------
>
> Key: SPARK-24666
> URL: https://issues.apache.org/jira/browse/SPARK-24666
> Project: Spark
> Issue Type: Bug
> Components: ML, MLlib
> Affects Versions: 2.3.1, 2.4.4
> Environment: 2.0.X, 2.1.X, 2.2.X, 2.3.X, 2.4.X
> Reporter: ZhongYu
> Priority: Critical
>
> We found that Word2Vec generate large absolute value vectors when
> numIterations are large, and if numIterations are large enough (>20), the
> vector's value many be *infinity(or -**infinity)***, resulting in useless
> vectors.
> In normal situations, vectors values are mainly around -1.0~1.0 when
> numIterations = 1.
> The bug is shown on spark 2.0.X, 2.1.X, 2.2.X, 2.3.X, 2.4.X
> There are already issues report this bug:
> https://issues.apache.org/jira/browse/SPARK-5261 , but the bug fix works
> seems missing.
> Other people's reports:
> [https://stackoverflow.com/questions/49741956/infinity-vectors-in-spark-mllib-word2vec]
> [http://apache-spark-user-list.1001560.n3.nabble.com/word2vec-outputs-Infinity-Infinity-vectors-with-increasing-iterations-td29020.html]
> =======================================================
> Here are the code to reproduce the issue. You can download title.akas.tsv
> from [https://datasets.imdbws.com/] and upload to hdfs.
>
> {code:java}
> import org.apache.spark.sql.SparkSession
> import org.apache.spark.ml.feature.Word2Vec
> case class Sentences(name: String, words: Array[String])
> // IMDB raw data title.akas.tsv download from https://datasets.imdbws.com/
> val dataset = spark.read
> .option("header", "true").option("sep", "\t")
> .option("quote", "").option("nullValue", "\\N")
> .csv("/tmp/word2vec/title.akas.tsv")
> .filter("region = 'US' or language = 'en'")
> .select("title")
> .as[String]
> .map(s => Sentences(s, s.split(' ')))
> .persist()
> println("Training model...")
> val word2Vec = new Word2Vec()
> .setInputCol("words")
> .setOutputCol("vector")
> .setVectorSize(64)
> .setWindowSize(4)
> .setNumPartitions(50)
> .setMinCount(5)
> .setMaxIter(30)
> val model = word2Vec.fit(dataset)
> model.getVectors.show()
> {code}
> When set maxIter to 30, you will get the result
> {code:java}
> scala> model.getVectors.show()
> +-------------+--------------------+
> | word| vector|
> +-------------+--------------------+
> | Unspoken|[-Infinity,-Infin...|
> | Talent|[Infinity,-Infini...|
> | Hourglass|[1.09657520526310...|
> |Nickelodeon's|[2.20436549446219...|
> | Priests|[-1.9625896848389...|
> | Religion:|[-3.8815759928213...|
> | Bu|[-7.9722236466752...|
> | Totoro:|[-4.1829056206528...|
> | Trouble,|[2.51985378203136...|
> | Hatter|[8.49108115961009...|
> | '79|[-5.4560309784650...|
> | Vile|[-1.2059769646379...|
> | 9/11|[Infinity,-Infini...|
> | Santino|[6.30405421282099...|
> | Motives|[1.96207712570869...|
> | '13|[-1.7641987324084...|
> | Fierce|[-Infinity,Infini...|
> | Stover|[5.10057474120744...|
> | 'It|[1.08629989605664...|
> | Butts|[Infinity,Infinit...|
> +-------------+--------------------+
> only showing top 20 rows
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
