[
https://issues.apache.org/jira/browse/FLINK-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952935#comment-15952935
]
ASF GitHub Bot commented on FLINK-2094:
---------------------------------------
Github user kalmanchapman commented on a diff in the pull request:
https://github.com/apache/flink/pull/2735#discussion_r109329636
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/nlp/Word2Vec.scala
---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.nlp
+
+import org.apache.flink.api.scala._
+import org.apache.flink.ml.common.{Parameter, ParameterMap}
+import org.apache.flink.ml.optimization.{Context, ContextEmbedder,
HSMWeightMatrix}
+import org.apache.flink.ml.pipeline.{FitOperation,
TransformDataSetOperation, Transformer}
+
+/**
+ * Implements Word2Vec as a transformer on a DataSet[Iterable[String]]
+ *
+ * Calculates valuable vectorizations of individual words given
+ * the context in which they appear
+ *
+ * @example
+ * {{{
+ * //constructed of 'sentences' - where each string in the iterable is
a word
+ * val stringsDS = DataSet[Iterable[String]] = ...
+ * val stringsDS2 = DataSet[Iterable[String]] = ...
+ *
+ * val w2V = Word2Vec()
+ * .setIterations(5)
+ * .setTargetCount(10)
+ * .setSeed(500)
+ *
+ * //internalizes an initial weightSet
+ * w2V.fit(stringsDS)
+ *
+ * //note that the same DS can be used to fit and optimize
+ * //the number of learned vectors is limted to the vocab built in fit
+ * val wordVectors : DataSet[(String, Vector[Double])] =
w2V.optimize(stringsDS2)
+ * }}}
+ *
+ * =Parameters=
+ *
+ * - [[org.apache.flink.ml.nlp.Word2Vec.WindowSize]]
+ * sets the size of window for skipGram formation: how far on either side
of
+ * a given word will we sample the context? (Default value: '''10''')
+ *
+ * - [[org.apache.flink.ml.nlp.Word2Vec.Iterations]]
+ * sets the number of global iterations the training set is passed
through - essentially looping on
+ * whole set, leveraging flink's iteration operator (Default value:
'''10''')
+ *
+ * - [[org.apache.flink.ml.nlp.Word2Vec.TargetCount]]
--- End diff --
hi @kateri1 - I'll be updating this to use a self-describing name
> Implement Word2Vec
> ------------------
>
> Key: FLINK-2094
> URL: https://issues.apache.org/jira/browse/FLINK-2094
> Project: Flink
> Issue Type: Improvement
> Components: Machine Learning Library
> Reporter: Nikolaas Steenbergen
> Assignee: Nikolaas Steenbergen
> Priority: Minor
> Labels: ML
>
> implement Word2Vec
> http://arxiv.org/pdf/1402.3722v1.pdf
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
