[jira] (FLINK-2094) Implement Word2Vec

[ASF GitHub Bot (JIRA)]

Title: Message Title

ASF GitHub Bot commented on FLINK-2094   Re: Implement Word2Vec Github user kateri1 commented on a diff in the pull request: https://github.com/apache/flink/pull/2735#discussion_r98613439 — Diff: flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/nlp/Word2Vec.scala — @@ -0,0 +1,243 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.nlp + +import org.apache.flink.api.scala._ +import org.apache.flink.ml.common. {Parameter, ParameterMap} +import org.apache.flink.ml.optimization. {Context, ContextEmbedder, HSMWeightMatrix} +import org.apache.flink.ml.pipeline. {FitOperation, TransformDataSetOperation, Transformer} + +/** + * Implements Word2Vec as a transformer on a DataSet[Iterable[String]] + * + * Calculates valuable vectorizations of individual words given + * the context in which they appear + * + * @example + * {{ { + * //constructed of 'sentences' - where each string in the iterable is a word + * val stringsDS = DataSet[Iterable[String]] = ... + * val stringsDS2 = DataSet[Iterable[String]] = ... + * + * val w2V = Word2Vec() + * .setIterations(5) + * .setTargetCount(10) + * .setSeed(500) + * + * //internalizes an initial weightSet + * w2V.fit(stringsDS) + * + * //note that the same DS can be used to fit and optimize + * //the number of learned vectors is limted to the vocab built in fit + * val wordVectors : DataSet[(String, Vector[Double])] = w2V.optimize(stringsDS2) + * } }} + * + * =Parameters= + * + * - [[org.apache.flink.ml.nlp.Word2Vec.WindowSize]] + * sets the size of window for skipGram formation: how far on either side of + * a given word will we sample the context? (Default value: '''10''') + * + * - [[org.apache.flink.ml.nlp.Word2Vec.Iterations]] + * sets the number of global iterations the training set is passed through - essentially looping on + * whole set, leveraging flink's iteration operator (Default value: '''10''') + * + * - [[org.apache.flink.ml.nlp.Word2Vec.TargetCount]] — End diff – the name of this fieald is not selfdescribing, please consider something like: MinWordFrequency instead of TargetCount, target count is misleading and corresponding get/set methods Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)