[
https://issues.apache.org/jira/browse/FLINK-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694886#comment-14694886
]
ASF GitHub Bot commented on FLINK-1901:
---------------------------------------
Github user thvasilo commented on a diff in the pull request:
https://github.com/apache/flink/pull/949#discussion_r36950634
--- Diff:
flink-scala/src/main/scala/org/apache/flink/api/scala/DataSet.scala ---
@@ -1182,6 +1184,60 @@ class DataSet[T: ClassTag](set: JavaDataSet[T]) {
getCallLocationName()))
//
--------------------------------------------------------------------------------------------
+ // Sample
+ //
--------------------------------------------------------------------------------------------
+ /**
+ * Generate a sample of DataSet by the probability fraction of each
element.
+ *
+ * @param withReplacement Whether element can be selected more than once.
+ * @param fraction Probability that each element is chosen,
should be [0,1] without
+ * replacement, and [0, ∞) with replacement.
While fraction is larger
+ * than 1, the elements are expected to be
selected multi times into
+ * sample on average.
+ * @param seed Random number generator seed.
+ * @return The sampled DataSet
+ */
+ def sample(
+ withReplacement: Boolean,
+ fraction: Double,
+ seed: Long = Utils.RNG.nextLong()): DataSet[T] = {
+
+ wrap(new MapPartitionOperator[T, T](javaSet,
+ getType(),
+ new SampleWithFraction(withReplacement, fraction, seed),
+ getCallLocationName()))
+ }
+
+ /**
+ * Generate a sample of DataSet by the probability fraction of each
element.
--- End diff --
Javadoc is from the fraction function.
> Create sample operator for Dataset
> ----------------------------------
>
> Key: FLINK-1901
> URL: https://issues.apache.org/jira/browse/FLINK-1901
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Reporter: Theodore Vasiloudis
> Assignee: Chengxiang Li
>
> In order to be able to implement Stochastic Gradient Descent and a number of
> other machine learning algorithms we need to have a way to take a random
> sample from a Dataset.
> We need to be able to sample with or without replacement from the Dataset,
> choose the relative or exact size of the sample, set a seed for
> reproducibility, and support sampling within iterations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
