[
https://issues.apache.org/jira/browse/FLINK-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046608#comment-15046608
]
ASF GitHub Bot commented on FLINK-2379:
---------------------------------------
Github user chiwanpark commented on the pull request:
https://github.com/apache/flink/pull/1032#issuecomment-162821875
How about base class with declaration of the both statistics values and
child class with the implementation of that values?
```scala
abstract class FieldStats {
// statistics values for discrete fields
def entropy: Double = throw NonImplementedError("entropy cannot be
accessed for continuous fields")
def gini: Double = throw NonImplementedError("gini cannot be accessed for
continuous fields")
def categoryCounts: Map[Double, Int] = throw
NonImplementedError("categoryCounts cannot be accessed for continuous fields")
// statistics values for continuous fields
def max: Double = throw NonImplementedError("max cannot be accessed for
discrete fields")
def min: Double = throw NonImplementedError("min cannot be accessed for
discrete fields")
def mean: Double = throw NonImplementedError("mean cannot be accessed for
discrete fields")
def variance: Double = throw NonImplementedError("variance cannot be
accessed for discrete fields")
}
class DiscreteFieldStats(
private val counts: Map[Double, Int]
) extends FieldStats {
override lazy val entropy = // calculation of entropy
override lazy val gini = // calculation of gini
override lazy val categoryCounts = // calculation of categoryCounts
override def toString = // implementation of toString
}
class ContinuousFieldStats(
override val max: Double,
override val min: Double,
override val mean: Double,
override val variance: Double
) extends FieldStats {
override def toString = // implementation of toString
}
```
If a user calls some non-implemented methods in the derived classes,
`scala.NotImplementedError` will be raised. Additionally, we can calculate all
values (including gini and entropy) once in constructor with this approach.
> Add methods to evaluate field wise statistics over DataSet of vectors.
> ----------------------------------------------------------------------
>
> Key: FLINK-2379
> URL: https://issues.apache.org/jira/browse/FLINK-2379
> Project: Flink
> Issue Type: New Feature
> Components: Machine Learning Library
> Reporter: Sachin Goel
> Assignee: Sachin Goel
>
> Design methods to evaluate statistics over dataset of vectors.
> For continuous fields, Minimum, maximum, mean, variance.
> For discrete fields, Class counts, Entropy, Gini Impurity.
> Further statistical measures can also be supported. For example, correlation
> between two series, computing the covariance matrix, etc.
> [These are currently the things Spark supports.]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
