[
https://issues.apache.org/jira/browse/SPARK-26158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-26158.
-------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23126
[https://github.com/apache/spark/pull/23126]
> Enhance the accuracy of covariance in RowMatrix for DenseVector
> ---------------------------------------------------------------
>
> Key: SPARK-26158
> URL: https://issues.apache.org/jira/browse/SPARK-26158
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.4.0
> Reporter: Liang Li
> Assignee: Liang Li
> Priority: Minor
> Fix For: 3.0.0
>
>
> Compare Spark computeCovariance function in RowMatrix for DenseVector and
> Numpy's function cov,
> *Find two problem, below is the result:*
> *1)The Spark function computeCovariance in RowMatrix is not accuracy*
> input data
> 1.0,2.0,3.0,4.0,5.0
> 2.0,3.0,1.0,2.0,6.0
> Numpy function cov result:
> [[2.5 1.75]
> [ 1.75 3.7 ]]
> RowMatrix function computeCovariance result:
> 2.5 1.75
> 1.75 3.700000000000001
>
> 2)For some input case, the result is not good
> generate input data by below logic
> data1 = np.random.normal(loc=100000, scale=0.000009, size=10000000)
> data2 = np.random.normal(loc=200000, scale=0.000002,size=10000000)
>
> Numpy function cov result:
> [[ 8.10536442e-11 -4.35439574e-15]
> [ -4.35439574e-15 3.99928264e-12]]
>
> RowMatrix function computeCovariance result:
> -0.0027484893798828125 0.001491546630859375
> 0.001491546630859375 8.087158203125E-4
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
