[
https://issues.apache.org/jira/browse/SPARK-58177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Ball updated SPARK-58177:
---------------------------------
Description:
When SVD++ combines two updates for the same node, it is supposed to add them
together. For one of the values, the code started from the wrong update, so
that value was never actually added up: it doubled one side and threw the other
away. Any node with two or more edges ended up with the wrong value.
The matching code just below already does it the right way (adds the two
updates), which shows this was a typo.
The fix is a single line: start from the first update instead of the second.
This is the same kind of one-line correctness fix as SPARK-6710.
was:
The Phase-2 aggregateMessages combiner in SVDPlusPlus.run merges two
(Array[Double], Array[Double], Double) messages destined for the same vertex. It
sums the _1 and _3 elements correctly but seeds the _2 accumulator from the
wrong
operand:
val out2 = g2._2.clone()
BLAS.nativeBLAS.daxpy(out2.length, 1.0, g2._2, 1, out2, 1)
Because daxpy computes y = x + y, this yields 2 * g2._2 and discards g1._2
entirely. Since aggregateMessages folds all messages arriving at a vertex
through
this function, the _2 term (the implicit-feedback / updateY accumulator)
collapses
to 2 * lastMessage for any vertex with two or more incident edges. The reduce
function is also non-commutative, so the result depends on the (undefined)
message
order.
The correct sibling merge in the same method (the outerJoinVertices step just
below) confirms the intended semantics: out2 = g1._2 + g2._2.
Fix: seed out2 from g1._2 instead of g2._2 (a one-character change).
> Fix wrong operand in SVD++ aggregateMessages combiner
> -----------------------------------------------------
>
> Key: SPARK-58177
> URL: https://issues.apache.org/jira/browse/SPARK-58177
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Affects Versions: 5.0.0
> Reporter: Matthew Ball
> Priority: Major
>
> When SVD++ combines two updates for the same node, it is supposed to add them
> together. For one of the values, the code started from the wrong update, so
> that value was never actually added up: it doubled one side and threw the
> other away. Any node with two or more edges ended up with the wrong value.
> The matching code just below already does it the right way (adds the two
> updates), which shows this was a typo.
> The fix is a single line: start from the first update instead of the second.
> This is the same kind of one-line correctness fix as SPARK-6710.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
