HeartSaVioR commented on a change in pull request #28975:
URL: https://github.com/apache/spark/pull/28975#discussion_r449297581
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala
##########
@@ -451,10 +451,25 @@ class SymmetricHashJoinStateManager(
}
private trait KeyWithIndexToValueRowConverter {
+ /** Defines the schema of the value row (the value side of K-V in state
store). */
def valueAttributes: Seq[Attribute]
+ /**
+ * Convert the value row to (actual value, match) pair.
+ *
+ * NOTE: implementations should ensure the result row is NOT reused during
execution, as
+ * caller may use the value to store without copy().
Review comment:
So the problem occurred here:
https://github.com/apache/spark/blob/42f01e314b4874236544cc8b94bef766269385ee/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala#L252-L281
Here we temporarily store `currentValue` via calling `convertValue` which is
candidate to remove (and return). After that we adjust the array a bit to not
having a hole in between, where it also triggers `convertValue` and "changes"
the value in `currentValue` silently if we don't copy.
It's also possible for caller to call copy() but the row to copy is inside
of ValueAndMatchPair so I feel it's pretty much easier to miss even with method
doc, so I took safer approach rather than performance-wise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
