HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values
which it shouldn't. (especially the case of rolling back)
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values (as
mutable Set instances) which it shouldn't. (especially the case of rolling back)
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
