sahnib commented on code in PR #45038:
URL: https://github.com/apache/spark/pull/45038#discussion_r1480416349
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##########
@@ -287,7 +332,13 @@ private[sql] class RocksDBStateStoreProvider
useColumnFamilies)
}
- @volatile private var encoder: RocksDBStateEncoder = _
+ private val encoderMapLock = new Object
Review Comment:
[nit] rename this to `columnFamilyEncoderMapLock` as it only applies to
column families. Encoders for default column family are picked at State store
init.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##########
@@ -287,7 +332,13 @@ private[sql] class RocksDBStateStoreProvider
useColumnFamilies)
}
- @volatile private var encoder: RocksDBStateEncoder = _
+ private val encoderMapLock = new Object
+
+ @GuardedBy("encoderMapLock")
+ @volatile private var keyEncoderMap = new mutable.HashMap[String,
RocksDBKeyStateEncoder]
+
+ @GuardedBy("encoderMapLock")
+ @volatile private var valueEncoderMap = new mutable.HashMap[String,
RocksDBValueStateEncoder]
Review Comment:
Can we just use a concurrentHashMap with signature `Map[String,
(RocksDBKeyStateEncoder, RocksDBValueStateEncoder)]`. Then, we don't need to
synchronize using a lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
