mridulm commented on code in PR #49479:
URL: https://github.com/apache/spark/pull/49479#discussion_r1917566034
##########
common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBSuite.java:
##########
@@ -422,6 +423,37 @@ public void testResourceCleaner() throws Exception {
}
}
+ @Test
+ public void testMultipleTypesWriteAll() throws Exception {
+
+ List<CustomType1> type1List = Arrays.asList(
+ createCustomType1(1),
+ createCustomType1(2),
+ createCustomType1(3),
+ createCustomType1(4)
+ );
+
+ List<CustomType2> type2List = Arrays.asList(
+ createCustomType2(10),
+ createCustomType2(11),
+ createCustomType2(12),
+ createCustomType2(13)
+ );
+
+ List fullList = new ArrayList();
+ fullList.addAll(type1List);
+ fullList.addAll(type2List);
+
+ db.writeAll(fullList);
+ for (CustomType1 value : type1List) {
Review Comment:
In this test, I purposely ensure both `CustomType1` and `CustomType2` are
written together in a call to `writeAll` - and subsequently try to read them -
which ends up with failing when deserializing `CustomType2` (as `CustomType1`
was written to those records).
Without the fix, it results in deserialization error during read.
For LevelDBSuite, we see this for example - since it is writing
`CustomType1` instead of `CustomType2` as value:
```
[ERROR] org.apache.spark.util.kvstore.LevelDBSuite.testMultipleTypesWriteAll
-- Time elapsed: 0.024 s <<< ERROR!
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
Unrecognized field "name" (class org.apache.spark.util.kvstore.CustomType2),
not marked as ignorable (3 known properties: "id", "parentId", "key"])
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION`
disabled); line: 1, column: 34] (through reference chain:
org.apache.spark.util.kvstore.CustomType2["name"])
at
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
at
com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1153)
at
com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2241)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1821)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1799)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at
com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4917)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3897)
at
org.apache.spark.util.kvstore.KVStoreSerializer.deserialize(KVStoreSerializer.java:70)
at org.apache.spark.util.kvstore.LevelDB.get(LevelDB.java:136)
at org.apache.spark.util.kvstore.LevelDB.read(LevelDB.java:148)
at
org.apache.spark.util.kvstore.LevelDBSuite.testMultipleTypesWriteAll(LevelDBSuite.java:452)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
