This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory.git
The following commit(s) were added to refs/heads/main by this push:
new ffdca675b refactor(java/python): refine collection header bitmap
(#2642)
ffdca675b is described below
commit ffdca675bf3968f312e317a4a7b7e9af2560c17b
Author: Shawn Yang <[email protected]>
AuthorDate: Tue Sep 23 10:27:59 2025 +0800
refactor(java/python): refine collection header bitmap (#2642)
## Why?
Refine collection header bitmap to make it more understandable.
## What does this PR do?
<!-- Describe the details of this PR. -->
## Related issues
Closes #
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fory/issues/new/choose) describing the
need to do so and update the document if necessary.
Delete section if not applicable.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
Delete section if not applicable.
-->
---
ci/run_ci.py | 3 +-
docs/specification/java_serialization_spec.md | 8 +--
docs/specification/xlang_serialization_spec.md | 10 ++--
.../fory/builder/BaseObjectCodecBuilder.java | 16 ++---
.../serializer/collection/CollectionFlags.java | 30 +++++-----
.../collection/CollectionLikeSerializer.java | 45 +++++++-------
.../fory/serializer/collection/MapFlags.java | 4 +-
.../test/java/org/apache/fory/RustXlangTest.java | 1 +
python/pyfory/_serialization.pyx | 70 ++++++++++++----------
python/pyfory/_serializer.py | 52 +++++++++-------
python/pyfory/tests/test_cross_language.py | 2 +-
11 files changed, 127 insertions(+), 114 deletions(-)
diff --git a/ci/run_ci.py b/ci/run_ci.py
index ed1a3f59b..cdd8f7fbc 100644
--- a/ci/run_ci.py
+++ b/ci/run_ci.py
@@ -293,7 +293,8 @@ def parse_args():
if USE_PYTHON_GO:
func()
else:
- run_shell_script("go")
+ # run_shell_script("go")
+ pass
elif command == "format":
if USE_PYTHON_FORMAT:
func()
diff --git a/docs/specification/java_serialization_spec.md
b/docs/specification/java_serialization_spec.md
index 283bc3078..dd2f57cce 100644
--- a/docs/specification/java_serialization_spec.md
+++ b/docs/specification/java_serialization_spec.md
@@ -392,8 +392,8 @@ which will be encoded by elements header, each use one bit:
- If track elements ref, use the first bit `0b1` of the header to flag it.
- If the collection has null, use the second bit `0b10` of the header to flag
it. If ref tracking is enabled for this
element type, this flag is invalid.
-- If the collection element types are not declared type, use the 3rd bit
`0b100` of the header to flag it.
-- If the collection element types are different, use the 4th bit `0b1000`
header to flag it.
+- If the collection element types are the declared type, use the 3rd bit
`0b100` of the header to flag it.
+- If the collection element types are same, use the 4th bit `0b1000` header to
flag it.
By default, all bits are unset, which means all elements won't track ref, all
elements are same type, not null and
the actual element is the declared type in the custom class field.
@@ -459,11 +459,11 @@ KV header:
- If track key ref, use the first bit `0b1` of the header to flag it.
- If the key has null, use the second bit `0b10` of the header to flag it. If
ref tracking is enabled for this
key type, this flag is invalid.
-- If the actual key type of map is not the declared key type, use the 3rd bit
`0b100` of the header to flag it.
+- If the actual key type of map is the declared key type, use the 3rd bit
`0b100` of the header to flag it.
- If track value ref, use the 4th bit `0b1000` of the header to flag it.
- If the value has null, use the 5th bit `0b10000` of the header to flag it.
If ref tracking is enabled for this
value type, this flag is invalid.
-- If the value type of map is not the declared value type, use the 6rd bit
`0b100000` of the header to flag it.
+- If the value type of map is the declared value type, use the 6rd bit
`0b100000` of the header to flag it.
- If key or value is null, that key and value will be written as a separate
chunk, and chunk size writing will be
skipped too.
diff --git a/docs/specification/xlang_serialization_spec.md
b/docs/specification/xlang_serialization_spec.md
index 83719872c..d44590c37 100644
--- a/docs/specification/xlang_serialization_spec.md
+++ b/docs/specification/xlang_serialization_spec.md
@@ -597,7 +597,7 @@ Which encoding to choose:
- If the string is encoded by `utf-8`, then fory will use `utf-8` to decode
the data. Cross-language string
serialization of fory uses `utf-8` by default.
-### list
+### collection/list
Format:
@@ -614,8 +614,8 @@ which will be encoded by elements header, each use one bit:
- If track elements ref, use the first bit `0b1` of the header to flag it.
- If the elements have null, use the second bit `0b10` of the header to flag
it. If ref tracking is enabled for this
element type, this flag is invalid.
-- If the element types are not the declared type, use the 3rd bit `0b100` of
the header to flag it.
-- If the element types are different, use the 4th bit `0b1000` header to flag
it.
+- If the element types are the declared type, use the 3rd bit `0b100` of the
header to flag it.
+- If the element types are smae, use the 4th bit `0b1000` header to flag it.
By default, all bits are unset, which means all elements won't track ref, all
elements are same type, not null and
the actual element is the declared type in the custom type field.
@@ -718,11 +718,11 @@ KV header:
- If track key ref, use the first bit `0b1` of the header to flag it.
- If the key has null, use the second bit `0b10` of the header to flag it. If
ref tracking is enabled for this
key type, this flag is invalid.
-- If the actual key type of map is not the declared key type, use the 3rd bit
`0b100` of the header to flag it.
+- If the actual key type of map is the declared key type, use the 3rd bit
`0b100` of the header to flag it.
- If track value ref, use the 4th bit `0b1000` of the header to flag it.
- If the value has null, use the 5th bit `0b10000` of the header to flag it.
If ref tracking is enabled for this
value type, this flag is invalid.
-- If the value type of map is not the declared value type, use the 6rd bit
`0b100000` of the header to flag it.
+- If the value type of map is the declared value type, use the 6rd bit
`0b100000` of the header to flag it.
- If key or value is null, that key and value will be written as a separate
chunk, and chunk size writing will be
skipped too.
diff --git
a/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java
b/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java
index 118297138..992dfee28 100644
---
a/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java
+++
b/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java
@@ -911,12 +911,12 @@ public abstract class BaseObjectCodecBuilder extends
CodecBuilder {
writeContainerElements(elementType, false, null, hasNull, buffer,
collection, size));
}
} else {
- Literal flag = ofInt(CollectionFlags.NOT_SAME_TYPE);
- Expression sameElementClass = neq(new BitAnd(flags, flag), flag,
"sameElementClass");
+ Literal flag = ofInt(CollectionFlags.IS_SAME_TYPE);
+ Expression sameElementClass = eq(new BitAnd(flags, flag), flag,
"sameElementClass");
builder.add(sameElementClass);
// if ((flags & Flags.NOT_DECL_ELEMENT_TYPE) ==
Flags.NOT_DECL_ELEMENT_TYPE)
- Literal notDeclTypeFlag = ofInt(CollectionFlags.NOT_DECL_ELEMENT_TYPE);
- Expression isDeclType = neq(new BitAnd(flags, notDeclTypeFlag),
notDeclTypeFlag);
+ Literal isDeclTypeFlag = ofInt(CollectionFlags.IS_DECL_ELEMENT_TYPE);
+ Expression isDeclType = eq(new BitAnd(flags, isDeclTypeFlag),
isDeclTypeFlag);
Expression elemSerializer; // make it in scope of `if(sameElementClass)`
boolean maybeDecl = typeResolver(r -> r.isSerializable(elemClass));
TypeRef<?> serializerType = getSerializerType(elementType);
@@ -1764,12 +1764,12 @@ public abstract class BaseObjectCodecBuilder extends
CodecBuilder {
readContainerElements(elementType, false, null, hasNull, buffer,
collection, size));
}
} else {
- Literal notSameTypeFlag = ofInt(CollectionFlags.NOT_SAME_TYPE);
+ Literal isSameTypeFlag = ofInt(CollectionFlags.IS_SAME_TYPE);
Expression sameElementClass =
- neq(new BitAnd(flags, notSameTypeFlag), notSameTypeFlag,
"sameElementClass");
+ eq(new BitAnd(flags, isSameTypeFlag), isSameTypeFlag,
"sameElementClass");
// if ((flags & Flags.NOT_DECL_ELEMENT_TYPE) ==
Flags.NOT_DECL_ELEMENT_TYPE)
- Literal notDeclTypeFlag = ofInt(CollectionFlags.NOT_DECL_ELEMENT_TYPE);
- Expression isDeclType = neq(new BitAnd(flags, notDeclTypeFlag),
notDeclTypeFlag);
+ Literal isDeclTypeFlag = ofInt(CollectionFlags.IS_DECL_ELEMENT_TYPE);
+ Expression isDeclType = eq(new BitAnd(flags, isDeclTypeFlag),
isDeclTypeFlag);
Invoke serializer =
inlineInvoke(readClassInfo(elemClass, buffer), "getSerializer",
SERIALIZER_TYPE);
TypeRef<?> serializerType = getSerializerType(elementType);
diff --git
a/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionFlags.java
b/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionFlags.java
index 6bfdc35fd..0d37b2de0 100644
---
a/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionFlags.java
+++
b/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionFlags.java
@@ -19,26 +19,26 @@
package org.apache.fory.serializer.collection;
-/**
- * Default unset bitmap flags.
- *
- * <ul>
- * <li>TRACKING_REF: false
- * <li>HAS_NULL: false
- * <li>NOT_DECL_ELEMENT_TYPE: false
- * <li>NOT_SAME_TYPE: false
- * </ul>
- */
+/** bitmap flags for collection serialization. */
public class CollectionFlags {
/** Whether track elements ref. */
public static int TRACKING_REF = 0b1;
- /** Whether collection has null. */
+ /** Whether collection has null elements. */
public static int HAS_NULL = 0b10;
- /** Whether collection elements type is not declare type. */
- public static int NOT_DECL_ELEMENT_TYPE = 0b100;
+ /** Whether collection elements type is declared type. */
+ public static int IS_DECL_ELEMENT_TYPE = 0b100;
+
+ /** Whether collection elements type are same. */
+ public static int IS_SAME_TYPE = 0b1000;
+
+ public static int DECL_SAME_TYPE_TRACKING_REF =
+ IS_DECL_ELEMENT_TYPE | IS_SAME_TYPE | TRACKING_REF;
+
+ public static int DECL_SAME_TYPE_NOT_TRACKING_REF = IS_DECL_ELEMENT_TYPE |
IS_SAME_TYPE;
+
+ public static int DECL_SAME_TYPE_HAS_NULL = IS_DECL_ELEMENT_TYPE |
IS_SAME_TYPE | HAS_NULL;
- /** Whether collection elements type different. */
- public static int NOT_SAME_TYPE = 0b1000;
+ public static int DECL_SAME_TYPE_NOT_HAS_NULL = IS_DECL_ELEMENT_TYPE |
IS_SAME_TYPE;
}
diff --git
a/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionLikeSerializer.java
b/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionLikeSerializer.java
index 1d845b652..c03028ac8 100644
---
a/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionLikeSerializer.java
+++
b/java/fory-core/src/main/java/org/apache/fory/serializer/collection/CollectionLikeSerializer.java
@@ -135,8 +135,8 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
boolean trackingRef = elemGenericType.trackingRef(typeResolver);
if (elemGenericType.isMonomorphic()) {
if (trackingRef) {
- buffer.writeByte(CollectionFlags.TRACKING_REF);
- return CollectionFlags.TRACKING_REF;
+ buffer.writeByte(CollectionFlags.DECL_SAME_TYPE_TRACKING_REF);
+ return CollectionFlags.DECL_SAME_TYPE_TRACKING_REF;
} else {
return writeNullabilityHeader(buffer, value);
}
@@ -151,8 +151,8 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
} else {
if (elemSerializer != null) {
if (elemSerializer.needToWriteRef()) {
- buffer.writeByte(CollectionFlags.TRACKING_REF);
- return CollectionFlags.TRACKING_REF;
+ buffer.writeByte(CollectionFlags.DECL_SAME_TYPE_TRACKING_REF);
+ return CollectionFlags.DECL_SAME_TYPE_TRACKING_REF;
} else {
return writeNullabilityHeader(buffer, value);
}
@@ -171,15 +171,18 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
public int writeNullabilityHeader(MemoryBuffer buffer, Collection value) {
for (Object elem : value) {
if (elem == null) {
- buffer.writeByte(CollectionFlags.HAS_NULL);
- return CollectionFlags.HAS_NULL;
+ buffer.writeByte(CollectionFlags.DECL_SAME_TYPE_HAS_NULL);
+ return CollectionFlags.DECL_SAME_TYPE_HAS_NULL;
}
}
- buffer.writeByte(0);
- return 0;
+ buffer.writeByte(CollectionFlags.DECL_SAME_TYPE_NOT_HAS_NULL);
+ return CollectionFlags.DECL_SAME_TYPE_NOT_HAS_NULL;
}
- /** Need to track elements ref, can't check elements nullability. */
+ /**
+ * Need to track elements ref, declared element type is not morphic, can't
check elements
+ * nullability.
+ */
@CodegenInvoke
public int writeTypeHeader(
MemoryBuffer buffer, Collection value, Class<?> declareElementType,
ClassInfoHolder cache) {
@@ -199,17 +202,17 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
}
}
if (hasDifferentClass) {
- bitmap |= CollectionFlags.NOT_SAME_TYPE |
CollectionFlags.NOT_DECL_ELEMENT_TYPE;
buffer.writeByte(bitmap);
} else {
if (elemClass == null) {
elemClass = void.class;
}
+ bitmap |= CollectionFlags.IS_SAME_TYPE;
// Write class in case peer doesn't have this class.
if (!fory.getConfig().isMetaShareEnabled() && elemClass ==
declareElementType) {
+ bitmap |= CollectionFlags.IS_DECL_ELEMENT_TYPE;
buffer.writeByte(bitmap);
} else {
- bitmap |= CollectionFlags.NOT_DECL_ELEMENT_TYPE;
buffer.writeByte(bitmap);
// Update classinfo, the caller will use it.
TypeResolver typeResolver = this.typeResolver;
@@ -222,7 +225,7 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
/** Maybe track elements ref, or write elements nullability. */
@CodegenInvoke
public int writeTypeHeader(MemoryBuffer buffer, Collection value,
ClassInfoHolder cache) {
- int bitmap = CollectionFlags.NOT_DECL_ELEMENT_TYPE;
+ int bitmap = 0;
boolean hasDifferentClass = false;
Class<?> elemClass = null;
boolean containsNull = false;
@@ -241,7 +244,7 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
bitmap |= CollectionFlags.HAS_NULL;
}
if (hasDifferentClass) {
- bitmap |= CollectionFlags.NOT_SAME_TYPE | CollectionFlags.TRACKING_REF;
+ bitmap |= CollectionFlags.TRACKING_REF;
buffer.writeByte(bitmap);
} else {
TypeResolver typeResolver = this.typeResolver;
@@ -250,6 +253,7 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
if (elemClass == null) {
elemClass = void.class;
}
+ bitmap |= CollectionFlags.IS_SAME_TYPE;
ClassInfo classInfo = typeResolver.getClassInfo(elemClass, cache);
if (classInfo.getSerializer().needToWriteRef()) {
bitmap |= CollectionFlags.TRACKING_REF;
@@ -286,8 +290,6 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
bitmap |= CollectionFlags.HAS_NULL;
}
if (hasDifferentClass) {
- // If collection contains null only, the type header will be meaningless
- bitmap |= CollectionFlags.NOT_SAME_TYPE |
CollectionFlags.NOT_DECL_ELEMENT_TYPE;
buffer.writeByte(bitmap);
} else {
// When serialize a collection with all elements null directly, the
declare type
@@ -295,11 +297,12 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
if (elemClass == null) {
elemClass = Object.class;
}
+ bitmap |= CollectionFlags.IS_SAME_TYPE;
// Write class in case peer doesn't have this class.
if (!fory.getConfig().isMetaShareEnabled() && elemClass ==
declareElementType) {
+ bitmap |= CollectionFlags.IS_DECL_ELEMENT_TYPE;
buffer.writeByte(bitmap);
} else {
- bitmap |= CollectionFlags.NOT_DECL_ELEMENT_TYPE;
buffer.writeByte(bitmap);
TypeResolver typeResolver = this.typeResolver;
ClassInfo classInfo = typeResolver.getClassInfo(elemClass, cache);
@@ -392,10 +395,9 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
Collection collection,
GenericType elemGenericType,
int flags) {
- if ((flags & CollectionFlags.NOT_SAME_TYPE) !=
CollectionFlags.NOT_SAME_TYPE) {
+ if ((flags & CollectionFlags.IS_SAME_TYPE) ==
CollectionFlags.IS_SAME_TYPE) {
Serializer serializer;
- if ((flags & CollectionFlags.NOT_DECL_ELEMENT_TYPE)
- != CollectionFlags.NOT_DECL_ELEMENT_TYPE) {
+ if ((flags & CollectionFlags.IS_DECL_ELEMENT_TYPE) ==
CollectionFlags.IS_DECL_ELEMENT_TYPE) {
Preconditions.checkNotNull(elemGenericType);
serializer = elemGenericType.getSerializer(typeResolver);
} else {
@@ -650,11 +652,10 @@ public abstract class CollectionLikeSerializer<T> extends
Serializer<T> {
int numElements,
int flags,
GenericType elemGenericType) {
- if ((flags & CollectionFlags.NOT_SAME_TYPE) !=
CollectionFlags.NOT_SAME_TYPE) {
+ if ((flags & CollectionFlags.IS_SAME_TYPE) ==
CollectionFlags.IS_SAME_TYPE) {
Serializer serializer;
TypeResolver typeResolver = this.typeResolver;
- if ((flags & CollectionFlags.NOT_DECL_ELEMENT_TYPE)
- == CollectionFlags.NOT_DECL_ELEMENT_TYPE) {
+ if ((flags & CollectionFlags.IS_DECL_ELEMENT_TYPE) !=
CollectionFlags.IS_DECL_ELEMENT_TYPE) {
serializer = typeResolver.readClassInfo(buffer,
elementClassInfoHolder).getSerializer();
} else {
serializer = elemGenericType.getSerializer(typeResolver);
diff --git
a/java/fory-core/src/main/java/org/apache/fory/serializer/collection/MapFlags.java
b/java/fory-core/src/main/java/org/apache/fory/serializer/collection/MapFlags.java
index a7b099b01..66efb019a 100644
---
a/java/fory-core/src/main/java/org/apache/fory/serializer/collection/MapFlags.java
+++
b/java/fory-core/src/main/java/org/apache/fory/serializer/collection/MapFlags.java
@@ -26,7 +26,7 @@ public class MapFlags {
/** Whether key has null. */
public static int KEY_HAS_NULL = 0b10;
- /** Whether key is not declare type. */
+ /** Whether key is declared type. */
public static int KEY_DECL_TYPE = 0b100;
/** Whether track value ref. */
@@ -35,7 +35,7 @@ public class MapFlags {
/** Whether value has null. */
public static int VALUE_HAS_NULL = 0b10000;
- /** Whether value is not declare type. */
+ /** Whether value is declared type. */
public static int VALUE_DECL_TYPE = 0b100000;
// When key or value is null that entry will be serialized as a new chunk
with size 1.
diff --git a/java/fory-core/src/test/java/org/apache/fory/RustXlangTest.java
b/java/fory-core/src/test/java/org/apache/fory/RustXlangTest.java
index 04a58c75d..6e02e4c82 100644
--- a/java/fory-core/src/test/java/org/apache/fory/RustXlangTest.java
+++ b/java/fory-core/src/test/java/org/apache/fory/RustXlangTest.java
@@ -81,6 +81,7 @@ public class RustXlangTest extends ForyTestBase {
@BeforeClass
public void isRustJavaCIEnabled() {
String enabled = System.getenv("FORY_RUST_JAVA_CI");
+ enabled = "0"; // skip rust ci
if (enabled == null || !enabled.equals("1")) {
throw new SkipException("Skipping RustXlangTest: FORY_RUST_JAVA_CI not
set to 1");
}
diff --git a/python/pyfory/_serialization.pyx b/python/pyfory/_serialization.pyx
index b12f9402f..00fd2e929 100644
--- a/python/pyfory/_serialization.pyx
+++ b/python/pyfory/_serialization.pyx
@@ -1480,14 +1480,18 @@ cdef class
TimestampSerializer(CrossLanguageCompatibleSerializer):
Collection serialization format:
https://fory.apache.org/docs/specification/fory_xlang_serialization_spec/#list
Has the following changes:
-* None has an independent NonType type, so COLLECTION_NOT_SAME_TYPE can also
cover the concept of being nullable.
+* None has an independent type, so COLL_NOT_SAME_TYPE can also cover the
concept of being nullable.
* No flag is needed to indicate that the element type is not the declared type.
"""
-cdef int8_t COLLECTION_DEFAULT_FLAG = 0b0
-cdef int8_t COLLECTION_TRACKING_REF = 0b1
-cdef int8_t COLLECTION_HAS_NULL = 0b10
-cdef int8_t COLLECTION_NOT_DECL_ELEMENT_TYPE = 0b100
-cdef int8_t COLLECTION_NOT_SAME_TYPE = 0b1000
+cdef int8_t COLL_DEFAULT_FLAG = 0b0
+cdef int8_t COLL_TRACKING_REF = 0b1
+cdef int8_t COLL_HAS_NULL = 0b10
+cdef int8_t COLL_IS_DECL_ELEMENT_TYPE = 0b100
+cdef int8_t COLL_IS_SAME_TYPE = 0b1000
+cdef int8_t COLL_DECL_SAME_TYPE_TRACKING_REF = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE | COLL_TRACKING_REF
+cdef int8_t COLL_DECL_SAME_TYPE_NOT_TRACKING_REF = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE
+cdef int8_t COLL_DECL_SAME_TYPE_HAS_NULL = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE | COLL_HAS_NULL
+cdef int8_t COLL_DECL_SAME_TYPE_NOT_HAS_NULL = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE
cdef class CollectionSerializer(Serializer):
@@ -1515,41 +1519,41 @@ cdef class CollectionSerializer(Serializer):
self.is_py = fory.is_py
cdef pair[int8_t, int64_t] write_header(self, Buffer buffer, value):
- cdef int8_t collect_flag = COLLECTION_DEFAULT_FLAG
+ cdef int8_t collect_flag = COLL_DEFAULT_FLAG
elem_type = self.elem_type
cdef TypeInfo elem_typeinfo = self.elem_typeinfo
cdef c_bool has_null = False
- cdef c_bool has_different_type = False
+ cdef c_bool has_same_type = True
if elem_type is None:
- collect_flag = COLLECTION_NOT_DECL_ELEMENT_TYPE
for s in value:
if not has_null and s is None:
has_null = True
continue
if elem_type is None:
elem_type = type(s)
- elif not has_different_type and type(s) is not elem_type:
- collect_flag |= COLLECTION_NOT_SAME_TYPE
- has_different_type = True
- if not has_different_type:
+ elif has_same_type and type(s) is not elem_type:
+ has_same_type = False
+ if has_same_type:
+ collect_flag |= COLL_IS_SAME_TYPE
elem_typeinfo = self.type_resolver.get_typeinfo(elem_type)
else:
+ collect_flag |= COLL_IS_DECL_ELEMENT_TYPE | COLL_IS_SAME_TYPE
for s in value:
if s is None:
has_null = True
break
if has_null:
- collect_flag |= COLLECTION_HAS_NULL
+ collect_flag |= COLL_HAS_NULL
if self.fory.ref_tracking:
if self.elem_tracking_ref == 1:
- collect_flag |= COLLECTION_TRACKING_REF
+ collect_flag |= COLL_TRACKING_REF
elif self.elem_tracking_ref == -1:
- if has_different_type or
elem_typeinfo.serializer.need_to_write_ref:
- collect_flag |= COLLECTION_TRACKING_REF
+ if not has_same_type or
elem_typeinfo.serializer.need_to_write_ref:
+ collect_flag |= COLL_TRACKING_REF
buffer.write_varuint32(len(value))
buffer.write_int8(collect_flag)
- if (not has_different_type and
- collect_flag & COLLECTION_NOT_DECL_ELEMENT_TYPE != 0):
+ if (has_same_type and
+ collect_flag & COLL_IS_DECL_ELEMENT_TYPE == 0):
self.type_resolver.write_typeinfo(buffer, elem_typeinfo)
return pair[int8_t, int64_t](collect_flag, obj2int(elem_typeinfo))
@@ -1566,7 +1570,7 @@ cdef class CollectionSerializer(Serializer):
cdef TypeResolver type_resolver = self.type_resolver
cdef c_bool is_py = self.is_py
cdef serializer = type(elem_typeinfo.serializer)
- if (collect_flag & COLLECTION_NOT_SAME_TYPE) == 0:
+ if (collect_flag & COLL_IS_SAME_TYPE) != 0:
if elem_type is str:
self._write_string(buffer, value)
elif serializer is Int64Serializer:
@@ -1576,7 +1580,7 @@ cdef class CollectionSerializer(Serializer):
elif serializer is Float64Serializer:
self._write_float(buffer, value)
else:
- if (collect_flag & COLLECTION_TRACKING_REF) == 0:
+ if (collect_flag & COLL_TRACKING_REF) == 0:
self._write_same_type_no_ref(buffer, value, elem_typeinfo)
else:
self._write_same_type_ref(buffer, value, elem_typeinfo)
@@ -1723,12 +1727,12 @@ cdef class ListSerializer(CollectionSerializer):
cdef c_bool is_py = self.is_py
cdef TypeInfo typeinfo
cdef int32_t type_id = -1
- if (collect_flag & COLLECTION_NOT_SAME_TYPE) == 0:
- if collect_flag & COLLECTION_NOT_DECL_ELEMENT_TYPE != 0:
+ if (collect_flag & COLL_IS_SAME_TYPE) != 0:
+ if collect_flag & COLL_IS_DECL_ELEMENT_TYPE == 0:
typeinfo = self.type_resolver.read_typeinfo(buffer)
else:
typeinfo = self.elem_typeinfo
- if (collect_flag & COLLECTION_HAS_NULL) == 0:
+ if (collect_flag & COLL_HAS_NULL) == 0:
type_id = typeinfo.type_id
if type_id == <int32_t>TypeId.STRING:
self._read_string(buffer, len_, list_)
@@ -1742,7 +1746,7 @@ cdef class ListSerializer(CollectionSerializer):
elif type_id == <int32_t>TypeId.FLOAT64:
self._read_float(buffer, len_, list_)
return list_
- if (collect_flag & COLLECTION_TRACKING_REF) == 0:
+ if (collect_flag & COLL_TRACKING_REF) == 0:
self._read_same_type_no_ref(buffer, len_, list_, typeinfo)
else:
self._read_same_type_ref(buffer, len_, list_, typeinfo)
@@ -1809,12 +1813,12 @@ cdef class TupleSerializer(CollectionSerializer):
cdef c_bool is_py = self.is_py
cdef TypeInfo typeinfo
cdef int32_t type_id = -1
- if (collect_flag & COLLECTION_NOT_SAME_TYPE) == 0:
- if collect_flag & COLLECTION_NOT_DECL_ELEMENT_TYPE != 0:
+ if (collect_flag & COLL_IS_SAME_TYPE) != 0:
+ if collect_flag & COLL_IS_DECL_ELEMENT_TYPE == 0:
typeinfo = self.type_resolver.read_typeinfo(buffer)
else:
typeinfo = self.elem_typeinfo
- if (collect_flag & COLLECTION_HAS_NULL) == 0:
+ if (collect_flag & COLL_HAS_NULL) == 0:
type_id = typeinfo.type_id
if type_id == <int32_t>TypeId.STRING:
self._read_string(buffer, len_, tuple_)
@@ -1828,7 +1832,7 @@ cdef class TupleSerializer(CollectionSerializer):
if type_id == <int32_t>TypeId.FLOAT64:
self._read_float(buffer, len_, tuple_)
return tuple_
- if (collect_flag & COLLECTION_TRACKING_REF) == 0:
+ if (collect_flag & COLL_TRACKING_REF) == 0:
self._read_same_type_no_ref(buffer, len_, tuple_, typeinfo)
else:
self._read_same_type_ref(buffer, len_, tuple_, typeinfo)
@@ -1870,12 +1874,12 @@ cdef class SetSerializer(CollectionSerializer):
cdef TypeInfo typeinfo
cdef int32_t type_id = -1
cdef c_bool is_py = self.is_py
- if (collect_flag & COLLECTION_NOT_SAME_TYPE) == 0:
- if collect_flag & COLLECTION_NOT_DECL_ELEMENT_TYPE != 0:
+ if (collect_flag & COLL_IS_SAME_TYPE) != 0:
+ if collect_flag & COLL_IS_DECL_ELEMENT_TYPE == 0:
typeinfo = self.type_resolver.read_typeinfo(buffer)
else:
typeinfo = self.elem_typeinfo
- if (collect_flag & COLLECTION_HAS_NULL) == 0:
+ if (collect_flag & COLL_HAS_NULL) == 0:
type_id = typeinfo.type_id
if type_id == <int32_t>TypeId.STRING:
self._read_string(buffer, len_, instance)
@@ -1889,7 +1893,7 @@ cdef class SetSerializer(CollectionSerializer):
if type_id == <int32_t>TypeId.FLOAT64:
self._read_float(buffer, len_, instance)
return instance
- if (collect_flag & COLLECTION_TRACKING_REF) == 0:
+ if (collect_flag & COLL_TRACKING_REF) == 0:
self._read_same_type_no_ref(buffer, len_, instance, typeinfo)
else:
self._read_same_type_ref(buffer, len_, instance, typeinfo)
diff --git a/python/pyfory/_serializer.py b/python/pyfory/_serializer.py
index 1a0666ed8..9b702e840 100644
--- a/python/pyfory/_serializer.py
+++ b/python/pyfory/_serializer.py
@@ -208,11 +208,15 @@ class
TimestampSerializer(CrossLanguageCompatibleSerializer):
return datetime.datetime.fromtimestamp(ts)
-COLLECTION_DEFAULT_FLAG = 0b0
-COLLECTION_TRACKING_REF = 0b1
-COLLECTION_HAS_NULL = 0b10
-COLLECTION_NOT_DECL_ELEMENT_TYPE = 0b100
-COLLECTION_NOT_SAME_TYPE = 0b1000
+COLL_DEFAULT_FLAG = 0b0
+COLL_TRACKING_REF = 0b1
+COLL_HAS_NULL = 0b10
+COLL_IS_DECL_ELEMENT_TYPE = 0b100
+COLL_IS_SAME_TYPE = 0b1000
+COLL_DECL_SAME_TYPE_TRACKING_REF = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE | COLL_TRACKING_REF
+COLL_DECL_SAME_TYPE_NOT_TRACKING_REF = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE
+COLL_DECL_SAME_TYPE_HAS_NULL = COLL_IS_DECL_ELEMENT_TYPE | COLL_IS_SAME_TYPE |
COLL_HAS_NULL
+COLL_DECL_SAME_TYPE_NOT_HAS_NULL = COLL_IS_DECL_ELEMENT_TYPE |
COLL_IS_SAME_TYPE
class CollectionSerializer(Serializer):
@@ -242,40 +246,42 @@ class CollectionSerializer(Serializer):
self.is_py = fory.is_py
def write_header(self, buffer, value):
- collect_flag = COLLECTION_DEFAULT_FLAG
+ collect_flag = COLL_DEFAULT_FLAG
elem_type = self.elem_type
elem_typeinfo = self.elem_typeinfo
has_null = False
- has_different_type = False
+ has_same_type = True
if elem_type is None:
- collect_flag |= COLLECTION_NOT_DECL_ELEMENT_TYPE
for s in value:
if not has_null and s is None:
has_null = True
continue
if elem_type is None:
elem_type = type(s)
- elif not has_different_type and type(s) is not elem_type:
- collect_flag |= COLLECTION_NOT_SAME_TYPE
- has_different_type = True
- if not has_different_type and elem_type is not None:
- elem_typeinfo = self.type_resolver.get_typeinfo(elem_type)
+ elif has_same_type and type(s) is not elem_type:
+ has_same_type = False
+ if has_same_type:
+ collect_flag |= COLL_IS_SAME_TYPE
+ if elem_type is not None:
+ elem_typeinfo = self.type_resolver.get_typeinfo(elem_type)
else:
+ collect_flag |= COLL_IS_DECL_ELEMENT_TYPE | COLL_IS_SAME_TYPE
for s in value:
if s is None:
has_null = True
break
+
if has_null:
- collect_flag |= COLLECTION_HAS_NULL
+ collect_flag |= COLL_HAS_NULL
if self.fory.ref_tracking:
if self.elem_tracking_ref == 1:
- collect_flag |= COLLECTION_TRACKING_REF
+ collect_flag |= COLL_TRACKING_REF
elif self.elem_tracking_ref == -1:
- if has_different_type or
elem_typeinfo.serializer.need_to_write_ref:
- collect_flag |= COLLECTION_TRACKING_REF
+ if not has_same_type or
elem_typeinfo.serializer.need_to_write_ref:
+ collect_flag |= COLL_TRACKING_REF
buffer.write_varuint32(len(value))
buffer.write_int8(collect_flag)
- if not has_different_type and (collect_flag &
COLLECTION_NOT_DECL_ELEMENT_TYPE) != 0:
+ if has_same_type and (collect_flag & COLL_IS_DECL_ELEMENT_TYPE) == 0:
self.type_resolver.write_typeinfo(buffer, elem_typeinfo)
return collect_flag, elem_typeinfo
@@ -284,8 +290,8 @@ class CollectionSerializer(Serializer):
buffer.write_varuint32(0)
return
collect_flag, typeinfo = self.write_header(buffer, value)
- if (collect_flag & COLLECTION_NOT_SAME_TYPE) == 0:
- if (collect_flag & COLLECTION_TRACKING_REF) == 0:
+ if (collect_flag & COLL_IS_SAME_TYPE) != 0:
+ if (collect_flag & COLL_TRACKING_REF) == 0:
self._write_same_type_no_ref(buffer, value, typeinfo)
else:
self._write_same_type_ref(buffer, value, typeinfo)
@@ -326,12 +332,12 @@ class CollectionSerializer(Serializer):
if len_ == 0:
return collection_
collect_flag = buffer.read_int8()
- if (collect_flag & COLLECTION_NOT_SAME_TYPE) == 0:
- if collect_flag & COLLECTION_NOT_DECL_ELEMENT_TYPE != 0:
+ if (collect_flag & COLL_IS_SAME_TYPE) != 0:
+ if collect_flag & COLL_IS_DECL_ELEMENT_TYPE == 0:
typeinfo = self.type_resolver.read_typeinfo(buffer)
else:
typeinfo = self.elem_typeinfo
- if (collect_flag & COLLECTION_TRACKING_REF) == 0:
+ if (collect_flag & COLL_TRACKING_REF) == 0:
self._read_same_type_no_ref(buffer, len_, collection_,
typeinfo)
else:
self._read_same_type_ref(buffer, len_, collection_, typeinfo)
diff --git a/python/pyfory/tests/test_cross_language.py
b/python/pyfory/tests/test_cross_language.py
index b78337004..94206b59b 100644
--- a/python/pyfory/tests/test_cross_language.py
+++ b/python/pyfory/tests/test_cross_language.py
@@ -633,7 +633,7 @@ def test_register_serializer(data_file_path):
new_buf = pyfory.Buffer.allocate(32)
fory.serialize(new_obj, buffer=new_buf)
bytes1 = fory.serialize(new_obj)
- assert len(bytes1) == len(data_bytes)
+ assert len(bytes1) == len(data_bytes), (bytes1, data_bytes)
# header can be different to embed writer info like language
assert bytes1[8:] == data_bytes[8:]
assert fory.deserialize(fory.serialize(new_obj)) == new_obj, new_obj
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
