This is an automated email from the ASF dual-hosted git repository.
wangweipeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-fury.git
The following commit(s) were added to refs/heads/main by this push:
new 3b61eca8 feat(spec): remove list/map header from type meta spec (#1590)
3b61eca8 is described below
commit 3b61eca87c7fb7b6332f05ef5a5b6ee1927e2118
Author: Shawn Yang <[email protected]>
AuthorDate: Sun Apr 28 16:38:14 2024 +0800
feat(spec): remove list/map header from type meta spec (#1590)
## What does this PR do?
This PR removes list/map header from type meta spec.
Such header may be computed ahead sometimes. But it may need to compute
based the data at runtime. If we write header into type meta, we may
still need to compute and write header when serialization. This will
introduce extra space cost.
Instead, we can write all such header at runtime, and when
creating/generating serializer, we can compute part of such header
ahead, and let remaing parts of header computed at serialization.
When deserialization, the deserializer can generate different
deserializer based on read header, which will be more efficient.
## Related issues
#1413
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/incubator-fury/issues/new/choose)
describing the need to do so and update the document if necessary.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
-->
---
docs/specification/java_serialization_spec.md | 26 ++++++--------------------
docs/specification/xlang_serialization_spec.md | 22 +++++-----------------
2 files changed, 11 insertions(+), 37 deletions(-)
diff --git a/docs/specification/java_serialization_spec.md
b/docs/specification/java_serialization_spec.md
index 242a1416..38dd5326 100644
--- a/docs/specification/java_serialization_spec.md
+++ b/docs/specification/java_serialization_spec.md
@@ -178,25 +178,8 @@ Meta header is a 64 bits number value encoded in little
endian order.
meta
for such types is written separately instead of inlining here is to
reduce meta space cost if object of this
type is serialized in current object graph multiple times, and the
field value may be null too.
- - Collection Type Info: collection type will have an extra byte for
elements info.
- Users can use annotation to provide those info.
- - elements type same
- - elements tracking ref
- - elements nullability
- - elements declared type
- - Map Type Info: map type will have an extra byte for kv items info.
- Users can use annotation to provide those info.
- - keys type same
- - keys tracking ref
- - keys nullability
- - keys declared type
- - values type same
- - values tracking ref
- - values nullability
- - values declared type
- Field name: If type id is set, type id will be used instead. Otherwise
meta string encoding length and data will
- be
- written instead.
+ be written instead.
Field order are left as implementation details, which is not exposed to
specification, the deserialization need to
resort fields based on Fury field comparator. In this way, fury can compute
statistics for field names or types and
@@ -394,6 +377,9 @@ which will be encoded by elements header, each use one bit:
By default, all bits are unset, which means all elements won't track ref, all
elements are same type, not null and
the actual element is the declared type in the custom class field.
+The implementation can generate different deserialization code based read
header, and look up the generated code from a
+linear map/list.
+
#### Elements data
Based on the elements header, the serialization of elements data may skip `ref
flag`/`null flag`/`element class info`.
@@ -469,8 +455,8 @@ format will be:
| KV header | N*2 objects |
```
-`KV header` will be a header marked by `MapFieldInfo` in java. For languages
such as golang, this can be computed in
-advance for non-interface types in most times.
+`KV header` will be a header marked by `MapFieldInfo` in java. The
implementation can generate different deserialization
+code based read header, and look up the generated code from a linear map/list.
### Enum
diff --git a/docs/specification/xlang_serialization_spec.md
b/docs/specification/xlang_serialization_spec.md
index bbfdc76e..0583e7e0 100644
--- a/docs/specification/xlang_serialization_spec.md
+++ b/docs/specification/xlang_serialization_spec.md
@@ -303,22 +303,6 @@ Meta header is a 64 bits number value encoded in little
endian order.
- Otherwise it will be encoded as `OBJECT_ID` if it isn't `final` and
`FINAL_OBJECT_ID` if it's `final`. The
meta for such types is written separately instead of inlining here
is to reduce meta space cost if object of
this type is serialized in the current object graph multiple times,
and the field value may be null too.
- - List Type Info: this type will have an extra byte for elements info.
- Users can use annotation to provide that info.
- - elements type same
- - elements tracking ref
- - elements nullability
- - elements declared type
- - Map Type Info: this type will have an extra byte for kv items info.
- Users can use annotation to provide that info.
- - keys type same
- - keys tracking ref
- - keys nullability
- - keys declared type
- - values type same
- - values tracking ref
- - values nullability
- - values declared type
- Field name: If tag id is set, tag id will be used instead. Otherwise
meta string encoding length and data will
be written instead.
@@ -489,6 +473,9 @@ which will be encoded by elements header, each use one bit:
By default, all bits are unset, which means all elements won't track ref, all
elements are same type, not null and
the actual element is the declared type in the custom type field.
+The implementation can generate different deserialization code based read
header, and look up the generated code from
+a linear map/list.
+
#### elements data
Based on the elements header, the serialization of elements data may skip `ref
flag`/`null flag`/`element type info`.
@@ -593,7 +580,8 @@ format will be:
```
`KV header` will be a header marked by `MapFieldInfo` in java. For languages
such as golang, this can be computed in
-advance for non-interface types most times.
+advance for non-interface types most times. The implementation can generate
different deserialization code based read
+header, and look up the generated code from a linear map/list.
#### Why serialize chunk by chunk?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
