This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory.git
The following commit(s) were added to refs/heads/main by this push:
new 4cdf1510f fix(xlang): reject malformed meta strings (#3885)
4cdf1510f is described below
commit 4cdf1510fcf292a08d79b8233a6c2dac8d4998be
Author: Shawn Yang <[email protected]>
AuthorDate: Mon Jul 27 01:30:47 2026 +0800
fix(xlang): reject malformed meta strings (#3885)
## Why?
Malformed meta strings and TypeDef name metadata could trigger
runtime-specific indexing failures or
allow decoding to continue with an incomplete error state. Go also
exposed both a shared `*Error`
parameter and a returned `error`, which made it easy for callers to drop
one error channel.
## What does this PR do?
- Hardens Go meta-string decoding for empty transformed output and
unmatched escape markers.
- Makes Go `MetaStringResolver.ReadMetaStringBytes` report failures only
through its supplied
`*Error`, and stops TypeInfo consumers before dereferencing failed
results.
- Treats Go name-hash lookups as candidates on TypeDef cache misses and
requires exact encoded
namespace/type identity before accepting them.
- Rejects malformed escape markers and invalid TypeDef name encodings or
overflowing name lengths
in the affected Java, Python, JavaScript, and Dart paths.
- Corrects JavaScript dynamic meta-string reference IDs and one-based
reference lookup.
- Adds focused regression coverage for the malformed cases and valid
JavaScript reference reuse.
Valid metadata bytes and the binary protocol are unchanged.
## Related issues
N/A.
## AI Contribution Checklist
### AI Usage Disclosure
## Does this PR introduce any user-facing change?
Go callers of `MetaStringResolver.ReadMetaStringBytes` now receive only
`*MetaStringBytes`; failures
are stored in the supplied `*Error`.
- [x] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Validation
- Go:
- `ENABLE_FORY_DEBUG_OUTPUT=1 go test -v ./...`
- focused `go test -race` coverage for malformed meta strings, resolver
errors, TypeDef names, and
TypeInfo consumers
- Java:
- `mvn -T16 spotless:check`
- focused metadata tests and full `ENABLE_FORY_DEBUG_OUTPUT=1 mvn -T16
test`
- Python:
- full pytest suites with `ENABLE_FORY_CYTHON_SERIALIZATION=0` and `=1`
- Java-driven `PythonXlangTest` in both modes
- `ruff format --check` passed; the changed lines introduce zero Ruff
findings
- JavaScript:
- focused and full Jest suites
- `npm run build`
- `npm run format-check`
- Dart:
- `dart analyze`
- focused xlang protocol test and full `dart test packages/fory/test`
## Benchmark
Not run, by explicit scope. The final diff was reviewed to ensure
validation remains on TypeDef
cache-miss or malformed-input paths. Checked metadata cache hits are
unchanged, and no new
successful-path allocation or wrapper was introduced.
---
dart/packages/fory/lib/src/meta/meta_string.dart | 7 +-
dart/packages/fory/test/xlang_protocol_test.dart | 19 +++++
go/fory/meta/meta_string_decoder.go | 10 ++-
go/fory/meta/meta_string_test.go | 16 ++++
go/fory/meta_string_resolver.go | 48 +++++++-----
go/fory/meta_string_resolver_test.go | 39 +++++-----
go/fory/optional_serializer.go | 6 +-
go/fory/pointer.go | 3 +
go/fory/tests/metastring_resolver_test.go | 12 +--
go/fory/type_def.go | 25 +++++-
go/fory/type_def_test.go | 37 +++++++++
go/fory/type_resolver.go | 23 +++++-
go/fory/type_resolver_test.go | 89 ++++++++++++++++++++++
.../org/apache/fory/meta/MetaStringDecoder.java | 4 +
.../org/apache/fory/meta/NativeTypeDefDecoder.java | 22 ++++--
.../java/org/apache/fory/meta/TypeDefDecoder.java | 6 +-
.../java/org/apache/fory/meta/MetaStringTest.java | 15 ++++
.../apache/fory/meta/NativeTypeDefEncoderTest.java | 74 ++++++++++++++++++
.../org/apache/fory/meta/TypeDefEncoderTest.java | 53 +++++++++++++
javascript/packages/core/lib/context.ts | 6 +-
javascript/packages/core/lib/meta/MetaString.ts | 3 +
javascript/test/metastring.test.ts | 65 ++++++++++++++++
python/pyfory/meta/metastring.py | 26 ++++---
python/pyfory/meta/typedef_decoder.py | 13 +++-
python/pyfory/tests/test_metastring.py | 26 ++++---
python/pyfory/tests/test_typedef_encoding.py | 17 ++++-
python/pyproject.toml | 1 +
27 files changed, 575 insertions(+), 90 deletions(-)
diff --git a/dart/packages/fory/lib/src/meta/meta_string.dart
b/dart/packages/fory/lib/src/meta/meta_string.dart
index 7089d0071..73a2bcf81 100644
--- a/dart/packages/fory/lib/src/meta/meta_string.dart
+++ b/dart/packages/fory/lib/src/meta/meta_string.dart
@@ -459,7 +459,12 @@ String _decodeAllToLowerSpecial(List<int> bytes) {
final buffer = StringBuffer();
for (var index = 0; index < decoded.length; index += 1) {
final character = decoded[index];
- if (character == '|' && index + 1 < decoded.length) {
+ if (character == '|') {
+ if (index + 1 >= decoded.length) {
+ throw StateError(
+ 'Invalid ALL_TO_LOWER_SPECIAL meta string: trailing escape.',
+ );
+ }
index += 1;
buffer.write(decoded[index].toUpperCase());
continue;
diff --git a/dart/packages/fory/test/xlang_protocol_test.dart
b/dart/packages/fory/test/xlang_protocol_test.dart
index 74b2bcd14..3251c3a30 100644
--- a/dart/packages/fory/test/xlang_protocol_test.dart
+++ b/dart/packages/fory/test/xlang_protocol_test.dart
@@ -25,6 +25,7 @@ import 'package:fory/src/codegen/generated_registry.dart';
import 'package:fory/src/context/meta_string_reader.dart';
import 'package:fory/src/context/meta_string_writer.dart';
import 'package:fory/src/meta/field_info.dart';
+import 'package:fory/src/meta/meta_string.dart';
import 'package:fory/src/meta/type_def.dart';
import 'package:fory/src/meta/type_meta.dart';
import 'package:fory/src/resolver/type_resolver.dart';
@@ -341,6 +342,24 @@ void main() {
);
});
+ test('rejects a trailing meta string escape', () {
+ expect(
+ () => decodeMetaString(
+ const <int>[0x74],
+ metaStringAllToLowerSpecialEncoding,
+ specialChar1: r'$',
+ specialChar2: '_',
+ ),
+ throwsA(
+ isA<StateError>().having(
+ (error) => error.toString(),
+ 'message',
+ contains('trailing escape'),
+ ),
+ ),
+ );
+ });
+
test('parsed TypeDef cache publishes beyond old implementation floor', () {
final cache = ParsedTypeMetaCache();
const oldImplementationFloor = 8192;
diff --git a/go/fory/meta/meta_string_decoder.go
b/go/fory/meta/meta_string_decoder.go
index 59273b032..3818120fd 100644
--- a/go/fory/meta/meta_string_decoder.go
+++ b/go/fory/meta/meta_string_decoder.go
@@ -38,8 +38,8 @@ func NewDecoder(specialCh1 byte, specialCh2 byte) *Decoder {
func (d *Decoder) Decode(data []byte, encoding Encoding) (result string, err
error) {
// we prepend one bit at the start to indicate whether strip last char
// so checking empty here will be convenient for decoding procedure
- if data == nil {
- return "", err
+ if len(data) == 0 {
+ return "", nil
}
var chars []byte
switch encoding {
@@ -50,6 +50,9 @@ func (d *Decoder) Decode(data []byte, encoding Encoding)
(result string, err err
case FIRST_TO_LOWER_SPECIAL:
chars, err = d.decodeGeneric(data, LOWER_SPECIAL)
if err == nil {
+ if len(chars) == 0 {
+ return "", fmt.Errorf("invalid
FIRST_TO_LOWER_SPECIAL meta string: missing first character")
+ }
chars[0] = chars[0] - 'a' + 'A'
}
case ALL_TO_LOWER_SPECIAL:
@@ -111,6 +114,9 @@ func (d *Decoder) decodeRepAllToLowerSpecial(data []byte,
algorithm Encoding) ([
j := 0
for i := 0; i < len(str); i++ {
if str[i] == '|' {
+ if i+1 >= len(str) {
+ return nil, fmt.Errorf("invalid
ALL_TO_LOWER_SPECIAL meta string: trailing escape")
+ }
chars[j] = str[i+1] - 'a' + 'A'
i++
} else {
diff --git a/go/fory/meta/meta_string_test.go b/go/fory/meta/meta_string_test.go
index 0d94561cc..7f2ad39ae 100644
--- a/go/fory/meta/meta_string_test.go
+++ b/go/fory/meta/meta_string_test.go
@@ -71,6 +71,22 @@ func TestEncodeAndDecodeMetaString(t *testing.T) {
require.Equal(t, "", dst)
}
+func TestDecodeMalformedMetaString(t *testing.T) {
+ decoder := NewDecoder('.', '_')
+
+ value, err := decoder.Decode(make([]byte, 0), LOWER_SPECIAL)
+ require.NoError(t, err)
+ require.Empty(t, value)
+
+ _, err = decoder.Decode([]byte{0x80}, FIRST_TO_LOWER_SPECIAL)
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "missing first character")
+
+ _, err = decoder.Decode([]byte{0x74}, ALL_TO_LOWER_SPECIAL)
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "trailing escape")
+}
+
func calcTotalBytes(src string, bitsPerChar int, encoding Encoding) int {
if encoding == UTF_8 {
return len(src)
diff --git a/go/fory/meta_string_resolver.go b/go/fory/meta_string_resolver.go
index d74d5aa2c..6134704bb 100644
--- a/go/fory/meta_string_resolver.go
+++ b/go/fory/meta_string_resolver.go
@@ -119,27 +119,31 @@ func (r *MetaStringResolver) WriteMetaStringBytes(buf
*ByteBuffer, m *MetaString
}
}
-// ReadMetaStringBytes reads a string from buffer, handling dynamic references
-func (r *MetaStringResolver) ReadMetaStringBytes(buf *ByteBuffer, ctxErr
*Error) (*MetaStringBytes, error) {
+// ReadMetaStringBytes reads a string from buffer, handling dynamic references.
+// All failures are stored in ctxErr so callers have one error channel.
+func (r *MetaStringResolver) ReadMetaStringBytes(buf *ByteBuffer, ctxErr
*Error) *MetaStringBytes {
// ReadData header containing length/reference info (uses
VarUint32Small7 to match Java)
header := buf.ReadVarUint32Small7(ctxErr)
if ctxErr.HasError() {
- return nil, *ctxErr
+ return nil
}
lengthValue := header >> 1
if header&1 != 0 {
if lengthValue == 0 || uint64(lengthValue) > uint64(MaxInt) {
- return nil, fmt.Errorf("invalid dynamic index: %d",
lengthValue)
+ ctxErr.SetError(fmt.Errorf("invalid dynamic index: %d",
lengthValue))
+ return nil
}
index := int(lengthValue) - 1
if index < 0 || index >= len(r.dynamicIDToEnumString) {
- return nil, fmt.Errorf("invalid dynamic index: %d",
index)
+ ctxErr.SetError(fmt.Errorf("invalid dynamic index: %d",
index))
+ return nil
}
- return r.dynamicIDToEnumString[index], nil
+ return r.dynamicIDToEnumString[index]
}
if lengthValue > uint32(MaxInt16) {
- return nil, fmt.Errorf("meta string length %d exceeds maximum
supported length %d", lengthValue, MaxInt16)
+ ctxErr.SetError(fmt.Errorf("meta string length %d exceeds
maximum supported length %d", lengthValue, MaxInt16))
+ return nil
}
length := int(lengthValue)
@@ -154,22 +158,24 @@ func (r *MetaStringResolver) ReadMetaStringBytes(buf
*ByteBuffer, ctxErr *Error)
if length <= SmallStringThreshold {
if length == 0 {
r.dynamicIDToEnumString =
append(r.dynamicIDToEnumString, emptyMetaStringBytes)
- return emptyMetaStringBytes, nil
+ return emptyMetaStringBytes
}
encByte := buf.ReadByte(ctxErr)
var encErr error
encoding, encErr = meta.EncodingFromByte(encByte)
if encErr != nil {
- return nil, encErr
+ ctxErr.SetError(encErr)
+ return nil
}
if !buf.CheckReadable(length, ctxErr) {
- return nil, *ctxErr
+ return nil
}
data = make([]byte, length)
_, err := buf.Read(data)
if err != nil {
- return nil, err
+ ctxErr.SetError(err)
+ return nil
}
words := smallMetaStringWords(data)
@@ -183,24 +189,28 @@ func (r *MetaStringResolver) ReadMetaStringBytes(buf
*ByteBuffer, ctxErr *Error)
// Large string handling
err := binary.Read(buf, binary.LittleEndian, &hashcode)
if err != nil {
- return nil, err
+ ctxErr.SetError(err)
+ return nil
}
var encErr error
encoding, encErr = meta.EncodingFromByte(byte(hashcode & 0xFF))
if encErr != nil {
- return nil, encErr
+ ctxErr.SetError(encErr)
+ return nil
}
if !buf.CheckReadable(length, ctxErr) {
- return nil, *ctxErr
+ return nil
}
data = make([]byte, length)
_, err = buf.Read(data)
if err != nil {
- return nil, err
+ ctxErr.SetError(err)
+ return nil
}
canonicalHashcode := ComputeMetaStringHash(data, encoding)
if canonicalHashcode != hashcode {
- return nil, fmt.Errorf("meta string body hash mismatch")
+ ctxErr.SetError(fmt.Errorf("meta string body hash
mismatch"))
+ return nil
}
}
@@ -208,12 +218,12 @@ func (r *MetaStringResolver) ReadMetaStringBytes(buf
*ByteBuffer, ctxErr *Error)
if length <= SmallStringThreshold {
if m, ok := r.smallHashToMetaStrBytes[key]; ok {
r.dynamicIDToEnumString =
append(r.dynamicIDToEnumString, m)
- return m, nil
+ return m
}
} else {
if m, ok := r.hashToMetaStrBytes[hashcode]; ok {
r.dynamicIDToEnumString =
append(r.dynamicIDToEnumString, m)
- return m, nil
+ return m
}
}
@@ -231,7 +241,7 @@ func (r *MetaStringResolver) ReadMetaStringBytes(buf
*ByteBuffer, ctxErr *Error)
}
r.dynamicIDToEnumString = append(r.dynamicIDToEnumString, m)
- return m, nil
+ return m
}
// GetMetaStrBytes converts MetaString to optimized MetaStringBytes
diff --git a/go/fory/meta_string_resolver_test.go
b/go/fory/meta_string_resolver_test.go
index 5a15be802..504ec0b43 100644
--- a/go/fory/meta_string_resolver_test.go
+++ b/go/fory/meta_string_resolver_test.go
@@ -40,10 +40,11 @@ func TestMetaStringResolverNegativeIndexPanic(t *testing.T)
{
var ctxErr Error
// This should NOT panic. The fix handles the negative index.
require.NotPanics(t, func() {
- _, err := resolver.ReadMetaStringBytes(buffer, &ctxErr)
- if err == nil {
+ result := resolver.ReadMetaStringBytes(buffer, &ctxErr)
+ if !ctxErr.HasError() {
t.Errorf("Expected error for negative index, got nil")
}
+ require.Nil(t, result)
}, "MetaStringResolver should not panic on negative index")
}
@@ -64,8 +65,8 @@ func TestMetaStringResolverBoundaryRegression(t *testing.T) {
buffer.SetReaderIndex(0)
var ctxErr Error
- result, err := resolver.ReadMetaStringBytes(buffer, &ctxErr)
- require.NoError(t, err)
+ result := resolver.ReadMetaStringBytes(buffer, &ctxErr)
+ require.False(t, ctxErr.HasError())
require.Equal(t, m, result, "Should correctly resolve the first dynamic
string (index 0)")
}
@@ -80,8 +81,8 @@ func TestMetaStringResolverRejectsLargeBodyHashMismatch(t
*testing.T) {
buffer.SetReaderIndex(0)
var ctxErr Error
- _, err := resolver.ReadMetaStringBytes(buffer, &ctxErr)
- require.Error(t, err)
+ require.Nil(t, resolver.ReadMetaStringBytes(buffer, &ctxErr))
+ require.True(t, ctxErr.HasError())
require.Empty(t, resolver.hashToMetaStrBytes)
require.Empty(t, resolver.dynamicIDToEnumString)
}
@@ -93,9 +94,9 @@ func
TestMetaStringResolverRejectsOversizedLengthBeforeAllocation(t *testing.T)
buffer.SetReaderIndex(0)
var ctxErr Error
- _, err := resolver.ReadMetaStringBytes(buffer, &ctxErr)
- require.Error(t, err)
- require.Contains(t, err.Error(), "meta string length")
+ require.Nil(t, resolver.ReadMetaStringBytes(buffer, &ctxErr))
+ require.True(t, ctxErr.HasError())
+ require.Contains(t, ctxErr.Error(), "meta string length")
require.Empty(t, resolver.hashToMetaStrBytes)
require.Empty(t, resolver.smallHashToMetaStrBytes)
require.Empty(t, resolver.dynamicIDToEnumString)
@@ -110,8 +111,8 @@ func
TestMetaStringResolverSmallCacheKeyIncludesLengthAndEncoding(t *testing.T)
oneByte.WriteByte(1)
oneByte.SetReaderIndex(0)
var oneErr Error
- first, err := resolver.ReadMetaStringBytes(oneByte, &oneErr)
- require.NoError(t, err)
+ first := resolver.ReadMetaStringBytes(oneByte, &oneErr)
+ require.False(t, oneErr.HasError())
require.Equal(t, []byte{1}, first.Data)
twoBytes := NewByteBuffer(nil)
@@ -120,8 +121,8 @@ func
TestMetaStringResolverSmallCacheKeyIncludesLengthAndEncoding(t *testing.T)
twoBytes.Write([]byte{1, 0})
twoBytes.SetReaderIndex(0)
var twoErr Error
- second, err := resolver.ReadMetaStringBytes(twoBytes, &twoErr)
- require.NoError(t, err)
+ second := resolver.ReadMetaStringBytes(twoBytes, &twoErr)
+ require.False(t, twoErr.HasError())
require.Equal(t, []byte{1, 0}, second.Data)
require.NotSame(t, first, second)
@@ -131,8 +132,8 @@ func
TestMetaStringResolverSmallCacheKeyIncludesLengthAndEncoding(t *testing.T)
differentEncoding.WriteByte(1)
differentEncoding.SetReaderIndex(0)
var encodingErr Error
- third, err := resolver.ReadMetaStringBytes(differentEncoding,
&encodingErr)
- require.NoError(t, err)
+ third := resolver.ReadMetaStringBytes(differentEncoding, &encodingErr)
+ require.False(t, encodingErr.HasError())
require.Equal(t, []byte{1}, third.Data)
require.Equal(t, meta.LOWER_SPECIAL, third.Encoding)
require.NotSame(t, first, third)
@@ -172,8 +173,8 @@ func TestMetaStringResolverReadCachesAreCapped(t
*testing.T) {
smallBuffer.WriteByte(1)
smallBuffer.SetReaderIndex(0)
var smallErr Error
- _, err := resolver.ReadMetaStringBytes(smallBuffer, &smallErr)
- require.NoError(t, err)
+ resolver.ReadMetaStringBytes(smallBuffer, &smallErr)
+ require.False(t, smallErr.HasError())
require.Len(t, resolver.smallHashToMetaStrBytes, maxCachedMetaStrings)
require.NotContains(t, resolver.smallHashToMetaStrBytes, smallKey)
@@ -185,8 +186,8 @@ func TestMetaStringResolverReadCachesAreCapped(t
*testing.T) {
largeBuffer.Write(largeData)
largeBuffer.SetReaderIndex(0)
var largeErr Error
- _, err = resolver.ReadMetaStringBytes(largeBuffer, &largeErr)
- require.NoError(t, err)
+ resolver.ReadMetaStringBytes(largeBuffer, &largeErr)
+ require.False(t, largeErr.HasError())
require.Len(t, resolver.hashToMetaStrBytes, maxCachedMetaStrings)
require.NotContains(t, resolver.hashToMetaStrBytes, largeHash)
}
diff --git a/go/fory/optional_serializer.go b/go/fory/optional_serializer.go
index 097176636..1f2a2c53f 100644
--- a/go/fory/optional_serializer.go
+++ b/go/fory/optional_serializer.go
@@ -293,7 +293,11 @@ func (s *optionalSerializer) Read(ctx *ReadContext,
refMode RefMode, readType bo
}
internalTypeID := TypeId(typeID)
if IsNamespacedType(internalTypeID) || internalTypeID ==
COMPATIBLE_STRUCT || internalTypeID == STRUCT {
- typeInfo :=
ctx.TypeResolver().readTypeInfoWithTypeID(buf, typeID, ctx.Err())
+ ctxErr := ctx.Err()
+ typeInfo :=
ctx.TypeResolver().readTypeInfoWithTypeID(buf, typeID, ctxErr)
+ if ctxErr.HasError() {
+ return
+ }
if structSer, ok :=
typeInfo.Serializer.(*structSerializer); ok && len(structSer.fieldDefs) > 0 {
valueField := s.valueField(value)
s.setHas(value, true)
diff --git a/go/fory/pointer.go b/go/fory/pointer.go
index e1debba3a..bfced966d 100644
--- a/go/fory/pointer.go
+++ b/go/fory/pointer.go
@@ -195,6 +195,9 @@ func (s *ptrToValueSerializer) Read(ctx *ReadContext,
refMode RefMode, readType
// Check if this is a struct type that needs type meta reading
if IsNamespacedType(internalTypeID) || internalTypeID ==
COMPATIBLE_STRUCT || internalTypeID == STRUCT {
typeInfo :=
ctx.TypeResolver().readTypeInfoWithTypeID(buf, typeID, ctxErr)
+ if ctxErr.HasError() {
+ return
+ }
// Use the serializer from TypeInfo which has the
remote field definitions
if structSer, ok :=
typeInfo.Serializer.(*structSerializer); ok && len(structSer.fieldDefs) > 0 {
// Allocate the pointer value if needed
diff --git a/go/fory/tests/metastring_resolver_test.go
b/go/fory/tests/metastring_resolver_test.go
index b0b820220..6adb1b013 100644
--- a/go/fory/tests/metastring_resolver_test.go
+++ b/go/fory/tests/metastring_resolver_test.go
@@ -39,7 +39,7 @@ func TestMetaStringResolver(t *testing.T) {
if bufErr.HasError() {
t.Fatalf("write failed: %v", bufErr.Error())
}
- got1, _ := resolver.ReadMetaStringBytes(buffer, &bufErr)
+ got1 := resolver.ReadMetaStringBytes(buffer, &bufErr)
if got1.Hashcode != metaBytes1.Hashcode || !bytes.Equal(got1.Data,
metaBytes1.Data) {
t.Errorf("Mismatch in English string")
}
@@ -51,7 +51,7 @@ func TestMetaStringResolver(t *testing.T) {
if bufErr.HasError() {
t.Fatalf("write failed: %v", bufErr.Error())
}
- got2, _ := resolver.ReadMetaStringBytes(buffer, &bufErr)
+ got2 := resolver.ReadMetaStringBytes(buffer, &bufErr)
if got2.Hashcode != metaBytes2.Hashcode || !bytes.Equal(got2.Data,
metaBytes2.Data) {
t.Errorf("Mismatch in custom data")
}
@@ -63,7 +63,7 @@ func TestMetaStringResolver(t *testing.T) {
if bufErr.HasError() {
t.Fatalf("write failed: %v", bufErr.Error())
}
- got3, _ := resolver.ReadMetaStringBytes(buffer, &bufErr)
+ got3 := resolver.ReadMetaStringBytes(buffer, &bufErr)
if got3.Hashcode != metaBytes3.Hashcode || !bytes.Equal(got3.Data,
metaBytes3.Data) {
t.Errorf("Mismatch in empty string")
}
@@ -75,7 +75,7 @@ func TestMetaStringResolver(t *testing.T) {
if bufErr.HasError() {
t.Fatalf("write failed: %v", bufErr.Error())
}
- got4, _ := resolver.ReadMetaStringBytes(buffer, &bufErr)
+ got4 := resolver.ReadMetaStringBytes(buffer, &bufErr)
if got4.Hashcode != metaBytes4.Hashcode || !bytes.Equal(got4.Data,
metaBytes4.Data) {
t.Errorf("Mismatch in Chinese string")
}
@@ -87,7 +87,7 @@ func TestMetaStringResolver(t *testing.T) {
if bufErr.HasError() {
t.Fatalf("write failed: %v", bufErr.Error())
}
- got5, _ := resolver.ReadMetaStringBytes(buffer, &bufErr)
+ got5 := resolver.ReadMetaStringBytes(buffer, &bufErr)
if got5.Hashcode != metaBytes5.Hashcode || !bytes.Equal(got5.Data,
metaBytes5.Data) {
t.Errorf("Mismatch in Japanese string")
}
@@ -100,7 +100,7 @@ func TestMetaStringResolver(t *testing.T) {
if bufErr.HasError() {
t.Fatalf("write failed: %v", bufErr.Error())
}
- got6, _ := resolver.ReadMetaStringBytes(buffer, &bufErr)
+ got6 := resolver.ReadMetaStringBytes(buffer, &bufErr)
if got6.Hashcode != metaBytes6.Hashcode || !bytes.Equal(got6.Data,
metaBytes6.Data) {
t.Errorf("Mismatch in long string")
}
diff --git a/go/fory/type_def.go b/go/fory/type_def.go
index a25f2a0f3..5c381efcb 100644
--- a/go/fory/type_def.go
+++ b/go/fory/type_def.go
@@ -18,6 +18,7 @@
package fory
import (
+ "bytes"
"fmt"
"reflect"
"strings"
@@ -1181,10 +1182,23 @@ func decodeTypeDef(fory *Fory, buffer *ByteBuffer,
header int64) (*TypeDef, erro
}
info, exists :=
fory.typeResolver.nsTypeToTypeInfo[nsTypeKey{nsBytes.Hashcode,
nameBytes.Hashcode}]
+ if exists &&
+ (!sameMetaStringBytes(nsBytes, info.PkgPathBytes) ||
+ !sameMetaStringBytes(nameBytes,
info.NameBytes)) {
+ // Hash lookup is only a candidate match; exact encoded
identity is required.
+ info = nil
+ exists = false
+ }
if !exists {
// Try fallback: decode strings and look up by name
- ns, _ :=
fory.typeResolver.namespaceDecoder.Decode(nsBytes.Data, nsBytes.Encoding)
- typeName, _ :=
fory.typeResolver.typeNameDecoder.Decode(nameBytes.Data, nameBytes.Encoding)
+ ns, err :=
fory.typeResolver.namespaceDecoder.Decode(nsBytes.Data, nsBytes.Encoding)
+ if err != nil {
+ return nil, fmt.Errorf("failed to decode
TypeDef namespace: %w", err)
+ }
+ typeName, err :=
fory.typeResolver.typeNameDecoder.Decode(nameBytes.Data, nameBytes.Encoding)
+ if err != nil {
+ return nil, fmt.Errorf("failed to decode
TypeDef typename: %w", err)
+ }
nameKey := [2]string{ns, typeName}
if fallbackInfo, fallbackExists :=
fory.typeResolver.namedTypeToTypeInfo[nameKey]; fallbackExists {
@@ -1271,6 +1285,13 @@ func decodeTypeDef(fory *Fory, buffer *ByteBuffer,
header int64) (*TypeDef, erro
return typeDef, nil
}
+func sameMetaStringBytes(left, right *MetaStringBytes) bool {
+ return left != nil &&
+ right != nil &&
+ left.Encoding == right.Encoding &&
+ bytes.Equal(left.Data, right.Data)
+}
+
func buildTypeDefEncoded(header int64, metaSizeBits, extraMetaSize int,
metaBytes []byte) []byte {
capacity := 8 + len(metaBytes) + 5
buffer := NewByteBuffer(make([]byte, 0, capacity))
diff --git a/go/fory/type_def_test.go b/go/fory/type_def_test.go
index c395bc2d3..822adc6de 100644
--- a/go/fory/type_def_test.go
+++ b/go/fory/type_def_test.go
@@ -23,6 +23,7 @@ import (
"reflect"
"testing"
+ "github.com/apache/fory/go/fory/meta"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -697,6 +698,42 @@ func TestTypeDefRejectsFieldNameLengthBeyondMetadata(t
*testing.T) {
require.Contains(t, err.Error(), "field name length")
}
+func TestTypeDefRejectsMalformedName(t *testing.T) {
+ tests := []struct {
+ name string
+ cacheHit bool
+ }{
+ {name: "cache miss"},
+ {name: "hash cache hit", cacheHit: true},
+ }
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ fory := NewFory(WithXlang(false), WithCompatible(false))
+ if test.cacheHit {
+ require.NoError(t,
fory.RegisterStructByName(SimpleStruct{}, "example.SimpleStruct"))
+ info :=
fory.typeResolver.namedTypeToTypeInfo[[2]string{"example", "SimpleStruct"}]
+ require.NotNil(t, info)
+ key := nsTypeKey{
+ ComputeMetaStringHash(nil,
meta.ALL_TO_LOWER_SPECIAL),
+ ComputeMetaStringHash([]byte{0x80},
meta.FIRST_TO_LOWER_SPECIAL),
+ }
+ fory.typeResolver.nsTypeToTypeInfo[key] = info
+ }
+
+ body := NewByteBuffer(nil)
+ body.WriteByte(StructTypeDefFlag | RegisterByNameFlag)
+ body.WriteByte(1) // Empty namespace using
ALL_TO_LOWER_SPECIAL.
+ body.WriteByte(1<<2 | 3) // One-byte typename using
FIRST_TO_LOWER_SPECIAL.
+ body.WriteByte(0x80)
+ frame, header := typeDefTestFrame(t, body.Bytes(),
false)
+
+ _, err := decodeTypeDef(fory, frame, header)
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "missing first
character")
+ })
+ }
+}
+
func bodyOnlyTypeDefHeaderHash(data []byte) uint64 {
hash := int64(Murmur3Sum64WithSeed(data, 47) << (64 - NUM_HASH_BITS))
if hash < 0 {
diff --git a/go/fory/type_resolver.go b/go/fory/type_resolver.go
index a3d3c5cb4..7f4280a88 100644
--- a/go/fory/type_resolver.go
+++ b/go/fory/type_resolver.go
@@ -2000,8 +2000,12 @@ func (r *TypeResolver) ReadTypeInfo(buffer *ByteBuffer,
err *Error) *TypeInfo {
return r.readSharedTypeMeta(buffer, err)
}
// ReadData namespace and type name metadata bytes
- nsBytes, _ := r.metaStringResolver.ReadMetaStringBytes(buffer,
err)
- typeBytes, _ :=
r.metaStringResolver.ReadMetaStringBytes(buffer, err)
+ nsBytes := r.metaStringResolver.ReadMetaStringBytes(buffer, err)
+ // Stop here because a following buffer read may replace the
first metadata error.
+ if err.HasError() {
+ return nil
+ }
+ typeBytes := r.metaStringResolver.ReadMetaStringBytes(buffer,
err)
if err.HasError() {
return nil
}
@@ -2053,8 +2057,12 @@ func (r *TypeResolver) readTypeInfoWithTypeID(buffer
*ByteBuffer, typeID uint32,
return r.readSharedTypeMeta(buffer, err)
}
// ReadData namespace and type name metadata bytes
- nsBytes, _ := r.metaStringResolver.ReadMetaStringBytes(buffer,
err)
- typeBytes, _ :=
r.metaStringResolver.ReadMetaStringBytes(buffer, err)
+ nsBytes := r.metaStringResolver.ReadMetaStringBytes(buffer, err)
+ // Stop here because a following buffer read may replace the
first metadata error.
+ if err.HasError() {
+ return nil
+ }
+ typeBytes := r.metaStringResolver.ReadMetaStringBytes(buffer,
err)
if err.HasError() {
return nil
}
@@ -2111,7 +2119,14 @@ func (r *TypeResolver) readTypeInfoForType(buffer
*ByteBuffer, expectedType refl
return nil
}
r.metaStringResolver.ReadMetaStringBytes(buffer, err)
+ // Stop here because a following buffer read may replace the
first metadata error.
+ if err.HasError() {
+ return nil
+ }
r.metaStringResolver.ReadMetaStringBytes(buffer, err)
+ if err.HasError() {
+ return nil
+ }
if internalTypeID == NAMED_STRUCT {
return r.typeToSerializers[expectedType]
}
diff --git a/go/fory/type_resolver_test.go b/go/fory/type_resolver_test.go
index f26ec4796..1738926e4 100644
--- a/go/fory/type_resolver_test.go
+++ b/go/fory/type_resolver_test.go
@@ -21,6 +21,7 @@ import (
"reflect"
"testing"
+ "github.com/apache/fory/go/fory/optional"
"github.com/stretchr/testify/require"
)
@@ -62,3 +63,91 @@ func TestGetTypeInfo_LazyInitPropagatesError(t *testing.T) {
"typePointerCache must not hold an entry with a nil
Serializer")
}
}
+
+func TestTypeInfoPropagatesMetaError(t *testing.T) {
+ expectedType := reflect.TypeOf(SimpleStruct{})
+ tests := []struct {
+ name string
+ read func(*TypeResolver, *ByteBuffer, *Error) any
+ }{
+ {
+ name: "with type id",
+ read: func(resolver *TypeResolver, buffer *ByteBuffer,
readErr *Error) any {
+ return resolver.ReadTypeInfo(buffer, readErr)
+ },
+ },
+ {
+ name: "known type id",
+ read: func(resolver *TypeResolver, buffer *ByteBuffer,
readErr *Error) any {
+ buffer.ReadUint8(readErr)
+ return resolver.readTypeInfoWithTypeID(buffer,
uint32(NAMED_STRUCT), readErr)
+ },
+ },
+ {
+ name: "expected type",
+ read: func(resolver *TypeResolver, buffer *ByteBuffer,
readErr *Error) any {
+ return resolver.readTypeInfoForType(buffer,
expectedType, readErr)
+ },
+ },
+ }
+
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ fory := NewFory(WithXlang(false), WithCompatible(false))
+ buffer := NewByteBuffer(nil)
+ buffer.WriteByte(byte(NAMED_STRUCT))
+ buffer.WriteVarUint32Small7(1)
+ buffer.SetReaderIndex(0)
+ var readErr Error
+
+ result := test.read(fory.typeResolver, buffer, &readErr)
+
+ require.Nil(t, result)
+ require.True(t, readErr.HasError())
+ require.Contains(t, readErr.Error(), "invalid dynamic
index")
+ })
+ }
+}
+
+func TestReadersStopOnTypeInfoError(t *testing.T) {
+ optionalType := reflect.TypeOf(optional.None[*SimpleStruct]())
+ info, ok := getOptionalInfo(optionalType)
+ require.True(t, ok)
+
+ tests := []struct {
+ name string
+ read func(*ReadContext)
+ }{
+ {
+ name: "pointer",
+ read: func(ctx *ReadContext) {
+ var value *SimpleStruct
+ serializer := &ptrToValueSerializer{}
+ serializer.Read(ctx, RefModeNone, true, false,
reflect.ValueOf(&value).Elem())
+ },
+ },
+ {
+ name: "optional",
+ read: func(ctx *ReadContext) {
+ value := reflect.New(optionalType).Elem()
+ serializer :=
newOptionalSerializer(optionalType, info, nil)
+ serializer.Read(ctx, RefModeNone, true, false,
value)
+ },
+ },
+ }
+
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ fory := NewFory(WithXlang(false), WithCompatible(false))
+ ctx := fory.readCtx
+ ctx.Reset()
+ ctx.SetData([]byte{byte(NAMED_STRUCT), 1})
+
+ require.NotPanics(t, func() {
+ test.read(ctx)
+ })
+ require.True(t, ctx.HasError())
+ require.Contains(t, ctx.Err().Error(), "invalid dynamic
index")
+ })
+ }
+}
diff --git
a/java/fory-core/src/main/java/org/apache/fory/meta/MetaStringDecoder.java
b/java/fory-core/src/main/java/org/apache/fory/meta/MetaStringDecoder.java
index 69650ee1c..6628f3b84 100644
--- a/java/fory-core/src/main/java/org/apache/fory/meta/MetaStringDecoder.java
+++ b/java/fory-core/src/main/java/org/apache/fory/meta/MetaStringDecoder.java
@@ -166,6 +166,10 @@ public class MetaStringDecoder {
char[] chars = str.toCharArray();
for (int i = 0; i < chars.length; i++) {
if (chars[i] == '|') {
+ if (i + 1 >= chars.length) {
+ throw new IllegalArgumentException(
+ "Invalid ALL_TO_LOWER_SPECIAL meta string: trailing escape");
+ }
char c = chars[++i];
builder.append(Character.toUpperCase(c));
} else {
diff --git
a/java/fory-core/src/main/java/org/apache/fory/meta/NativeTypeDefDecoder.java
b/java/fory-core/src/main/java/org/apache/fory/meta/NativeTypeDefDecoder.java
index 60bf518dd..a69089586 100644
---
a/java/fory-core/src/main/java/org/apache/fory/meta/NativeTypeDefDecoder.java
+++
b/java/fory-core/src/main/java/org/apache/fory/meta/NativeTypeDefDecoder.java
@@ -329,7 +329,11 @@ class NativeTypeDefDecoder {
boolean useTagID = encodingFlags == 3;
int size = header >>> 4;
if (size == 7) {
- size += buffer.readVarUInt32Small7();
+ int extendedSize = buffer.readVarUInt32Small7();
+ if (extendedSize < 0 || extendedSize > Integer.MAX_VALUE - size - 1) {
+ throw new DeserializationException("Invalid TypeDef field name
size");
+ }
+ size += extendedSize;
}
size += 1;
@@ -369,7 +373,7 @@ class NativeTypeDefDecoder {
// The `6 bits size: 0~63` will be used to indicate size `0~63`,
// the value `63` the size need more byte to read, the encoding will
encode `size - 63` as
// a varint next.
- return readName(Encoders.PACKAGE_DECODER, buffer, pkgEncodings);
+ return readName(Encoders.PACKAGE_DECODER, buffer, pkgEncodings,
"namespace");
}
static String readTypeName(MemoryBuffer buffer) {
@@ -380,17 +384,25 @@ class NativeTypeDefDecoder {
// The `6 bits size: 0~63` will be used to indicate size `0~63`,
// the value `63` the size need more byte to read, the encoding will
encode `size - 63` as
// a varint next.
- return readName(Encoders.TYPE_NAME_DECODER, buffer, typeNameEncodings);
+ return readName(Encoders.TYPE_NAME_DECODER, buffer, typeNameEncodings,
"type name");
}
private static String readName(
- MetaStringDecoder decoder, MemoryBuffer buffer, Encoding[] encodings) {
+ MetaStringDecoder decoder, MemoryBuffer buffer, Encoding[] encodings,
String nameKind) {
int header = buffer.readByte() & 0xff;
int encodingFlags = header & 0b11;
+ if (encodingFlags >= encodings.length) {
+ throw new DeserializationException(
+ "Invalid TypeDef " + nameKind + " encoding " + encodingFlags);
+ }
Encoding encoding = encodings[encodingFlags];
int size = header >> 2;
if (size == BIG_NAME_THRESHOLD) {
- size = buffer.readVarUInt32Small7() + BIG_NAME_THRESHOLD;
+ int extendedSize = buffer.readVarUInt32Small7();
+ if (extendedSize < 0 || extendedSize > Integer.MAX_VALUE -
BIG_NAME_THRESHOLD) {
+ throw new DeserializationException("Invalid TypeDef " + nameKind + "
size");
+ }
+ size = extendedSize + BIG_NAME_THRESHOLD;
}
return decoder.decode(buffer.readBytes(size), encoding);
}
diff --git
a/java/fory-core/src/main/java/org/apache/fory/meta/TypeDefDecoder.java
b/java/fory-core/src/main/java/org/apache/fory/meta/TypeDefDecoder.java
index 587e36919..9801842e4 100644
--- a/java/fory-core/src/main/java/org/apache/fory/meta/TypeDefDecoder.java
+++ b/java/fory-core/src/main/java/org/apache/fory/meta/TypeDefDecoder.java
@@ -222,7 +222,11 @@ class TypeDefDecoder {
boolean useTagID = encodingFlags == 3;
int fieldNameSize = (header >>> 2) & 0b1111;
if (fieldNameSize == FIELD_NAME_SIZE_THRESHOLD) {
- fieldNameSize += buffer.readVarUInt32Small7();
+ int extendedSize = buffer.readVarUInt32Small7();
+ if (extendedSize < 0 || extendedSize > Integer.MAX_VALUE -
fieldNameSize - 1) {
+ throw new DeserializationException("Invalid TypeDef field name
size");
+ }
+ fieldNameSize += extendedSize;
}
fieldNameSize += 1;
boolean nullable = (header & 0b10) != 0;
diff --git
a/java/fory-core/src/test/java/org/apache/fory/meta/MetaStringTest.java
b/java/fory-core/src/test/java/org/apache/fory/meta/MetaStringTest.java
index ebc943867..1c5f1ae0f 100644
--- a/java/fory-core/src/test/java/org/apache/fory/meta/MetaStringTest.java
+++ b/java/fory-core/src/test/java/org/apache/fory/meta/MetaStringTest.java
@@ -23,6 +23,7 @@ import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertNotSame;
import static org.testng.Assert.assertTrue;
+import static org.testng.Assert.expectThrows;
import static org.testng.AssertJUnit.assertSame;
import org.apache.fory.util.StringUtils;
@@ -143,6 +144,20 @@ public class MetaStringTest {
assertEquals(decoded, "");
}
+ @Test
+ public void testTrailingEscape() {
+ MetaStringEncoder encoder = new MetaStringEncoder('_', '$');
+ MetaStringDecoder decoder = new MetaStringDecoder('_', '$');
+ byte[] encoded = encoder.encodeLowerSpecial("|");
+
+ IllegalArgumentException exception =
+ expectThrows(
+ IllegalArgumentException.class,
+ () -> decoder.decode(encoded,
MetaString.Encoding.ALL_TO_LOWER_SPECIAL));
+ assertEquals(
+ exception.getMessage(), "Invalid ALL_TO_LOWER_SPECIAL meta string:
trailing escape");
+ }
+
@Test
public void testEncodeCharactersOutsideOfLowerSpecial() {
// Contains characters outside LOWER_SPECIAL
diff --git
a/java/fory-core/src/test/java/org/apache/fory/meta/NativeTypeDefEncoderTest.java
b/java/fory-core/src/test/java/org/apache/fory/meta/NativeTypeDefEncoderTest.java
index c18fcf20e..86dcc1a34 100644
---
a/java/fory-core/src/test/java/org/apache/fory/meta/NativeTypeDefEncoderTest.java
+++
b/java/fory-core/src/test/java/org/apache/fory/meta/NativeTypeDefEncoderTest.java
@@ -425,6 +425,80 @@ public class NativeTypeDefEncoderTest {
RuntimeException.class, () ->
TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
}
+ @Test
+ public void testRejectsNamespaceEncoding() {
+ Fory fory =
+ Fory.builder()
+ .withXlang(false)
+ .withMetaShare(true)
+ .requireClassRegistration(false)
+ .withCompatible(false)
+ .build();
+ MemoryBuffer body = MemoryBuffer.newHeapBuffer(3);
+ body.writeByte(NativeTypeDefEncoder.nativeKindCode(Types.NAMED_STRUCT) <<
4);
+ body.writeVarUInt32Small7(0);
+ body.writeByte(0b11);
+ MemoryBuffer encoded = NativeTypeDefEncoder.prependHeader(body, false);
+
+ DeserializationException exception =
+ Assert.expectThrows(
+ DeserializationException.class,
+ () -> TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
+ Assert.assertTrue(exception.getMessage().contains("namespace encoding"));
+ }
+
+ @Test
+ public void testRejectsExtendedNameSize() {
+ Fory fory =
+ Fory.builder()
+ .withXlang(false)
+ .withMetaShare(true)
+ .requireClassRegistration(false)
+ .withCompatible(false)
+ .build();
+ for (int extendedSize : new int[] {-1, Integer.MAX_VALUE}) {
+ MemoryBuffer body = MemoryBuffer.newHeapBuffer(8);
+ body.writeByte(NativeTypeDefEncoder.nativeKindCode(Types.NAMED_STRUCT)
<< 4);
+ body.writeVarUInt32Small7(0);
+ body.writeByte(NativeTypeDefEncoder.BIG_NAME_THRESHOLD << 2);
+ body.writeVarUInt32Small7(extendedSize);
+ MemoryBuffer encoded = NativeTypeDefEncoder.prependHeader(body, false);
+
+ DeserializationException exception =
+ Assert.expectThrows(
+ DeserializationException.class,
+ () -> TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
+ Assert.assertTrue(exception.getMessage().contains("namespace size"));
+ }
+ }
+
+ @Test
+ public void testRejectsExtendedFieldNameSize() {
+ Fory fory =
+ Fory.builder()
+ .withXlang(false)
+ .withMetaShare(true)
+ .requireClassRegistration(false)
+ .withCompatible(false)
+ .build();
+ for (int extendedSize : new int[] {-1, Integer.MAX_VALUE}) {
+ MemoryBuffer body = MemoryBuffer.newHeapBuffer(16);
+ body.writeByte(NativeTypeDefEncoder.nativeKindCode(Types.STRUCT) << 4);
+ body.writeVarUInt32Small7(3);
+ body.writeUInt8(Types.STRUCT);
+ body.writeVarUInt32(0);
+ body.writeByte(7 << 4);
+ body.writeVarUInt32Small7(extendedSize);
+ MemoryBuffer encoded = NativeTypeDefEncoder.prependHeader(body, false);
+
+ DeserializationException exception =
+ Assert.expectThrows(
+ DeserializationException.class,
+ () -> TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
+ Assert.assertTrue(exception.getMessage().contains("field name size"));
+ }
+ }
+
private static byte[] corruptEncodedBody(TypeDef typeDef, String needle) {
byte[] malformed = typeDef.getEncoded().clone();
byte[] needleBytes = Encoders.encodeFieldName(needle).getBytes();
diff --git
a/java/fory-core/src/test/java/org/apache/fory/meta/TypeDefEncoderTest.java
b/java/fory-core/src/test/java/org/apache/fory/meta/TypeDefEncoderTest.java
index 3221d176a..8f4bff69f 100644
--- a/java/fory-core/src/test/java/org/apache/fory/meta/TypeDefEncoderTest.java
+++ b/java/fory-core/src/test/java/org/apache/fory/meta/TypeDefEncoderTest.java
@@ -710,6 +710,59 @@ public class TypeDefEncoderTest {
RuntimeException.class, () ->
TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
}
+ @Test
+ public void testRejectsNamespaceEncoding() {
+ Fory fory =
Fory.builder().withXlang(true).withCompatible(false).withMetaShare(true).build();
+ MemoryBuffer body = MemoryBuffer.newHeapBuffer(2);
+ body.writeByte(TypeDefEncoder.STRUCT_FLAG |
TypeDefEncoder.REGISTER_BY_NAME_FLAG);
+ body.writeByte(0b11);
+ MemoryBuffer encoded = NativeTypeDefEncoder.prependHeader(body, false);
+
+ DeserializationException exception =
+ Assert.expectThrows(
+ DeserializationException.class,
+ () -> TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
+ Assert.assertTrue(exception.getMessage().contains("namespace encoding"));
+ }
+
+ @Test
+ public void testRejectsExtendedNameSize() {
+ Fory fory =
Fory.builder().withXlang(true).withCompatible(false).withMetaShare(true).build();
+ for (int extendedSize : new int[] {-1, Integer.MAX_VALUE}) {
+ MemoryBuffer body = MemoryBuffer.newHeapBuffer(8);
+ body.writeByte(TypeDefEncoder.STRUCT_FLAG |
TypeDefEncoder.REGISTER_BY_NAME_FLAG);
+ body.writeByte(NativeTypeDefEncoder.BIG_NAME_THRESHOLD << 2);
+ body.writeVarUInt32Small7(extendedSize);
+ MemoryBuffer encoded = NativeTypeDefEncoder.prependHeader(body, false);
+
+ DeserializationException exception =
+ Assert.expectThrows(
+ DeserializationException.class,
+ () -> TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
+ Assert.assertTrue(exception.getMessage().contains("namespace size"));
+ }
+ }
+
+ @Test
+ public void testRejectsExtendedFieldNameSize() {
+ Fory fory =
Fory.builder().withXlang(true).withCompatible(false).withMetaShare(true).build();
+ for (int extendedSize : new int[] {-1, Integer.MAX_VALUE}) {
+ MemoryBuffer body = MemoryBuffer.newHeapBuffer(16);
+ body.writeByte(TypeDefEncoder.STRUCT_FLAG | 1);
+ body.writeVarUInt32(0);
+ body.writeByte(TypeDefEncoder.FIELD_NAME_SIZE_THRESHOLD << 2);
+ body.writeVarUInt32Small7(extendedSize);
+ body.writeUInt8(Types.INT32);
+ MemoryBuffer encoded = NativeTypeDefEncoder.prependHeader(body, false);
+
+ DeserializationException exception =
+ Assert.expectThrows(
+ DeserializationException.class,
+ () -> TypeDef.readTypeDef(fory.getTypeResolver(), encoded));
+ Assert.assertTrue(exception.getMessage().contains("field name size"));
+ }
+ }
+
@Test
public void testSkipTypeDefRejectsExtendedSizeOverflow() {
MemoryBuffer buffer = MemoryBuffer.newHeapBuffer(8);
diff --git a/javascript/packages/core/lib/context.ts
b/javascript/packages/core/lib/context.ts
index f2613c7bc..0eab53138 100644
--- a/javascript/packages/core/lib/context.ts
+++ b/javascript/packages/core/lib/context.ts
@@ -263,7 +263,7 @@ export class MetaStringWriter {
writeBytes(writer: BinaryWriter, bytes: MetaStringBytes) {
if (bytes.dynamicWriteStringId !== -1) {
- writer.writeVarUInt32(((this.dynamicNameId + 1) << 1) | 1);
+ writer.writeVarUInt32(((bytes.dynamicWriteStringId + 1) << 1) | 1);
} else {
bytes.dynamicWriteStringId = this.dynamicNameId;
this.dynamicNameId += 1;
@@ -301,7 +301,7 @@ export class MetaStringReader {
readTypeName(reader: BinaryReader) {
const idOrLen = reader.readVarUInt32();
if (idOrLen & 1) {
- return this.names[idOrLen >> 1];
+ return this.names[(idOrLen >>> 1) - 1];
}
const len = idOrLen >> 1;
if (len === 0) {
@@ -317,7 +317,7 @@ export class MetaStringReader {
readNamespace(reader: BinaryReader) {
const idOrLen = reader.readVarUInt32();
if (idOrLen & 1) {
- return this.names[idOrLen >> 1];
+ return this.names[(idOrLen >>> 1) - 1];
}
const len = idOrLen >> 1;
if (len === 0) {
diff --git a/javascript/packages/core/lib/meta/MetaString.ts
b/javascript/packages/core/lib/meta/MetaString.ts
index 940b1aed8..6b5d442ab 100644
--- a/javascript/packages/core/lib/meta/MetaString.ts
+++ b/javascript/packages/core/lib/meta/MetaString.ts
@@ -221,6 +221,9 @@ export class MetaStringDecoder {
const chars = [...str];
for (let i = 0; i < chars.length; i++) {
if (chars[i] === "|") {
+ if (i + 1 >= chars.length) {
+ throw new Error("Malformed ALL_TO_LOWER_SPECIAL encoding: trailing
uppercase marker");
+ }
const c = chars[++i];
builder.push(c.toUpperCase());
} else {
diff --git a/javascript/test/metastring.test.ts
b/javascript/test/metastring.test.ts
new file mode 100644
index 000000000..6157ead85
--- /dev/null
+++ b/javascript/test/metastring.test.ts
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { MetaStringReader, MetaStringWriter } from
"../packages/core/lib/context";
+import {
+ Encoding,
+ MetaStringDecoder,
+ MetaStringEncoder,
+} from "../packages/core/lib/meta/MetaString";
+import { BinaryReader } from "../packages/core/lib/reader";
+import { BinaryWriter } from "../packages/core/lib/writer";
+import { describe, expect, test } from "@jest/globals";
+
+function readerFor(bytes: Uint8Array): BinaryReader {
+ const reader = new BinaryReader({});
+ reader.reset(bytes);
+ return reader;
+}
+
+describe("meta string", () => {
+ test("rejects a trailing uppercase marker", () => {
+ const encoded = new MetaStringEncoder("$", "_").encodeByEncoding(
+ "|",
+ Encoding.ALL_TO_LOWER_SPECIAL,
+ );
+ const reader = readerFor(encoded.getBytes());
+ const decoder = new MetaStringDecoder("$", "_");
+
+ expect(() =>
+ decoder.decode(reader, encoded.getBytes().length,
Encoding.ALL_TO_LOWER_SPECIAL),
+ ).toThrow("Malformed ALL_TO_LOWER_SPECIAL encoding: trailing uppercase
marker");
+ });
+
+ test("round trips dynamic references", () => {
+ const metaStringWriter = new MetaStringWriter();
+ const first = metaStringWriter.encodeTypeName("first");
+ const second = metaStringWriter.encodeTypeName("second");
+ const writer = new BinaryWriter({});
+ metaStringWriter.writeBytes(writer, first);
+ metaStringWriter.writeBytes(writer, second);
+ metaStringWriter.writeBytes(writer, first);
+
+ const reader = readerFor(writer.dump());
+ const metaStringReader = new MetaStringReader();
+ expect(metaStringReader.readTypeName(reader)).toBe("first");
+ expect(metaStringReader.readTypeName(reader)).toBe("second");
+ expect(metaStringReader.readTypeName(reader)).toBe("first");
+ });
+});
diff --git a/python/pyfory/meta/metastring.py b/python/pyfory/meta/metastring.py
index a3832be9e..7cde8f46a 100644
--- a/python/pyfory/meta/metastring.py
+++ b/python/pyfory/meta/metastring.py
@@ -17,7 +17,9 @@
from collections import namedtuple
from enum import Enum
-from typing import List
+from typing import List, Optional
+
+# Python 3.8 must be able to evaluate these public annotations at runtime.
class Encoding(Enum):
@@ -249,6 +251,8 @@ class MetaStringDecoder:
skip = False
continue
if char == "|":
+ if i + 1 >= len(decoded_str):
+ raise ValueError("Invalid ALL_TO_LOWER_SPECIAL meta
string: trailing escape")
result.append(decoded_str[i + 1].upper())
skip = True
else:
@@ -268,7 +272,7 @@ class MetaStringEncoder:
self.special_char1 = special_char1
self.special_char2 = special_char2
- def encode(self, input_string: str, encodings: List[Encoding] = None) ->
MetaString:
+ def encode(self, input_string: str, encodings: Optional[List[Encoding]] =
None) -> MetaString:
"""
Encodes the input string into a MetaString object.
@@ -285,7 +289,7 @@ class MetaStringEncoder:
return MetaString(
input_string,
Encoding.UTF_8,
- bytes(),
+ b"",
0,
self.special_char1,
self.special_char2,
@@ -312,7 +316,7 @@ class MetaStringEncoder:
return MetaString(
input_string,
Encoding.UTF_8,
- bytes(),
+ b"",
0,
self.special_char1,
self.special_char2,
@@ -372,7 +376,7 @@ class MetaStringEncoder:
self.special_char2,
)
- def compute_encoding(self, input_string: str, encodings: List[Encoding] =
None) -> Encoding:
+ def compute_encoding(self, input_string: str, encodings:
Optional[List[Encoding]] = None) -> Encoding:
"""
Determines the encoding type of the input string.
@@ -422,12 +426,12 @@ class MetaStringEncoder:
digit_count = 0
upper_count = 0
for c in chars:
- if can_lower_upper_digit_special_encoded:
- if not (c.islower() or c.isupper() or c.isdigit() or c in
{self.special_char1, self.special_char2}):
- can_lower_upper_digit_special_encoded = False
- if can_lower_special_encoded:
- if not (c.islower() or c in {".", "_", "$", "|"}):
- can_lower_special_encoded = False
+ if can_lower_upper_digit_special_encoded and not (
+ c.islower() or c.isupper() or c.isdigit() or c in
{self.special_char1, self.special_char2}
+ ):
+ can_lower_upper_digit_special_encoded = False
+ if can_lower_special_encoded and not (c.islower() or c in {".",
"_", "$", "|"}):
+ can_lower_special_encoded = False
if c.isdigit():
digit_count += 1
if c.isupper():
diff --git a/python/pyfory/meta/typedef_decoder.py
b/python/pyfory/meta/typedef_decoder.py
index 4c410d833..a1be8a443 100644
--- a/python/pyfory/meta/typedef_decoder.py
+++ b/python/pyfory/meta/typedef_decoder.py
@@ -22,7 +22,10 @@ This module implements the decoding of TypeDef objects
according to the xlang se
"""
from dataclasses import make_dataclass
-from typing import List, Any
+from typing import Any, List
+
+# Python 3.8 must be able to evaluate these annotations at runtime.
+
from pyfory.serialization import Buffer
from pyfory.meta.typedef import TypeDef, FieldInfo, FieldType
from pyfory.meta.typedef import (
@@ -226,15 +229,15 @@ def _validate_parsed_typedef_hash(header: int,
encoded_meta_data: bytes) -> None
def read_namespace(buffer: Buffer) -> str:
"""Read namespace from the buffer."""
- return read_meta_string(buffer, NAMESPACE_DECODER, NAMESPACE_ENCODINGS)
+ return read_meta_string(buffer, NAMESPACE_DECODER, NAMESPACE_ENCODINGS,
"namespace")
def read_typename(buffer: Buffer) -> str:
"""Read typename from the buffer."""
- return read_meta_string(buffer, TYPENAME_DECODER, TYPE_NAME_ENCODINGS)
+ return read_meta_string(buffer, TYPENAME_DECODER, TYPE_NAME_ENCODINGS,
"type name")
-def read_meta_string(buffer: Buffer, decoder: MetaStringDecoder, encodings:
List[Encoding]) -> str:
+def read_meta_string(buffer: Buffer, decoder: MetaStringDecoder, encodings:
List[Encoding], name_kind: str) -> str:
"""Read a big meta string (namespace/typename) from the buffer using 6-bit
size field."""
# Read encoding and length combined in first byte
header = buffer.read_uint8()
@@ -243,6 +246,8 @@ def read_meta_string(buffer: Buffer, decoder:
MetaStringDecoder, encodings: List
encoding_value = header & 0b11
size_value = (header >> 2) & 0b111111
+ if encoding_value >= len(encodings):
+ raise ValueError(f"Invalid TypeDef {name_kind} encoding
{encoding_value}")
encoding = encodings[encoding_value]
# Read length - same logic as encoder
diff --git a/python/pyfory/tests/test_metastring.py
b/python/pyfory/tests/test_metastring.py
index d470de299..32199a9d0 100644
--- a/python/pyfory/tests/test_metastring.py
+++ b/python/pyfory/tests/test_metastring.py
@@ -81,16 +81,13 @@ def test_metastring():
decoder = MetaStringDecoder(special_char1=".", special_char2="_")
for i in range(1, 128):
- try:
- string = create_string(i)
- metastring = encoder.encode(string)
- assert metastring.encoding != Encoding.UTF_8
- assert metastring.original == string
+ string = create_string(i)
+ metastring = encoder.encode(string)
+ assert metastring.encoding != Encoding.UTF_8, f"Failed at {i}"
+ assert metastring.original == string, f"Failed at {i}"
- new_string = decoder.decode(metastring.encoded_data,
metastring.encoding)
- assert new_string == string
- except Exception as e:
- pytest.fail(f"Failed at {i} with exception: {str(e)}")
+ new_string = decoder.decode(metastring.encoded_data,
metastring.encoding)
+ assert new_string == string, f"Failed at {i}"
def test_encode_empty_string():
@@ -110,6 +107,15 @@ def test_encode_empty_string():
assert decoded == ""
+def test_trailing_escape():
+ encoder = MetaStringEncoder(special_char1=".", special_char2="_")
+ decoder = MetaStringDecoder(special_char1=".", special_char2="_")
+ encoded = encoder._encode_lower_special("|")
+
+ with pytest.raises(ValueError, match="ALL_TO_LOWER_SPECIAL"):
+ decoder.decode(encoded, Encoding.ALL_TO_LOWER_SPECIAL)
+
+
def test_encode_characters_outside_of_lower_special():
encoder = MetaStringEncoder(special_char1=".", special_char2="_")
@@ -168,7 +174,7 @@ def test_empty_string():
decoder = MetaStringDecoder(special_char1=".", special_char2="_")
metastring = encoder.encode("")
- assert metastring.encoded_data == bytes()
+ assert metastring.encoded_data == b""
decoded = decoder.decode(metastring.encoded_data, metastring.encoding)
assert decoded == ""
diff --git a/python/pyfory/tests/test_typedef_encoding.py
b/python/pyfory/tests/test_typedef_encoding.py
index 2fed75b8a..0416bb6ed 100644
--- a/python/pyfory/tests/test_typedef_encoding.py
+++ b/python/pyfory/tests/test_typedef_encoding.py
@@ -22,12 +22,14 @@ Tests for xlang TypeDef implementation.
import array
import enum
from dataclasses import dataclass, make_dataclass
-from typing import List, Dict, Optional
+from typing import Dict, List, Optional
+
+# Fory resolves these model annotations at runtime, so keep Python
3.8-compatible typing aliases.
import pytest
import pyfory
-import pyfory.meta.typedef_decoder as typedef_decoder
+from pyfory.meta import typedef_decoder
from pyfory.serialization import Buffer
from pyfory.meta.typedef import (
TypeDef,
@@ -38,6 +40,8 @@ from pyfory.meta.typedef import (
DynamicFieldType,
FIELD_NAME_ENCODINGS,
COMPRESS_META_FLAG,
+ REGISTER_BY_NAME_FLAG,
+ STRUCT_TYPEDEF_FLAG,
META_SIZE_MASKS,
TYPEDEF_HASH_MASK,
TYPEDEF_HASH_SHIFT,
@@ -344,6 +348,15 @@ def
test_decode_typedef_rejects_hash_consistent_malformed_body():
decode_typedef(Buffer(encoded), fory.type_resolver)
+def test_namespace_encoding():
+ fory = Fory(xlang=True, compatible=False)
+ body = bytes([STRUCT_TYPEDEF_FLAG | REGISTER_BY_NAME_FLAG, 0b11])
+ encoded = prepend_header(body, False)
+
+ with pytest.raises(ValueError, match="Invalid TypeDef namespace encoding"):
+ decode_typedef(Buffer(encoded), fory.type_resolver)
+
+
def test_decode_typedef_rejects_compressed_xlang_metadata():
fory = Fory(xlang=True, compatible=False)
fory.register(SimpleTypeDef, name="example.SimpleTypeDef")
diff --git a/python/pyproject.toml b/python/pyproject.toml
index e0d4d0352..4fdedcc19 100644
--- a/python/pyproject.toml
+++ b/python/pyproject.toml
@@ -97,6 +97,7 @@ ignore = [
"C408", "E203", "E225", "E226", "E227", "E402", "E24",
"W605", "I", "N", "B002", "B003", "B004", "B005",
"B007", "B008", "B009", "B010", "B011", "B012", "B013", "B014", "B015",
"B016", "B017",
+ "UP006", "UP035", "UP045",
]
[tool.ruff.lint.per-file-ignores]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]