[
https://issues.apache.org/jira/browse/AVRO-3225?focusedWorklogId=663170&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-663170
]
ASF GitHub Bot logged work on AVRO-3225:
----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Oct/21 11:57
Start Date: 09/Oct/21 11:57
Worklog Time Spent: 10m
Work Description: blachniet commented on a change in pull request #1357:
URL: https://github.com/apache/avro/pull/1357#discussion_r725478365
##########
File path: lang/csharp/src/apache/test/IO/BinaryCodecTests.cs
##########
@@ -216,21 +217,98 @@ public void TestString(string n, int overhead)
#if NETCOREAPP3_1
[Test]
- public void TestLargeString()
+ public void TestStringReadIntoArrayPool()
{
+ const int maxFastReadLength = 4096;
+
// Create a 16KB buffer in the Array Pool
var largeBufferToSeedPool = ArrayPool<byte>.Shared.Rent(2 << 14);
ArrayPool<byte>.Shared.Return(largeBufferToSeedPool);
- // Create a slightly less than 16KB buffer, which will use the
16KB buffer in the pool
- var n = string.Concat(Enumerable.Repeat("1234567890", 1600));
- var overhead = 3;
+ var n = string.Concat(Enumerable.Repeat("A", maxFastReadLength));
+ var overhead = 2;
TestRead(n, (Decoder d) => d.ReadString(), (Encoder e, string t)
=> e.WriteString(t), overhead + n.Length);
- TestSkip(n, (Decoder d) => d.SkipString(), (Encoder e, string t)
=> e.WriteString(t), overhead + n.Length);
+ }
+
+ [Test]
+ public void TestStringReadByBinaryReader()
+ {
+ const int overhead = 2;
+ const int maxFastReadLength = 4096;
+ const int expectedStringLength = maxFastReadLength + 1;
+ var n = string.Concat(Enumerable.Repeat("A",
expectedStringLength));
+
+ TestRead(n, (Decoder d) => d.ReadString(), (Encoder e, string t)
=> e.WriteString(t), expectedStringLength + overhead);
}
#endif
+ [Test]
+ public void TestInvalidInputWithNegativeStringLength()
+ {
+ using (MemoryStream iostr = new MemoryStream())
+ {
+ Encoder e = new BinaryEncoder(iostr);
+
+ e.WriteLong(-1);
+
+ iostr.Flush();
+ iostr.Position = 0;
+ Decoder d = new BinaryDecoder(iostr);
+
+ var exception = Assert.Throws<AvroException>(() =>
d.ReadString());
+
+ Assert.NotNull(exception);
+ Assert.AreEqual("Can not deserialize a string with negative
length!", exception.Message);
+ iostr.Close();
+ }
+ }
+
+ [Test]
+ public void TestInvalidInputWithMaxIntAsStringLength()
+ {
+ using (MemoryStream iostr = new MemoryStream())
+ {
+ Encoder e = new BinaryEncoder(iostr);
+
+ e.WriteLong(int.MaxValue);
+ e.WriteBytes(Encoding.UTF8.GetBytes("SomeSmallString"));
+
+ iostr.Flush();
+ iostr.Position = 0;
+ Decoder d = new BinaryDecoder(iostr);
+
+ var exception = Assert.Throws<AvroException>(() =>
d.ReadString());
+
+ Assert.NotNull(exception);
+ Assert.AreEqual("String length is not supported!",
exception.Message);
+ iostr.Close();
+ }
+ }
+
+ [Test]
+ public void TestInvalidInputWithMaxArrayLengthAsStringLength()
Review comment:
This test is failing when running on .NET Framework 4.6.1. It looks like
we aren't running the Windows builds/tests in GitHub actions anymore, which
would explain why that didn't catch it.
```
+ dotnet test --configuration Release --no-build --filter
'TestCategory!=Interop' Avro.sln
Test run for
C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\bin\Release\net461\Avro.test.dll
(.NETFramework,Version=v4.6.1)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestInvalidInputWithMaxArrayLengthAsStringLength [75 ms]
Error Message:
Expected: <Avro.AvroException>
But was: <System.OutOfMemoryException: Exception of type
'System.OutOfMemoryException' was thrown.
at System.IO.BinaryReader.ReadBytes(Int32 count)
at Avro.IO.BinaryDecoder.ReadString() in
C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\main\IO\BinaryDecoder.netstandard2.0.cs:line
90
at
Avro.Test.BinaryCodecTests.<>c__DisplayClass11_0.<TestInvalidInputWithMaxArrayLengthAsStringLength>b__0()
in
C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\IO\BinaryCodecTests.cs:line
304
at NUnit.Framework.Assert.Throws(IResolveConstraint expression,
TestDelegate code, String message, Object[] args)>
Stack Trace:
at
Avro.Test.BinaryCodecTests.TestInvalidInputWithMaxArrayLengthAsStringLength()
in
C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\IO\BinaryCodecTests.cs:line
304
Failed! - Failed: 1, Passed: 626, Skipped: 0, Total: 627,
Duration: 3 s - Avro.test.dll (net461)
Test run for
C:\Users\Brian.Lachniet\code\github.com\apache\avro\lang\csharp\src\apache\test\bin\Release\netcoreapp2.1\Avro.test.dll
(.NETCoreApp,Version=v2.1)
```
The [Remarks on the Array
Class](https://docs.microsoft.com/en-us/dotnet/api/system.array?view=net-5.0#remarks)
indicate that the .NET Framework may have a smaller size limit. However, even
after setting `MaxDotNetArrayLength = 0x77359400;`, I'm getting the same test
failure (only on the net461 tests).
@RyanSkraba , how does the Java implementation handle large strings? Does it
specify some arbitrary limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 663170)
Time Spent: 40m (was: 0.5h)
> StackOverflowException on invalid input for BinaryDecoder.ReadString on
> NetStandard 2.1+
> ----------------------------------------------------------------------------------------
>
> Key: AVRO-3225
> URL: https://issues.apache.org/jira/browse/AVRO-3225
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
> Reporter: Philip Sanetra
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The BinaryDecoder.ReadString() method on NetStandard2.1+ produces a stack
> overflow exception if there is invalid input caused by this code:
>
> {code:java}
> int length = ReadInt();
> Span<byte> buffer = length <= StackallocThreshold ? stackalloc byte[length] :
> (bufferArray = ArrayPool<byte>.Shared.Rent(length)).AsSpan(0,
> length);
> {code}
>
> This code fails if ReadInt() returns a negative value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
