[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[James E. King, III (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258259#comment-16258259
 ] 

James E. King, III commented on THRIFT-4372:


I added csharp as a multiplexed client for cross testing in some work I am 
doing and the "huge" send interacts with the rs server badly.  I'll disable the 
tests for now; it seems to be the rs server, since the csharp client appears to 
be okay against other languages.

> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
> Fix For: 0.11.0
>
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221324#comment-16221324
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user jeking3 commented on the issue:

https://github.com/apache/thrift/pull/1402
  
Makes sense, thanks for the info.  Every little bit helps us all improve 
what we do next.


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
> Fix For: 0.11.0
>
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221023#comment-16221023
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user Jens-G commented on the issue:

https://github.com/apache/thrift/pull/1402
  
> Why not actually use (2^16)-1 which is the limit?

Several reasons. First, aligned memory access is always faster. If we 
subtract 1 byte, we get the worst case. Next, at least on Windows a number of 
system resources use multiples of 4096, so it is probably not a bad idea to 
follow that model. That's why I picked this particular value.


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
> Fix For: 0.11.0
>
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220978#comment-16220978
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user asfgit closed the pull request at:

https://github.com/apache/thrift/pull/1402


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220653#comment-16220653
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user jeking3 commented on a diff in the pull request:

https://github.com/apache/thrift/pull/1402#discussion_r147185064
  
--- Diff: lib/csharp/src/Transport/TNamedPipeClientTransport.cs ---
@@ -88,7 +89,18 @@ public override void Write(byte[] buf, int off, int len)
 throw new 
TTransportException(TTransportException.ExceptionType.NotOpen);
 }
 
-client.Write(buf, off, len);
+// if necessary, send the data in chunks
+// there's a system limit around 0x1 bytes that we hit 
otherwise
+// MSDN: "Pipe write operations across a network are limited 
to 65,535 bytes per write. For more information regarding pipes, see the 
Remarks section."
+var nBytes = Math.Min(len, 15 * 4096);  // 16 would exceed the 
limit
--- End diff --

Why not actually use `(2^16)-1` which is the limit?


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220655#comment-16220655
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user jeking3 commented on a diff in the pull request:

https://github.com/apache/thrift/pull/1402#discussion_r147185494
  
--- Diff: lib/csharp/src/Transport/TNamedPipeServerTransport.cs ---
@@ -239,40 +239,51 @@ public override void Write(byte[] buf, int off, int 
len)
 throw new 
TTransportException(TTransportException.ExceptionType.NotOpen);
 }
 
-if (asyncMode)
+// if necessary, send the data in chunks
+// there's a system limit around 0x1 bytes that we hit 
otherwise
+// MSDN: "Pipe write operations across a network are 
limited to 65,535 bytes per write. For more information regarding pipes, see 
the Remarks section."
+var nBytes = Math.Min(len, 15 * 4096);  // 16 would exceed 
the limit
--- End diff --

Why not actually use (2^16)-1 which is the limit?


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220656#comment-16220656
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user jeking3 commented on a diff in the pull request:

https://github.com/apache/thrift/pull/1402#discussion_r147185326
  
--- Diff: lib/csharp/src/Transport/TNamedPipeClientTransport.cs ---
@@ -88,7 +89,18 @@ public override void Write(byte[] buf, int off, int len)
 throw new 
TTransportException(TTransportException.ExceptionType.NotOpen);
 }
 
-client.Write(buf, off, len);
+// if necessary, send the data in chunks
+// there's a system limit around 0x1 bytes that we hit 
otherwise
+// MSDN: "Pipe write operations across a network are limited 
to 65,535 bytes per write. For more information regarding pipes, see the 
Remarks section."
+var nBytes = Math.Min(len, 15 * 4096);  // 16 would exceed the 
limit
+while (nBytes > 0)
+{
+client.Write(buf, off, nBytes);
+
+off += nBytes;
+len -= nBytes;
+nBytes = Math.Min(len, nBytes);
--- End diff --

Shouldn't this be the same calculation as the one outside the loop?  
Technically it doesn't have to be, however I found this confusing to read.


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the writing end  acts as if it had written all the bytes (in fact, it did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the reading end
> Consequently, the process at the reading end of the pipe gets stuck while 
> waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220654#comment-16220654
 ] 

ASF GitHub Bot commented on THRIFT-4372:


Github user jeking3 commented on a diff in the pull request:

https://github.com/apache/thrift/pull/1402#discussion_r147185736
  
--- Diff: lib/csharp/src/Transport/TNamedPipeServerTransport.cs ---
@@ -239,40 +239,51 @@ public override void Write(byte[] buf, int off, int 
len)
 throw new 
TTransportException(TTransportException.ExceptionType.NotOpen);
 }
 
-if (asyncMode)
+// if necessary, send the data in chunks
+// there's a system limit around 0x1 bytes that we hit 
otherwise
+// MSDN: "Pipe write operations across a network are 
limited to 65,535 bytes per write. For more information regarding pipes, see 
the Remarks section."
+var nBytes = Math.Min(len, 15 * 4096);  // 16 would exceed 
the limit
+while (nBytes > 0)
 {
-Exception eOuter = null;
-var evt = new ManualResetEvent(false);
 
-stream.BeginWrite(buf, off, len, asyncResult =>
+if (asyncMode)
 {
-try
-{
-if (stream != null)
-stream.EndWrite(asyncResult);
-else
-eOuter = new 
TTransportException(TTransportException.ExceptionType.Interrupted);
-}
-catch (Exception e)
-{
-if (stream != null)
-eOuter = e;
-else
-eOuter = new 
TTransportException(TTransportException.ExceptionType.Interrupted, e.Message);
-}
-evt.Set();
-}, null);
+Exception eOuter = null;
+var evt = new ManualResetEvent(false);
 
-evt.WaitOne();
+stream.BeginWrite(buf, off, nBytes, asyncResult =>
+{
+try
+{
+if (stream != null)
+stream.EndWrite(asyncResult);
+else
+eOuter = new 
TTransportException(TTransportException.ExceptionType.Interrupted);
+}
+catch (Exception e)
+{
+if (stream != null)
+eOuter = e;
+else
+eOuter = new 
TTransportException(TTransportException.ExceptionType.Interrupted, e.Message);
+}
+evt.Set();
+}, null);
+
+evt.WaitOne();
+
+if (eOuter != null)
+throw eOuter; // rethrow exception
+}
+else
+{
+stream.Write(buf, off, nBytes);
+}
 
-if (eOuter != null)
-throw eOuter; // rethrow exception
+off += nBytes;
+len -= nBytes;
+nBytes = Math.Min(len, nBytes);
--- End diff --

Shouldn't this be the same calculation as the one outside the loop?  
Technically it doesn't have to be, however I found this confusing to read.


> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library, Delphi - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into 

[jira] [Commented] (THRIFT-4372) Pipe write operations across a network are limited to 65,535 bytes per write.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/THRIFT-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219478#comment-16219478
 ] 

ASF GitHub Bot commented on THRIFT-4372:


GitHub user Jens-G opened a pull request:

https://github.com/apache/thrift/pull/1402

THRIFT-4372 Pipe write operations across a network are limited to 65,…

…535 bytes per write

Client: C#
Patch: Jens Geyer

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Jens-G/thrift THRIFT-4372

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/thrift/pull/1402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1402


commit ccfc002a480c9ed23b50674c3fd76a280750bbb0
Author: Jens Geyer 
Date:   2017-10-25T20:30:23Z

THRIFT-4372 Pipe write operations across a network are limited to 65,535 
bytes per write
Client: C#
Patch: Jens Geyer




> Pipe write operations across a network are limited to 65,535 bytes per write. 
> --
>
> Key: THRIFT-4372
> URL: https://issues.apache.org/jira/browse/THRIFT-4372
> Project: Thrift
>  Issue Type: Bug
>  Components: C# - Library
>Reporter: Jens Geyer
>Assignee: Jens Geyer
>Priority: Critical
>
> {quote}Pipe write operations across a network are limited to 65,535 bytes per 
> write. For more information regarding pipes, see the Remarks section.{quote}
> Source: [WriteFileEx 
> function|https://msdn.microsoft.com/en-us/library/windows/desktop/aa365748(v=vs.85).aspx]
> I managed to run into exactly that limit today. Patch follows.
> Symptom is that
>  * the server acts as if he had written all the bytes (in fact, he did)
>  * but the remainder of ~ 65535 bytes is just lost somewhere and never 
> reaches the client
> Consequently, the client get stuck while waiting for the remaining bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)