[GitHub] flink pull request #2690: [FLINK-4894] [network] Don't request buffer after ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2690


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2690: [FLINK-4894] [network] Don't request buffer after ...

[uce]

Github user uce commented on a diff in the pull request:

https://github.com/apache/flink/pull/2690#discussion_r85299971
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
 ---
@@ -108,15 +108,25 @@ private void sendToTarget(T record, int 
targetChannel) throws IOException, Inter
 
synchronized (serializer) {
SerializationResult result = 
serializer.addRecord(record);
+
while (result.isFullBuffer()) {
Buffer buffer = serializer.getCurrentBuffer();
 
if (buffer != null) {
-   writeBuffer(buffer, targetChannel, 
serializer);
+   writeAndClearBuffer(buffer, 
targetChannel, serializer);
+
+   // If this was a full record, we are 
done. Not breaking
+   // out of the loop at this point will 
lead to another
+   // buffer request before breaking out 
(that would not be
+   // a problem per se, but it can lead to 
stalls in the
+   // pipeline).
+   if (result.isFullRecord()) {
+   break;
--- End diff --

Yes, exactly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2690: [FLINK-4894] [network] Don't request buffer after ...

[StephanEwen]

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/2690#discussion_r85199848
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
 ---
@@ -108,15 +108,25 @@ private void sendToTarget(T record, int 
targetChannel) throws IOException, Inter
 
synchronized (serializer) {
SerializationResult result = 
serializer.addRecord(record);
+
while (result.isFullBuffer()) {
Buffer buffer = serializer.getCurrentBuffer();
 
if (buffer != null) {
-   writeBuffer(buffer, targetChannel, 
serializer);
+   writeAndClearBuffer(buffer, 
targetChannel, serializer);
+
+   // If this was a full record, we are 
done. Not breaking
+   // out of the loop at this point will 
lead to another
+   // buffer request before breaking out 
(that would not be
+   // a problem per se, but it can lead to 
stalls in the
+   // pipeline).
+   if (result.isFullRecord()) {
+   break;
--- End diff --

Can this only happen when the end of the record and the end of the buffer 
are exactly aligned?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2690: [FLINK-4894] [network] Don't request buffer after ...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/2690

[FLINK-4894] [network] Don't request buffer after writing to partition

After emitting a record via the RecordWriter, we eagerly requested a new 
buffer for the next emit on that channel (although it's not clear that we will 
immediately need it). With this change, we request that buffer lazily when an 
emit call requires it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink 4894-late_request_buffer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2690.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2690


commit a4dbe0c8db4bd8fee1a508ae212f80fa7c8bb824
Author: Ufuk Celebi 
Date:   2016-10-24T16:01:28Z

[FLINK-4894] [network] Don't request buffer after writing to partition

After emitting a record via the RecordWriter, we eagerly requested
a new buffer for the next emit on that channel (although it's not clear
that we will immediately need it). With this change, we request that
buffer lazily when an emit call requires it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---