[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244087#comment-16244087 ] ASF subversion and git services commented on DISPATCH-767: -- Commit 74cecc7d115fdf9f6a0a241efcbb0185cec73f71 in qpid-dispatch's branch refs/heads/1.0.x from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=74cecc7 ] DISPATCH-767 - Minor fix to qdr_deliver_continue_peers_CT. Activate the connection if the work is processing > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: New Feature > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244081#comment-16244081 ] ASF subversion and git services commented on DISPATCH-767: -- Commit c39d6ba64199e5c33b17b5463088c6bcd06604dd in qpid-dispatch's branch refs/heads/master from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=c39d6ba ] DISPATCH-767 - Minor fix to qdr_deliver_continue_peers_CT. Activate the connection if the work is processing > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: New Feature > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240449#comment-16240449 ] ASF subversion and git services commented on DISPATCH-767: -- Commit b9ce5b4a2c8f0634a703ef9e1e4558a2ae9cdf52 in qpid-dispatch's branch refs/heads/1.0.x from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=b9ce5b4 ] DISPATCH-767 - Backed out previous commit that was stalling links. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: New Feature > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240446#comment-16240446 ] ASF subversion and git services commented on DISPATCH-767: -- Commit a9e1147697930654f49f6111c508efdfa1581069 in qpid-dispatch's branch refs/heads/master from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=a9e1147 ] DISPATCH-767 - Backed out previous commit that was stalling links. The commit that was backed out was fff38466a6a0e8860531cf44e4074bbb1cc1 > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: New Feature > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225012#comment-16225012 ] ASF subversion and git services commented on DISPATCH-767: -- Commit 97f37b2762b43aca5d43043335d7212ec110b204 in qpid-dispatch's branch refs/heads/1.0.x from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=97f37b2 ] DISPATCH-767 - Additional fix to regression introduced in a previous checkin for this fix. Add the out_link to links_with_work only on new work creation > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: New Feature > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225001#comment-16225001 ] ASF subversion and git services commented on DISPATCH-767: -- Commit fff38466a6a0e8860531cf44e4074bbb1cc1 in qpid-dispatch's branch refs/heads/master from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=fff3846 ] DISPATCH-767 - Additional fix to regression introduced in a previous checkin for this fix. Add the out_link to links_with_work only on new work creation > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: New Feature > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122268#comment-16122268 ] ASF GitHub Bot commented on DISPATCH-767: - Github user asfgit closed the pull request at: https://github.com/apache/qpid-dispatch/pull/186 > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122267#comment-16122267 ] ASF subversion and git services commented on DISPATCH-767: -- Commit c9262728dc1a6d1bcdc5a68f4b19f1b9d2221d53 in qpid-dispatch's branch refs/heads/master from [~ganeshmurthy] [ https://git-wip-us.apache.org/repos/asf?p=qpid-dispatch.git;h=c926272 ] DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122241#comment-16122241 ] ASF GitHub Bot commented on DISPATCH-767: - GitHub user ganeshmurthy opened a pull request: https://github.com/apache/qpid-dispatch/pull/186 DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ganeshmurthy/qpid-dispatch DISPATCH-767-SQUASH-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/qpid-dispatch/pull/186.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #186 commit c9262728dc1a6d1bcdc5a68f4b19f1b9d2221d53 Author: Ganesh Murthy Date: 2017-07-05T15:51:06Z DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122210#comment-16122210 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ganeshmurthy closed the pull request at: https://github.com/apache/qpid-dispatch/pull/182 > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105600#comment-16105600 ] ASF GitHub Bot commented on DISPATCH-767: - GitHub user ganeshmurthy opened a pull request: https://github.com/apache/qpid-dispatch/pull/182 DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ganeshmurthy/qpid-dispatch DISPATCH-767-SQUASH Alternatively you can review and apply these changes as the patch at: https://github.com/apache/qpid-dispatch/pull/182.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #182 commit e387a6be363c7fed1cbaa8dac36f4bd7122adc95 Author: Ganesh Murthy Date: 2017-07-05T15:51:06Z DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105582#comment-16105582 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ganeshmurthy closed the pull request at: https://github.com/apache/qpid-dispatch/pull/177 > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100160#comment-16100160 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ganeshmurthy commented on a diff in the pull request: https://github.com/apache/qpid-dispatch/pull/177#discussion_r129333458 --- Diff: include/qpid/dispatch/message.h --- @@ -296,6 +296,61 @@ qd_parsed_field_t *qd_message_get_trace (qd_message_t *msg); */ intqd_message_get_phase_val (qd_message_t *msg); +/* + * Should the message be discarded. + * A message can be discarded if the disposition is released or rejected. + * + * @param msg A pointer to the message. + **/ +bool qd_message_is_discard(qd_message_t *msg); + +/** + *Set the discard field on the message to to the passed in boolean value. + * + * @param msg A pointer to the message. + * @param discard - the boolean value of discard. + */ +void qd_message_set_discard(qd_message_t *msg, bool discard); + +/** + * Has the message been completely received? + * Return true if the message is fully received + * Returns false if only the partial message has been received, if there is more of the message to be received. + * + * @param msg A pointer to the message. + */ +bool qd_message_receive_complete(qd_message_t *msg); + +/** + * Returns true if the message has been completely received AND the message has been completely sent. + */ +bool qd_message_send_complete(qd_message_t *msg); + +/** + * Returns true if the delivery tag has already been sent. + */ +bool qd_message_tag_sent(qd_message_t *msg); + + +/** + * Returns true if the delivery tag has already been sent. --- End diff -- Fixed the wrong comment produced due to cut and paste > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100159#comment-16100159
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user ganeshmurthy commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/177#discussion_r12919
--- Diff: src/message.c ---
@@ -999,11 +1014,84 @@ void qd_message_set_ingress_annotation(qd_message_t
*in_msg, qd_composed_field_t
qd_compose_free(ingress_field);
}
+bool qd_message_is_discard(qd_message_t *msg)
+{
+if (!msg)
+return false;
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+return pvt_msg->content->discard;
+}
+
+void qd_message_set_discard(qd_message_t *msg, bool discard)
+{
+if (!msg)
+return;
+
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+pvt_msg->content->discard = discard;
+}
+
+size_t qd_message_fanout(qd_message_t *in_msg)
+{
+if (!in_msg)
+return 0;
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->content->fanout;
+}
+
+void qd_message_add_fanout(qd_message_t *in_msg)
+{
+assert(in_msg);
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+msg->content->fanout++;
--- End diff --
yes, agreed, atomic inc
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100158#comment-16100158 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ganeshmurthy commented on a diff in the pull request: https://github.com/apache/qpid-dispatch/pull/177#discussion_r129333194 --- Diff: src/message.c --- @@ -846,6 +853,8 @@ qd_message_t *qd_message() } ZERO(msg->content); +msg->content->receive_complete = false; +msg->content->discard = false; --- End diff -- Agreed. Calling ZERO(msg->content) will set those flags to 0 (false) already. So, why bother setting them to false again. I will remove this. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100156#comment-16100156
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user ganeshmurthy commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/177#discussion_r129332913
--- Diff: src/buffer.c ---
@@ -83,7 +85,26 @@ size_t qd_buffer_size(qd_buffer_t *buf)
void qd_buffer_insert(qd_buffer_t *buf, size_t len)
{
buf->size += len;
-assert(buf->size <= buffer_size);
+assert(buf->size <= BUFFER_SIZE);
+}
+
+void qd_buffer_add_fanout(qd_buffer_t *buf)
+{
+buf->fanout++;
--- End diff --
Currently, the only function calling qd_buffer_add_fanout() is
qdr_forward_new_delivery_CT(in forwarder.c) which is only called by the core
thread. So, there should be no contention. But I am happy to change the fanout
to atomic so that anybody else calling it outside of the core thread will not
be affected.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100124#comment-16100124 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ChugR commented on a diff in the pull request: https://github.com/apache/qpid-dispatch/pull/177#discussion_r129306348 --- Diff: include/qpid/dispatch/message.h --- @@ -296,6 +296,61 @@ qd_parsed_field_t *qd_message_get_trace (qd_message_t *msg); */ intqd_message_get_phase_val (qd_message_t *msg); +/* + * Should the message be discarded. + * A message can be discarded if the disposition is released or rejected. + * + * @param msg A pointer to the message. + **/ +bool qd_message_is_discard(qd_message_t *msg); + +/** + *Set the discard field on the message to to the passed in boolean value. + * + * @param msg A pointer to the message. + * @param discard - the boolean value of discard. + */ +void qd_message_set_discard(qd_message_t *msg, bool discard); + +/** + * Has the message been completely received? + * Return true if the message is fully received + * Returns false if only the partial message has been received, if there is more of the message to be received. + * + * @param msg A pointer to the message. + */ +bool qd_message_receive_complete(qd_message_t *msg); + +/** + * Returns true if the message has been completely received AND the message has been completely sent. + */ +bool qd_message_send_complete(qd_message_t *msg); + +/** + * Returns true if the delivery tag has already been sent. + */ +bool qd_message_tag_sent(qd_message_t *msg); + + +/** + * Returns true if the delivery tag has already been sent. --- End diff -- cut-paste issue. This function sets something and returns nothing. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100126#comment-16100126
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user ChugR commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/177#discussion_r129308722
--- Diff: src/message.c ---
@@ -999,11 +1014,84 @@ void qd_message_set_ingress_annotation(qd_message_t
*in_msg, qd_composed_field_t
qd_compose_free(ingress_field);
}
+bool qd_message_is_discard(qd_message_t *msg)
+{
+if (!msg)
+return false;
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+return pvt_msg->content->discard;
+}
+
+void qd_message_set_discard(qd_message_t *msg, bool discard)
+{
+if (!msg)
+return;
+
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+pvt_msg->content->discard = discard;
+}
+
+size_t qd_message_fanout(qd_message_t *in_msg)
+{
+if (!in_msg)
+return 0;
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->content->fanout;
+}
+
+void qd_message_add_fanout(qd_message_t *in_msg)
+{
+assert(in_msg);
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+msg->content->fanout++;
+}
+
+bool qd_message_receive_complete(qd_message_t *in_msg)
+{
+if (!in_msg)
+return false;
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->content->receive_complete;
+}
+
+bool qd_message_send_complete(qd_message_t *in_msg)
+{
+if (!in_msg)
+return false;
+
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->send_complete;
+}
+
+bool qd_message_tag_sent(qd_message_t *in_msg)
+{
+if (!in_msg)
+return false;
+
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->tag_sent;
+}
+
+void qd_message_set_tag_sent(qd_message_t *in_msg, bool tag_sent)
+{
+if (!in_msg)
+return;
+
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+msg->tag_sent = tag_sent;
+}
+
+qd_field_location_t qd_message_cursor(qd_message_pvt_t *in_msg)
--- End diff --
prefer a 'qd_field_location_t *' return and not a copy of the whole struct
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100123#comment-16100123
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user ChugR commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/177#discussion_r129318215
--- Diff: src/message_private.h ---
@@ -107,11 +106,21 @@ typedef struct {
qd_parsed_field_t *ma_pf_to_override;
qd_parsed_field_t *ma_pf_trace;
int ma_int_phase;
+//qd_parsed_field_t *parsed_message_annotations;
+
+bool discard;// Should this
message be discarded?
+bool receive_complete; // true if the
message has been completely received, false otherwise
+unsigned int fanout; // The number of
receivers for this message. This number does not include in-process subscribers.
} qd_message_content_t;
typedef struct {
DEQ_LINKS(qd_message_t); // Deque linkage that overlays the
qd_message_t
-qd_message_content_t *content;
+qd_field_location_t cursor; // A pointer to the current
location of the outgoing byte stream.
--- End diff --
qd_iterator_pointer_t might be more appropriate for a pointer into buffers.
field_location has baggage that is not needed.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100127#comment-16100127
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user ChugR commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/177#discussion_r129307766
--- Diff: src/message.c ---
@@ -999,11 +1014,84 @@ void qd_message_set_ingress_annotation(qd_message_t
*in_msg, qd_composed_field_t
qd_compose_free(ingress_field);
}
+bool qd_message_is_discard(qd_message_t *msg)
+{
+if (!msg)
+return false;
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+return pvt_msg->content->discard;
+}
+
+void qd_message_set_discard(qd_message_t *msg, bool discard)
+{
+if (!msg)
+return;
+
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+pvt_msg->content->discard = discard;
+}
+
+size_t qd_message_fanout(qd_message_t *in_msg)
+{
+if (!in_msg)
+return 0;
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->content->fanout;
+}
+
+void qd_message_add_fanout(qd_message_t *in_msg)
+{
+assert(in_msg);
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+msg->content->fanout++;
--- End diff --
atomic inc?
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100128#comment-16100128 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ChugR commented on a diff in the pull request: https://github.com/apache/qpid-dispatch/pull/177#discussion_r129305989 --- Diff: src/message.c --- @@ -846,6 +853,8 @@ qd_message_t *qd_message() } ZERO(msg->content); +msg->content->receive_complete = false; +msg->content->discard = false; --- End diff -- after ZEROing the content setting any value to 0 or false (seven places) is a waste. Maybe change the assignments to asserts if you don't trust the ZERO. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100125#comment-16100125
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user ChugR commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/177#discussion_r129306967
--- Diff: src/buffer.c ---
@@ -83,7 +85,26 @@ size_t qd_buffer_size(qd_buffer_t *buf)
void qd_buffer_insert(qd_buffer_t *buf, size_t len)
{
buf->size += len;
-assert(buf->size <= buffer_size);
+assert(buf->size <= BUFFER_SIZE);
+}
+
+void qd_buffer_add_fanout(qd_buffer_t *buf)
+{
+buf->fanout++;
--- End diff --
Is this called with a lock held? Otherwise probably needs to be atomic
increment.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090521#comment-16090521 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ganeshmurthy commented on a diff in the pull request: https://github.com/apache/qpid-dispatch/pull/177#discussion_r127819616 --- Diff: src/router_core/transfer.c --- @@ -851,6 +977,144 @@ static void qdr_delete_delivery_CT(qdr_core_t *core, qdr_action_t *action, bool qdr_delete_delivery_internal_CT(core, action->args.delivery.delivery); } +static bool qdr_delivery_is_all_peers_unlinked_CT(qdr_delivery_t *dlv) --- End diff -- @ted-ross this function (qdr_delivery_is_all_peers_unlinked_CT) needs a little bit of your extra attention. Please take a look and let me know if you have questions. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090358#comment-16090358 ] ASF GitHub Bot commented on DISPATCH-767: - GitHub user ganeshmurthy opened a pull request: https://github.com/apache/qpid-dispatch/pull/177 DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ganeshmurthy/qpid-dispatch DISPATCH-767-3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/qpid-dispatch/pull/177.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #177 commit 54bd583053bc125a178e006611740e73bbe99a63 Author: Ganesh Murthy Date: 2017-07-05T15:51:06Z DISPATCH-767 - Added code to support multi frame message streaming 1. Added new core API function qdr_deliver_continue() that will continue delivering a large message 2. Modified qdr_link_process_deliveries() to not remove deliveries from the undelivered list until they are fully delivered 3. Modified qd_message_receive() to recieve partial messages. 4. Modified qd_message_send() to be able to handle streaming send. This function can now be called multiple times for the same message. It keeps internal pointers to the point upto which the message has been sent and is able to continue where it left off. Message content buffers are freed as soon as the message has been sent to all recipients. 5. Added peer linkage for large settled deliveries and added a settled list to handle with abrupt connection terminations when large messages are being transmitted. 6. Added unit tests to test large messages. > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080322#comment-16080322 ] ASF GitHub Bot commented on DISPATCH-767: - Github user ganeshmurthy closed the pull request at: https://github.com/apache/qpid-dispatch/pull/172 > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078262#comment-16078262
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126168949
--- Diff: src/message.c ---
@@ -1151,89 +1287,140 @@ void qd_message_send(qd_message_t *in_msg,
{
qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
qd_message_content_t *content = msg->content;
-qd_buffer_t *buf = DEQ_HEAD(content->buffers);
-unsigned char*cursor;
+qd_buffer_t *buf = 0;
pn_link_t*pnl = qd_link_pn(link);
-qd_buffer_list_t new_ma;
-DEQ_INIT(new_ma);
+// How many receivers does this message have?
+int fanout = qd_message_fanout(in_msg);
-// Process the message annotations if any
-compose_message_annotations(msg, &new_ma, strip_annotations);
+if (msg->sent_depth < QD_DEPTH_MESSAGE_ANNOTATIONS) {
+//
+// Start with the very first buffer;
+//
+buf = DEQ_HEAD(content->buffers);
--- End diff --
No lock here? What's the protocol for protecting buffers between
send/receive threads? We can't let threads work concurrently on the same buffer
(in particular change the buffer size) We could lock the use of buffers, not
just the enqueue/deque, or there are some other options.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078264#comment-16078264
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126164203
--- Diff: src/message.c ---
@@ -1026,16 +1150,24 @@ qd_message_t *qd_message_receive(pn_delivery_t
*delivery)
// will only happen if the size of the message content is an
exact multiple
// of the buffer size.
//
-
-if (qd_buffer_size(buf) == 0) {
+if (buf && qd_buffer_size(buf) == 0) {
+sys_mutex_lock(msg->content->lock);
DEQ_REMOVE_TAIL(msg->content->buffers);
+sys_mutex_unlock(msg->content->lock);
qd_buffer_free(buf);
}
+//
+// We have received the entire message since rc == PN_EOS, set
the receive_complete flag to false
--- End diff --
Comment typo: says "false" should say "true"
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078265#comment-16078265
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126170901
--- Diff: src/message.c ---
@@ -1151,89 +1287,140 @@ void qd_message_send(qd_message_t *in_msg,
{
qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
qd_message_content_t *content = msg->content;
-qd_buffer_t *buf = DEQ_HEAD(content->buffers);
-unsigned char*cursor;
+qd_buffer_t *buf = 0;
pn_link_t*pnl = qd_link_pn(link);
-qd_buffer_list_t new_ma;
-DEQ_INIT(new_ma);
+// How many receivers does this message have?
+int fanout = qd_message_fanout(in_msg);
-// Process the message annotations if any
-compose_message_annotations(msg, &new_ma, strip_annotations);
+if (msg->sent_depth < QD_DEPTH_MESSAGE_ANNOTATIONS) {
+//
+// Start with the very first buffer;
+//
+buf = DEQ_HEAD(content->buffers);
-//
-// This is the case where the message annotations have been modified.
-// The message send must be divided into sections: The existing
header;
-// the new message annotations; the rest of the existing message.
-// Note that the original message annotations that are still in the
-// buffer chain must not be sent.
-//
-// Start by making sure that we've parsed the message sections through
-// the message annotations
-//
-// ??? NO LONGER NECESSARY???
-if (!qd_message_check(in_msg, QD_DEPTH_MESSAGE_ANNOTATIONS)) {
-qd_log(log_source, QD_LOG_ERROR, "Cannot send: %s",
qd_error_message);
-return;
-}
+if (!qd_message_check(in_msg, QD_DEPTH_MESSAGE_ANNOTATIONS)) {
+qd_log(log_source, QD_LOG_ERROR, "Cannot send: %s",
qd_error_message);
+return;
+}
-//
-// Send header if present
-//
-cursor = qd_buffer_base(buf);
-if (content->section_message_header.length > 0) {
-buf= content->section_message_header.buffer;
-cursor = content->section_message_header.offset +
qd_buffer_base(buf);
-advance(&cursor, &buf,
-content->section_message_header.length +
content->section_message_header.hdr_length,
-send_handler, (void*) pnl);
-}
+//
+// Send header if present
+//
+unsigned char *cursor = qd_buffer_base(buf);
+int header_consume = content->section_message_header.length +
content->section_message_header.hdr_length;
+if (content->section_message_header.length > 0) {
+buf= content->section_message_header.buffer;
+cursor = content->section_message_header.offset +
qd_buffer_base(buf);
+advance(&cursor, &buf, header_consume, send_handler, (void*)
pnl);
+}
-//
-// Send delivery annotation if present
-//
-if (content->section_delivery_annotation.length > 0) {
-buf= content->section_delivery_annotation.buffer;
-cursor = content->section_delivery_annotation.offset +
qd_buffer_base(buf);
-advance(&cursor, &buf,
-content->section_delivery_annotation.length +
content->section_delivery_annotation.hdr_length,
-send_handler, (void*) pnl);
-}
+//
+// Send delivery annotation if present
+//
+int da_consume = content->section_delivery_annotation.length +
content->section_delivery_annotation.hdr_length;
+if (content->section_delivery_annotation.length > 0) {
+buf= content->section_delivery_annotation.buffer;
+cursor = content->section_delivery_annotation.offset +
qd_buffer_base(buf);
+advance(&cursor, &buf, da_consume, send_handler, (void*) pnl);
+}
-//
-// Send new message annotations
-//
-qd_buffer_t *da_buf = DEQ_HEAD(new_ma);
-while (da_buf) {
-char *to_send = (char*) qd_buffer_base(da_buf);
-pn_link_send(pnl, to_send, qd_buffer_size(da_buf));
-da_buf = DEQ_NEXT(da_buf);
-}
-qd_buffer_list_free_buffers(&new_ma);
+qd_buffer_list_t new_ma;
+DEQ_INIT(new_ma);
-//
-// Skip over replaced message annotations
-//
-if (content->section_message_a
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078267#comment-16078267
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126161795
--- Diff: src/message.c ---
@@ -996,21 +1094,47 @@ qd_message_t *qd_message_receive(pn_delivery_t
*delivery)
}
//
+// The discard flag indicates if we should continue receiving the
message.
+// This is pertinent in the case of large messages. When large
messages are being received, we try to send out part of the
+// message that has been received so far. If we not able to send it
anywhere, there is no need to keep creating buffers
+//
+bool discard = qd_message_is_discard((qd_message_t*)msg);
+
+//
// Get a reference to the tail buffer on the message. This is the
buffer into which
-// we will store incoming message data. If there is no buffer in the
message, allocate
-// an empty one and add it to the message.
+// we will store incoming message data. If there is no buffer in the
message, this is the
+// first time we are here and we need to allocate an empty one and add
it to the message.
//
-buf = DEQ_TAIL(msg->content->buffers);
-if (!buf) {
-buf = qd_buffer();
-DEQ_INSERT_TAIL(msg->content->buffers, buf);
+if (!discard) {
+buf = DEQ_TAIL(msg->content->buffers);
+if (!buf) {
+buf = qd_buffer();
+DEQ_INSERT_TAIL(msg->content->buffers, buf);
+}
}
while (1) {
-//
-// Try to receive enough data to fill the remaining space in the
tail buffer.
-//
-rc = pn_link_recv(link, (char*) qd_buffer_cursor(buf),
qd_buffer_capacity(buf));
+if (discard) {
+char dummy[BUFFER_SIZE];
+rc = pn_link_recv(link, dummy, BUFFER_SIZE);
+}
+else {
+
+//
--- End diff --
Remove commented-out code
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078268#comment-16078268
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126172968
--- Diff: src/router_core/forwarder.c ---
@@ -395,9 +415,22 @@ int qdr_forward_closest_CT(qdr_core_t *core,
// Forward to an in-process subscriber if there is one.
//
if (!exclude_inprocess) {
+bool receive_complete = qd_message_receive_complete(msg);
qdr_subscription_t *sub = DEQ_HEAD(addr->subscriptions);
if (sub) {
-qdr_forward_on_message_CT(core, sub, in_delivery ?
in_delivery->link : 0, msg);
+
+//
+// Only if the message has been completely received, forward
it.
+// Subscriptions, at the moment, dont have the ability to deal
with partial messages
+//
+if (receive_complete)
--- End diff --
For streaming we need to start forwarding before completion.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078270#comment-16078270
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126161603
--- Diff: src/message.c ---
@@ -996,21 +1094,47 @@ qd_message_t *qd_message_receive(pn_delivery_t
*delivery)
}
//
+// The discard flag indicates if we should continue receiving the
message.
+// This is pertinent in the case of large messages. When large
messages are being received, we try to send out part of the
+// message that has been received so far. If we not able to send it
anywhere, there is no need to keep creating buffers
+//
+bool discard = qd_message_is_discard((qd_message_t*)msg);
+
+//
// Get a reference to the tail buffer on the message. This is the
buffer into which
-// we will store incoming message data. If there is no buffer in the
message, allocate
-// an empty one and add it to the message.
+// we will store incoming message data. If there is no buffer in the
message, this is the
+// first time we are here and we need to allocate an empty one and add
it to the message.
//
-buf = DEQ_TAIL(msg->content->buffers);
-if (!buf) {
-buf = qd_buffer();
-DEQ_INSERT_TAIL(msg->content->buffers, buf);
+if (!discard) {
+buf = DEQ_TAIL(msg->content->buffers);
+if (!buf) {
+buf = qd_buffer();
+DEQ_INSERT_TAIL(msg->content->buffers, buf);
+}
}
while (1) {
-//
-// Try to receive enough data to fill the remaining space in the
tail buffer.
-//
-rc = pn_link_recv(link, (char*) qd_buffer_cursor(buf),
qd_buffer_capacity(buf));
+if (discard) {
+char dummy[BUFFER_SIZE];
--- End diff --
Pity that we have to copy out data only to throw it away, but I think you
are correct that we do need to do it. Maybe the AMQP spec has something we can
use to abort an incoming message, but even if it does this code path is
probably needed as a fall-back.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078266#comment-16078266
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126155671
--- Diff: include/qpid/dispatch/buffer.h ---
@@ -32,10 +32,14 @@ typedef struct qd_buffer_t qd_buffer_t;
DEQ_DECLARE(qd_buffer_t, qd_buffer_list_t);
+extern size_t BUFFER_SIZE;
+extern size_t MAX_BUFFER_LENGTH;
+
/** A raw byte buffer .*/
struct qd_buffer_t {
DEQ_LINKS(qd_buffer_t);
unsigned int size; ///< Size of data content
+unsigned int fanout;// The number of receivers for this buffer
--- End diff --
Q: Why record fanout per-buffer? Fanout is per-message and buffers can be
re-used with different fanouts in different messages. Can we avoid duplicating
the fanout value from the message onto all its buffers? Maybe not - just
wondering, it seems out of place.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078263#comment-16078263
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126156980
--- Diff: src/buffer.c ---
@@ -83,7 +86,28 @@ size_t qd_buffer_size(qd_buffer_t *buf)
void qd_buffer_insert(qd_buffer_t *buf, size_t len)
{
buf->size += len;
-assert(buf->size <= buffer_size);
+assert(buf->size <= BUFFER_SIZE);
+}
+
+void qd_buffer_add_fanout(qd_buffer_t *buf)
+{
+buf->fanout++;
+}
+
+size_t qd_buffer_fanout(qd_buffer_t *buf)
+{
+return buf->fanout;
+}
+
+
+unsigned char *qd_buffer_at(qd_buffer_t *buf, size_t len)
+{
+assert(len >=0);
--- End diff --
Redundant, size_t is unsigned, it can never be < 0
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078269#comment-16078269 ] ASF GitHub Bot commented on DISPATCH-767: - Github user alanconway commented on a diff in the pull request: https://github.com/apache/qpid-dispatch/pull/172#discussion_r126155845 --- Diff: include/qpid/dispatch/message.h --- @@ -203,9 +203,9 @@ int qd_message_get_phase_annotation(const qd_message_t *msg); void qd_message_set_ingress_annotation(qd_message_t *msg, qd_composed_field_t *ingress_field); /** - * Receive message data via a delivery. This function may be called more than once on the same - * delivery if the message spans multiple frames. Once a complete message has been received, this - * function shall return the message. + * Receive message data frame by frame via a delivery. This function may be called more than once on the same + * delivery if the message spans multiple frames. Always returns a message. The message buffers are filled up to the point with the data that was been received so far. + * The buffer keeps filling up on successive calls to this function. * * @param delivery An incoming delivery from a link * @return A pointer to the complete message or 0 if the message is not yet complete. --- End diff -- This comment is out of date > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078261#comment-16078261
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126158278
--- Diff: src/message_private.h ---
@@ -94,10 +94,19 @@ typedef struct {
unsigned char *parse_cursor;
qd_message_depth_t parse_depth;
qd_parsed_field_t *parsed_message_annotations;
+
+bool discard;// Should this
message be discarded?
+bool receive_complete; // true if the
message has been completely received, false otherwise
+unsigned int fanout; // The number of
receivers for this message. This number does not include in-process subscribers.
} qd_message_content_t;
typedef struct {
DEQ_LINKS(qd_message_t); // Deque linkage that overlays the
qd_message_t
+qd_field_location_t cursor; // A pointer to the current
location of the outgoing byte stream.
+qd_message_depth_tmessage_depth;
--- End diff --
Add a comment to explain these here - not obvious (to me) what they mean.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078260#comment-16078260
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126160704
--- Diff: src/message.c ---
@@ -975,11 +990,94 @@ void qd_message_set_ingress_annotation(qd_message_t
*in_msg, qd_composed_field_t
qd_compose_free(ingress_field);
}
+bool qd_message_is_discard(qd_message_t *msg)
+{
+if (!msg)
+return false;
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+return pvt_msg->content->discard;
+}
+
+void qd_message_set_discard(qd_message_t *msg, bool discard)
+{
+if (!msg)
+return;
+
+qd_message_pvt_t *pvt_msg = (qd_message_pvt_t*) msg;
+pvt_msg->content->discard = discard;
+}
+
+size_t qd_message_fanout(qd_message_t *in_msg)
+{
+if (!in_msg)
+return 0;
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->content->fanout;
+}
+
+void qd_message_add_fanout(qd_message_t *in_msg)
+{
+assert(in_msg);
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+msg->content->fanout++;
+}
+
+bool qd_message_receive_complete(qd_message_t *in_msg)
+{
+if (!in_msg)
+return false;
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->content->receive_complete;
+}
+
+bool qd_message_send_complete(qd_message_t *in_msg)
+{
+if (!in_msg)
+return false;
+
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->send_complete;
+}
+
+bool qd_message_tag_sent(qd_message_t *in_msg)
+{
+if (!in_msg)
+return false;
+
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+return msg->tag_sent;
+}
+
+void qd_message_set_tag_sent(qd_message_t *in_msg, bool tag_sent)
+{
+if (!in_msg)
+return;
+
+qd_message_pvt_t *msg = (qd_message_pvt_t*) in_msg;
+msg->tag_sent = tag_sent;
+}
+
+qd_buffer_t *qd_message_cursor_buffer(qd_message_pvt_t *in_msg)
+{
+return in_msg->cursor.buffer;
+}
+
+int qd_message_cursor_offset(qd_message_pvt_t *in_msg)
+{
+return in_msg->cursor.offset;
+}
+
--- End diff --
Delete qd_message_cursor_buffer|offset - you don't need them with
qd_message_cursor()
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[
https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078271#comment-16078271
]
ASF GitHub Bot commented on DISPATCH-767:
-
Github user alanconway commented on a diff in the pull request:
https://github.com/apache/qpid-dispatch/pull/172#discussion_r126163415
--- Diff: src/message.c ---
@@ -996,21 +1094,47 @@ qd_message_t *qd_message_receive(pn_delivery_t
*delivery)
}
//
+// The discard flag indicates if we should continue receiving the
message.
+// This is pertinent in the case of large messages. When large
messages are being received, we try to send out part of the
+// message that has been received so far. If we not able to send it
anywhere, there is no need to keep creating buffers
+//
+bool discard = qd_message_is_discard((qd_message_t*)msg);
+
+//
// Get a reference to the tail buffer on the message. This is the
buffer into which
-// we will store incoming message data. If there is no buffer in the
message, allocate
-// an empty one and add it to the message.
+// we will store incoming message data. If there is no buffer in the
message, this is the
+// first time we are here and we need to allocate an empty one and add
it to the message.
//
-buf = DEQ_TAIL(msg->content->buffers);
-if (!buf) {
-buf = qd_buffer();
-DEQ_INSERT_TAIL(msg->content->buffers, buf);
+if (!discard) {
+buf = DEQ_TAIL(msg->content->buffers);
+if (!buf) {
+buf = qd_buffer();
+DEQ_INSERT_TAIL(msg->content->buffers, buf);
+}
}
while (1) {
-//
-// Try to receive enough data to fill the remaining space in the
tail buffer.
-//
-rc = pn_link_recv(link, (char*) qd_buffer_cursor(buf),
qd_buffer_capacity(buf));
+if (discard) {
+char dummy[BUFFER_SIZE];
+rc = pn_link_recv(link, dummy, BUFFER_SIZE);
+}
+else {
+
+//
+// Make sure our buffer chain length is always less than
MAX_BUFFER_LENGTH. We don't want to add any more buffers beyond
MAX_BUFFER_LENGTH.
+//
+//
+//sys_mutex_lock(msg->content->lock);
+//if (DEQ_SIZE(msg->content->buffers) >
MAX_BUFFER_CHAIN_LENGTH) {
+//return (qd_message_t*) msg;
+//}
+//sys_mutex_unlock(msg->content->lock);
+
+//
+// Try to receive enough data to fill the remaining space in
the tail buffer.
+//
+rc = pn_link_recv(link, (char*) qd_buffer_cursor(buf),
qd_buffer_capacity(buf));
+}
//
// If we receive PN_EOS, we have come to the end of the message.
--- End diff --
FYI: instead of watching for PN_EOS, you can also check for
(pn_delivery_pending(d) == 0 && !pn_delivery_partial(d)). They are equivalent,
PN_EOS is correct - sometimes the pending/partial test is more convenient
because it stays true on the delivery so you can check it any time, not just at
return from pn_link_recv(). Only FYI, no need to change this code.
> Message Cut-Through/Streaming for efficient handling of large messages
> --
>
> Key: DISPATCH-767
> URL: https://issues.apache.org/jira/browse/DISPATCH-767
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
>Reporter: Ted Ross
>Assignee: Ganesh Murthy
> Fix For: 1.0.0
>
>
> When large, multi-frame messages are sent through the router, there is no
> need to wait for the entire message to arrive before starting to send it
> onward.
> This feature causes the router to route the first frame and allow subsequent
> frames in a delivery to be streamed out in pipeline fashion. Ideally, the
> memory usage in the router should only involve pending frames. This would
> allow the router to handle arbitrary numbers of concurrent arbitrarily large
> messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-767) Message Cut-Through/Streaming for efficient handling of large messages
[ https://issues.apache.org/jira/browse/DISPATCH-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075693#comment-16075693 ] ASF GitHub Bot commented on DISPATCH-767: - GitHub user ganeshmurthy opened a pull request: https://github.com/apache/qpid-dispatch/pull/172 DISPATCH-767 - Added code to support multi frame message streaming You can merge this pull request into a Git repository by running: $ git pull https://github.com/ganeshmurthy/qpid-dispatch DISPATCH-767-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/qpid-dispatch/pull/172.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #172 commit 7dfbed4355c8e626a441b70c3429f792ec712729 Author: Ganesh Murthy Date: 2017-07-05T15:51:06Z DISPATCH-767 - Added code to support multi frame message streaming > Message Cut-Through/Streaming for efficient handling of large messages > -- > > Key: DISPATCH-767 > URL: https://issues.apache.org/jira/browse/DISPATCH-767 > Project: Qpid Dispatch > Issue Type: Improvement > Components: Router Node >Reporter: Ted Ross >Assignee: Ganesh Murthy > Fix For: 1.0.0 > > > When large, multi-frame messages are sent through the router, there is no > need to wait for the entire message to arrive before starting to send it > onward. > This feature causes the router to route the first frame and allow subsequent > frames in a delivery to be streamed out in pipeline fashion. Ideally, the > memory usage in the router should only involve pending frames. This would > allow the router to handle arbitrary numbers of concurrent arbitrarily large > messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
