[jira] [Commented] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896504#comment-15896504 ] Pramod Immaneni commented on APEXCORE-581: -- Do the streams need to be control tuple aware? It is still a single stream isn't it and the nodes do the de-multiplexing isn't it? > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716101#comment-15716101 ] Vlad Rozov commented on APEXCORE-581: - Agree, GenericNode needs to orchestrate tuples the same way what it currently does. The same applies to the Buffer Server. I foresee code changes both to GenericNode and to the Buffer Server needed to support custom control tuples. > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716077#comment-15716077 ] David Yan commented on APEXCORE-581: Yes, I think that makes sense. [~vrozov] thoughts? > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711794#comment-15711794 ] Bhupesh Chawda commented on APEXCORE-581: - Do we need to store the control tuples in the buffer server. I was thinking that the buffer server flow for all the tuples (including control tuples) would be as it is. We can intercept the tuples in GenericNode and buffer them there. Once consolidated, we could put them into the sinks at the window boundaries. > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
