[
https://issues.apache.org/jira/browse/FLINK-11256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16733025#comment-16733025
]
Haibo Suen commented on FLINK-11256:
------------------------------------
The simplest solution is to put the ID of StreamNode in StreamEdge. But as Till
said, introducing a runtime StreamEdge type is cleaner and more reasonable.
Also, I think the StreamConfig type is confusing. The chained head StreamConfig
contains the information about task-wide, while the non-head StreamConfig
contains the information about operator-wide. So I consider adopting the
simplest solution to make a pull request for this JIRA, then creating another
JIRA issue to introduce a runtime StreamEdge type and to rework StreamConfig.
That issue is bigger than this. What do you think?
> Referencing StreamNode objects directly in StreamEdge causes the sizes of
> JobGraph and TDD to become unnecessarily large
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-11256
> URL: https://issues.apache.org/jira/browse/FLINK-11256
> Project: Flink
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.7.0, 1.7.1
> Reporter: Haibo Suen
> Assignee: Haibo Suen
> Priority: Major
>
> When a job graph is generated from StreamGraph, StreamEdge(s) on the stream
> graph are serialized to StreamConfig and stored into the job graph. After
> that, the serialized bytes will be included in the TDD and distributed to TM.
> Because StreamEdge directly reference to StreamNode objects including
> sourceVertex and targetVertex, these objects are also written transitively on
> serializing StreamEdge. But these StreamNode objects are not needed in JM and
> Task. For a large size topology, this will causes JobGraph/TDD to become much
> larger than that actually need, and more likely to occur rpc timeout when
> transmitted.
> In StreamEdge, only the ID of StreamNode should be stored to avoid this
> situation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
