[ https://issues.apache.org/jira/browse/FLINK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204554#comment-16204554 ]
Xingcan Cui commented on FLINK-7799: ------------------------------------ Hi [~fhueske], I've reconsidered this problem and found a drawback of the time block solution. Specifically, if we merge all the rows belonging to the same time block in an entry (the value of which is either a {{Map}} or a {{List}}) of the {{MapState}}, the minimum operating unit of the state becomes a collection. That means everytime we store/remove a single row, all the data in the same block must also be rewritten, which will definitely bring a lot of extra cost. If that drawback cannot be eliminated, I wonder if we could improve the join performance from another point of view. Since the rocksdb backend should be widely used in real applications and the {{MapState}} entries are ordered in it, can we provide something like a hint mechanism in the state API, so that the join function can be aware of the ordering? > Improve performance of windowed joins > ------------------------------------- > > Key: FLINK-7799 > URL: https://issues.apache.org/jira/browse/FLINK-7799 > Project: Flink > Issue Type: Sub-task > Components: Table API & SQL > Affects Versions: 1.4.0 > Reporter: Fabian Hueske > Assignee: Xingcan Cui > Priority: Critical > > The performance of windowed joins can be improved by changing the state > access patterns. > Right now, rows are inserted into a MapState with their timestamp as key. > Since we use a time resolution of 1ms, this means that the full key space of > the state must be iterated and many map entries must be accessed when joining > or evicting rows. > A better strategy would be to block the time into larger intervals and > register the rows in their respective interval. Another benefit would be that > we can directly access the state entries because we know exactly which > timestamps to look up. Hence, we can limit the state access to the relevant > section during joining and state eviction. > The good size for intervals needs to be identified and might depend on the > size of the window. -- This message was sent by Atlassian JIRA (v6.4.14#64029)