[
https://issues.apache.org/jira/browse/FLINK-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272762#comment-17272762
]
HideOnBush commented on FLINK-21145:
------------------------------------
I tried this method, but it doesn’t seem to work well. OOM also appears. There
is another problem. If I make a Kafka table into a view, my view has a
PROCTIME() field. When I use the view and Hive Temp join of dimension table is
not possible, but the original kafka table can be temp join hive dimension table
> Flink Temporal Join Hive optimization
> -------------------------------------
>
> Key: FLINK-21145
> URL: https://issues.apache.org/jira/browse/FLINK-21145
> Project: Flink
> Issue Type: Wish
> Components: Connectors / Hive
> Affects Versions: 1.12.0
> Reporter: HideOnBush
> Priority: Major
>
> When flink temporal join hive dimension table, the latest partition data will
> be loaded into task memory in full, which will lead to high memory overhead.
> In fact, sometimes the latest full data is not required. You can add options
> like options in future versions. Is the dimension table data filtered?
> For example, select * from dim /*'streaming-source.partition.include'
> ='latest' condition='fild1=ab'*/ filter the latest partition data as long as
> fild1=ab
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
