[
https://issues.apache.org/jira/browse/FLINK-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20827:
-----------------------------------
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Just read record correlating to join key in FilesystemLookUpFunc
> ----------------------------------------------------------------
>
> Key: FLINK-20827
> URL: https://issues.apache.org/jira/browse/FLINK-20827
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Connectors / Hive
> Reporter: zoucao
> Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> When using Temporal table join, all hive tables' records will be loaded into
> cache. But sometimes, the size of hive temporal table is larger than
> expected, and users can't know exactly how big it is in memory. In this
> situation, some error will occur, for example, `GC overhead limit exceeded`,
> `the heartbeat of TaskManager timeout (caused by gc)`.
> Maybe we can optimize the number of records readed from hive table? If the
> upstream records can be hashed only by using `Join key`, then we only need
> to load the part of records into cache, whose value of join key after being
> hashed, is equal to one fixed hash value. If it can be done, the whole table
> can be divided by the number of parallelism. I don't know whether it could
> come true in the upstream under the existing framework, but It is easy to
> support in `FileSystemLookupFunction`
> If not, we can add some logs to tell others the size of cache to help them to
> set MemorySize or other parameter of TM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
