[
https://issues.apache.org/jira/browse/KYLIN-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107017#comment-17107017
]
Harvey Yue commented on KYLIN-3849:
-----------------------------------
Hi ShaoFeng,
3000w+ records mr need about 10+ minutes, but with flink 1.10 version need
25min, I have tried to adjust flink TaskExecutor parameters with task heap,
flink managed memory (could help to improve sorting operator), maybe I still
missing other things, please pending this pr, I will do more tunning work.
Refer XinTong Song's TaskExecutor memory mode sharing
[https://www.bilibili.com/s/video/BV1At4y1U7vH]
And please refer the snapshot with flink hfile step
> Flink cubing step : convert to HFile
> ------------------------------------
>
> Key: KYLIN-3849
> URL: https://issues.apache.org/jira/browse/KYLIN-3849
> Project: Kylin
> Issue Type: Sub-task
> Components: Flink Engine
> Reporter: vinoyang
> Assignee: Harvey Yue
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
