[
https://issues.apache.org/jira/browse/FLINK-25318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461158#comment-17461158
]
Shammon commented on FLINK-25318:
---------------------------------
Hi [~pnowojski] Thanks for your comment. The suggestion about tests sounds
good, I like it and I agree that it's very important. I have read [various
micro
benchmarks|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847]
and clone flink-benchmarks
code, it's wonderful! I will think about the tests for flink olap, discuss it
with [~xtsong], we will give the design and plan then. Thanks for [~xtsong] to
push this in the community, and I also hope you [~pnowojski] can help to review
them when you are free, thanks
In briefly we use flink session cluster in our htap system in bytedance for
olap, and this system serves some businesses in production. We met some
bottlenecks about qps and latency on job scheduler and query execution when
flink cluster executes multiple jobs in parallel. We want to improve flink on
them, and I think some features are good for flink streaming && batch too. But
indeed it should be careful and the tests on these improvements is necessary
and important!
We mentioned this issue and hope that more OLAP and flink devs will participate
in and promote the progress of Flink in OLAP. THX :)
> Improvement of scheduler and execution for Flink OLAP
> -----------------------------------------------------
>
> Key: FLINK-25318
> URL: https://issues.apache.org/jira/browse/FLINK-25318
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
> Affects Versions: 1.14.0, 1.12.5, 1.13.3
> Reporter: Shammon
> Priority: Major
> Labels: Umbrella
> Fix For: 1.15.0
>
>
> We use flink to perform OLAP queries. We launch flink session cluster, submit
> batch jobs to the cluster as OLAP queries, and fetch the jobs' results. OLAP
> jobs are generally small queries which will finish at the seconds or
> milliseconds, and users always submit multiple jobs to the session cluster
> concurrently. We found the qps and latency of jobs will be greatly affected
> when there're tens jobs are running, even when there's little data in each
> query. We will give the result of benchmark for the latest version later.
> After discussed with [~xtsong], and thanks for his advice, we create this
> issue to trace and manager Flink OLAP related improvements. More users and
> developers are welcome and feel free to create Flink OLAP related subtasks
> here, thanks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
