[
https://issues.apache.org/jira/browse/KUDU-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699530#comment-17699530
]
shenxingwuying commented on KUDU-3455:
--------------------------------------
I have patched the information.
> Improve space complexity about prune hash partitions for in-list predicate
> --------------------------------------------------------------------------
>
> Key: KUDU-3455
> URL: https://issues.apache.org/jira/browse/KUDU-3455
> Project: Kudu
> Issue Type: Task
> Reporter: shenxingwuying
> Assignee: shenxingwuying
> Priority: Major
> Attachments: image-2023-03-06-17-23-35-119.png,
> image-2023-03-11-16-57-16-589.png
>
>
> My partner(Chenbo Lu) has countered an oom problem when in his application
> which uses kudu java client. And he collects some information and do a lot of
> analytics for this problem, I shared his work for this issue.
> Application program was killed by OS very frequently because of oom. When
> java heap memory 8GB(inner heap 5.5GB available), more than 10000 rows
> in-list predicate would not work(oom happens). The kudu table in his case
> exists about 1500 columns. His scan requests like '{*}select * from
> profile_wos where id in (...){*}'.
>
> The problem only happened when KuduScanPredicate is In-List predicate, other
> predicates have no problem.
> He found the memory consumption is positive correlation to count of (ids *
> count of columns). In fact, I think it's also a very important key factor
> that the count of every in-list columns' values.
>
> When using kudu api to build a scanner, the memory will reach a very high
> watermark and multi-thread will make the problem worse. A picture can explain
> this and prove in-list predicate consumes very high memory.
>
> !image-2023-03-11-16-57-16-589.png!
>
>
>
> Reduce space complexity about prune hash partitions for in-list predicate
> Pruning hash partitions for in-list predicate at java-client, the logic
> codes has a high space complexity, and it may cause java-client out
> of memory. And at the same time, PartialRow has many deep copy, it may
> be slow.
>
> !image-2023-03-06-17-23-35-119.png!
>
>
> So, we need to fix the problem to improve the space complexity and speed
> optimization.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
