[
https://issues.apache.org/jira/browse/PHOENIX-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Feng updated PHOENIX-5987:
-------------------------------
Description:
When all rowkeys are covered in where conditions, Phoenix uses point look up to
switch "a huge range scan" to "multi single key scans".
However, when #scans are too huge, Phoenix quick exhausts the memory.
We meet such condition in hour product environment as follow:
We have a table with five primary keys like k1, k2, ..., k5, all key types are
UNSIGNED_LONG.
The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and
k4 in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
We have l=600, m=800 and n=1000, #scans of look up is
1*1*600*800*1000=480,000,000.
Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is
480,000,000 * 50bytes = 25GB.
25GB exceeds the JMX configuration and causes OOM exception.
was:
When all rowkeys are covered in where conditions, Phoenix use point look up to
switch "a huge range scan" to "multi single key scans".
However, the number single key scans are too huge, it quick exhausts the memory.
We meet such condition in hour product environment as follow:
We have a table with five primary keys like k1, k2, ..., k5, all key types are
UNSIGNED_LONG.
The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and k4
in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
We have l=600, m=800 and n=1000, so the possible number of look up scans is
1*1*600*800*1000=480,000,000.
Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is
480,000,000 * 50bytes = 25GB.
25GB exceeds the JMX configuration and causes OOM exception.
> PointLookup may cost too much memory
> ------------------------------------
>
> Key: PHOENIX-5987
> URL: https://issues.apache.org/jira/browse/PHOENIX-5987
> Project: Phoenix
> Issue Type: Bug
> Reporter: Chen Feng
> Priority: Major
>
> When all rowkeys are covered in where conditions, Phoenix uses point look up
> to switch "a huge range scan" to "multi single key scans".
> However, when #scans are too huge, Phoenix quick exhausts the memory.
> We meet such condition in hour product environment as follow:
> We have a table with five primary keys like k1, k2, ..., k5, all key types
> are UNSIGNED_LONG.
> The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and
> k4 in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
> We have l=600, m=800 and n=1000, #scans of look up is
> 1*1*600*800*1000=480,000,000.
> Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is
> 480,000,000 * 50bytes = 25GB.
> 25GB exceeds the JMX configuration and causes OOM exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
