Hi,

I'm working on the JIRA-749(https://issues.apache.org/jira/browse/IOTDB-749). 


Although we already have the fetch_size to limit the max rows of one query, we 
don't have any limitation about the max columns. So, if we have too many 
timeseries in iotdb, then once we do query like 'select * from root', it will 
cause OOM.


So, I want to add a property `max_deduplicated_path_num` in the 
iotdb-engine.properties, default to 10000.


It is the allowed max numbers of deduplicated path in one query. However, it's 
just an advised value, the real limitation will be the smaller one between this 
and the one we calculated. 


For example, we have 10GB total memory size. The Read can be allocated 3GB as 
default configuration. In addition to the memory allocated for cache, we will 
have about 2GB for query processing. Then our calculation will be 2GB / 16B / 
fetch_size = 64000. So the actual limitation will be min(10000, 64000). 16B is 
an estimated size for one point memory usage.


If there more than one query at the same time, for query1, it will have full 
free memory 2GB, for query2, it will only have the 2GB - estimated memory usage 
of query1.


In this way, we can get the column limitation before generating the logical 
plan, we can pass it into the `MManager.getInstance().getAllTimeseriesPath()`, 
if the deduplicated paths have been over the limitation, we can throw an 
exception in the logical plan phase.

Reply via email to