Hello everyone,

I'm trying to read data from a Phoenix Table using apache Spark. I actually use the suggested method: sc.phoenixTableAsRDD without issuing any query (e.g. reading the whole table) and I noticed that the number of partitions that spark creates is equal to the number of regionServers. Is there a way to use a custom number of regions?

The problem we actually face is that if a region is bigger than the available memory of the spark executor, it goes in OOM. Being able to tune the number of regions, we might use a higher number of partitions reducing the memory footprint of the processing (and also slowing it down, i know :( ).

Thank you in advance


Reply via email to