Re: Partitions and the number of cores/executors

[Stefan Richter]

Hi,

Your assumption is right. Parallel processing is based in splitting inputs and 
each split is only processed by one task instance at a time.

Best,
Stefan

> On 13. Mar 2019, at 09:52, mbilalce@gmail.com wrote:
> 
> Hi,
> 
> I am working with Gelly graph library but I think the question is applicable 
> in general. I just want to confirm if a single data partition in Flink is 
> executed by only a single executor/core? i.e. multiple executors can't be 
> utilized to process a single partition in parallel. So, if I need to have 
> utilize higher parallelism I should simply have an equal number of 
> partitions. 
> 
> I suppose the reason for this is to avoid any kind of locking that might be 
> required to process a data partition with multiple executors. 
> 
> 
> Thanks
> 
> - Bilal
> 

Partitions and the number of cores/executors

[mbilalce . dev]

Hi,

I am working with Gelly graph library but I think the question is applicable in 
general. I just want to confirm if a single data partition in Flink is executed 
by only a single executor/core? i.e. multiple executors can't be utilized to 
process a single partition in parallel. So, if I need to have utilize higher 
parallelism I should simply have an equal number of partitions. 

I suppose the reason for this is to avoid any kind of locking that might be 
required to process a data partition with multiple executors. 


Thanks

- Bilal