Hi Fabian and Stephan, back to work :)
I finally managed to find the problem of the parallelism encountered by my
colleague!
Basically that was introduced by this API change. Before I was using
env.setConfiguration() to merge the default params with some custom ones.
Now, after the API change I was using the following code:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
if (env instanceof LocalEnvironment) {
Configuration c = new Configuration();
c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR);
c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
env = ExecutionEnvironment.createLocalEnvironment(c);
}
However, the first env and the reassigned one doesn't behave in the same
manner.
If I don't reassign env I have parallelism=8, otherwise it's 1 :(
Am I using the wrong APIs or the execution environment doesn't allow now to
configure such parameters anymore?
Thanks in advance,
Flavio
On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <pomperma...@okkam.it>
wrote:
> That makes sense: what can be configured should be differentiated between
> local and remote envs (obviously this is a minor issue/improvement)
>
> Thanks again,
> Flavio
>
> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>
>> We can think about that, but I think it may be quite confusing. The
>> configurations actually mean something different for local and remote
>> environments:
>>
>> - For the local environment, the config basically describes the entire
>> Flink cluster setup (for the local execution cluster in the background)
>> - For the remote environment, the config describes the parameters for
>> the client that connects to the cluster (akka paramters, optimizer
>> parameters, ...), but not parameters of the cluster itself (like
>> taskmanager slots and memory).
>>
>> Greetings,
>> Stephan
>>
>>
>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <pomperma...@okkam.it
>> > wrote:
>>
>>> However it could be a good idea to overload also
>>> the getExecutionEnvironment() to be able to pass a custom
>>> configuration..what do you think?
>>> Otherwise I have to know a priori if I'm working in a local deployment
>>> or in a remote one, or check if getExecutionEnvironment() returned an
>>> instance of LocalEnvironment/RemoteEnvironment..
>>>
>>>
>>>
>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>> pomperma...@okkam.it> wrote:
>>>
>>>> Yes Stephan!
>>>> I usually work with the master version, at least in development ;)
>>>> Thanks for the quick support!
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> Are you on the SNAPSHOT master version?
>>>>>
>>>>> You can pass the configuration to the constructor of the execution
>>>>> environment, or create one via
>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>
>>>>> The change of the signature was part of an API cleanup for the next
>>>>> release. Sorry for the inconvenience...
>>>>>
>>>>> Stephan
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>> pomperma...@okkam.it> wrote:
>>>>>
>>>>>> Hi to all,
>>>>>>
>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>> parameters in my unit tests?
>>>>>>
>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>
>>>>>> Best,
>>>>>> Flavio
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
