I think it's not a nice solution to check for the type of the returned
execution environment to determine whether it is a local or a remote
execution environment.
Wouldn't it be better to add a method isLocal() to ExecutionEnvironment?
Cheers, Fabian
2015-10-14 19:14 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>:
> of course,I tried to configure the task slot during a debug test and I
> forgot to remove it..
> Just for curiosity, is there any good reason why you've changed the
> default parallellelism that way?and moreover, is it the only unexpected
> changed behaviour wrt the previous API version?
> On 14 Oct 2015 18:35, "Stephan Ewen" <se...@apache.org> wrote:
>
>> Hi Flavio!
>>
>> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment()
>> by default picks up the number of cores as the parallelism, while the
>> manual environments do not do that.
>> You can still set it manually set the parallelism
>> "env.setParallelism(Runtime.getRuntime().availableProcessors());"
>>
>> I would not configure the slots for the local execution, they should be
>> automatically configured based on the max parallelism.
>>
>> Greetings,
>> Stephan
>>
>>
>> On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <pomperma...@okkam.it
>> > wrote:
>>
>>> Hi Fabian and Stephan, back to work :)
>>>
>>> I finally managed to find the problem of the parallelism encountered by
>>> my colleague!
>>> Basically that was introduced by this API change. Before I was using
>>> env.setConfiguration() to merge the default params with some custom ones.
>>> Now, after the API change I was using the following code:
>>>
>>> ExecutionEnvironment env =
>>> ExecutionEnvironment.getExecutionEnvironment();
>>> if (env instanceof LocalEnvironment) {
>>> Configuration c = new Configuration();
>>> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY,
>>> FLINK_TEST_TMP_DIR);
>>>
>>> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
>>> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 *
>>> 2);
>>> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
>>> env = ExecutionEnvironment.createLocalEnvironment(c);
>>> }
>>>
>>> However, the first env and the reassigned one doesn't behave in the same
>>> manner.
>>> If I don't reassign env I have parallelism=8, otherwise it's 1 :(
>>> Am I using the wrong APIs or the execution environment doesn't allow now
>>> to configure such parameters anymore?
>>>
>>> Thanks in advance,
>>> Flavio
>>>
>>>
>>> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <
>>> pomperma...@okkam.it> wrote:
>>>
>>>> That makes sense: what can be configured should be differentiated
>>>> between local and remote envs (obviously this is a minor issue/improvement)
>>>>
>>>> Thanks again,
>>>> Flavio
>>>>
>>>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>>>>
>>>>> We can think about that, but I think it may be quite confusing. The
>>>>> configurations actually mean something different for local and remote
>>>>> environments:
>>>>>
>>>>> - For the local environment, the config basically describes the
>>>>> entire Flink cluster setup (for the local execution cluster in the
>>>>> background)
>>>>> - For the remote environment, the config describes the parameters
>>>>> for the client that connects to the cluster (akka paramters, optimizer
>>>>> parameters, ...), but not parameters of the cluster itself (like
>>>>> taskmanager slots and memory).
>>>>>
>>>>> Greetings,
>>>>> Stephan
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <
>>>>> pomperma...@okkam.it> wrote:
>>>>>
>>>>>> However it could be a good idea to overload also
>>>>>> the getExecutionEnvironment() to be able to pass a custom
>>>>>> configuration..what do you think?
>>>>>> Otherwise I have to know a priori if I'm working in a local
>>>>>> deployment or in a remote one, or check if getExecutionEnvironment()
>>>>>> returned an instance of LocalEnvironment/RemoteEnvironment..
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>>>>> pomperma...@okkam.it> wrote:
>>>>>>
>>>>>>> Yes Stephan!
>>>>>>> I usually work with the master version, at least in development ;)
>>>>>>> Thanks for the quick support!
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> Are you on the SNAPSHOT master version?
>>>>>>>>
>>>>>>>> You can pass the configuration to the constructor of the execution
>>>>>>>> environment, or create one via
>>>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>>>>
>>>>>>>> The change of the signature was part of an API cleanup for the next
>>>>>>>> release. Sorry for the inconvenience...
>>>>>>>>
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>>
>>>>>>>>> Hi to all,
>>>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>>>>> parameters in my unit tests?
>>>>>>>>>
>>>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Flavio
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
