of course,I tried to configure the task slot during a debug test and I
forgot to remove it..
Just for curiosity, is there any good reason why you've changed the default
parallellelism that way?and moreover, is it the only unexpected changed
behaviour wrt the previous API version?
On 14 Oct 2015 18:35, "Stephan Ewen" <se...@apache.org> wrote:
> Hi Flavio!
>
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment()
> by default picks up the number of cores as the parallelism, while the
> manual environments do not do that.
> You can still set it manually set the parallelism
> "env.setParallelism(Runtime.getRuntime().availableProcessors());"
>
> I would not configure the slots for the local execution, they should be
> automatically configured based on the max parallelism.
>
> Greetings,
> Stephan
>
>
> On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <pomperma...@okkam.it>
> wrote:
>
>> Hi Fabian and Stephan, back to work :)
>>
>> I finally managed to find the problem of the parallelism encountered by
>> my colleague!
>> Basically that was introduced by this API change. Before I was using
>> env.setConfiguration() to merge the default params with some custom ones.
>> Now, after the API change I was using the following code:
>>
>> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
>> if (env instanceof LocalEnvironment) {
>> Configuration c = new Configuration();
>> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR);
>>
>> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
>> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
>> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
>> env = ExecutionEnvironment.createLocalEnvironment(c);
>> }
>>
>> However, the first env and the reassigned one doesn't behave in the same
>> manner.
>> If I don't reassign env I have parallelism=8, otherwise it's 1 :(
>> Am I using the wrong APIs or the execution environment doesn't allow now
>> to configure such parameters anymore?
>>
>> Thanks in advance,
>> Flavio
>>
>>
>> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <pomperma...@okkam.it
>> > wrote:
>>
>>> That makes sense: what can be configured should be differentiated
>>> between local and remote envs (obviously this is a minor issue/improvement)
>>>
>>> Thanks again,
>>> Flavio
>>>
>>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> We can think about that, but I think it may be quite confusing. The
>>>> configurations actually mean something different for local and remote
>>>> environments:
>>>>
>>>> - For the local environment, the config basically describes the
>>>> entire Flink cluster setup (for the local execution cluster in the
>>>> background)
>>>> - For the remote environment, the config describes the parameters for
>>>> the client that connects to the cluster (akka paramters, optimizer
>>>> parameters, ...), but not parameters of the cluster itself (like
>>>> taskmanager slots and memory).
>>>>
>>>> Greetings,
>>>> Stephan
>>>>
>>>>
>>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <
>>>> pomperma...@okkam.it> wrote:
>>>>
>>>>> However it could be a good idea to overload also
>>>>> the getExecutionEnvironment() to be able to pass a custom
>>>>> configuration..what do you think?
>>>>> Otherwise I have to know a priori if I'm working in a local deployment
>>>>> or in a remote one, or check if getExecutionEnvironment() returned an
>>>>> instance of LocalEnvironment/RemoteEnvironment..
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>>>> pomperma...@okkam.it> wrote:
>>>>>
>>>>>> Yes Stephan!
>>>>>> I usually work with the master version, at least in development ;)
>>>>>> Thanks for the quick support!
>>>>>>
>>>>>> Best,
>>>>>> Flavio
>>>>>>
>>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> Are you on the SNAPSHOT master version?
>>>>>>>
>>>>>>> You can pass the configuration to the constructor of the execution
>>>>>>> environment, or create one via
>>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>>>
>>>>>>> The change of the signature was part of an API cleanup for the next
>>>>>>> release. Sorry for the inconvenience...
>>>>>>>
>>>>>>> Stephan
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>>>> pomperma...@okkam.it> wrote:
>>>>>>>
>>>>>>>> Hi to all,
>>>>>>>>
>>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>>>> parameters in my unit tests?
>>>>>>>>
>>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Flavio
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
