[ 
https://issues.apache.org/jira/browse/FLINK-9777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541745#comment-16541745
 ] 

ASF GitHub Bot commented on FLINK-9777:
---------------------------------------

Github user dawidwys commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6297#discussion_r202059575
  
    --- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
    @@ -386,10 +386,10 @@ private ClusterSpecification 
createClusterSpecification(Configuration configurat
                }
     
                // JobManager Memory
    -           final int jobManagerMemoryMB = 
MemorySize.parse(configuration.getString(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY)).getMebiBytes();
    +           final int jobManagerMemoryMB = 
MemorySize.parse(configuration.getString(JobManagerOptions.JOB_MANAGER_HEAP_MEMORY),
 MemorySize.MemoryUnit.MEGA_BYTES).getMebiBytes();
    --- End diff --
    
    Imagine a case when previously we had in configuration:
    
        jobmanager.heap.mb: 1024
    
    which meant 1024 MB, in the new code we will pass 1024 as value for 
`jobmanager.heap.size`, which was introduced solely to treat numbers without 
units as bytes. Therefore it will be parsed as 1024 bytes rather than 1024 MB.
    
    If we switch the default unit to MB, there is no point for having a new 
`ConfigOption`.


> YARN: JM and TM Memory must be specified with Units 
> ----------------------------------------------------
>
>                 Key: FLINK-9777
>                 URL: https://issues.apache.org/jira/browse/FLINK-9777
>             Project: Flink
>          Issue Type: Bug
>          Components: Documentation, YARN
>    Affects Versions: 1.6.0
>         Environment: commit 9f736d1927c62d220a82931c4f5ffa4955910f27
>            Reporter: Gary Yao
>            Assignee: vinoyang
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> FLINK-6469 breaks backwards compatibility because the JobManager and 
> TaskManager memory must be specified with units (otherwise bytes are 
> assumed). The command to start a YARN session as documented 
> ([https://github.com/apache/flink/blob/9f736d1927c62d220a82931c4f5ffa4955910f27/docs/ops/deployment/yarn_setup.md)
>  
> |https://github.com/apache/flink/blob/9f736d1927c62d220a82931c4f5ffa4955910f27/docs/ops/deployment/yarn_setup.md]
>  would not work because 1024 bytes and 4096 bytes are not enough for the heap 
> size. The command finishes with the following exception:
> {noformat}
> java.lang.reflect.UndeclaredThrowableException
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1854)
>       at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>       at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:802)
> Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: 
> Couldn't deploy Yarn session cluster
>       at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:420)
>       at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:599)
>       at 
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:802)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:422)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>       ... 2 more
> Caused by: org.apache.flink.util.FlinkException: Cannot fulfill the minimum 
> memory requirements with the provided cluster specification. Please increase 
> the memory of the cluster.
>       at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.validateClusterSpecification(AbstractYarnClusterDescriptor.java:453)
>       at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:475)
>       at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:413)
>       ... 7 more
> Caused by: java.lang.IllegalArgumentException
>       at 
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:123)
>       at 
> org.apache.flink.runtime.clusterframework.ContaineredTaskManagerParameters.calculateCutoffMB(ContaineredTaskManagerParameters.java:115)
>       at 
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.validateClusterSpecification(AbstractYarnClusterDescriptor.java:450)
>       ... 9 more
> {noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to