Re: Flink with Yarn on MapR

[ani.desh1512]

While it is difficult to pinpoint exactly what is wrong without Yarn logs,
there are a couple of things that I want you to ensure:
1. Is the YARN_CONF_DIR environment variable set?
2. Is the "yarn.nodemanager.resource.cpu-vcores" property set in
yarn-site.xml?

Let me know if these two things are in order and if they are, and you are
still getting the error, then  we will need to look at Yarn logs.



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-with-Yarn-on-MapR-tp15448p16249.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: Flink with Yarn on MapR

[ani.desh1512]

While it is difficult to pinpoint exactly what is wrong without Yarn logs,
there are a couple of things that I want you to ensure:
1. Is the YARN_CONF_DIR environment variable set?
2. Is the "yarn.nodemanager.resource.cpu-vcores" property set in
yarn-site.xml?

Let me know if these two things are in order and if they are, and you are
still getting the error, then  we will need to look at Yarn logs.



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-with-Yarn-on-MapR-tp15448p16248.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: Flink with Yarn on MapR

[ani.desh1512]

I was able to resolve this issue by referring to  this issue
  

In short, I had to modify flink's pom.xml to change the zookeeper version to
the zookeeper mapr version and this error the disappears.

But, I do have ran into a separate issue, which I will be starting a new
thread on.

Thanks for all the help Robert.





--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-with-Yarn-on-MapR-tp15448p15727.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: Flink with Yarn on MapR

[ani.desh1512]

Following is the output when I execute *./bin/yarn-session.sh -n 2*


/2017-01-25 15:53:57,805 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.address, 10.101.2.111
2017-01-25 15:53:57,807 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.port, 6123
2017-01-25 15:53:57,807 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.heap.mb, 8192
2017-01-25 15:53:57,807 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.heap.mb, 40960
2017-01-25 15:53:57,807 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.numberOfTaskSlots, 5
2017-01-25 15:53:57,807 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.memory.preallocate, false
2017-01-25 15:53:57,808 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: parallelism.default, 1
2017-01-25 15:53:57,808 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.web.port, 8081
2017-01-25 15:53:57,808 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.tmp.dirs, /tmp/flink
2017-01-25 15:53:57,809 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: fs.hdfs.hadoopconf,
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
2017-01-25 15:53:57,809 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: fs.default-scheme, maprfs:///
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.address, 10.101.2.111
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.port, 6123
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.heap.mb, 8192
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.heap.mb, 40960
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.numberOfTaskSlots, 5
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.memory.preallocate, false
2017-01-25 15:53:57,839 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: parallelism.default, 1
2017-01-25 15:53:57,840 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.web.port, 8081
2017-01-25 15:53:57,840 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.tmp.dirs, /tmp/flink
2017-01-25 15:53:57,841 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: fs.hdfs.hadoopconf,
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
2017-01-25 15:53:57,841 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: fs.default-scheme, maprfs:///
2017-01-25 15:53:59,356 WARN  org.apache.hadoop.util.NativeCodeLoader   
   
- Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2017-01-25 15:53:59,628 INFO 
org.apache.flink.runtime.security.modules.HadoopModule- Hadoop user
set to ubuntu (auth:SIMPLE)
2017-01-25 15:53:59,649 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.address, 10.101.2.111
2017-01-25 15:53:59,649 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.rpc.port, 6123
2017-01-25 15:53:59,650 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.heap.mb, 8192
2017-01-25 15:53:59,650 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.heap.mb, 40960
2017-01-25 15:53:59,650 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.numberOfTaskSlots, 5
2017-01-25 15:53:59,650 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: taskmanager.memory.preallocate, false
2017-01-25 15:53:59,650 INFO 
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: parallelism.default, 1
2017-01-25 15:

Re: Flink with Yarn on MapR

[ani.desh1512]

Robert Metzger wrote
> The second one is tougher to fix. It seems that there is an issue with
> loading the Hadoop configuration correctly.
> Can you post the contents of the client log file from the "log/"
> directory?
> It contains for example the Hadoop version being used (Maybe it didn't
> correctly pick up the custom Hadoop version) and maybe some helpful WARN
> log messages (because our YARN client is doing some checks before
> starting)

That is the problem. There are NO logs under "log/" folder.

P.S : Sorry for the dual post. I messed up on my part.




--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-with-Yarn-on-MapR-tp15448p15589.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.