[jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
[ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763460#comment-15763460 ] Rohith Sharma K S commented on YARN-5585: - bq. Another option could be that if more than one row is encountered for a single entity read, we send some sort of error message indicating multiple idprefixes in backend which can alert the user/application of some issue on the write side. bq. the storage may throw exceptions or return errors when multiple prefixes are found for the same entity. It make sense to me. Indicating user with exception is better so that user would look into their code and fix it. [~gtCarrera9] For Entities, I think we can support below filters # fromId="idPrefix:entityId" where scan will start from this row key. # fromId="idPrefix" where scan will start from this row key. # But for supporting "*:entityId", the backend it is 2 queries which need to make where in first query is for finding prefix which need to do range scan, and secondly again range scan after finding entityPrefixId. And IMO, if we support only entityId then user always use entityId fromId. The entityIdPrefix will become unusable. For Entity, filter can be only idPrefix. # Filter name would be *idPrefix* as-is. If this filter is passed then use Get API else, range scan with SingleColumnValueFilter for matching entityId. > [Atsv2] Reader side changes for entity prefix and support for pagination via > additional filters > --- > > Key: YARN-5585 > URL: https://issues.apache.org/jira/browse/YARN-5585 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Labels: yarn-5355-merge-blocker > Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, > YARN-5585-YARN-5355.0002.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch > > > TimelineReader REST API's provides lot of filters to retrieve the > applications. Along with those, it would be good to add new filter i.e fromId > so that entities can be retrieved after the fromId. > Current Behavior : Default limit is set to 100. If there are 1000 entities > then REST call gives first/last 100 entities. How to retrieve next set of 100 > entities i.e 101 to 200 OR 900 to 801? > Example : If applications are stored database, app-1 app-2 ... app-10. > *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is > no way to achieve this. > So proposal is to have fromId in the filter like > *getApps?limit=5&=app-5* which gives list of apps from app-6 to > app-10. > Since ATS is targeting large number of entities storage, it is very common > use case to get next set of entities using fromId rather than querying all > the entites. This is very useful for pagination in web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5877) Allow all env's from yarn.nodemanager.env-whitelist to get overridden during launch
[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763457#comment-15763457
]
Sunil G commented on YARN-5877:
---
This patch will not be backported to branch-2 as environment variable
inheritance is not broken in branch-2.
> Allow all env's from yarn.nodemanager.env-whitelist to get overridden during
> launch
> ---
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 3.0.0-alpha2
>
> Attachments: 5877 nm logs of test.log, Dockerfile,
> YARN-5877.0001.patch, YARN-5877.0002.patch, YARN-5877.0003.patch,
> YARN-5877.0004.patch, YARN-5877.0005.patch, YARN-5877.0006.patch,
> bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
>
> Environment variables that containers may override rather
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
>
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5877) Allow all env's from yarn.nodemanager.env-whitelist to get overridden during launch
[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763447#comment-15763447
]
Hudson commented on YARN-5877:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11014 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11014/])
YARN-5877. Allow all env's from yarn.nodemanager.env-whitelist to get (sunilg:
rev 575773a3570b85293fdf7b8aeb8467580ec7f896)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
> Allow all env's from yarn.nodemanager.env-whitelist to get overridden during
> launch
> ---
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 3.0.0-alpha2
>
> Attachments: 5877 nm logs of test.log, Dockerfile,
> YARN-5877.0001.patch, YARN-5877.0002.patch, YARN-5877.0003.patch,
> YARN-5877.0004.patch, YARN-5877.0005.patch, YARN-5877.0006.patch,
> bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
>
> Environment variables that containers may override rather
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
>
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5877) Allow all env's from yarn.nodemanager.env-whitelist to get overridden during launch
[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated YARN-5877:
--
Summary: Allow all env's from yarn.nodemanager.env-whitelist to get
overridden during launch (was: Allow all nm-whitelist-env to get overridden
during launch)
> Allow all env's from yarn.nodemanager.env-whitelist to get overridden during
> launch
> ---
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 5877 nm logs of test.log, Dockerfile,
> YARN-5877.0001.patch, YARN-5877.0002.patch, YARN-5877.0003.patch,
> YARN-5877.0004.patch, YARN-5877.0005.patch, YARN-5877.0006.patch,
> bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
>
> Environment variables that containers may override rather
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
>
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6013) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
[
https://issues.apache.org/jira/browse/YARN-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763232#comment-15763232
]
Jian He commented on YARN-6013:
---
[~Steven Rand], do you have server side log where this exception happens?
> ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when
> RPC privacy is enabled
> --
>
> Key: YARN-6013
> URL: https://issues.apache.org/jira/browse/YARN-6013
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, yarn
>Affects Versions: 2.8.0
>Reporter: Steven Rand
>Priority: Critical
>
> When privacy is enabled for RPC (hadoop.rpc.protection = privacy),
> {{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always)
> fails with an EOFException. I've reproduced this with Spark 2.0.2 built
> against latest branch-2.8 and with a simple distcp job on latest branch-2.8.
> Steps to reproduce using distcp:
> 1. Set hadoop.rpc.protection equal to privacy
> 2. Write data to HDFS. I did this with Spark as follows:
> {code}
> sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k,
> org.apache.commons.lang.RandomStringUtils.random(1024,
> "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData")
> {code}
> 3. Attempt to distcp that data to another location in HDFS. For example:
> {code}
> hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData
> hdfs:///tmp/testDataCopy
> {code}
> I observed this error in the ApplicationMaster's syslog:
> {code}
> 2016-12-19 19:13:50,097 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1482189777425_0004, File:
> hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist
> 2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1482189777425_0004: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit= knownNMs=3
> 2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator]
> org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking
> ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after
> sleeping for 3ms.
> java.io.EOFException: End of File Exception between local host is:
> "/"; destination host is: "":8030;
> : java.io.EOFException; For more details see:
> http://wiki.apache.org/hadoop/EOFException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486)
> at org.apache.hadoop.ipc.Client.call(Client.java:1428)
> at org.apache.hadoop.ipc.Client.call(Client.java:1338)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy80.allocate(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> at
>
[jira] [Updated] (YARN-6014) Followup fix for slider core module findbugs
[ https://issues.apache.org/jira/browse/YARN-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6014: -- Attachment: YARN-6014-yarn-native-services.01.patch > Followup fix for slider core module findbugs > > > Key: YARN-6014 > URL: https://issues.apache.org/jira/browse/YARN-6014 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6014-yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6014) Followup fix for slider core module findbugs
Jian He created YARN-6014: - Summary: Followup fix for slider core module findbugs Key: YARN-6014 URL: https://issues.apache.org/jira/browse/YARN-6014 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763042#comment-15763042
]
zhengchenyu commented on YARN-5936:
---
I don't care the performance loss of cpu bandwith, because I know it must
happen. I only wanna know the method of keep fair without cpu bandwith.
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763023#comment-15763023
]
zhengchenyu commented on YARN-5936:
---
But your "time" command is only related to it's own program, every program is
single thread.
My question is below:
two process that has different numbers of threads has different ability of
schedule, though they have the same cpu.share.
I know the reason, but I don't have a proper suggestion to avoid this problem.
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengchenyu updated YARN-5936:
--
Comment: was deleted
(was: But your "time" command is only related to it's own program, every
program is single thread.
My question is below:
two process that has different numbers of threads has different ability of
schedule, though they have the same cpu.share.
I know the reason, but I don't have a proper suggestion to avoid this problem.)
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengchenyu updated YARN-5936:
--
Comment: was deleted
(was: But your "time" command is only related to it's own program, every
program is single thread.
My question is below:
two process that has different numbers of threads has different ability of
schedule, though they have the same cpu.share.
I know the reason, but I don't have a proper suggestion to avoid this problem.)
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[
https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5938:
--
Attachment: YARN-5938-YARN-5085.004.patch
Updating patch based on [~leftnoteasy]'s suggestions.
bq. ApplicationMasterService#allocate -> internalAllocate
I renamed it to {{allocateInternal()}} to maintain consistency with other
places such as ContainerManager#startContainerInternal()
> Minor refactoring to OpportunisticContainerAllocatorAMService
> -
>
> Key: YARN-5938
> URL: https://issues.apache.org/jira/browse/YARN-5938
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5938-YARN-5085.001.patch,
> YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch,
> YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch
>
>
> Minor code re-organization to do the following:
> # The OpportunisticContainerAllocatorAMService currently allocates outside
> the ApplicationAttempt lock maintained by the ApplicationMasterService. This
> should happen inside the lock.
> # Refactored out some code to simplify the allocate() method.
> # Removed some unused fields inside the OpportunisticContainerAllocator.
> # Re-organized some of the code in the
> OpportunisticContainerAllocatorAMService::allocate method to make it a bit
> more readable.
> # Moved SchedulerRequestKey to a new package, so it can be used by the
> OpportunisticContainerAllocator/Context.
> # Moved all usages of Priority in the OpportunisticContainerAllocator ->
> SchedulerRequestKey.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763021#comment-15763021
]
zhengchenyu commented on YARN-5936:
---
But your "time" command is only related to it's own program, every program is
single thread.
My question is below:
two process that has different numbers of threads has different ability of
schedule, though they have the same cpu.share.
I know the reason, but I don't have a proper suggestion to avoid this problem.
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763020#comment-15763020
]
zhengchenyu commented on YARN-5936:
---
But your "time" command is only related to it's own program, every program is
single thread.
My question is below:
two process that has different numbers of threads has different ability of
schedule, though they have the same cpu.share.
I know the reason, but I don't have a proper suggestion to avoid this problem.
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5709) Cleanup leader election configs and pluggability
[
https://issues.apache.org/jira/browse/YARN-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-5709:
-
Priority: Critical (was: Blocker)
> Cleanup leader election configs and pluggability
>
>
> Key: YARN-5709
> URL: https://issues.apache.org/jira/browse/YARN-5709
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: yarn-5709-wip.2.patch, yarn-5709.1.patch,
> yarn-5709.2.patch, yarn-5709.3.patch, yarn-5709.4.patch
>
>
> While reviewing YARN-5677 and YARN-5694, I noticed we could make the
> curator-based election code cleaner. It is nicer to get this fixed in 2.8
> before we ship it, but this can be done at a later time as well.
> # By EmbeddedElector, we meant it was running as part of the RM daemon. Since
> the Curator-based elector is also running embedded, I feel the code should be
> checking for {{!curatorBased}} instead of {{isEmbeddedElector}}
> # {{LeaderElectorService}} should probably be named
> {{CuratorBasedEmbeddedElectorService}} or some such.
> # The code that initializes the elector should be at the same place
> irrespective of whether it is curator-based or not.
> # We seem to be caching the CuratorFramework instance in RM. It makes more
> sense for it to be in RMContext. If others are okay with it, we might even be
> better of having {{RMContext#getCurator()}} method to lazily create the
> curator framework and then cache it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5709) Cleanup leader election configs and pluggability
[
https://issues.apache.org/jira/browse/YARN-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762999#comment-15762999
]
Junping Du commented on YARN-5709:
--
Hi [~kasha], do you get chance to look at it recently? If not, I am OK to defer
it to next release given arguably this refactor work doesn't belongs to real
blocker for release.
> Cleanup leader election configs and pluggability
>
>
> Key: YARN-5709
> URL: https://issues.apache.org/jira/browse/YARN-5709
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Blocker
> Fix For: 3.0.0-alpha2
>
> Attachments: yarn-5709-wip.2.patch, yarn-5709.1.patch,
> yarn-5709.2.patch, yarn-5709.3.patch, yarn-5709.4.patch
>
>
> While reviewing YARN-5677 and YARN-5694, I noticed we could make the
> curator-based election code cleaner. It is nicer to get this fixed in 2.8
> before we ship it, but this can be done at a later time as well.
> # By EmbeddedElector, we meant it was running as part of the RM daemon. Since
> the Curator-based elector is also running embedded, I feel the code should be
> checking for {{!curatorBased}} instead of {{isEmbeddedElector}}
> # {{LeaderElectorService}} should probably be named
> {{CuratorBasedEmbeddedElectorService}} or some such.
> # The code that initializes the elector should be at the same place
> irrespective of whether it is curator-based or not.
> # We seem to be caching the CuratorFramework instance in RM. It makes more
> sense for it to be in RMContext. If others are okay with it, we might even be
> better of having {{RMContext#getCurator()}} method to lazily create the
> curator framework and then cache it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762998#comment-15762998
]
Miklos Szegedi commented on YARN-5936:
--
Indeed, I see some 6% performance loss in the second case above compared to the
first. This is when I move away from the root cgroup and use the hierarchy but
do not use the limit, yet. This happens, when I am running 10 processes in
different cgroups with 100 threads each.
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4675) Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762940#comment-15762940 ] Varun Saxena commented on YARN-4675: Yes, I think we can have new interface as TimelineV2Client and impl as TimelineV2ClientImpl. Also TimelineV2Client.createTimelineClient should be fine to get the v2 instance > Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: YARN-4675-YARN-2928.v1.001.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762880#comment-15762880 ] Arun Suresh commented on YARN-5938: --- Thanks for the review [~leftnoteasy] bq. OpportunisticContainerAllocator#allocateContainers, why change from AllocateRequest to several separated param? Is it possible that we need some other fields in AllocateRequest? So, Firslty, I wanted to decouple the OpportunisticContainerAllocator from the AllocateRequest object and Secondly.. we were doing 1. allocateRequest.setAskList() 2. then OpportunisticContainerAllocator::allocate() 3. then doing a allocateRequest.setAskList() 4. super.allocate() The above sequence seemed a bit "hacky"... and made a code flow a bit difficult to follow.. the refactoring fixes that. It is possible we might need other things later from the AllocateRequest, but we can address that as and when we need to (there are better ways like updating the OpportunisticContainerContext etc.) Will update the patch addressing your remaining comments shortly.. > Minor refactoring to OpportunisticContainerAllocatorAMService > - > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762877#comment-15762877
]
Hudson commented on YARN-5524:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11012 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11012/])
YARN-5524. Yarn live log aggregation does not throw if command line arg
(naganarasimha_gr: rev 358341398adfe8f59bb2165472b50751de3ffb98)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Xuan Gong
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch,
> YARN-5524.3.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4675) Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl
[
https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762866#comment-15762866
]
Naganarasimha G R commented on YARN-4675:
-
Thanks for the discussion [~sjlee0], [~gtCarrera9] and [~varun_saxena] and
sorry for the delay.
few clarifications in the finalized approach:
# So we have finalized that there will be 2 different interfaces for ATSv1(&
1.5) and ATSV2, So shall i name the new interface for V2 as
{{TimelineV2Client}} and impl as {{TimelineV2ClientImpl}} ?
# {{TimelineV2Client.createTimelineClient}} would this interface be fine to get
the v2 instance ?
> Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl
>
>
> Key: YARN-4675
> URL: https://issues.apache.org/jira/browse/YARN-4675
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Labels: YARN-5355, yarn-5355-merge-blocker
> Attachments: YARN-4675-YARN-2928.v1.001.patch
>
>
> We need to reorganize TimeClientImpl into TimeClientV1Impl ,
> TimeClientV2Impl and if required a base class, so that its clear which part
> of the code belongs to which version and thus better maintainable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762856#comment-15762856 ] Wangda Tan commented on YARN-5938: -- [~asuresh], Thanks for working on this patch, generally looks good, several minor comments: 1) OpportunisticContainerAllocator#allocateContainers, why change from AllocateRequest to several separated param? Is it possible that we need some other fields in AllocateRequest? 2) ApplicationMasterService#allocate -> internalAllocate 3) OpportunisticContainerContext: Remove unused imports > Minor refactoring to OpportunisticContainerAllocatorAMService > - > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762814#comment-15762814
]
Naganarasimha G R commented on YARN-5524:
-
Thanks [~vrushalic], as the approach and the patch was given out by [~xgong], i
have assigned the jira to him and as the patch LGTM will commit it shortly!
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch,
> YARN-5524.3.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naganarasimha G R updated YARN-5524:
Assignee: Xuan Gong (was: Vrushali C)
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Xuan Gong
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch,
> YARN-5524.3.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6013) ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled
Steven Rand created YARN-6013:
-
Summary: ApplicationMasterProtocolPBClientImpl.allocate fails with
EOFException when RPC privacy is enabled
Key: YARN-6013
URL: https://issues.apache.org/jira/browse/YARN-6013
Project: Hadoop YARN
Issue Type: Bug
Components: client, yarn
Affects Versions: 2.8.0
Reporter: Steven Rand
Priority: Critical
When privacy is enabled for RPC (hadoop.rpc.protection = privacy),
{{ApplicationMasterProtocolPBClientImpl.allocate}} sometimes (but not always)
fails with an EOFException. I've reproduced this with Spark 2.0.2 built against
latest branch-2.8 and with a simple distcp job on latest branch-2.8.
Steps to reproduce using distcp:
1. Set hadoop.rpc.protection equal to privacy
2. Write data to HDFS. I did this with Spark as follows:
{code}
sc.parallelize(1 to (5*1024*1024)).map(k => Seq(k,
org.apache.commons.lang.RandomStringUtils.random(1024,
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWxyZ0123456789")).mkString("|")).toDF().repartition(100).write.parquet("hdfs:///tmp/testData")
{code}
3. Attempt to distcp that data to another location in HDFS. For example:
{code}
hadoop distcp -Dmapreduce.framework.name=yarn hdfs:///tmp/testData
hdfs:///tmp/testDataCopy
{code}
I observed this error in the ApplicationMaster's syslog:
{code}
2016-12-19 19:13:50,097 INFO [eventHandlingThread]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
setup for JobId: job_1482189777425_0004, File:
hdfs://:8020/tmp/hadoop-yarn/staging//.staging/job_1482189777425_0004/job_1482189777425_0004_1.jhist
2016-12-19 19:13:51,004 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling:
PendingReds:0 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2016-12-19 19:13:51,031 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for
application_1482189777425_0004: ask=1 release= 0 newContainers=0
finishedContainers=0 resourcelimit= knownNMs=3
2016-12-19 19:13:52,043 INFO [RMCommunicator Allocator]
org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking
ApplicationMasterProtocolPBClientImpl.allocate over null. Retrying after
sleeping for 3ms.
java.io.EOFException: End of File Exception between local host is:
"/"; destination host is: "":8030; :
java.io.EOFException; For more details see:
http://wiki.apache.org/hadoop/EOFException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1486)
at org.apache.hadoop.ipc.Client.call(Client.java:1428)
at org.apache.hadoop.ipc.Client.call(Client.java:1338)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy80.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335)
at com.sun.proxy.$Proxy81.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:204)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:735)
at
[jira] [Commented] (YARN-5756) Add state-machine implementation for queues
[
https://issues.apache.org/jira/browse/YARN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762758#comment-15762758
]
Wangda Tan commented on YARN-5756:
--
Thanks [~xgong], generally looks good, only one comment, could you add some
test cases to make sure {{initializeQueueState}} works well?
I'm afraid regression could easily happen for any change to the method.
> Add state-machine implementation for queues
> ---
>
> Key: YARN-5756
> URL: https://issues.apache.org/jira/browse/YARN-5756
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-5756.1.patch, YARN-5756.2.patch, YARN-5756.3.patch,
> YARN-5756.4.patch, YARN-5756.5.patch, YARN-5756.6.patch, YARN-5756.6.patch,
> YARN-5756.7.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5774) MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set yarn.scheduler.minimum-allocation-mb to 0.
[
https://issues.apache.org/jira/browse/YARN-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762745#comment-15762745
]
Wangda Tan commented on YARN-5774:
--
Hi [~yufeigu], [~templedf],
I noticed this change while reviewing YARN-5959. This change looks like an
incompatible change to me, it removed public/stable method from
ResourceRequest, which makes API doc different. And now ResourceRequest
(public/stable) inherits from a public/unstable class.
I can understand it is added for making normalizeResource to handle both of
updateContainerRequest / ResourceRequest. But changing user facing API to
simplify internal implementation doesn't look like a good idea to me.
Instead, I suggest to revert the AbstractResourceRequest change, and update
{{void normalizeRequest(AbstractResourceRequest request);}} to {{Resource
getNormalizedResource(Resource askedResource)}} to make "what to normalize"
more clear.
Thoughts?
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler if set
> yarn.scheduler.minimum-allocation-mb to 0.
>
>
> Key: YARN-5774
> URL: https://issues.apache.org/jira/browse/YARN-5774
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Labels: oct16-easy
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: YARN-5774.001.patch, YARN-5774.002.patch,
> YARN-5774.003.patch, YARN-5774.004.patch, YARN-5774.005.patch,
> YARN-5774.006.patch, YARN-5774.007.patch
>
>
> MR Job stuck in ACCEPTED status without any progress in Fair Scheduler
> because there is no resource request for the AM. This happened when you
> configure {{yarn.scheduler.minimum-allocation-mb}} to zero.
> The problem is in the code used by both Capacity Scheduler and Fair
> Scheduler. {{scheduler.increment-allocation-mb}} is a concept in FS, but not
> CS. So the common code in class RMAppManager passes the
> {{yarn.scheduler.minimum-allocation-mb}} as incremental one because there is
> no incremental one for CS when it tried to normalize the resource requests.
> {code}
> SchedulerUtils.normalizeRequest(amReq, scheduler.getResourceCalculator(),
> scheduler.getClusterResource(),
> scheduler.getMinimumResourceCapability(),
> scheduler.getMaximumResourceCapability(),
> scheduler.getMinimumResourceCapability()); --> incrementResource
> should be passed here.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5529) Create new DiskValidator class with metrics
[
https://issues.apache.org/jira/browse/YARN-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762738#comment-15762738
]
Hadoop QA commented on YARN-5529:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 43s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.io.file.tfile.TestVLong |
| | hadoop.io.file.tfile.TestTFileByteArrays |
| | hadoop.util.TestJarFinder |
| | hadoop.fs.contract.localfs.TestLocalFSContractSetTimes |
| | hadoop.fs.contract.rawlocal.TestRawlocalContractAppend |
| | hadoop.fs.TestFsShellCopy |
| | hadoop.fs.contract.localfs.TestLocalFSContractOpen |
| | hadoop.crypto.TestCryptoStreamsForLocalFS |
| | hadoop.fs.shell.TestTextCommand |
| | hadoop.fs.viewfs.TestFcCreateMkdirLocalFs |
| | hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal |
| | hadoop.fs.contract.localfs.TestLocalFSContractDelete |
| | hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem |
| | hadoop.fs.TestLocalFSFileContextMainOperations |
| | hadoop.fs.contract.localfs.TestLocalFSContractAppend |
| | hadoop.fs.TestDFCachingGetSpaceUsed |
| | hadoop.io.file.tfile.TestTFileNoneCodecsStreams |
| | hadoop.fs.TestSymlinkLocalFSFileContext |
| | hadoop.fs.contract.rawlocal.TestRawlocalContractOpen |
| | hadoop.fs.TestLocalFileSystemPermission |
| | hadoop.fs.TestFcLocalFsPermission |
| | hadoop.fs.viewfs.TestChRootedFileSystem |
| | hadoop.fs.TestLocalFileSystem |
| | hadoop.fs.contract.rawlocal.TestRawlocalContractGetFileStatus |
| | hadoop.io.TestArrayFile |
| | hadoop.fs.TestFsShellList |
| | hadoop.fs.TestFileContextDeleteOnExit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5529 |
| JIRA Patch URL |
[jira] [Commented] (YARN-6010) Fix findbugs, site warnings in yarn-services-api module
[ https://issues.apache.org/jira/browse/YARN-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762735#comment-15762735 ] Billie Rinaldi commented on YARN-6010: -- +1, committing with whitespace=fix. > Fix findbugs, site warnings in yarn-services-api module > --- > > Key: YARN-6010 > URL: https://issues.apache.org/jira/browse/YARN-6010 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6010-yarn-native-services.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
[ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762691#comment-15762691 ] Li Lu commented on YARN-5585: - Thanks [~rohithsharma] for the update! With regards to the APIs, I think we can pretty much reuse the current set of APIs. IMO, we should not force a prefix all the time. Of course, if the user knows the exact entity prefix it's certainly beneficial to include it in the query (so that we can save a range scan and just use a get). When referring to timeline entity ids, how about the following patterns: 1. !: string 1 is the prefix and string 2 is the id 2. or *\!: string is the entity id and the storage needs to query the entity prefix. If we have problems distinguishing from the above case maybe we can use *\! bq. If we plan to reuse same API's, then we need to handle one scenario where same entityId is published with 2 entityIdPrefix. This sounds like a really messy situation. Semantically, we've got two ways to decide this: 1) we explicitly claim that entity prefix id is a part of the id system. This means two entities are different even if they only only differ in entity prefixes and 2) we claim that entity prefix is _not_ a part of the id system. Under this assumption, it is up to the storage system to decide how to deal the case which prefixes are updated. Therefore the behavior when one entity is associated with two prefixes, from the API level, is undefined. As [~varun_saxena] suggested, the storage may throw exceptions or return errors when multiple prefixes are found for the same entity. > [Atsv2] Reader side changes for entity prefix and support for pagination via > additional filters > --- > > Key: YARN-5585 > URL: https://issues.apache.org/jira/browse/YARN-5585 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Labels: yarn-5355-merge-blocker > Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, > YARN-5585-YARN-5355.0002.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch > > > TimelineReader REST API's provides lot of filters to retrieve the > applications. Along with those, it would be good to add new filter i.e fromId > so that entities can be retrieved after the fromId. > Current Behavior : Default limit is set to 100. If there are 1000 entities > then REST call gives first/last 100 entities. How to retrieve next set of 100 > entities i.e 101 to 200 OR 900 to 801? > Example : If applications are stored database, app-1 app-2 ... app-10. > *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is > no way to achieve this. > So proposal is to have fromId in the filter like > *getApps?limit=5&=app-5* which gives list of apps from app-6 to > app-10. > Since ATS is targeting large number of entities storage, it is very common > use case to get next set of entities using fromId rather than querying all > the entites. This is very useful for pagination in web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5529) Create new DiskValidator class with metrics
[
https://issues.apache.org/jira/browse/YARN-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762690#comment-15762690
]
Hadoop QA commented on YARN-5529:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 18s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestBasicDiskValidator |
| | hadoop.http.TestSSLHttpServer |
| | hadoop.util.TestGenericOptionsParser |
| | hadoop.util.TestApplicationClassLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5529 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843964/YARN-5529.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 87f9415a2030 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ef2dd7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14368/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14368/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14368/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14368/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create new DiskValidator class with metrics
[jira] [Updated] (YARN-5529) Create new DiskValidator class with metrics
[ https://issues.apache.org/jira/browse/YARN-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-5529: --- Attachment: YARN-5529.002.patch > Create new DiskValidator class with metrics > --- > > Key: YARN-5529 > URL: https://issues.apache.org/jira/browse/YARN-5529 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ray Chiang >Assignee: Yufei Gu > Labels: supportability > Attachments: YARN-5529.001.patch, YARN-5529.002.patch > > > With really large clusters, the basic DiskValidator isn't sufficient for some > of the less common types of disk failures. > Look at a new DiskValidator that could do one or more of the following: > - Add new tests to find more problems > - Add new metrics to at least characterize problems that we haven't predicted -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5529) Create new DiskValidator class with metrics
[ https://issues.apache.org/jira/browse/YARN-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-5529: --- Attachment: (was: YARN-5529.002.patch) > Create new DiskValidator class with metrics > --- > > Key: YARN-5529 > URL: https://issues.apache.org/jira/browse/YARN-5529 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ray Chiang >Assignee: Yufei Gu > Labels: supportability > Attachments: YARN-5529.001.patch > > > With really large clusters, the basic DiskValidator isn't sufficient for some > of the less common types of disk failures. > Look at a new DiskValidator that could do one or more of the following: > - Add new tests to find more problems > - Add new metrics to at least characterize problems that we haven't predicted -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned YARN-5304: Assignee: Vrushali C > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5529) Create new DiskValidator class with metrics
[ https://issues.apache.org/jira/browse/YARN-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-5529: --- Attachment: YARN-5529.002.patch > Create new DiskValidator class with metrics > --- > > Key: YARN-5529 > URL: https://issues.apache.org/jira/browse/YARN-5529 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ray Chiang >Assignee: Yufei Gu > Labels: supportability > Attachments: YARN-5529.001.patch, YARN-5529.002.patch > > > With really large clusters, the basic DiskValidator isn't sufficient for some > of the less common types of disk failures. > Look at a new DiskValidator that could do one or more of the following: > - Add new tests to find more problems > - Add new metrics to at least characterize problems that we haven't predicted -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762438#comment-15762438 ] Li Lu commented on YARN-4061: - Thanks [~jrottinghuis]! bq. For our usecase, that makes the puts idempotent. Other use-cases may not need this requirement, but they do need to deal with duplicate puts. That makes sense to me. However do we think this is too specific (at least for now) for our use case in timeline v2? I can understand if there are concerns in the HBase community if we'd like to put this immediately into HBase codebase... Maybe what we can do is to expose buffered mutators from Hbase, and implement our own spooling buffered mutator in timeline code? > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Joep Rottinghuis > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5647) [Security] Collector and reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762433#comment-15762433 ] Sangjin Lee commented on YARN-5647: --- Related to this, there is the scenario of using a reader instance to access other clusters' data than the one the reader belongs to. It's not clear how to implement/enforce authentication for that. Perhaps accessing other clusters' data should be reconsidered if security is enabled? > [Security] Collector and reader side changes for loading auth filters and > principals > > > Key: YARN-5647 > URL: https://issues.apache.org/jira/browse/YARN-5647 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-5647-YARN-5355.wip.002.patch, > YARN-5647-YARN-5355.wip.003.patch, YARN-5647-YARN-5355.wip.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4990) Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion
[ https://issues.apache.org/jira/browse/YARN-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762386#comment-15762386 ] Junping Du commented on YARN-4990: -- Thanks [~xgong] for uploading a patch to fix this. The patch looks good in general. However, may be we can add some sanity UT here? just like what we do in TestAMFilter, etc. > Re-direction of a particular log file within in a container in NM UI does not > redirect properly to Log Server ( history ) on container completion > - > > Key: YARN-4990 > URL: https://issues.apache.org/jira/browse/YARN-4990 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Xuan Gong > Attachments: YARN-4990.1.patch > > > The NM does the redirection to the history server correctly. However if the > user is viewing or has a link to a particular specific file, the redirect > ends up going to the top level page for the container and not redirecting to > the specific file. Additionally, the start param to show logs from the offset > 0 also goes missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5993) Allow native services quicklinks to be exported for each component
[
https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762325#comment-15762325
]
Hadoop QA commented on YARN-5993:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
in yarn-native-services has 6 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services
failed. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core:
The patch generated 5 new + 170 unchanged - 6 fixed = 175 total (was 176)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5993 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843950/YARN-5993-yarn-native-services.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b57915817408 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / af649d2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/14367/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core-warnings.html
|
| javadoc |
[jira] [Updated] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
[ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-4061: -- Labels: YARN-5355 yarn-5355-merge-blocker (was: YARN-5355) > [Fault tolerance] Fault tolerant writer for timeline v2 > --- > > Key: YARN-4061 > URL: https://issues.apache.org/jira/browse/YARN-4061 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Joep Rottinghuis > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: FaulttolerantwriterforTimelinev2.pdf > > > We need to build a timeline writer that can be resistant to backend storage > down time and timeline collector failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-5304: --- Assignee: (was: Joep Rottinghuis) > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5993) Allow native services quicklinks to be exported for each component
[ https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-5993: - Attachment: YARN-5993-yarn-native-services.003.patch > Allow native services quicklinks to be exported for each component > -- > > Key: YARN-5993 > URL: https://issues.apache.org/jira/browse/YARN-5993 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-5993-yarn-native-services.001.patch, > YARN-5993-yarn-native-services.002.patch, > YARN-5993-yarn-native-services.003.patch > > > The quicklinks export capability changed in switching from the agent provider > to the docker provider, and currently the docker provider only allows one > component to export links. We should improve this capability to be more in > line with the previous agent capability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5993) Allow native services quicklinks to be exported for each component
[
https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762261#comment-15762261
]
Hadoop QA commented on YARN-5993:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-5993 does not apply to yarn-native-services. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5993 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843949/YARN-5993-yarn-native-services.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14366/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Allow native services quicklinks to be exported for each component
> --
>
> Key: YARN-5993
> URL: https://issues.apache.org/jira/browse/YARN-5993
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5993-yarn-native-services.001.patch,
> YARN-5993-yarn-native-services.002.patch
>
>
> The quicklinks export capability changed in switching from the agent provider
> to the docker provider, and currently the docker provider only allows one
> component to export links. We should improve this capability to be more in
> line with the previous agent capability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5993) Allow native services quicklinks to be exported for each component
[ https://issues.apache.org/jira/browse/YARN-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-5993: - Attachment: YARN-5993-yarn-native-services.002.patch > Allow native services quicklinks to be exported for each component > -- > > Key: YARN-5993 > URL: https://issues.apache.org/jira/browse/YARN-5993 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-5993-yarn-native-services.001.patch, > YARN-5993-yarn-native-services.002.patch > > > The quicklinks export capability changed in switching from the agent provider > to the docker provider, and currently the docker provider only allows one > component to export links. We should improve this capability to be more in > line with the previous agent capability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6011) Add a new web service to list the files on a container in AHSWebService
[
https://issues.apache.org/jira/browse/YARN-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762176#comment-15762176
]
Hadoop QA commented on YARN-6011:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new +
30 unchanged - 1 fixed = 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 52s{color}
| {color:red} hadoop-yarn-server-applicationhistoryservice in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.timeline.webapp.TestTimelineWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843937/YARN-6011.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1ca4afdb2648 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ef2dd7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14365/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
|
| unit |
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762142#comment-15762142
]
Vrushali C commented on YARN-5524:
--
Sorry for the delay, please feel free to reassign the jira to yourself, [~xgong]
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation
>Affects Versions: 2.9.0
>Reporter: Prasanth Jayachandran
>Assignee: Vrushali C
> Attachments: YARN-5524.001.patch, YARN-5524.002.patch,
> YARN-5524.3.patch
>
>
> When we used wrong command line arg for specify log file pattern, yarn did
> not throw any exception instead it pulled entire log onto the client.
> {code}
> [hive@ctr-e20-1468887904486-0007-01-03 ~]$ yarn logs -applicationId
> application_1470931023753_0001 -logFiles hive.*2016-08-11-21.* > logs
> {code}
> NOTE: we are using -logFiles instead of -log_files
> This query should have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5524) Yarn live log aggregation does not throw if command line arg is wrong
[
https://issues.apache.org/jira/browse/YARN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762120#comment-15762120
]
Hadoop QA commented on YARN-5524:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 3 new +
99 unchanged - 0 fixed = 102 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
25s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5524 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843778/YARN-5524.3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 60164caa3a43 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ef2dd7b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14364/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14364/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14364/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Yarn live log aggregation does not throw if command line arg is wrong
> -
>
> Key: YARN-5524
> URL: https://issues.apache.org/jira/browse/YARN-5524
> Project: Hadoop YARN
> Issue Type: Sub-task
>
[jira] [Updated] (YARN-6011) Add a new web service to list the files on a container in AHSWebService
[ https://issues.apache.org/jira/browse/YARN-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6011: Attachment: YARN-6011.2.patch > Add a new web service to list the files on a container in AHSWebService > --- > > Key: YARN-6011 > URL: https://issues.apache.org/jira/browse/YARN-6011 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-6011.1.patch, YARN-6011.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5647) [Security] Collector and reader side changes for loading auth filters and principals
[
https://issues.apache.org/jira/browse/YARN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762056#comment-15762056
]
Sangjin Lee commented on YARN-5647:
---
{quote}
As of now, we will have one level db storage per NM (per NM Collector Manager).
This, however, would need a change once collector (not collector manager)
itself is launched as a separate process i.e. as a system container if we ever
do it. We need to see how to handle failures for such a container. And that
would require more discussion.
Anyways token recovery related stuff will be handled in a separate JIRA. So we
can have more discussion in detail there once that is opened.
{quote}
Thanks for the clarification [~varun_saxena]. I understand that having timeline
collectors as out-of-process containers would be different in this regard. I
think it is acceptable to capture that as part of the container work.
I also think having multiple reader instances might complicate things here.
Today it is still a daemon, so there would be a single instance per cluster, so
we might be OK with using a local LevelDB attached to it.
Something to think about and answer, however: how do we ensure we don't get
mixed up with the timeline state store from v.1 on the same node? Is it not a
concern because in v.1 the state store would be on the RM machine? But that
would also mean we need to be careful with running the reader on the RM
machine? Should we at least see if we can differentiate the v.1 timeline state
store and v.2 so they don't get mixed up?
> [Security] Collector and reader side changes for loading auth filters and
> principals
>
>
> Key: YARN-5647
> URL: https://issues.apache.org/jira/browse/YARN-5647
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-5647-YARN-5355.wip.002.patch,
> YARN-5647-YARN-5355.wip.003.patch, YARN-5647-YARN-5355.wip.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3053) [Security] Review and implement security in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761939#comment-15761939 ] Varun Saxena commented on YARN-3053: Only authentication. Will change the title > [Security] Review and implement security in ATS v.2 > --- > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5378) Accomodate app-id->cluster mapping
[ https://issues.apache.org/jira/browse/YARN-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5378: -- Labels: yarn-5355-merge-blocker (was: ) > Accomodate app-id->cluster mapping > -- > > Key: YARN-5378 > URL: https://issues.apache.org/jira/browse/YARN-5378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: yarn-5355-merge-blocker > > In discussion with [~sjlee0], [~vrushalic], [~subru], and [~curino] a > use-case came up to be able to map from application-id to cluster-id in > context of federation for Yarn. > What happens is that a "random" cluster in the federation is asked to > generate an app-id and then potentially a different cluster can be the "home" > cluster for the AM. Furthermore, tasks can then run in yet other clusters. > In order to be able to pull up the logical home cluster on which the > application ran, there needs to be a mapping from application-id to > cluster-id. This mapping is available in the federated Yarn case only during > the active live of the application. > A similar situation is common in our larger production environment. Somebody > will complain about a slow job, some failure or whatever. If we're lucky we > have an application-id. When we ask the user which cluster they ran on, > they'll typically answer with the machine from where they launched the job > (many users are unaware of the underlying physical clusters). This leaves us > to spelunk through various RM ui's to find a matching epoch in the > application ID. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4675) Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-4675: -- Labels: YARN-5355 yarn-5355-merge-blocker (was: YARN-5355 oct16-medium) > Reorganize TimeClientImpl into TimeClientV1Impl and TimeClientV2Impl > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: YARN-4675-YARN-2928.v1.001.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5980) Update documentation for single node hbase deploy
[ https://issues.apache.org/jira/browse/YARN-5980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5980: -- Labels: yarn-5355-merge-blocker (was: ) > Update documentation for single node hbase deploy > - > > Key: YARN-5980 > URL: https://issues.apache.org/jira/browse/YARN-5980 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Attachments: YARN-5980.001.patch > > > Per HBASE-17272, a single node hbase deployment (single jvm running daemons + > hdfs writes) will be added to hbase shortly. > We should update the timeline service documentation in the setup/deployment > context accordingly, this will help users who are a bit wary of hbase > deployments help get started with timeline service more easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5976) Update hbase version to 1.2
[ https://issues.apache.org/jira/browse/YARN-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5976: -- Labels: yarn-5355-merge-blocker (was: ) > Update hbase version to 1.2 > --- > > Key: YARN-5976 > URL: https://issues.apache.org/jira/browse/YARN-5976 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-5976.001.wip.patch, YARN-5976.002.wip.patch > > > I believe phoenix now works with hbase 1.2. We should now upgrade timeline > service to use hbase 1.2 now. > And also update documentation in timelineservice to reflect that hbase mode > of all daemons in single jvm but writing to hdfs is supported. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
[ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5585: -- Labels: yarn-5355-merge-blocker (was: oct16-hard) > [Atsv2] Reader side changes for entity prefix and support for pagination via > additional filters > --- > > Key: YARN-5585 > URL: https://issues.apache.org/jira/browse/YARN-5585 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Labels: yarn-5355-merge-blocker > Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, > YARN-5585-YARN-5355.0002.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch > > > TimelineReader REST API's provides lot of filters to retrieve the > applications. Along with those, it would be good to add new filter i.e fromId > so that entities can be retrieved after the fromId. > Current Behavior : Default limit is set to 100. If there are 1000 entities > then REST call gives first/last 100 entities. How to retrieve next set of 100 > entities i.e 101 to 200 OR 900 to 801? > Example : If applications are stored database, app-1 app-2 ... app-10. > *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is > no way to achieve this. > So proposal is to have fromId in the filter like > *getApps?limit=5&=app-5* which gives list of apps from app-6 to > app-10. > Since ATS is targeting large number of entities storage, it is very common > use case to get next set of entities using fromId rather than querying all > the entites. This is very useful for pagination in web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5647) [Security] Collector and reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5647: -- Labels: yarn-5355-merge-blocker (was: oct16-hard) > [Security] Collector and reader side changes for loading auth filters and > principals > > > Key: YARN-5647 > URL: https://issues.apache.org/jira/browse/YARN-5647 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-5647-YARN-5355.wip.002.patch, > YARN-5647-YARN-5355.wip.003.patch, YARN-5647-YARN-5355.wip.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5648) [Security] Client side changes for authentication
[ https://issues.apache.org/jira/browse/YARN-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5648: -- Labels: yarn-5355-merge-blocker (was: ) > [Security] Client side changes for authentication > - > > Key: YARN-5648 > URL: https://issues.apache.org/jira/browse/YARN-5648 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-5648-YARN-5355.wip.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3053) [Security] Review and implement security in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-3053: -- Labels: YARN-5355 yarn-5355-merge-blocker (was: YARN-5355) > [Security] Review and implement security in ATS v.2 > --- > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-5304: -- Labels: YARN-5355 yarn-5355-merge-blocker (was: YARN-5355) > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5132) Exclude generated protobuf sources from YARN Javadoc build
[ https://issues.apache.org/jira/browse/YARN-5132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761892#comment-15761892 ] Jian He commented on YARN-5132: --- [~subru], [~kasha], [~billie.rinaldi] found this, could you please confirm ? seems like an issue. bq. It looks like the maven-javadoc-plugin is not configured properly for the hadoop-yarn module. There are YARN exclusions in the top level pom: https://github.com/apache/hadoop/blob/trunk/pom.xml#L443, and these are overridden in the hadoop-yarn pom: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/pom.xml#L78 by a subset of the package names. I am not sure if the lack of exclusion of the yarn server and yarn webapp packages was intentional or not. > Exclude generated protobuf sources from YARN Javadoc build > -- > > Key: YARN-5132 > URL: https://issues.apache.org/jira/browse/YARN-5132 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Subru Krishnan >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-5132-v1.patch > > > Currently YARN build includes Javadoc from generated protobuf sources which > is causing CI to fail. This JIRA proposes to exclude generated protobuf > sources from YARN Javadoc build -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5936) when cpu strict mode is closed, yarn couldn't assure scheduling fairness between containers
[
https://issues.apache.org/jira/browse/YARN-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761877#comment-15761877
]
Miklos Szegedi commented on YARN-5936:
--
I run 10 of them in parallel.
{quote}
First，The linux kernel source code of cpu bandwidth will add too many timer,
and add more function to be called.
{quote}
Are you saying you see this only if an app is running multiple threads and not
multiple processes?
> when cpu strict mode is closed, yarn couldn't assure scheduling fairness
> between containers
> ---
>
> Key: YARN-5936
> URL: https://issues.apache.org/jira/browse/YARN-5936
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.7.1
> Environment: CentOS7.1
>Reporter: zhengchenyu
>Priority: Critical
> Fix For: 2.7.1
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> When using LinuxContainer, the setting that
> "yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage" is
> true could assure scheduling fairness with the cpu bandwith of cgroup. But
> the cpu bandwidth of cgroup would lead to bad performance in our experience.
> Without cpu bandwidth of cgroup, cpu.share of cgroup is our only way to
> assure scheduling fairness, but it is not completely effective. For example,
> There are two container that have same vcore(means same cpu.share), one
> container is single-threaded, the other container is multi-thread. the
> multi-thread will have more CPU time, It's unreasonable!
> Here is my test case, I submit two distributedshell application. And two
> commmand are below:
> {code}
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 10 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> hadoop jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_script ./run.sh -shell_args 1 -num_containers 1 -container_memory
> 1024 -container_vcores 1 -master_memory 1024 -master_vcores 1 -priority 10
> {code}
> here show the cpu time of the two container:
> {code}
> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 15448 yarn 20 0 9059592 28336 9180 S 998.7 0.1 24:09.30 java
> 15026 yarn 20 0 9050340 27480 9188 S 100.0 0.1 3:33.97 java
> 13767 yarn 20 0 1799816 381208 18528 S 4.6 1.2 0:30.55 java
>77 root rt 0 0 0 0 S 0.3 0.0 0:00.74
> migration/1
> {code}
> We find the cpu time of Muliti-Thread are ten times than the cpu time of
> Single-Thread, though the two container have same cpu.share.
> notes:
> run.sh
> {code}
> java -cp /home/yarn/loop.jar:$CLASSPATH loop.loop $1
> {code}
> loop.java
> {code}
> package loop;
> public class loop {
> public static void main(String[] args) {
> // TODO Auto-generated method stub
> int loop = 1;
> if(args.length>=1) {
> System.out.println(args[0]);
> loop = Integer.parseInt(args[0]);
> }
> for(int i=0;i System.out.println("start thread " + i);
> new Thread(new Runnable() {
> @Override
> public void run() {
> // TODO Auto-generated method stub
> int j=0;
> while(true){j++;}
> }
> }).start();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
[
https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761743#comment-15761743
]
Varun Saxena commented on YARN-5585:
Thanks [~rohithsharma] for the patch.
bq. For single entity retrieval, when IdPrefix is not known, need to match
column value for entityType by doing range scan. Any other way this can achieve
this?
I am wondering that can we utilize setting the start and stop row in Scan for
this. Reason being we know idprefix can have a range of 0 to max value of long.
Thus, our start row can be
{{cluster!user!flow!runid!appid!entitytype!0!entityid}} and as stop row in not
inclusive, we can call
TimelineStorageUtils#calculateTheClosestNextRowKeyForPrefix for
{{cluster!user!flow!runid!appid!entitytype!LONG_MAX!entityid}}. This would mean
that typically only one row will be scanned. We can anyways break out of the
loop as soon as first row (which will be true for almost all the cases) is
found. We can use PageFilter of 1 to keep the Scan and result retrieved via it
as small. Thoughts ?
bq. FromId can be passed as filter where in fromId=idPrefix!entityId
As idPrefix is numeric any separator should be fine as we won't have to encode
it. Prefer to use those separators which do not require URL encoding.
bq. If we plan to reuse same API's.
I think we can reuse same APIs'. We can add a new query param, say idprefix and
we can document that query retrieval will be slightly faster if idprefix is
provided. Would like to know what others think about this though.
bq. we need to handle one scenario where same entityId is published with 2
entityIdPrefix. entityIdPrefix is mandatorily written even though user do not
provide any idPrefix while publishing entities. So, if case of idPrefix is not
known, should we use default idPrefix to get a row?
This will be tricky. We can follow what I mentioned in point 1 (if feasible)
and break out of the loop on first row.
If we just use 0 (default idprefix) we wont be able to support direct queries
by user based on say, container id, task id, etc. where the user may not know
about the corresponding prefix.
Another option could be that if more than one row is encountered for a single
entity read, we send some sort of error message indicating multiple idprefixes
in backend which can alert the user/application of some issue on the write side.
> [Atsv2] Reader side changes for entity prefix and support for pagination via
> additional filters
> ---
>
> Key: YARN-5585
> URL: https://issues.apache.org/jira/browse/YARN-5585
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Critical
> Labels: oct16-hard
> Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch,
> YARN-5585-YARN-5355.0002.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the
> applications. Along with those, it would be good to add new filter i.e fromId
> so that entities can be retrieved after the fromId.
> Current Behavior : Default limit is set to 100. If there are 1000 entities
> then REST call gives first/last 100 entities. How to retrieve next set of 100
> entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is
> no way to achieve this.
> So proposal is to have fromId in the filter like
> *getApps?limit=5&=app-5* which gives list of apps from app-6 to
> app-10.
> Since ATS is targeting large number of entities storage, it is very common
> use case to get next set of entities using fromId rather than querying all
> the entites. This is very useful for pagination in web UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5647) [Security] Collector and reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761573#comment-15761573 ] Varun Saxena commented on YARN-5647: Thanks Li and Sangjin for sharing your views. Sorry for the long delay in replying to comments made on the JIRA. As discussed offline in the meeting preceding the last one and pointed out in preceding comments, we can go with delegation token for now. bq. So we will change collector discovery once again? We will send tokens as part of AppCollectorData. That should be fine I guess. bq. It appears that TimelineDelegationTokenSecretManagerService uses a (local) LevelDB to store tokens and manage renewals, etc. That will work fine with a single-instance-based v.1.x, but how would that work with Timeline Service v.2? As of now, we will have one level db storage per NM (per NM Collector Manager). This, however, would need a change once collector (not collector manager) itself is launched as a separate process i.e. as a system container if we ever do it. We need to see how to handle failures for such a container. And that would require more discussion. Anyways token recovery related stuff will be handled in a separate JIRA. So we can have more discussion in detail there once that is opened. > [Security] Collector and reader side changes for loading auth filters and > principals > > > Key: YARN-5647 > URL: https://issues.apache.org/jira/browse/YARN-5647 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: oct16-hard > Attachments: YARN-5647-YARN-5355.wip.002.patch, > YARN-5647-YARN-5355.wip.003.patch, YARN-5647-YARN-5355.wip.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5968) Fix slider core module javadocs
[ https://issues.apache.org/jira/browse/YARN-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761527#comment-15761527 ] Billie Rinaldi commented on YARN-5968: -- Yes, I think that's what it means. > Fix slider core module javadocs > --- > > Key: YARN-5968 > URL: https://issues.apache.org/jira/browse/YARN-5968 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Billie Rinaldi > Attachments: YARN-5968-yarn-native-services.01.patch, > YARN-5968-yarn-native-services.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
[ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-5585: Attachment: YARN-5585-YARN-5355.0002.patch It is quite long time, apologies for this. I worked on this JIRA and tested for pagination, initial testing for pagination mode looks fine in real cluster. Attaching incremental patch(but it does not contain REST API changes), but it still required to modify few code patch based on the discussion. There are couple of things to discuss and get a consensus # *REST API definition* : There are set of pre-defined REST API for retrieval of entities/entity. *Entities* : I would like to keep same API's for pagination mode. FromId can be passed as filter where in *fromId=idPrefix!entityId*. User need to input combination idPrefix and entityId separated by *\!* or *+* symbol in filters. Reader server parses this combination and start scan from given idPrefix!entityId row key. Separator to be decided whether *\!* or any other symbols. *Entity* : Current entity API's are defined without idPrefix. Should we provide new API's for entityIdPrefix ? If we plan to reuse same API's, then we need to handle one scenario where same entityId is published with 2 entityIdPrefix. entityIdPrefix is mandatorily written even though user do not provide any idPrefix while publishing entities. So, if case of idPrefix is not known, should we use default idPrefix to get a row? # For single entity retrieval, when IdPrefix is not known, need to match column value for entityType by doing range scan. Any other way this can achieve this? Thoughts? cc:/ [~gtCarrera9] [~varun_saxena] [~sjlee0] [~vrushalic] > [Atsv2] Reader side changes for entity prefix and support for pagination via > additional filters > --- > > Key: YARN-5585 > URL: https://issues.apache.org/jira/browse/YARN-5585 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Critical > Labels: oct16-hard > Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, > YARN-5585-YARN-5355.0002.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch > > > TimelineReader REST API's provides lot of filters to retrieve the > applications. Along with those, it would be good to add new filter i.e fromId > so that entities can be retrieved after the fromId. > Current Behavior : Default limit is set to 100. If there are 1000 entities > then REST call gives first/last 100 entities. How to retrieve next set of 100 > entities i.e 101 to 200 OR 900 to 801? > Example : If applications are stored database, app-1 app-2 ... app-10. > *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is > no way to achieve this. > So proposal is to have fromId in the filter like > *getApps?limit=5&=app-5* which gives list of apps from app-6 to > app-10. > Since ATS is targeting large number of entities storage, it is very common > use case to get next set of entities using fromId rather than querying all > the entites. This is very useful for pagination in web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5650) Render Application Timeout value in web UI
[ https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760759#comment-15760759 ] Sunil G commented on YARN-5650: --- Thanks. +1 Committing to branch-2. > Render Application Timeout value in web UI > -- > > Key: YARN-5650 > URL: https://issues.apache.org/jira/browse/YARN-5650 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Rohith Sharma K S >Assignee: Akhil PB > Attachments: YARN-5650-branch-2.001.patch, YARN-5650.001.patch, > YARN-5650.002.patch, YARN-5650.003.patch, YARN-5650.004.patch, > YARN-5650.005.patch, screenshot-1.png, screenshot-2.png > > > This is to track timeout rendering in old and new yarn web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5650) Render Application Timeout value in web UI
[ https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760683#comment-15760683 ] Akhil PB commented on YARN-5650: Test case failures are unrelated and already tracked via another jira. This is a UI fix, hence test cases are not needed. AppBlock related javadoc errors are for the whole file. This patch follows the same syntax. cc [~sunilg] > Render Application Timeout value in web UI > -- > > Key: YARN-5650 > URL: https://issues.apache.org/jira/browse/YARN-5650 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Rohith Sharma K S >Assignee: Akhil PB > Attachments: YARN-5650-branch-2.001.patch, YARN-5650.001.patch, > YARN-5650.002.patch, YARN-5650.003.patch, YARN-5650.004.patch, > YARN-5650.005.patch, screenshot-1.png, screenshot-2.png > > > This is to track timeout rendering in old and new yarn web UI -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5877) Allow all nm-whitelist-env to get overridden during launch
[
https://issues.apache.org/jira/browse/YARN-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760627#comment-15760627
]
Sunil G commented on YARN-5877:
---
Thanks [~bibinchundatt] for the logs. Looks fine for me.
Newly added test case is also covering various cases as expected. I will commit
the patch later today.
> Allow all nm-whitelist-env to get overridden during launch
> --
>
> Key: YARN-5877
> URL: https://issues.apache.org/jira/browse/YARN-5877
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 5877 nm logs of test.log, Dockerfile,
> YARN-5877.0001.patch, YARN-5877.0002.patch, YARN-5877.0003.patch,
> YARN-5877.0004.patch, YARN-5877.0005.patch, YARN-5877.0006.patch,
> bootstrap.sh, yarn-site.xml
>
>
> As per the {{yarn.nodemanager.env-whitelist}} for the configured values
> should containers may override rather than use NodeManager's default.
> {code}
>
> Environment variables that containers may override rather
> than use NodeManager's default.
> yarn.nodemanager.env-whitelist
>
> JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME
>
> {code}
> But only the following containers can override
> {code}
> whitelist.add(ApplicationConstants.Environment.HADOOP_YARN_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_COMMON_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_HDFS_HOME.name());
> whitelist.add(ApplicationConstants.Environment.HADOOP_CONF_DIR.name());
> whitelist.add(ApplicationConstants.Environment.JAVA_HOME.name());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-3091) [Umbrella] Improve and fix locks of RM scheduler
[ https://issues.apache.org/jira/browse/YARN-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengchenyu updated YARN-3091: -- Comment: was deleted (was: Sorry, although this issue is completed, I still wanna ask for advice. I think the improvement of performance is not enough, if used read-write lock. Though read-write lock is not better than synchronized, and solved the problem of dead lock, but it's coarse-grained. For example, FairScheduler.update() hold a write block, but it still block the FairScheduler.checkAccess(), we know most code of FairScheduler.update() doesn't matter FairScheduler.checkAccess(). I think we should associate the lock with resources instead of function. For exmaple, demand is resource of the queue, we could operate it by CAS. So can we optimize with other way?) > [Umbrella] Improve and fix locks of RM scheduler > > > Key: YARN-3091 > URL: https://issues.apache.org/jira/browse/YARN-3091 > Project: Hadoop YARN > Issue Type: Task > Components: capacityscheduler, fairscheduler, resourcemanager, > scheduler >Reporter: Wangda Tan > > In existing YARN RM scheduler, there're some issues of using locks. For > example: > - Many unnecessary synchronized locks, we have seen several cases recently > that too frequent access of scheduler makes scheduler hang. Which could be > addressed by using read/write lock. Components include scheduler, CS queues, > apps > - Some fields not properly locked (Like clusterResource) > We can address them together in this ticket. > (More details see comments below) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3091) [Umbrella] Improve and fix locks of RM scheduler
[ https://issues.apache.org/jira/browse/YARN-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760615#comment-15760615 ] zhengchenyu commented on YARN-3091: --- Sorry, although this issue is completed, I still wanna ask for advice. I think the improvement of performance is not enough, if used read-write lock. Though read-write lock is not better than synchronized, and solved the problem of dead lock, but it's coarse-grained. For example, FairScheduler.update() hold a write block, but it still block the FairScheduler.checkAccess(), we know most code of FairScheduler.update() doesn't matter FairScheduler.checkAccess(). I think we should associate the lock with resources instead of function. For exmaple, demand is resource of the queue, we could operate it by CAS. So can we optimize with other way? > [Umbrella] Improve and fix locks of RM scheduler > > > Key: YARN-3091 > URL: https://issues.apache.org/jira/browse/YARN-3091 > Project: Hadoop YARN > Issue Type: Task > Components: capacityscheduler, fairscheduler, resourcemanager, > scheduler >Reporter: Wangda Tan > > In existing YARN RM scheduler, there're some issues of using locks. For > example: > - Many unnecessary synchronized locks, we have seen several cases recently > that too frequent access of scheduler makes scheduler hang. Which could be > addressed by using read/write lock. Components include scheduler, CS queues, > apps > - Some fields not properly locked (Like clusterResource) > We can address them together in this ticket. > (More details see comments below) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3091) [Umbrella] Improve and fix locks of RM scheduler
[ https://issues.apache.org/jira/browse/YARN-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760613#comment-15760613 ] zhengchenyu commented on YARN-3091: --- Sorry, although this issue is completed, I still wanna ask for advice. I think the improvement of performance is not enough, if used read-write lock. Though read-write lock is not better than synchronized, and solved the problem of dead lock, but it's coarse-grained. For example, FairScheduler.update() hold a write block, but it still block the FairScheduler.checkAccess(), we know most code of FairScheduler.update() doesn't matter FairScheduler.checkAccess(). I think we should associate the lock with resources instead of function. For exmaple, demand is resource of the queue, we could operate it by CAS. So can we optimize with other way? > [Umbrella] Improve and fix locks of RM scheduler > > > Key: YARN-3091 > URL: https://issues.apache.org/jira/browse/YARN-3091 > Project: Hadoop YARN > Issue Type: Task > Components: capacityscheduler, fairscheduler, resourcemanager, > scheduler >Reporter: Wangda Tan > > In existing YARN RM scheduler, there're some issues of using locks. For > example: > - Many unnecessary synchronized locks, we have seen several cases recently > that too frequent access of scheduler makes scheduler hang. Which could be > addressed by using read/write lock. Components include scheduler, CS queues, > apps > - Some fields not properly locked (Like clusterResource) > We can address them together in this ticket. > (More details see comments below) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5988) RM unable to start in secure setup
[ https://issues.apache.org/jira/browse/YARN-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760603#comment-15760603 ] Naganarasimha G R commented on YARN-5988: - [~ajithshetty], One Test case(TestRMAdminService.testSecureRMBecomeActive) seems to be failing from the modified classes, can you test locally and confirm ? Checkstyle issue can avoided during the commit but if you are modifying for the testcase then as well update this too. > RM unable to start in secure setup > -- > > Key: YARN-5988 > URL: https://issues.apache.org/jira/browse/YARN-5988 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha1 >Reporter: Ajith S >Assignee: Ajith S >Priority: Blocker > Attachments: YARN-5988.01.patch, YARN-5988.02.patch, > YARN-5988.03.patch, YARN-5988.04.patch > > > When CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION=true > RM is unable to start -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5650) Render Application Timeout value in web UI
[
https://issues.apache.org/jira/browse/YARN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760588#comment-15760588
]
Hadoop QA commented on YARN-5650:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new +
80 unchanged - 1 fixed = 81 total (was 81) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdk1.8.0_111
with JDK v1.8.0_111 generated 2 new + 159 unchanged - 0 fixed = 161 total (was
159) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} hadoop-yarn-server-common in the patch passed with
JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 49s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
