[jira] [Commented] (YARN-5396) YARN large file broadcast service
[ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078630#comment-16078630 ] Zhiyuan Yang commented on YARN-5396: [~elgoiri] Thanks for your interest! Please refer to Spark broadcast variable implementation and this [paper|https://pdfs.semanticscholar.org/7b0e/6a3dc18babb19daddb63890e763795943485.pdf]. > YARN large file broadcast service > - > > Key: YARN-5396 > URL: https://issues.apache.org/jira/browse/YARN-5396 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: slides-prototype.pdf, YARN-broadcast-prototype.patch, > YARNFileTransferService-prototype.pdf > > > In Hadoop and related softwares, there are demands of broadcasting large > files. For example, YARN application may localize large jar files on each > node; Hive may distribute large tables in fragment-replicate joins; docker > integration may broadcast large container image. The current local resource > based solution is to put the files on HDFS and let each node download from > HDFS, which is inefficient and not scalable. So we want to build a better > file transfer service in YARN so that all applications can use it broadcast > large file efficiently. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982066#comment-15982066 ] Zhiyuan Yang edited comment on YARN-5007 at 4/24/17 10:58 PM: -- This change breaks Tez and possibly many other downstream projects. In 2.7.0, the only way to enable ATS is through ctor parameters; now the only way is configuration. How do you expect downstream projects to support both 2.7.0 and 3.0.0? was (Author: aplusplus): This change breaks Tez and possibly many other downstream projects. In 2.7.0, the only way to enable ATS is through ctor parameters; now the only way is configuration. How do you expect downstream projects to support both support both 2.7.0 and 3.0.0? > Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster > --- > > Key: YARN-5007 > URL: https://issues.apache.org/jira/browse/YARN-5007 > Project: Hadoop YARN > Issue Type: Test >Reporter: Andras Bokor >Assignee: Andras Bokor > Labels: oct16-easy > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5007.01.patch, YARN-5007.02.patch, > YARN-5007.03.patch > > > MiniYarnCluster has a deprecated constructor which is called by the other > constructors and it causes javac warnings during the build. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982066#comment-15982066 ] Zhiyuan Yang edited comment on YARN-5007 at 4/24/17 10:58 PM: -- This change breaks Tez and possibly many other downstream projects. In 2.7.0, the only way to enable ATS is through ctor parameters; now the only way is configuration. How do you expect downstream projects to support both 2.7.0 and 3.0.0? Even a hadoop shim layer cannot fix the problem. was (Author: aplusplus): This change breaks Tez and possibly many other downstream projects. In 2.7.0, the only way to enable ATS is through ctor parameters; now the only way is configuration. How do you expect downstream projects to support both 2.7.0 and 3.0.0? > Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster > --- > > Key: YARN-5007 > URL: https://issues.apache.org/jira/browse/YARN-5007 > Project: Hadoop YARN > Issue Type: Test >Reporter: Andras Bokor >Assignee: Andras Bokor > Labels: oct16-easy > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5007.01.patch, YARN-5007.02.patch, > YARN-5007.03.patch > > > MiniYarnCluster has a deprecated constructor which is called by the other > constructors and it causes javac warnings during the build. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5007) Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/YARN-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982066#comment-15982066 ] Zhiyuan Yang commented on YARN-5007: This change breaks Tez and possibly many other downstream projects. In 2.7.0, the only way to enable ATS is through ctor parameters; now the only way is configuration. How do you expect downstream projects to support both support both 2.7.0 and 3.0.0? > Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster > --- > > Key: YARN-5007 > URL: https://issues.apache.org/jira/browse/YARN-5007 > Project: Hadoop YARN > Issue Type: Test >Reporter: Andras Bokor >Assignee: Andras Bokor > Labels: oct16-easy > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5007.01.patch, YARN-5007.02.patch, > YARN-5007.03.patch > > > MiniYarnCluster has a deprecated constructor which is called by the other > constructors and it causes javac warnings during the build. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5835) NoClassDefFoundError when reading ATS v1.5
[
https://issues.apache.org/jira/browse/YARN-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated YARN-5835:
---
Description:
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in
$HADOOP_HOME/share/hadoop/tools/lib but is never added in timeline server's
classpath. Reading data from LevelDBCacheTimelineStore will get
NoClassDefFoundError. Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
at
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
A temporary fix would be adding $HADOOP_HOME/share/hadoop/tools/lib/* into
HADOOP_CLASSPATH manually.
was:
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath.
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError.
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at
[jira] [Updated] (YARN-5835) NoClassDefFoundError when reading ATS v1.5
[
https://issues.apache.org/jira/browse/YARN-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated YARN-5835:
---
Description:
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath.
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError.
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
at
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
A temporary fix would be adding /hadoop/shared/hadoop/tools into
HADOOP_CLASSPATH manually.
was:
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath.
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError.
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at
[jira] [Commented] (YARN-5835) NoClassDefFoundError when reading ATS v1.5
[
https://issues.apache.org/jira/browse/YARN-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634846#comment-15634846
]
Zhiyuan Yang commented on YARN-5835:
Thanks [~gtCarrera9] for helping debug on this!
[~hitesh], [~Sreenath], have you ever gotten this issue? This occurs when I run
Tez UI on hadoop branch-2.8.
> NoClassDefFoundError when reading ATS v1.5
> --
>
> Key: YARN-5835
> URL: https://issues.apache.org/jira/browse/YARN-5835
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Zhiyuan Yang
>
> LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in
> /hadoop/shared/hadoop/tools but is never added in timeline server's
> classpath. Reading data from LevelDBCacheTimelineStore will get
> NoClassDefFoundError. Here's the stacktrace:
> {code:java}
> java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
> at
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
> at
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
> at
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
> at
> org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
> at
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
> at
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
> at
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
> at
> org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
> at
> org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
> at
> org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
> at
> org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
> at
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
> at
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
> at
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
> at
> org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
> at
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
> at
> org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5835) NoClassDefFoundError when reading ATS v1.5
Zhiyuan Yang created YARN-5835:
--
Summary: NoClassDefFoundError when reading ATS v1.5
Key: YARN-5835
URL: https://issues.apache.org/jira/browse/YARN-5835
Project: Hadoop YARN
Issue Type: Task
Reporter: Zhiyuan Yang
LevelDBCacheTimelineStore requires jackson-databind-2.2.3.jar which is in
/hadoop/shared/hadoop/tools but is never added in timeline server's classpath.
Reading data from LevelDBCacheTimelineStore will get NoClassDefFoundError.
Here's the stacktrace:
{code:java}
java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.getEntityForKey(LevelDBCacheTimelineStore.java:296)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:161)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore$LevelDBMapAdapter.get(LevelDBCacheTimelineStore.java:140)
at
org.apache.hadoop.yarn.server.timeline.KeyValueBasedTimelineStore.getEntity(KeyValueBasedTimelineStore.java:199)
at
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.getEntity(LevelDBCacheTimelineStore.java:58)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:349)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)
at
org.apache.hadoop.yarn.server.timeline.EntityLogInfo.doParse(LogInfo.java:204)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parsePath(LogInfo.java:156)
at
org.apache.hadoop.yarn.server.timeline.LogInfo.parseForStore(LogInfo.java:113)
at
org.apache.hadoop.yarn.server.timeline.EntityCacheItem.refreshCache(EntityCacheItem.java:143)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getCachedStore(EntityGroupFSTimelineStore.java:938)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresFromCacheIds(EntityGroupFSTimelineStore.java:853)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getTimelineStoresForRead(EntityGroupFSTimelineStore.java:906)
at
org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getEntities(EntityGroupFSTimelineStore.java:959)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doGetEntities(TimelineDataManager.java:169)
at
org.apache.hadoop.yarn.server.timeline.TimelineDataManager.getEntities(TimelineDataManager.java:139)
at
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.getEntities(TimelineWebServices.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5396) YARN large file broadcast service
[ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5396: --- Attachment: YARNFileTransferService-prototype.pdf Attached documentation, including some background, design decisions, implementation notes and usage doc. > YARN large file broadcast service > - > > Key: YARN-5396 > URL: https://issues.apache.org/jira/browse/YARN-5396 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-broadcast-prototype.patch, > YARNFileTransferService-prototype.pdf, slides-prototype.pdf > > > In Hadoop and related softwares, there are demands of broadcasting large > files. For example, YARN application may localize large jar files on each > node; Hive may distribute large tables in fragment-replicate joins; docker > integration may broadcast large container image. The current local resource > based solution is to put the files on HDFS and let each node download from > HDFS, which is inefficient and not scalable. So we want to build a better > file transfer service in YARN so that all applications can use it broadcast > large file efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5396) YARN large file broadcast service
[ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5396: --- Attachment: slides-prototype.pdf Attach the slides which explains the big picture. > YARN large file broadcast service > - > > Key: YARN-5396 > URL: https://issues.apache.org/jira/browse/YARN-5396 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-broadcast-prototype.patch, slides-prototype.pdf > > > In Hadoop and related softwares, there are demands of broadcasting large > files. For example, YARN application may localize large jar files on each > node; Hive may distribute large tables in fragment-replicate joins; docker > integration may broadcast large container image. The current local resource > based solution is to put the files on HDFS and let each node download from > HDFS, which is inefficient and not scalable. So we want to build a better > file transfer service in YARN so that all applications can use it broadcast > large file efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5396) YARN large file broadcast service
[ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5396: --- Attachment: YARN-broadcast-prototype.patch Attach the hacky prototype of my last year's internship work. I've made it work on most recent branch-2 revision so that people can try it out(although not recommend until the doc is uploaded). It contains the followings: 1. BitTorrent-based broadcast service as aux service 2. Modified resource localization that makes use of broadcast service and computes md5 of localized file. 3. An example yarn app that simply localizes resource via broadcast service. I'll attach some documentation about design,implementation and usage later. > YARN large file broadcast service > - > > Key: YARN-5396 > URL: https://issues.apache.org/jira/browse/YARN-5396 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-broadcast-prototype.patch > > > In Hadoop and related softwares, there are demands of broadcasting large > files. For example, YARN application may localize large jar files on each > node; Hive may distribute large tables in fragment-replicate joins; docker > integration may broadcast large container image. The current local resource > based solution is to put the files on HDFS and let each node download from > HDFS, which is inefficient and not scalable. So we want to build a better > file transfer service in YARN so that all applications can use it broadcast > large file efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397739#comment-15397739 ] Zhiyuan Yang commented on YARN-5436: [~rohithsharma] Thanks for reviewing! You are right in the sense this patch is mostly letting DrainDispatcher not reuse AsyncDispatcher's drained field, but the fix for YARN-2991 is still there. bq. does small tiny race is causing TEZ test failures? Yes. In Tez UT tests, invocation of dispatcher.await() finished without handling all events and assertion after dispatcher.await() failed. This race condition only happens when queue is almost empty, which is exactly the case in Tez UT tests. bq. If so would it be good to fix in AsyncDispatcher rather adding full duplicate code. The root cause of race is we cannot guarantee we enqueue event and update drained atomically. I didn't find a way to fix this without adding more synchronization which is a very expensive fix for a minimum benefit. YARN-3878 discussed about this race and decided to ignore it for the same reason. bq. How about adding additional check before adding into event queue to avoid a race? While this may avoid enqueuing last event, race can still happen without invoking dispatcher.serviceStop(). Actually in Tez UT test, we never invoke dispatcher.serviceStop(). > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch, > YARN-5436.4.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.4.patch Uploaded the patch that doesn't use java 8 feature for branch-2 sake. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch, > YARN-5436.4.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.3.patch > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: (was: YARN-5436.3.patch) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396514#comment-15396514 ] Zhiyuan Yang edited comment on YARN-5436 at 7/27/16 10:23 PM: -- Thanks [~gtCarrera9] for reviewing the patch. Sorry for misusing the term 'data race'. Already rephrased the comments. was (Author: aplusplus): Thanks [~gtCarrera9] for review the patch. Sorry for misusing the term 'data race'. Already rephrased the comments. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.3.patch Thanks [~gtCarrera9] for review the patch. Sorry for misusing the term 'data race'. Already rephrased the comments. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch, YARN-5436.3.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.2.patch > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch, YARN-5436.2.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396345#comment-15396345 ] Zhiyuan Yang commented on YARN-5436: Upload the patch that fixes problems only in DrainDispatcher and documents minor race condition in AsyncDispatcher. Please help review. [~jianhe], [~rohithsharma], [~varun_saxena]. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396348#comment-15396348 ] Zhiyuan Yang commented on YARN-5436: Race in AsyncDispatcher has been found and ignored in YARN-3887. Leave it there for now. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Attachment: YARN-5436.1.patch > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: YARN-5436.1.patch > > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Description: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never documented...). In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher reuse some AsyncDispatcher method because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also). (was: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher reuse some AsyncDispatcher method because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also).) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher (this was found and ignored in YARN-3878 but never > documented...). In YARN-2991, another DrainDispatcher bug was fixed by > letting DrainDispatcher reuse some AsyncDispatcher method because > AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and > now similar race reappears in Tez unit tests (probably also YARN unit tests > also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Description: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher reuse some AsyncDispatcher method because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also). (was: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher extend AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also).) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher reuse some > AsyncDispatcher method because AsyncDispatcher doesn't have such issue. > However, this shadows YARN-2264, and now similar race reappears in Tez unit > tests (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Comment: was deleted (was: Data race can cause RM stop without handling last enqueued event.) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396054#comment-15396054 ] Zhiyuan Yang commented on YARN-5436: Data race can cause RM stop without handling last enqueued event. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396052#comment-15396052 ] Zhiyuan Yang commented on YARN-5436: Data race can cause RM stop without handling last enqueued event. > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
[ https://issues.apache.org/jira/browse/YARN-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-5436: --- Description: In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainDispatcher bug was fixed by letting DrainDispatcher extend AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also). (was: In YARN-2264, a race in DrainedDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainedDispatcher bug was fixed by letting DrainedDispatcher extend AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also).) > Race in AsyncDispatcher can cause random test failures in Tez(probably YARN > also ) > -- > > Key: YARN-5436 > URL: https://issues.apache.org/jira/browse/YARN-5436 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In YARN-2264, a race in DrainDispatcher was fixed. Unfortunately, it also > exists in AsyncDispatcher but wasn't found. In YARN-2991, another > DrainDispatcher bug was fixed by letting DrainDispatcher extend > AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, > this shadows YARN-2264, and now similar race reappears in Tez unit tests > (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5436) Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also )
Zhiyuan Yang created YARN-5436: -- Summary: Race in AsyncDispatcher can cause random test failures in Tez(probably YARN also ) Key: YARN-5436 URL: https://issues.apache.org/jira/browse/YARN-5436 Project: Hadoop YARN Issue Type: Bug Reporter: Zhiyuan Yang Assignee: Zhiyuan Yang In YARN-2264, a race in DrainedDispatcher was fixed. Unfortunately, it also exists in AsyncDispatcher but wasn't found. In YARN-2991, another DrainedDispatcher bug was fixed by letting DrainedDispatcher extend AsyncDispatcher because AsyncDispatcher doesn't have such issue. However, this shadows YARN-2264, and now similar race reappears in Tez unit tests (probably also YARN unit tests also). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5396) YARN large file broadcast service
[ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang reassigned YARN-5396: -- Assignee: Zhiyuan Yang > YARN large file broadcast service > - > > Key: YARN-5396 > URL: https://issues.apache.org/jira/browse/YARN-5396 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > > In Hadoop and related softwares, there are demands of broadcasting large > files. For example, YARN application may localize large jar files on each > node; Hive may distribute large tables in fragment-replicate joins; docker > integration may broadcast large container image. The current local resource > based solution is to put the files on HDFS and let each node download from > HDFS, which is inefficient and not scalable. So we want to build a better > file transfer service in YARN so that all applications can use it broadcast > large file efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5396) YARN large file broadcast service
Zhiyuan Yang created YARN-5396: -- Summary: YARN large file broadcast service Key: YARN-5396 URL: https://issues.apache.org/jira/browse/YARN-5396 Project: Hadoop YARN Issue Type: New Feature Reporter: Zhiyuan Yang In Hadoop and related softwares, there are demands of broadcasting large files. For example, YARN application may localize large jar files on each node; Hive may distribute large tables in fragment-replicate joins; docker integration may broadcast large container image. The current local resource based solution is to put the files on HDFS and let each node download from HDFS, which is inefficient and not scalable. So we want to build a better file transfer service in YARN so that all applications can use it broadcast large file efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4956) findbug issue on LevelDBCacheTimelineStore
[
https://issues.apache.org/jira/browse/YARN-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259376#comment-15259376
]
Zhiyuan Yang commented on YARN-4956:
Console output shows jenkins did findbugs check before building hadoop. That's
why findbugs warning is still there.
> findbug issue on LevelDBCacheTimelineStore
> --
>
> Key: YARN-4956
> URL: https://issues.apache.org/jira/browse/YARN-4956
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.8.0
>Reporter: Xuan Gong
>Assignee: Zhiyuan Yang
> Attachments: YARN-4956-trunk.000.patch
>
>
> {code}
> Multithreaded correctness Warnings
> Code Warning IS Inconsistent synchronization of
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration;
> locked 66% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore
> Field
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration
> Synchronized 66% of the time
> Unsynchronized access at LevelDBCacheTimelineStore.java:[line 82]
> Synchronized access at LevelDBCacheTimelineStore.java:[line 117]
> Synchronized access at LevelDBCacheTimelineStore.java:[line 122]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4956) findbug issue on LevelDBCacheTimelineStore
[
https://issues.apache.org/jira/browse/YARN-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang updated YARN-4956:
---
Attachment: YARN-4956-trunk.000.patch
Fixed by adding synchronized to serviceInit
> findbug issue on LevelDBCacheTimelineStore
> --
>
> Key: YARN-4956
> URL: https://issues.apache.org/jira/browse/YARN-4956
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.8.0
>Reporter: Xuan Gong
>Assignee: Zhiyuan Yang
> Attachments: YARN-4956-trunk.000.patch
>
>
> {code}
> Multithreaded correctness Warnings
> Code Warning IS Inconsistent synchronization of
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration;
> locked 66% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore
> Field
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration
> Synchronized 66% of the time
> Unsynchronized access at LevelDBCacheTimelineStore.java:[line 82]
> Synchronized access at LevelDBCacheTimelineStore.java:[line 117]
> Synchronized access at LevelDBCacheTimelineStore.java:[line 122]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-4956) findbug issue on LevelDBCacheTimelineStore
[
https://issues.apache.org/jira/browse/YARN-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiyuan Yang reassigned YARN-4956:
--
Assignee: Zhiyuan Yang (was: Li Lu)
> findbug issue on LevelDBCacheTimelineStore
> --
>
> Key: YARN-4956
> URL: https://issues.apache.org/jira/browse/YARN-4956
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 2.8.0
>Reporter: Xuan Gong
>Assignee: Zhiyuan Yang
>
> {code}
> Multithreaded correctness Warnings
> Code Warning IS Inconsistent synchronization of
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration;
> locked 66% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore
> Field
> org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration
> Synchronized 66% of the time
> Unsynchronized access at LevelDBCacheTimelineStore.java:[line 82]
> Synchronized access at LevelDBCacheTimelineStore.java:[line 117]
> Synchronized access at LevelDBCacheTimelineStore.java:[line 122]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1427) yarn-env.cmd should have the analog comments that are in yarn-env.sh
[ https://issues.apache.org/jira/browse/YARN-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-1427: --- Attachment: YARN-1427-trunk.4.patch re-submit original patch to get debug info yarn-env.cmd should have the analog comments that are in yarn-env.sh Key: YARN-1427 URL: https://issues.apache.org/jira/browse/YARN-1427 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhiyuan Yang Labels: BB2015-05-TBR, newbie, windows Attachments: YARN-1427-trunk.2.patch, YARN-1427-trunk.3.patch, YARN-1427-trunk.4.patch, YARN-1427.1.patch There're the paragraphs of about RM/NM env vars (probably AHS as well soon) in yarn-env.sh. Should the windows version script provide the similar comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1427) yarn-env.cmd should have the analog comments that are in yarn-env.sh
[ https://issues.apache.org/jira/browse/YARN-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594987#comment-14594987 ] Zhiyuan Yang commented on YARN-1427: Attached patch.Thanks yarn-env.cmd should have the analog comments that are in yarn-env.sh Key: YARN-1427 URL: https://issues.apache.org/jira/browse/YARN-1427 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhiyuan Yang Labels: BB2015-05-TBR, newbie, windows Attachments: YARN-1427-trunk.2.patch, YARN-1427-trunk.3.patch, YARN-1427.1.patch There're the paragraphs of about RM/NM env vars (probably AHS as well soon) in yarn-env.sh. Should the windows version script provide the similar comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-1427) yarn-env.cmd should have the analog comments that are in yarn-env.sh
[ https://issues.apache.org/jira/browse/YARN-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated YARN-1427: --- Attachment: YARN-1427-trunk.3.patch yarn-env.cmd should have the analog comments that are in yarn-env.sh Key: YARN-1427 URL: https://issues.apache.org/jira/browse/YARN-1427 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhiyuan Yang Labels: BB2015-05-TBR, newbie, windows Attachments: YARN-1427-trunk.2.patch, YARN-1427-trunk.3.patch, YARN-1427.1.patch There're the paragraphs of about RM/NM env vars (probably AHS as well soon) in yarn-env.sh. Should the windows version script provide the similar comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-1427) yarn-env.cmd should have the analog comments that are in yarn-env.sh
[ https://issues.apache.org/jira/browse/YARN-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang reassigned YARN-1427: -- Assignee: Zhiyuan Yang yarn-env.cmd should have the analog comments that are in yarn-env.sh Key: YARN-1427 URL: https://issues.apache.org/jira/browse/YARN-1427 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.4.0 Reporter: Zhijie Shen Assignee: Zhiyuan Yang Labels: BB2015-05-TBR, newbie, windows Attachments: YARN-1427-trunk.2.patch, YARN-1427.1.patch There're the paragraphs of about RM/NM env vars (probably AHS as well soon) in yarn-env.sh. Should the windows version script provide the similar comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
