[jira] [Commented] (MYRIAD-171) Hadoop-2.7.1 UpdateFIle

2015-12-08 Thread Xuan Gong (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1504#comment-1504
 ] 

Xuan Gong commented on MYRIAD-171:
--

Actually, we should open a jira in YARN Community. This issue should be fixed 
in YARN side. That change apparently breaks the compatibility. 

> Hadoop-2.7.1 UpdateFIle
> ---
>
> Key: MYRIAD-171
> URL: https://issues.apache.org/jira/browse/MYRIAD-171
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Brandon Gulla
>Assignee: Brandon Gulla
>Priority: Blocker
>
> Hadoop 2.7.1 introduced an API changed that breaks Myriad when using HA 
> ResourceManager. They added an extra argument to the method that we are 
> currently using. 
> To get around this, switch hadoop version to 2.7.0 or wait for the impending 
> code change. 
> Hadoop 2.7.1
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> Hadoop 2.7.0
> https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> I1106 19:26:55.098060  5135 sched.cpp:164] Version: 0.25.0
> 15/11/06 19:26:55 INFO scheduler.MyriadDriver: Driver started with status: 
> DRIVER_RUNNING
> 15/11/06 19:26:55 INFO scheduler.MyriadDriverManager: Driver started with 
> status: DRIVER_RUNNING
> 15/11/06 19:26:55 INFO myriad.Main: started mesosDriver..
> 15/11/06 19:26:55 INFO interceptor.CompositeInterceptor: Registered 
> org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
> I1106 19:26:55.100406  5185 sched.cpp:262] New master detected at 
> master@172.31.45.229:5050
> I1106 19:26:55.100591  5185 sched.cpp:272] No credentials provided. 
> Attempting to register without authentication
> 15/11/06 19:26:55 INFO myriad.Main: Launching 1 NM(s) with profile medium
> I1106 19:26:55.102533  5179 sched.cpp:641] Framework registered with 
> 2f8adca0-12da-4200-867d-2b742955cabd-0004
> 15/11/06 19:26:55 INFO scheduler.MyriadOperations: Adding 1 NM instances to 
> cluster
> 15/11/06 19:26:55 INFO handlers.RegisteredEventHandler: Received event: 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5 with frameworkId: 
> value: "2f8adca0-12da-4200-867d-2b742955cabd-0004"
> Nov 06, 2015 7:26:55 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 0 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.storeMyriadState(MyriadFileSystemRMStateStore.java:95)
>   at 
> org.apache.myriad.state.SchedulerState.updateStateStore(SchedulerState.java:374)
>   at 
> org.apache.myriad.state.SchedulerState.setFrameworkId(SchedulerState.java:362)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:42)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:30)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> 15/11/06 19:26:55 INFO handlers.ResourceOffersEventHandler: Received offers 3
> 15/11/06 19:26:55 ERROR yarn.YarnUncaughtExceptionHandler: Thread 
> Thread[pool-6-thread-1,5,main] threw an Exception.
> java.lang.RuntimeException: java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
>

Re: [jira] [Commented] (MYRIAD-171) Hadoop-2.7.1 UpdateFIle

2015-11-07 Thread Jim Klucar 
I wonder if we're not the only ones to find this issue. We should search
and post to Hadoop lists. Maybe they don't think of this as a public API,
but it could be good to let them know what we use so they think twice about
changes.

On Fri, Nov 6, 2015 at 6:52 PM, DarinJ (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/MYRIAD-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994731#comment-14994731
> ]
>
> DarinJ commented on MYRIAD-171:
> ---
>
> I agree we should stick to 2.7.0 for this release.  Shimloaders take time
> to do right.  However, some of the docs says 2.7.1, it's not a blocked but
> should be fixed as promptly as we can.
>
> > Hadoop-2.7.1 UpdateFIle
> > ---
> >
> > Key: MYRIAD-171
> > URL: https://issues.apache.org/jira/browse/MYRIAD-171
> > Project: Myriad
> >  Issue Type: Bug
> >Affects Versions: Myriad 0.1.0
> >Reporter: Brandon Gulla
> >Assignee: Brandon Gulla
> >Priority: Blocker
> >
> > Hadoop 2.7.1 introduced an API changed that breaks Myriad when using HA
> ResourceManager. They added an extra argument to the method that we are
> currently using.
> > To get around this, switch hadoop version to 2.7.0 or wait for the
> impending code change.
> > Hadoop 2.7.1
> >
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> > Hadoop 2.7.0
> >
> https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> > I1106 19:26:55.098060  5135 sched.cpp:164] Version: 0.25.0
> > 15/11/06 19:26:55 INFO scheduler.MyriadDriver: Driver started with
> status: DRIVER_RUNNING
> > 15/11/06 19:26:55 INFO scheduler.MyriadDriverManager: Driver started
> with status: DRIVER_RUNNING
> > 15/11/06 19:26:55 INFO myriad.Main: started mesosDriver..
> > 15/11/06 19:26:55 INFO interceptor.CompositeInterceptor: Registered
> org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
> > I1106 19:26:55.100406  5185 sched.cpp:262] New master detected at
> master@172.31.45.229:5050
> > I1106 19:26:55.100591  5185 sched.cpp:272] No credentials provided.
> Attempting to register without authentication
> > 15/11/06 19:26:55 INFO myriad.Main: Launching 1 NM(s) with profile medium
> > I1106 19:26:55.102533  5179 sched.cpp:641] Framework registered with
> 2f8adca0-12da-4200-867d-2b742955cabd-0004
> > 15/11/06 19:26:55 INFO scheduler.MyriadOperations: Adding 1 NM instances
> to cluster
> > 15/11/06 19:26:55 INFO handlers.RegisteredEventHandler: Received event:
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5 with
> frameworkId: value: "2f8adca0-12da-4200-867d-2b742955cabd-0004"
> > Nov 06, 2015 7:26:55 PM com.lmax.disruptor.FatalExceptionHandler
> handleEventException
> > SEVERE: Exception processing: 0
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5
> > java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
> >   at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.storeMyriadState(MyriadFileSystemRMStateStore.java:95)
> >   at
> org.apache.myriad.state.SchedulerState.updateStateStore(SchedulerState.java:374)
> >   at
> org.apache.myriad.state.SchedulerState.setFrameworkId(SchedulerState.java:362)
> >   at
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:42)
> >   at
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:30)
> >   at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
> >   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >   at java.lang.Thread.run(Thread.java:745)
> > 15/11/06 19:26:55 INFO handlers.ResourceOffersEventHandler: Received
> offers 3
> > 15/11/06 19:26:55 ERROR yarn.YarnUncaughtExceptionHandler: Thread
> Thread[pool-6-thread-1,5,main] threw an Exception.
> > java.lang.RuntimeException: java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
> >   at
> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
> >   at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
> >   at
>

[jira] [Commented] (MYRIAD-171) Hadoop-2.7.1 UpdateFIle

2015-11-06 Thread Santosh Marella (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994655#comment-14994655
 ] 

Santosh Marella commented on MYRIAD-171:


Since Myriad has a compile dependency on hadoop 2.7.0, it'd work best with 
hadoop 2.7.0 and may not work well with other hadoop versions with API changes 
like this. 

Are you ok if we do a myriad 0.1.0 release with this restriction (i.e. Myriad 
works just with hadoop 2.7.0), and fix the binary compatibility issues with 
other hadoop versions in Myriad 0.2.0? 

> Hadoop-2.7.1 UpdateFIle
> ---
>
> Key: MYRIAD-171
> URL: https://issues.apache.org/jira/browse/MYRIAD-171
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Brandon Gulla
>Assignee: Brandon Gulla
>Priority: Blocker
>
> Hadoop 2.7.1 introduced an API changed that breaks Myriad when using HA 
> ResourceManager. They added an extra argument to the method that we are 
> currently using. 
> To get around this, switch hadoop version to 2.7.0 or wait for the impending 
> code change. 
> Hadoop 2.7.1
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> Hadoop 2.7.0
> https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> I1106 19:26:55.098060  5135 sched.cpp:164] Version: 0.25.0
> 15/11/06 19:26:55 INFO scheduler.MyriadDriver: Driver started with status: 
> DRIVER_RUNNING
> 15/11/06 19:26:55 INFO scheduler.MyriadDriverManager: Driver started with 
> status: DRIVER_RUNNING
> 15/11/06 19:26:55 INFO myriad.Main: started mesosDriver..
> 15/11/06 19:26:55 INFO interceptor.CompositeInterceptor: Registered 
> org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
> I1106 19:26:55.100406  5185 sched.cpp:262] New master detected at 
> master@172.31.45.229:5050
> I1106 19:26:55.100591  5185 sched.cpp:272] No credentials provided. 
> Attempting to register without authentication
> 15/11/06 19:26:55 INFO myriad.Main: Launching 1 NM(s) with profile medium
> I1106 19:26:55.102533  5179 sched.cpp:641] Framework registered with 
> 2f8adca0-12da-4200-867d-2b742955cabd-0004
> 15/11/06 19:26:55 INFO scheduler.MyriadOperations: Adding 1 NM instances to 
> cluster
> 15/11/06 19:26:55 INFO handlers.RegisteredEventHandler: Received event: 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5 with frameworkId: 
> value: "2f8adca0-12da-4200-867d-2b742955cabd-0004"
> Nov 06, 2015 7:26:55 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 0 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.storeMyriadState(MyriadFileSystemRMStateStore.java:95)
>   at 
> org.apache.myriad.state.SchedulerState.updateStateStore(SchedulerState.java:374)
>   at 
> org.apache.myriad.state.SchedulerState.setFrameworkId(SchedulerState.java:362)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:42)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:30)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> 15/11/06 19:26:55 INFO handlers.ResourceOffersEventHandler: Received offers 3
> 15/11/06 19:26:55 ERROR yarn.YarnUncaughtExceptionHandler: Thread 
> Thread[pool-6-thread-1,5,main] threw an Exception.
> java.lang.RuntimeException: java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by:

[jira] [Commented] (MYRIAD-171) Hadoop-2.7.1 UpdateFIle

2015-11-06 Thread Brandon Gulla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994648#comment-14994648
 ] 

Brandon Gulla commented on MYRIAD-171:
--

Bingo.

Looks like I should be able to hit the RM rest-endpoint and pull the version 
from there. Hopefully there is another way that doesn't pose some race 
conditions, but I'll explore that further. 

https://github.com/apache/hadoop/blob/f8adeb712dc834c27cec15c04a986f2f635aba83/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md

> Hadoop-2.7.1 UpdateFIle
> ---
>
> Key: MYRIAD-171
> URL: https://issues.apache.org/jira/browse/MYRIAD-171
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Brandon Gulla
>Assignee: Brandon Gulla
>Priority: Blocker
>
> Hadoop 2.7.1 introduced an API changed that breaks Myriad when using HA 
> ResourceManager. They added an extra argument to the method that we are 
> currently using. 
> To get around this, switch hadoop version to 2.7.0 or wait for the impending 
> code change. 
> Hadoop 2.7.1
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> Hadoop 2.7.0
> https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> I1106 19:26:55.098060  5135 sched.cpp:164] Version: 0.25.0
> 15/11/06 19:26:55 INFO scheduler.MyriadDriver: Driver started with status: 
> DRIVER_RUNNING
> 15/11/06 19:26:55 INFO scheduler.MyriadDriverManager: Driver started with 
> status: DRIVER_RUNNING
> 15/11/06 19:26:55 INFO myriad.Main: started mesosDriver..
> 15/11/06 19:26:55 INFO interceptor.CompositeInterceptor: Registered 
> org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
> I1106 19:26:55.100406  5185 sched.cpp:262] New master detected at 
> master@172.31.45.229:5050
> I1106 19:26:55.100591  5185 sched.cpp:272] No credentials provided. 
> Attempting to register without authentication
> 15/11/06 19:26:55 INFO myriad.Main: Launching 1 NM(s) with profile medium
> I1106 19:26:55.102533  5179 sched.cpp:641] Framework registered with 
> 2f8adca0-12da-4200-867d-2b742955cabd-0004
> 15/11/06 19:26:55 INFO scheduler.MyriadOperations: Adding 1 NM instances to 
> cluster
> 15/11/06 19:26:55 INFO handlers.RegisteredEventHandler: Received event: 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5 with frameworkId: 
> value: "2f8adca0-12da-4200-867d-2b742955cabd-0004"
> Nov 06, 2015 7:26:55 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 0 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.storeMyriadState(MyriadFileSystemRMStateStore.java:95)
>   at 
> org.apache.myriad.state.SchedulerState.updateStateStore(SchedulerState.java:374)
>   at 
> org.apache.myriad.state.SchedulerState.setFrameworkId(SchedulerState.java:362)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:42)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:30)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> 15/11/06 19:26:55 INFO handlers.ResourceOffersEventHandler: Received offers 3
> 15/11/06 19:26:55 ERROR yarn.YarnUncaughtExceptionHandler: Thread 
> Thread[pool-6-thread-1,5,main] threw an Exception.
> java.lang.RuntimeException: java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by:

[jira] [Commented] (MYRIAD-171) Hadoop-2.7.1 UpdateFIle

2015-11-06 Thread Brandon Gulla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MYRIAD-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994663#comment-14994663
 ] 

Brandon Gulla commented on MYRIAD-171:
--

2.7.0, Absolutely. The defined hadoop version in the docker build is 2.7.1 
currently but I can that patch tonight. 

I think taking our time with the reflection and setting up a ShimLoader will 
benefit us greater in the long run. 



> Hadoop-2.7.1 UpdateFIle
> ---
>
> Key: MYRIAD-171
> URL: https://issues.apache.org/jira/browse/MYRIAD-171
> Project: Myriad
>  Issue Type: Bug
>Affects Versions: Myriad 0.1.0
>Reporter: Brandon Gulla
>Assignee: Brandon Gulla
>Priority: Blocker
>
> Hadoop 2.7.1 introduced an API changed that breaks Myriad when using HA 
> ResourceManager. They added an extra argument to the method that we are 
> currently using. 
> To get around this, switch hadoop version to 2.7.0 or wait for the impending 
> code change. 
> Hadoop 2.7.1
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> Hadoop 2.7.0
> https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
> I1106 19:26:55.098060  5135 sched.cpp:164] Version: 0.25.0
> 15/11/06 19:26:55 INFO scheduler.MyriadDriver: Driver started with status: 
> DRIVER_RUNNING
> 15/11/06 19:26:55 INFO scheduler.MyriadDriverManager: Driver started with 
> status: DRIVER_RUNNING
> 15/11/06 19:26:55 INFO myriad.Main: started mesosDriver..
> 15/11/06 19:26:55 INFO interceptor.CompositeInterceptor: Registered 
> org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
> I1106 19:26:55.100406  5185 sched.cpp:262] New master detected at 
> master@172.31.45.229:5050
> I1106 19:26:55.100591  5185 sched.cpp:272] No credentials provided. 
> Attempting to register without authentication
> 15/11/06 19:26:55 INFO myriad.Main: Launching 1 NM(s) with profile medium
> I1106 19:26:55.102533  5179 sched.cpp:641] Framework registered with 
> 2f8adca0-12da-4200-867d-2b742955cabd-0004
> 15/11/06 19:26:55 INFO scheduler.MyriadOperations: Adding 1 NM instances to 
> cluster
> 15/11/06 19:26:55 INFO handlers.RegisteredEventHandler: Received event: 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5 with frameworkId: 
> value: "2f8adca0-12da-4200-867d-2b742955cabd-0004"
> Nov 06, 2015 7:26:55 PM com.lmax.disruptor.FatalExceptionHandler 
> handleEventException
> SEVERE: Exception processing: 0 
> org.apache.myriad.scheduler.event.RegisteredEvent@6a005de5
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.storeMyriadState(MyriadFileSystemRMStateStore.java:95)
>   at 
> org.apache.myriad.state.SchedulerState.updateStateStore(SchedulerState.java:374)
>   at 
> org.apache.myriad.state.SchedulerState.setFrameworkId(SchedulerState.java:362)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:42)
>   at 
> org.apache.myriad.scheduler.event.handlers.RegisteredEventHandler.onEvent(RegisteredEventHandler.java:30)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> 15/11/06 19:26:55 INFO handlers.ResourceOffersEventHandler: Received offers 3
> 15/11/06 19:26:55 ERROR yarn.YarnUncaughtExceptionHandler: Thread 
> Thread[pool-6-thread-1,5,main] threw an Exception.
> java.lang.RuntimeException: java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore.updateFile(Lorg/apache/hadoop/fs/Path;[B)V
>   at 
> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
>   at 
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: 
>