date:20140526

[jira] [Commented] (YARN-1911) NullPointerException in ResourceOption.getOverCommitTimeout()

2014-05-26 Thread Binglin Chang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008608#comment-14008608
 ] 

Binglin Chang commented on YARN-1911:
-

+1, I also got this bug when doing YARN-2051, hope it will get committed soon.


 NullPointerException in ResourceOption.getOverCommitTimeout()
 -

 Key: YARN-1911
 URL: https://issues.apache.org/jira/browse/YARN-1911
 Project: Hadoop YARN
  Issue Type: Bug
  Components: api
Affects Versions: 2.3.0
Reporter: Kenji Kikushima
Assignee: Kenji Kikushima
 Attachments: YARN-1911.patch


 I'm trying to implement REST API for Dynamic configuration. But 
 NullPointerException occurred when ResourceOption.getOverCommitTimeout() 
 called. Here is a sample code with error message.
 {code}
 @Test
 public void testGetOverCommitTimeout() {
   Resource capability = Resource.newInstance(4096, 4);
   ResourceOption resourceOption = ResourceOption.newInstance(capability,
 RMNode.OVER_COMMIT_TIMEOUT_MILLIS_DEFAULT);
   Assert.assertEquals(RMNode.OVER_COMMIT_TIMEOUT_MILLIS_DEFAULT,
 resourceOption.getOverCommitTimeout());
 }
 {code}
 {code}
 java.lang.NullPointerException
 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
 at 
 org.apache.hadoop.yarn.api.records.impl.pb.ResourceOptionPBImpl.getOverCommitTimeout(ResourceOptionPBImpl.java:63)
 at 
 org.apache.hadoop.yarn.server.resourcemanager.TestResourceOption.testGetOverCommitTimeout(TestResourceOption.java:33)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
 at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
 at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
 at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
 at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
 at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
 at 
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
 at 
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
 at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
 at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
 at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
 at 
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (YARN-2103) Fix code bug in SerializedExceptionPBImpl

2014-05-26 Thread Binglin Chang (JIRA)

Binglin Chang created YARN-2103:
---

 Summary: Fix code bug in SerializedExceptionPBImpl
 Key: YARN-2103
 URL: https://issues.apache.org/jira/browse/YARN-2103
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang


{code}
  SerializedExceptionProto proto = SerializedExceptionProto
  .getDefaultInstance();
  SerializedExceptionProto.Builder builder = null;
  boolean viaProto = false;
{code}

Since viaProto is false, we should initiate build rather than proto




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (YARN-2051) Fix code bug and add more unit tests for PBImpls

2014-05-26 Thread Binglin Chang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Binglin Chang updated YARN-2051:


Summary: Fix code bug and add more unit tests for PBImpls  (was: Add more 
unit tests for PBImpl that didn't get covered)

 Fix code bug and add more unit tests for PBImpls
 

 Key: YARN-2051
 URL: https://issues.apache.org/jira/browse/YARN-2051
 Project: Hadoop YARN
  Issue Type: Test
Reporter: Junping Du
Assignee: Binglin Chang
Priority: Critical

 From YARN-2016, we can see some bug could exist in PB implementation of 
 protocol. The bad news is most of these PBImpl don't have any unit test to 
 verify the info is not lost or changed after serialization/deserialization. 
 We should add more tests for it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2051) Fix code bug and add more unit tests for PBImpls

2014-05-26 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008623#comment-14008623
 ] 

Wangda Tan commented on YARN-2051:
--

Hi [~decster],
I think in the future, we can consider YARN-1871 for resolving PBImpl hard to 
write and add tests, please feel free to leave some thoughts in YARN-1871 if 
you feel interested :)

 Fix code bug and add more unit tests for PBImpls
 

 Key: YARN-2051
 URL: https://issues.apache.org/jira/browse/YARN-2051
 Project: Hadoop YARN
  Issue Type: Test
Reporter: Junping Du
Assignee: Binglin Chang
Priority: Critical

 From YARN-2016, we can see some bug could exist in PB implementation of 
 protocol. The bad news is most of these PBImpl don't have any unit test to 
 verify the info is not lost or changed after serialization/deserialization. 
 We should add more tests for it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-1871) We should eliminate writing *PBImpl code in YARN

2014-05-26 Thread Binglin Chang (JIRA)

[
https://issues.apache.org/jira/browse/YARN-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008650#comment-14008650
]

Binglin Chang commented on YARN-1871:
-

Good idea on eliminating PBImpl code, some comments:
bq. Make record class become a non-abstract class, add simple getters/setters
implementation.
I like we use simple getter/setter implementation, rather than current complex
builder/proto/field mixed cache
It's better to remain the old API unchanged, and hide implementation. if we can
generate PBImpl toProto/toRecord, why can't we generate simple getters/setters?

bq. serialization a record to Proto type using reflection
We'd better to generate code, reflection can be used on test code, but I'm
afraid using reflection to ser/de-ser in rpc code is not acceptable.

bq. There are only some limited patterns in current *PBImpl
There are some complex situations: read only type/property, generic types,
recursive types and name mismatch in record/proto. When doing YARN-2051, I
found some situations hard to automate and need special treatment.

We should eliminate writing *PBImpl code in YARN

Key: YARN-1871
URL: https://issues.apache.org/jira/browse/YARN-1871
Project: Hadoop YARN
Issue Type: Improvement
Components: api
Affects Versions: 2.4.0
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: YARN-1871.demo.patch

Currently, We need write PBImpl classes one by one. After running find .
-name *PBImpl*.java | xargs wc -l under hadoop source code directory, we
can see, there're more than 25,000 LOC. I think we should improve this, which
will be very helpful for YARN developers to make changes for YARN protocols.
There're only some limited patterns in current *PBImpl,
* Simple types, like string, int32, float.
* List? types
* Map? types
* Enum types
Code generation should be enough to generate such PBImpl classes.
Some other requirements are,
* Leave other related code alone, like service implemention (e.g.
ContainerManagerImpl).
* (If possible) Forward compatibility, developpers can write their own PBImpl
or genereate them.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2051) Fix code bug and add more unit tests for PBImpls

2014-05-26 Thread Binglin Chang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008654#comment-14008654
 ] 

Binglin Chang commented on YARN-2051:
-

Hi [~wangda], thanks for the notice.

 Fix code bug and add more unit tests for PBImpls
 

 Key: YARN-2051
 URL: https://issues.apache.org/jira/browse/YARN-2051
 Project: Hadoop YARN
  Issue Type: Test
Reporter: Junping Du
Assignee: Binglin Chang
Priority: Critical

 From YARN-2016, we can see some bug could exist in PB implementation of 
 protocol. The bad news is most of these PBImpl don't have any unit test to 
 verify the info is not lost or changed after serialization/deserialization. 
 We should add more tests for it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2083) In fair scheduler, Queue should not been assigned more containers when its usedResource had reach the maxResource limit

2014-05-26 Thread Yi Tian (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008671#comment-14008671
 ] 

Yi Tian commented on YARN-2083:
---

[~sandyr] could you help me to find out how is these case failed? I ran the 
test without my patch, it failed too.

 In fair scheduler, Queue should not been assigned more containers when its 
 usedResource had reach the maxResource limit
 ---

 Key: YARN-2083
 URL: https://issues.apache.org/jira/browse/YARN-2083
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Affects Versions: 2.3.0
Reporter: Yi Tian
  Labels: assignContainer, fair, scheduler
 Fix For: 2.3.0

 Attachments: YARN-2083.patch


 In fair scheduler, FSParentQueue and FSLeafQueue do an 
 assignContainerPreCheck to guaranty this queue is not over its limit.
 But the fitsIn function in Resource.java did not return false when the 
 usedResource equals the maxResource.
 I think we should create a new Function fitsInWithoutEqual instead of 
 fitsIn in this case.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

33 matches

Mail list logo