subject:"\[jira\] \[Commented\] \(HBASE\-7904\) Make mapreduce jobs pass based on 2.0.4\-alpha"


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619155#comment-13619155
 ] 

Andrew Purtell commented on HBASE-7904:
---

Does this change mean that coprocessors will have a different view of 
configuration than the HRegion itself? I.e. not see updates to configuration 
via CompoundConfiguration from the schema? 
{noformat}
Index: 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
===
--- 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(revision 1461235)
+++ 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(working copy)
@@ -484,7 +484,7 @@
   this.rsAccounting = this.rsServices.getRegionServerAccounting();
   // don't initialize coprocessors if not running within a regionserver
   // TODO: revisit if coprocessors should load in other cases
-  this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, conf);
+  this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, 
baseConf);
   this.metricsRegionWrapper = new MetricsRegionWrapperImpl(this);
   this.metricsRegion = new MetricsRegion(this.metricsRegionWrapper);
 } else {
{noformat}

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619158#comment-13619158
 ] 

Andrew Purtell commented on HBASE-7904:
---

Wait, I just saw the v9 patch has been committed to trunk and 0.95. Why is this 
JIRA still open? See my above comment. Should this be addressed as an addendum 
here or as a new JIRA?

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-04-01 Thread Siddharth Seth (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619159#comment-13619159
]

Siddharth Seth commented on HBASE-7904:
---

Posted a summary at
https://issues.apache.org/jira/browse/YARN-449?focusedCommentId=13619145page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13619145

bq. Did something change in the hadoop2 snapshot? A yarn patch got committed?
Nothing that should've affected HBase (given the previous comment about
clusters not running in parallel)

bq. I suppose I could go look myself but I ask because supposedly there was an
incompatibility issue.
Do you happen to know what this incompatibility was (and against which version)
?

bq. Regarding TestImportExport - I'm not sure why this test is using the
HBaseTestingUtility differently than other tests. It was missing setting YARN
conf in certain places.

bq. We just want something that worked like minimr in hadoop1. We start the
cluster do our tests, not in //, then shut down the cluster. I misunderstood
you when you said // above. Yes we run our tests in //. We do not set up a
minimr cluster once and run tests in // against it.
I'm still not clear about this. (not so relevant now that the tests are
passing, but would be nice to know).
Yes we run our tests in // - this is a little confusing. As an example, can
TestTableMapReduce run in parallel with TestHFileOutputFormat ?, or can
multiple tests within TestTableMapReduce run in parallel, or something else.

Make mapreduce jobs pass based on 2.0.4-alpha
-

Key: HBASE-7904
URL: https://issues.apache.org/jira/browse/HBASE-7904
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
Fix For: 0.95.0, 0.98.0

Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt,
7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt,
7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt,
7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt,
7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt

2.0.3-alpha has been released.
We should upgrade the dependency.

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619162#comment-13619162
 ] 

Andrew Purtell commented on HBASE-7904:
---

Since I am the owner of coprocessors and this patch changes how 
RegionCoprocessorHost is instantiated, I would really have appreciated a ping. 
Thanks.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619175#comment-13619175
]

Ted Yu commented on HBASE-7904:
---

bq. Why is this JIRA still open?
0.95 should not ship with 2.0.4-SNAPSHOT build. That was why I left this open.
If this JIRA is resolved, we can open another JIRA to update pom.xml when
2.0.4-ALPHA is released.

bq. this patch changes how RegionCoprocessorHost is instantiated
I exchanged a few emails with Nicolas who was the author of
CompoundConfiguration.
My initial plan was to allow CompoundConfiguration.setClass() to be called. He
advised against that approach because CompoundConfiguration was designed to be
immutable.

I understand that the patch changed behavior for RegionCoprocessorHost
instantiation. If there is a better approach to solving the issue of running
mapreduce job on hadoop 2.0 *and* maintaining immutability of
CompoundConfiguration, I would be more than happy to adopt.

Make mapreduce jobs pass based on 2.0.4-alpha
-

Key: HBASE-7904
URL: https://issues.apache.org/jira/browse/HBASE-7904
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
Fix For: 0.95.0, 0.98.0

2.0.3-alpha has been released.
We should upgrade the dependency.

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619180#comment-13619180
 ] 

Andrew Purtell commented on HBASE-7904:
---

bq. I understand that the patch changed behavior for RegionCoprocessorHost 
instantiation. 

So you are ok with coprocessors having a different configuration than HRegion? 
Don't you forsee issues with that? 

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619182#comment-13619182
 ] 

Andrew Purtell commented on HBASE-7904:
---

bq. 0.95 should not ship with 2.0.4-SNAPSHOT build. That was why I left this 
open.

I think it is confusing, because the patch on this issue has already been 
committed to the target branches.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619186#comment-13619186
 ] 

Andrew Purtell commented on HBASE-7904:
---

bq. I exchanged a few emails with Nicolas who was the author of 
CompoundConfiguration.

All interactions should be on JIRA or on the dev@ or user@ email lists. 
Decisions made in private communication are against the Apache Way.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-04-01 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619229#comment-13619229
]

stack commented on HBASE-7904:
--

bq. Do you happen to know what this incompatibility was (and against which
version) ?
[~sseth]

No. This issue started out as an incompability issue (see original [~ted_yu]
comments at head of issue) and then ended w/ some configuration manipulations
that seem gratiuitous. I ask the author what fixed the issue and I get that
they think the config. manipulations are all that is necessary. I have spent
a bunch of time trying to follow what is going on here, but gave up, and +1'd
it as a means of putting the issue behind us since the patch seemed relatively
harmless.

bq. As an example, can TestTableMapReduce run in parallel with
TestHFileOutputFormat ?, or can multiple tests within TestTableMapReduce run in
parallel, or something else.

Yes, hbase tests can be run in parallel: e.g. TestTableMapReduce and
TestHFileOutputFormat run at same time. Our surefire runs our test suite by
firing off N JVMs each running a distinct, long-running test.

What we do not do is set up a mini mr cluster and then run many unit tests
against.

bq. I exchanged a few emails with Nicolas who was the author of
CompoundConfiguration.

[~ted_yu] And you say nothing in here about it? Why do it in private? They
call it open source for a reason.

Suggest we revert this patch. Seems broke that CPs see another config. because
Chane to HRegion to not use a 'CompoundConfiguration' when talking to HDFS,
due to the inability to change the RPC engine with CompoundConfiguration.
(Causes several *SecureLoad tests to hang) according to
https://issues.apache.org/jira/browse/YARN-449 (a conclusion that is absent
here).

Make mapreduce jobs pass based on 2.0.4-alpha
-

Key: HBASE-7904
URL: https://issues.apache.org/jira/browse/HBASE-7904
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
Fix For: 0.95.0, 0.98.0

2.0.3-alpha has been released.
We should upgrade the dependency.

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619234#comment-13619234
 ] 

Ted Yu commented on HBASE-7904:
---

I tried the following change:
{code}
Index: 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
===
--- 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(revision 1463235)
+++ 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(working copy)
@@ -484,7 +484,7 @@
   this.rsAccounting = this.rsServices.getRegionServerAccounting();
   // don't initialize coprocessors if not running within a regionserver
   // TODO: revisit if coprocessors should load in other cases
-  this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, 
baseConf);
+  this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, conf);
   this.metricsRegionWrapper = new MetricsRegionWrapperImpl(this);
   this.metricsRegion = new MetricsRegion(this.metricsRegionWrapper);
 } else {
Index: 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
===
--- 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
  (revision 1463235)
+++ 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
  (working copy)
@@ -121,7 +121,7 @@
 baseStagingDir = SecureBulkLoadUtil.getBaseStagingDir(conf);

 try {
-  fs = FileSystem.get(conf);
+  fs = FileSystem.get(new Configuration(conf));
   fs.mkdirs(baseStagingDir, PERM_HIDDEN);
   fs.setPermission(baseStagingDir, PERM_HIDDEN);
   //no sticky bit in hadoop-1.0, making directory nonempty so it never 
gets erased
{code}
But I still got:
{code}
2013-04-01 14:43:38,337 ERROR [IPC Server handler 0 on 52052] 
access.SecureBulkLoadEndpoint$1(240): Failed to complete bulk load
java.lang.UnsupportedOperationException: Immutable Configuration
  at 
org.apache.hadoop.hbase.CompoundConfiguration.setClass(CompoundConfiguration.java:474)
  at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:193)
  at 
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:249)
  at 
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:168)
  at 
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
  at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:421)
  at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:388)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:126)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2312)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2346)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2328)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:164)
  at 
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:224)
  at 
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:218)
{code}
I also tried this:
{code}
Index: 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
===
--- 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(revision 1463235)
+++ 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(working copy)
@@ -484,7 +484,7 @@
   this.rsAccounting = this.rsServices.getRegionServerAccounting();
   // don't initialize coprocessors if not running within a regionserver
   // TODO: revisit if coprocessors should load in other cases
-  this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, 
baseConf);
+  this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, conf);
   this.metricsRegionWrapper = new MetricsRegionWrapperImpl(this);
   this.metricsRegion = new MetricsRegion(this.metricsRegionWrapper);
 } else {
Index: 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
===
--- 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
  (revision 1463235)
+++ 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
  (working copy)
@@ -31,6 +31,7 @@
 import org.apache.hadoop.fs.permission.FsPermission;
 import org.apache.hadoop.hbase.Coprocessor;

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619238#comment-13619238
 ] 

Andrew Purtell commented on HBASE-7904:
---

bq. Suggest we revert this patch. Seems broke that CPs see another config.

+1 for revert

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619240#comment-13619240
 ] 

Ted Yu commented on HBASE-7904:
---

bq. Suggest we revert this patch
I agree. We should consider a better design first.
What about the notification that checkin to 0.95 is on hold ?

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619246#comment-13619246
 ] 

Ted Yu commented on HBASE-7904:
---

Another solution I can think of is to expose two Configurations from 
CoprocessorEnvironment: one that is immutable, the other mutable. The second 
one would be used for this call:
{code}
  fs = FileSystem.get(conf);
{code}
BTW, here is javadoc for FileSystem.get():
{code}
   * Get a filesystem instance based on the uri, the passed
   * configuration and the user
   * @param uri of the filesystem
   * @param conf the configuration to use
{code}
I don't see mentioning that conf cannot be immutable.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-04-01 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619251#comment-13619251
 ] 

stack commented on HBASE-7904:
--

[~ted_yu] Please back this out of trunk and 0.95 (I tried but cannot figure 
what has been committed w/o doing a bunch of work)

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619252#comment-13619252
]

Andrew Purtell commented on HBASE-7904:
---

bq. Another solution I can think of is to expose two Configurations from
CoprocessorEnvironment: one that is immutable, the other mutable. The second
one would be used for this call

I think that's closer to a solution. I think what you want is to take the
immutable CompoundConfiguration and create a new mutable Configuration with it
to pass to the RegionCoprocessorHost constructor, one initialized with the
overrides provided by the compound view.

But I suggest reverting this patch and tackling the issues here with focused
JIRAs.

Make mapreduce jobs pass based on 2.0.4-alpha
-

Key: HBASE-7904
URL: https://issues.apache.org/jira/browse/HBASE-7904
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
Fix For: 0.95.0, 0.98.0

2.0.3-alpha has been released.
We should upgrade the dependency.

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619259#comment-13619259
 ] 

Ted Yu commented on HBASE-7904:
---

Backed out of 0.95 and trunk.

Will attach patch v10 soon.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619261#comment-13619261
 ] 

Andrew Purtell commented on HBASE-7904:
---

I'm going to close this issue. Let's deal with the individual items here with 
JIRAs that focus on them individually.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619284#comment-13619284
 ] 

Ted Yu commented on HBASE-7904:
---

YARN-449, marked Resolved, is linked to this issue.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619286#comment-13619286
 ] 

Andrew Purtell commented on HBASE-7904:
---

Unlinked from YARN-449.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619354#comment-13619354
 ] 

Ted Yu commented on HBASE-7904:
---

I plan to introduce the following methods to CoprocessorEnvironment and :
{code}
  /** @return the mutable configuration */
  public Configuration getMutableConfiguration();

  /** @return the immutable configuration */
  public Configuration getImmutableConfiguration();
{code}
w.r.t. the existing method:
{code}
  /** @return the configuration */
  public Configuration getConfiguration();
{code}
We can either remove it or let it return the immutable config (current 
behavior).

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619356#comment-13619356
 ] 

Andrew Purtell commented on HBASE-7904:
---

{quote}
I plan to introduce the following methods to CoprocessorEnvironment and :
{noformat}
  /** @return the mutable configuration */
  public Configuration getMutableConfiguration();

  /** @return the immutable configuration */
  public Configuration getImmutableConfiguration();
{noformat}
{quote}

-1, see my above comment. I know you read it, why are you ignoring it?

-1, this issue is closed.


 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619363#comment-13619363
 ] 

Ted Yu commented on HBASE-7904:
---

I think you were referring to 
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13619252page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13619252

SecureBulkLoadEndpoint needs a way to get access to both Configuration's, right 
?
See the following in SecureBulkLoadEndpoint#start():
{code}
conf = env.getConfiguration();
{code}

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619366#comment-13619366
 ] 

Andrew Purtell commented on HBASE-7904:
---

-1 I won't discuss this further here, this issue is closed. Kindly open a JIRA 
to address the SecureBulkLoadEndpoint issue.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619367#comment-13619367
 ] 

Ted Yu commented on HBASE-7904:
---

@Andy:
Maybe you're clear about the new JIRA (title, description, etc)

If you can open the JIRA, that would be great.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619370#comment-13619370
 ] 

Andrew Purtell commented on HBASE-7904:
---

bq. If you can open the JIRA, that would be great.

Surely this is an April Fool's joke and not a display of petulance.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-04-01 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619438#comment-13619438
 ] 

Hudson commented on HBASE-7904:
---

Integrated in HBase-TRUNK #4005 (See 
[https://builds.apache.org/job/HBase-TRUNK/4005/])
HBASE-7904 Revert, due to concern over coprocessor exposure (Revision 
1463326)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
* /hbase/trunk/pom.xml


 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
  Components: mapreduce
Reporter: Ted Yu
Priority: Critical
 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-03-31 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618532#comment-13618532
 ] 

Hudson commented on HBASE-7904:
---

Integrated in hbase-0.95-on-hadoop2 #49 (See 
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/49/])
HBASE-7904 Make mapreduce jobs pass based on 2.0.4-alpha (Ted Yu) (Revision 
1462873)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
* /hbase/branches/0.95/pom.xml


 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-03-30 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618127#comment-13618127
 ] 

Hudson commented on HBASE-7904:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #470 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/470/])
HBASE-7904 Addendum turns off vmem check (Revision 1462701)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java


 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-03-30 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618220#comment-13618220
 ] 

Ted Yu commented on HBASE-7904:
---

From trunk build results on Apache and EC2 Jenkins, mapreduce jobs passed.

Integrated to 0.95 as well.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-03-30 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618247#comment-13618247
 ] 

Hudson commented on HBASE-7904:
---

Integrated in hbase-0.95 #115 (See 
[https://builds.apache.org/job/hbase-0.95/115/])
HBASE-7904 Make mapreduce jobs pass based on 2.0.4-alpha (Ted Yu) (Revision 
1462873)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
* /hbase/branches/0.95/pom.xml


 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0, 0.98.0

 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 
 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 
 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 
 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 
 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617363#comment-13617363
 ] 

Ted Yu commented on HBASE-7904:
---

bq. Why this change?
When SecureBulkLoadEndpoint tries to access hdfs, 
org.apache.hadoop.ipc.RPC.setProtocolEngine() would call 
Configuration.setClass(). Without the change, we would get:
{code}
2013-03-25 22:36:19,493 ERROR [IPC Server handler 2 on 40900] 
access.SecureBulkLoadEndpoint$1(240): Failed to complete bulk load
java.lang.UnsupportedOperationException: Immutable Configuration
  at 
org.apache.hadoop.hbase.CompoundConfiguration.setClass(CompoundConfiguration.java:474)
  at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:193)
{code}
bq. This seems to be only real addition
I followed your suggestion. Since debugging output from copyNewConfEntries() is 
no longer needed, I removed that method.

bq. Did something change in the hadoop2 snapshot?
I would have to go into recent checkins in more detail. This may be answered by 
a YARN developer.

bq. Why we have to do this now?
We have been doing that for the remaining tests. See one example from 
testSimpleCase():
{code}
Configuration conf = opts.getConfiguration();

// copy or add the necessary configuration values from the map reduce 
config to the hbase config
copyConfigurationValues(UTIL.getConfiguration(), conf);
args = opts.getRemainingArgs();
{code}
bq. Why create a new HBaseConfiguration instance?
I don't see reference to HBaseConfiguration in TestImportExport. The patch 
doesn't introduce HBaseConfiguration either.

bq. How does cluster config relate to UTIL.getConfiguration?
cluster is a MiniHBaseCluster instance. UTIL is an HBaseTestingUtility where we 
store all yarn-related conf parameters. So they're not the same.
copyConfigurationValues() passes all yarn-related conf parameters from 
UTIL.getConfiguration() to conf.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0

 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 
 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 
 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 
 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, 
 hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-03-29 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617686#comment-13617686
]

stack commented on HBASE-7904:
--

bq. I would have to go into recent checkins in more detail. This may be
answered by a YARN developer.

I suppose I could go look myself but I ask because supposedly there was an
incompatibility issue. If tests are now passing, is it because the snapshot
has incompatibilities fixed now? (I suppose I could look for myself but you
are supposed to know what is going on here since you are 'fixing' it). Or is
it just this config. manipulation that was needed and there is no compatibility
problem.

Would think we would want to answer the above before we had confidence in the
commit.

bq. I don't see reference to HBaseConfiguration in TestImportExport. The patch
doesn't introduce HBaseConfiguration either.

UTIL.getConfiguration is an HBaseConfiguration.

bq. cluster is a MiniHBaseCluster instance. UTIL is an HBaseTestingUtility
where we store all yarn-related conf parameters.

Where do you think cluster gets its config from? Isn't it using
UTIL.getConfiguration?

I was asking questions about the config juggling because to me it looked like
the test was doing a bunch of config copying to no useful end; that a little
thought might make it so there was less juggling going on. Would help the
developer who comes along after trying to make sense of this test.

Make mapreduce jobs pass based on 2.0.4-alpha
-

Key: HBASE-7904
URL: https://issues.apache.org/jira/browse/HBASE-7904
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
Fix For: 0.95.0

Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt,
7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt,
7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt,
7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt,
hbase-7904-v3.txt

2.0.3-alpha has been released.
We should upgrade the dependency.

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617700#comment-13617700
 ] 

Ted Yu commented on HBASE-7904:
---

I looked at the changes in hadoop-2 branch in the past week.
There were fixes to CapacityScheduler but I think they may not be directly 
related to the test failures we saw.

bq. Or is it just this config. manipulation that was needed
I think so.

bq. Where do you think cluster gets its config from? Isn't it using 
UTIL.getConfiguration?
We obtain cluster at the beginning of the test:
{code}
  public static void beforeClass() throws Exception {
cluster = UTIL.startMiniCluster();
{code}
Config manipulation happens in UTIL.startMiniMapReduceCluster().
But startMiniCluster() only calls startMiniDFSCluster(), startMiniZKCluster() 
and startMiniHBaseCluster(), in that order.

This is why a little extra manipulation is needed in the tests of 
TestImportExport.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0

 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 
 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 
 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 
 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, 
 hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha

2013-03-29 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617841#comment-13617841
 ] 

stack commented on HBASE-7904:
--

I don't follow at all what you are saying above.  You are not sure what fixed 
the problem and you present a spaghetti list of how we start up clusters in our 
test util to conclude a little extra manipulation is needed

+1 on patch.   I want to put this issue out of its misery.  Can open a new one 
if still incompatibility issues running against hadoop2.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0

 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 
 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 
 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 
 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, 
 hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha


[ 
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617844#comment-13617844
 ] 

Ted Yu commented on HBASE-7904:
---

bq. But startMiniCluster() only calls startMiniDFSCluster(), 
startMiniZKCluster() and startMiniHBaseCluster(), in that order.
I was just saying that cluster (a MiniHBaseCluster) wouldn't get its config 
from UTIL.getConfiguration because the effective config is updated in 
UTIL.startMiniMapReduceCluster().

bq. Can open a new one if still incompatibility issues running against hadoop2.
True. Another JIRA would be needed when 2.0.4-ALPHA is released.
In the mean time, if we see mapreduce errors using 2.0 SNAPSHOT build, we can 
provide feedback upstream.

 Make mapreduce jobs pass based on 2.0.4-alpha
 -

 Key: HBASE-7904
 URL: https://issues.apache.org/jira/browse/HBASE-7904
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
 Fix For: 0.95.0

 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 
 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 
 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 
 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, 
 hbase-7904-v3.txt


 2.0.3-alpha has been released.
 We should upgrade the dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha