[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623091#comment-13623091
]
Ted Yu commented on HBASE-8258:
-------------------------------
[~jmhsieh]:
bq. I'd like to get this fixed without having to upgrade hadoop versions to a
snapshot version.
Did you mean that we should wait for 2.0.4-alpha to come out ? I think for
0.95, we should use 2.0.4-alpha which is to be released soon.
If we use 2.0.3-alpha, we would get (due to capacity-scheduler.xml missing in
artifact):
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13584497&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13584497
In case you haven't read comments from Siddharth, you can find them starting
with this one :
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13611080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611080
That was how the HBaseConfiguration.merge() call in HBaseTestingUtility.java
came about.
Changes in TestImportExport.java were to align with remaining tests in that
file. The comment below would provide some more context:
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13617700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13617700
bq. vague statements "necessary configuration values" without explanation
I think you were referring to the following:
{code}
// copy or add the necessary configuration values from the map reduce
config to the hbase config
{code}
Such comments have been in TestImportExport. This command can show us the
history:
{code}
$ svn blame
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
| grep 'necessary configurati'
{code}
For SecureBulkLoadEndpoint.java, here is the background:
Secure bulk loading needs to establish staging directory on hdfs. So it calls:
{code}
fs = FileSystem.get(conf);
{code}
The conf above is an instance of CompoundConfiguration. NameNodeProxies would
set up RPC engine by modifying conf.
This results in:
{code}
2013-03-25 22:36:19,493 ERROR [IPC Server handler 2 on 40900]
access.SecureBulkLoadEndpoint$1(240): Failed to complete bulk load
java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.CompoundConfiguration.setClass(CompoundConfiguration.java:474)
at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:193)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:249)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:168)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:421)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:388)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2277)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2311)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2293)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:317)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:224)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:218)
{code}
In 8258-v1-hadoop-2.0.txt, I tried to limit the scope of changes while
satisfying requirement from hadoop 2.0
That is, clone a mutable Configuration (an HBaseConfiguration to be more exact)
from the CompoundConfiguration and pass it to FileSystem.get().
I am open to discussion about proposed changes. I can upload the patch onto
review board so that opinion on each change can be expressed better.
I am open to other proposals as well.
> Make mapreduce tests pass on hadoop2
> ------------------------------------
>
> Key: HBASE-8258
> URL: https://issues.apache.org/jira/browse/HBASE-8258
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Reporter: stack
> Priority: Blocker
> Fix For: 0.95.1
>
> Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt
>
>
> HBASE-7904 was a first attempt at making this work but it got lost in the
> weeds.
> This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
> breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
