[jira] [Commented] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[
https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110481#comment-15110481
]
jack liuquan commented on HADOOP-11828:
---
Hi [~zhz],
Thanks for your review!
bq.For an util class with all static methods, we don't need a constructor.
I add a private constructor for checkstyle issue, just reference to code of
{{DumpUtil}} class in {{rawcoder}}
bq.The getPiggyBacksFromInput method is fairly complex and deserves a unit
test. A ASCII illustration would also be very helpful, similar to Figure 4 in
the Hitchhiker paper .
Although {{getPiggyBacksFromInput }} is fairly complex, there only one running
branch in it. I think current unit test cases are good to cover it. I will add
a ASCII illustration for it.
bq.3.Maybe I'm missing something, but how do we guarantee the length of inputs
passed to performCoding is always numDataUnits * subPacketSize?
As Rashmi said, subPacketSize is always 2 in Hitchhiker. I think we can
guarantee it when we preparing block chunks.
> Implement the Hitchhiker erasure coding algorithm
> -
>
> Key: HADOOP-11828
> URL: https://issues.apache.org/jira/browse/HADOOP-11828
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: jack liuquan
> Attachments: 7715-hitchhikerXOR-v2-testcode.patch,
> 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch,
> HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch,
> HADOOP-11828-hitchhikerXOR-V6.patch, HADOOP-11828-hitchhikerXOR-V7.patch,
> HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch
>
>
> [Hitchhiker |
> http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is
> a new erasure coding algorithm developed as a research project at UC
> Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45%
> during data reconstruction while retaining the same storage capacity and
> failure tolerance capability as RS codes. This JIRA aims to introduce
> Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12726) Unsupported operations should throw UnsupportedOperationException
Daniel Templeton created HADOOP-12726:
-
Summary: Unsupported operations should throw
UnsupportedOperationException
Key: HADOOP-12726
URL: https://issues.apache.org/jira/browse/HADOOP-12726
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.7.1
Reporter: Daniel Templeton
Assignee: Daniel Templeton
In the {{FileSystem}} implementation classes, unsupported operations throw
{{new IOException("Not supported")}}, which makes it needlessly difficult to
distinguish an actual error from an unsupported operation. They should instead
throw {{new UnsupportedOperationException()}}.
It's possible that this anti-idiom is used elsewhere in the code base. This
JIRA should include finding and cleaning up those instances as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12712) Fix some cmake plugin and native build warnings
[ https://issues.apache.org/jira/browse/HADOOP-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110401#comment-15110401 ] Varun Vasudev commented on HADOOP-12712: Thanks for catching this [~cmccabe]. My apologies for letting this slip through. Like Andrew, I suspect some implicit return was carried out by the compiler. I've cherry-picked the patch to 2.8 as well. > Fix some cmake plugin and native build warnings > --- > > Key: HADOOP-12712 > URL: https://issues.apache.org/jira/browse/HADOOP-12712 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.4.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.8.0, 2.9.0 > > Attachments: HADOOP-12712.001.patch, HADOOP-12712.002.patch, > HADOOP-12712.003.patch > > > Fix some native build warnings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12712) Fix some cmake plugin and native build warnings
[ https://issues.apache.org/jira/browse/HADOOP-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated HADOOP-12712: --- Fix Version/s: 2.8.0 > Fix some cmake plugin and native build warnings > --- > > Key: HADOOP-12712 > URL: https://issues.apache.org/jira/browse/HADOOP-12712 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.4.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.8.0, 2.9.0 > > Attachments: HADOOP-12712.001.patch, HADOOP-12712.002.patch, > HADOOP-12712.003.patch > > > Fix some native build warnings -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12605) Fix intermittent failure of TestIPC.testIpcWithReaderQueuing
[ https://issues.apache.org/jira/browse/HADOOP-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-12605: -- Resolution: Fixed Fix Version/s: 2.9.0 Target Version/s: 2.9.0 Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks, [~xiaochen] and [~ste...@apache.org]. > Fix intermittent failure of TestIPC.testIpcWithReaderQueuing > > > Key: HADOOP-12605 > URL: https://issues.apache.org/jira/browse/HADOOP-12605 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 2.9.0 > > Attachments: HADOOP-12605.001.patch, HADOOP-12605.002.patch, > HADOOP-12605.003.patch, HADOOP-12605.004.patch, HADOOP-12605.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12605) Fix intermittent failure of TestIPC.testIpcWithReaderQueuing
[ https://issues.apache.org/jira/browse/HADOOP-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110328#comment-15110328 ] Hudson commented on HADOOP-12605: - FAILURE: Integrated in Hadoop-trunk-Commit #9146 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9146/]) HADOOP-12605. Fix intermittent failure of (iwasakims: rev 9eec6cbedcc53e6dd306a4a578a22937ae239260) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java > Fix intermittent failure of TestIPC.testIpcWithReaderQueuing > > > Key: HADOOP-12605 > URL: https://issues.apache.org/jira/browse/HADOOP-12605 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HADOOP-12605.001.patch, HADOOP-12605.002.patch, > HADOOP-12605.003.patch, HADOOP-12605.004.patch, HADOOP-12605.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11505) Various native parts use bswap incorrectly and unportably
[ https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110257#comment-15110257 ] Alan Burlison commented on HADOOP-11505: Ah, it's been pushed since my last sync, I'll resync - thanks. > Various native parts use bswap incorrectly and unportably > - > > Key: HADOOP-11505 > URL: https://issues.apache.org/jira/browse/HADOOP-11505 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Alan Burlison > Fix For: 3.0.0 > > Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch, > HADOOP-11505.004.patch, HADOOP-11505.005.patch, HADOOP-11505.006.patch, > HADOOP-11505.007.patch > > > hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some > cases. Also, on some alternate, non-x86, non-ARM architectures the generated > code is incorrect. Thanks to Steve Loughran and Edward Nevill for finding > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12727) Minor cleanups needed for CMake 3.X
Alan Burlison created HADOOP-12727:
--
Summary: Minor cleanups needed for CMake 3.X
Key: HADOOP-12727
URL: https://issues.apache.org/jira/browse/HADOOP-12727
Project: Hadoop Common
Issue Type: Sub-task
Components: native
Affects Versions: 2.7.1
Reporter: Alan Burlison
Assignee: Alan Burlison
Priority: Minor
On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
\\
\\
* There's a syntax error in
{{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
warning in 3.X
* {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
{{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
incorrect flags, 3.3.2 doesn't and building static libraries fails as a result.
See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12043) Display warning if defaultFs is not set when running fs commands.
[
https://issues.apache.org/jira/browse/HADOOP-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110717#comment-15110717
]
Kazuho Fujii commented on HADOOP-12043:
---
Hi,
This function is useful. But, I think "fs.defaultFs" in the warning message
should be the correct name "fs.defaultFS". A user who does not set fs.defaultFS
property or mistypes the property name gets confused when seeing the message.
At least I was misled.
How do you think?
> Display warning if defaultFs is not set when running fs commands.
> -
>
> Key: HADOOP-12043
> URL: https://issues.apache.org/jira/browse/HADOOP-12043
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.7.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8322.000.patch, HDFS-8322.001.patch,
> HDFS-8322.002.patch, HDFS-8322.003.patch, HDFS-8322.003.patch,
> HDFS-8322.004.patch, HDFS-8322.005.patch, HDFS-8322.006.patch
>
>
> Using {{LocalFileSystem}} is rarely the intention of running {{hadoop fs
> -ls}}.
> This JIRA proposes displaying a warning message if hadoop fs -ls is showing
> the local filesystem or using default fs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110802#comment-15110802 ] Allen Wittenauer commented on HADOOP-11683: --- I think it is important to recognize that principal -> username conversion happens all over the stack. For example, every single web UI is going to be needing this functionality. Unless I missed something, the way this code is written will require the mapping code+configuration to be present on every single node in a way that every single process is going to need access. To make matters worse, if a non-Java AM decides to provide user auth (think Slider), it doesn't appear to have a way to access this functionality without using JNI. > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch, HADOOP-11683.002.patch, > HADOOP-11683.003.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12423) Handle failure of registering shutdownhook by ShutdownHookManager in static block
[
https://issues.apache.org/jira/browse/HADOOP-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110731#comment-15110731
]
Hudson commented on HADOOP-12423:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #9148 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9148/])
HADOOP-12423. Handle failure of registering shutdownhook by (vinayakumarb: rev
446987e20aacd870e1bc3f950b2bd4bbda1f9571)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ShutdownHookManager.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> Handle failure of registering shutdownhook by ShutdownHookManager in static
> block
> -
>
> Key: HADOOP-12423
> URL: https://issues.apache.org/jira/browse/HADOOP-12423
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-12423.patch
>
>
> If JVM is under shutdown, static method in ShutdownHookManager will throw an
> IllegalStateException. This exception should be caught and ignored while
> registering the hooks.
> Stack trace:
> {noformat}
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.util.ShutdownHookManager
>at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2639)
> ~[stormjar.jar:1.4.0-SNAPSHOT]
>at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> ~[stormjar.jar:1.4.0-SNAPSHOT]
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> ~[stormjar.jar:1.4.0-SNAPSHOT]
>...
>...
>at
> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
> ~[storm-core-0.9.5.jar:0.9.5]
>at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461)
> ~[storm-core-0.9.5.jar:0.9.5]
>at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12423) Handle failure of registering shutdownhook by ShutdownHookManager in static block
[
https://issues.apache.org/jira/browse/HADOOP-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HADOOP-12423:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-2 and branch-2.8
thanks for the contribution [~abhishek.agarwal]
thanks [~lakshman] for review.
> Handle failure of registering shutdownhook by ShutdownHookManager in static
> block
> -
>
> Key: HADOOP-12423
> URL: https://issues.apache.org/jira/browse/HADOOP-12423
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-12423.patch
>
>
> If JVM is under shutdown, static method in ShutdownHookManager will throw an
> IllegalStateException. This exception should be caught and ignored while
> registering the hooks.
> Stack trace:
> {noformat}
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.util.ShutdownHookManager
>at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2639)
> ~[stormjar.jar:1.4.0-SNAPSHOT]
>at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
> ~[stormjar.jar:1.4.0-SNAPSHOT]
>at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
> ~[stormjar.jar:1.4.0-SNAPSHOT]
>...
>...
>at
> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
> ~[storm-core-0.9.5.jar:0.9.5]
>at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461)
> ~[storm-core-0.9.5.jar:0.9.5]
>at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-7161) Remove unnecessary oro dependency
[ https://issues.apache.org/jira/browse/HADOOP-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110966#comment-15110966 ] Akira AJISAKA commented on HADOOP-7161: --- +1, committing this. > Remove unnecessary oro dependency > - > > Key: HADOOP-7161 > URL: https://issues.apache.org/jira/browse/HADOOP-7161 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Sean Busbey >Priority: Minor > Labels: beginner > Fix For: 3.0.0 > > Attachments: HADOOP-7161.1.patch > > > Best I can tell we never use the "oro" dependency, but it's been in ivy.xml > forever. Does anyone know any reason we might need it? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-7161) Remove unnecessary oro package from dependency management section
[ https://issues.apache.org/jira/browse/HADOOP-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-7161: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed this to trunk, branch-2, and branch-2.8. Thanks [~busbey] for the contribution! > Remove unnecessary oro package from dependency management section > - > > Key: HADOOP-7161 > URL: https://issues.apache.org/jira/browse/HADOOP-7161 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 2.7.1 >Reporter: Todd Lipcon >Assignee: Sean Busbey >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: HADOOP-7161.1.patch > > > Best I can tell we never use the "oro" dependency, but it's been in ivy.xml > forever. Does anyone know any reason we might need it? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111085#comment-15111085 ] Hudson commented on HADOOP-11828: - FAILURE: Integrated in Hadoop-trunk-Commit #9153 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9153/]) HADOOP-11828. Implement the Hitchhiker erasure coding algorithm. (zhz: rev 1bb31fb22e6f8e6df8e9ff4e94adf20308b4c743) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/util/HHUtil.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecodingStep.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/AbstractHHErasureCodingStep.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestErasureCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncodingStep.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/coder/TestHHXORErasureCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/HHXORErasureEncoder.java > Implement the Hitchhiker erasure coding algorithm > - > > Key: HADOOP-11828 > URL: https://issues.apache.org/jira/browse/HADOOP-11828 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Zhe Zhang >Assignee: jack liuquan > Fix For: 3.0.0 > > Attachments: 7715-hitchhikerXOR-v2-testcode.patch, > 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch, > HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch, > HADOOP-11828-hitchhikerXOR-V6.patch, HADOOP-11828-hitchhikerXOR-V7.patch, > HADOOP-11828-v8.patch, HDFS-7715-hhxor-decoder.patch, > HDFS-7715-hhxor-encoder.patch > > > [Hitchhiker | > http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is > a new erasure coding algorithm developed as a research project at UC > Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% > during data reconstruction while retaining the same storage capacity and > failure tolerance capability as RS codes. This JIRA aims to introduce > Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. > The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-11828: --- Affects Version/s: 3.0.0 > Implement the Hitchhiker erasure coding algorithm > - > > Key: HADOOP-11828 > URL: https://issues.apache.org/jira/browse/HADOOP-11828 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Zhe Zhang >Assignee: jack liuquan > Fix For: 3.0.0 > > Attachments: 7715-hitchhikerXOR-v2-testcode.patch, > 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch, > HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch, > HADOOP-11828-hitchhikerXOR-V6.patch, HADOOP-11828-hitchhikerXOR-V7.patch, > HADOOP-11828-v8.patch, HDFS-7715-hhxor-decoder.patch, > HDFS-7715-hhxor-encoder.patch > > > [Hitchhiker | > http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is > a new erasure coding algorithm developed as a research project at UC > Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% > during data reconstruction while retaining the same storage capacity and > failure tolerance capability as RS codes. This JIRA aims to introduce > Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. > The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-11828: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Target Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Implement the Hitchhiker erasure coding algorithm > - > > Key: HADOOP-11828 > URL: https://issues.apache.org/jira/browse/HADOOP-11828 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Zhe Zhang >Assignee: jack liuquan > Fix For: 3.0.0 > > Attachments: 7715-hitchhikerXOR-v2-testcode.patch, > 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch, > HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch, > HADOOP-11828-hitchhikerXOR-V6.patch, HADOOP-11828-hitchhikerXOR-V7.patch, > HADOOP-11828-v8.patch, HDFS-7715-hhxor-decoder.patch, > HDFS-7715-hhxor-encoder.patch > > > [Hitchhiker | > http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is > a new erasure coding algorithm developed as a research project at UC > Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% > during data reconstruction while retaining the same storage capacity and > failure tolerance capability as RS codes. This JIRA aims to introduce > Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. > The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-7161) Remove unnecessary oro package from dependency management section
[ https://issues.apache.org/jira/browse/HADOOP-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111029#comment-15111029 ] Hudson commented on HADOOP-7161: FAILURE: Integrated in Hadoop-trunk-Commit #9152 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9152/]) HADOOP-7161. Remove unnecessary oro package from dependency management (aajisaka: rev 2ac39ca762fd7e47b7b9a7b3308b31138b0f59be) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-project/pom.xml > Remove unnecessary oro package from dependency management section > - > > Key: HADOOP-7161 > URL: https://issues.apache.org/jira/browse/HADOOP-7161 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 2.7.1 >Reporter: Todd Lipcon >Assignee: Sean Busbey >Priority: Minor > Labels: newbie > Fix For: 2.8.0 > > Attachments: HADOOP-7161.1.patch > > > Best I can tell we never use the "oro" dependency, but it's been in ivy.xml > forever. Does anyone know any reason we might need it? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[
https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HADOOP-11828:
---
Attachment: HADOOP-11828-v8.patch
Thanks for the discussions Rashmi and Jack.
I just added the license header to {{HHXORErasureDecodingStep}} and committed
the v8 patch.
This is really exciting work! Let's plan on getting some performance results.
> Implement the Hitchhiker erasure coding algorithm
> -
>
> Key: HADOOP-11828
> URL: https://issues.apache.org/jira/browse/HADOOP-11828
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: jack liuquan
> Attachments: 7715-hitchhikerXOR-v2-testcode.patch,
> 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch,
> HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch,
> HADOOP-11828-hitchhikerXOR-V6.patch, HADOOP-11828-hitchhikerXOR-V7.patch,
> HADOOP-11828-v8.patch, HDFS-7715-hhxor-decoder.patch,
> HDFS-7715-hhxor-encoder.patch
>
>
> [Hitchhiker |
> http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is
> a new erasure coding algorithm developed as a research project at UC
> Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45%
> during data reconstruction while retaining the same storage capacity and
> failure tolerance capability as RS codes. This JIRA aims to introduce
> Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-7161) Remove unnecessary oro package from dependency management section
[ https://issues.apache.org/jira/browse/HADOOP-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-7161: -- Affects Version/s: 2.7.1 Labels: newbie (was: beginner) Hadoop Flags: Reviewed Fix Version/s: (was: 3.0.0) Summary: Remove unnecessary oro package from dependency management section (was: Remove unnecessary oro dependency) > Remove unnecessary oro package from dependency management section > - > > Key: HADOOP-7161 > URL: https://issues.apache.org/jira/browse/HADOOP-7161 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.23.0, 2.7.1 >Reporter: Todd Lipcon >Assignee: Sean Busbey >Priority: Minor > Labels: newbie > Attachments: HADOOP-7161.1.patch > > > Best I can tell we never use the "oro" dependency, but it's been in ivy.xml > forever. Does anyone know any reason we might need it? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12726) Unsupported operations should throw UnsupportedOperationException
[
https://issues.apache.org/jira/browse/HADOOP-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111051#comment-15111051
]
Steve Loughran commented on HADOOP-12726:
-
in favour for all FS operations. In an ideal world, you'd even be able to tell
in advance what was unsupported, but as it may depend on the specific FS
instance, that's a tough one, especially for rawfs
> Unsupported operations should throw UnsupportedOperationException
> -
>
> Key: HADOOP-12726
> URL: https://issues.apache.org/jira/browse/HADOOP-12726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>
> In the {{FileSystem}} implementation classes, unsupported operations throw
> {{new IOException("Not supported")}}, which makes it needlessly difficult to
> distinguish an actual error from an unsupported operation. They should
> instead throw {{new UnsupportedOperationException()}}.
> It's possible that this anti-idiom is used elsewhere in the code base. This
> JIRA should include finding and cleaning up those instances as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12728) "hadoop jar my.jar" should probably prioritize my.jar in the classpath by default
[ https://issues.apache.org/jira/browse/HADOOP-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12728: -- Target Version/s: 3.0.0 Hadoop Flags: Incompatible change > "hadoop jar my.jar" should probably prioritize my.jar in the classpath by > default > - > > Key: HADOOP-12728 > URL: https://issues.apache.org/jira/browse/HADOOP-12728 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 2.7.1 >Reporter: Ovidiu Gheorghioiu >Assignee: Allen Wittenauer >Priority: Minor > > Found this surprising behavior when testing a dev version of a jar that was > already in the hadoop classpath:"hadoop jar ./my.jar" used the system > my.jar, which was an old version that did not contain my bug fix. Since > "hadoop jar" is the rough equivalent of running an executable, it should use > the version passed on the command line. > Even worse than my case (which took a while to figure out with log messages) > is when I'd be testing that the new version works the same as the old in some > use case. Then I'd think it did, even though the new version was completely > broken. > Allen mentioned verbally that there are some tricky aspects to this, but to > open this issue for tracking / brainstorming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12651) Replace dev-support with wrappers to Yetus
[ https://issues.apache.org/jira/browse/HADOOP-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111225#comment-15111225 ] Allen Wittenauer commented on HADOOP-12651: --- Assuming the +1 is still good, I'll go ahead and commit, update the wiki pages, and drop a note to common-dev so people aren't taken by surprise. Thanks for the review! > Replace dev-support with wrappers to Yetus > -- > > Key: HADOOP-12651 > URL: https://issues.apache.org/jira/browse/HADOOP-12651 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-12651.00.patch, HADOOP-12651.01.patch, > HADOOP-12651.02.patch, HADOOP-12651.03.patch, HADOOP-12651.04.patch > > > Now that Yetus has had a release, we should rip out the components that make > it up from dev-support and replace them with wrappers. The wrappers should: > * default to a sane version > * allow for version overrides via an env var > * download into patchprocess > * execute with the given parameters > Marking this as an incompatible change, since we should also remove the > filename extensions and move these into a bin directory for better > maintenance towards the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12651) Replace dev-support with wrappers to Yetus
[ https://issues.apache.org/jira/browse/HADOOP-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-12651: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Replace dev-support with wrappers to Yetus > -- > > Key: HADOOP-12651 > URL: https://issues.apache.org/jira/browse/HADOOP-12651 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 3.0.0 > > Attachments: HADOOP-12651.00.patch, HADOOP-12651.01.patch, > HADOOP-12651.02.patch, HADOOP-12651.03.patch, HADOOP-12651.04.patch > > > Now that Yetus has had a release, we should rip out the components that make > it up from dev-support and replace them with wrappers. The wrappers should: > * default to a sane version > * allow for version overrides via an env var > * download into patchprocess > * execute with the given parameters > Marking this as an incompatible change, since we should also remove the > filename extensions and move these into a bin directory for better > maintenance towards the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12659) Incorrect usage of config parameters in token manager of KMS
[
https://issues.apache.org/jira/browse/HADOOP-12659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-12659:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Thanks [~liuml07] for the contribution and all for the reviews. I've committed
the patch to trunk, branch-2 and branch-2.8.
> Incorrect usage of config parameters in token manager of KMS
>
>
> Key: HADOOP-12659
> URL: https://issues.apache.org/jira/browse/HADOOP-12659
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.1, 2.6.2
>Reporter: Tianyin Xu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12659.000.patch
>
>
> Hi, the usage of the following configs of Key Management Server (KMS) are
> problematic:
> {{hadoop.kms.authentication.delegation-token.renew-interval.sec}}
> {{hadoop.kms.authentication.delegation-token.removal-scan-interval.sec}}
> The name indicates that the units are {{sec}}, and the online doc shows that
> the default values are {{86400}} and {{3600}}, respectively.
> https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
> which is also defined in
> {code:title=DelegationTokenManager.java|borderStyle=solid}
> 55 public static final String RENEW_INTERVAL = PREFIX +
> "renew-interval.sec";
> 56 public static final long RENEW_INTERVAL_DEFAULT = 24 * 60 * 60;
> ...
> 58 public static final String REMOVAL_SCAN_INTERVAL = PREFIX +
> 59 "removal-scan-interval.sec";
> 60 public static final long REMOVAL_SCAN_INTERVAL_DEFAULT = 60 * 60;
> {code}
> However, in {{DelegationTokenManager.java}} and
> {{ZKDelegationTokenSecretManager.java}}, these two parameters are used
> incorrectly.
> 1. *{{DelegationTokenManager.java}}*
> {code}
> 70 conf.getLong(RENEW_INTERVAL, RENEW_INTERVAL_DEFAULT) * 1000,
> 71 conf.getLong(REMOVAL_SCAN_INTERVAL,
> 72 REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
> {code}
> Apparently, at Line 72, {{REMOVAL_SCAN_INTERVAL}} should be used in the same
> way as {{RENEW_INTERVAL}}, like
> {code}
> 72c72
> < REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
> ---
> > REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
> {code}
> Currently, the unit of
> {{hadoop.kms.authentication.delegation-token.removal-scan-interval.sec}} is
> not {{sec}} but {{millisec}}.
> 2. *{{ZKDelegationTokenSecretManager.java}}*
> {code}
> 142 conf.getLong(DelegationTokenManager.RENEW_INTERVAL,
> 143 DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
> 144 conf.getLong(DelegationTokenManager.REMOVAL_SCAN_INTERVAL,
> 145 DelegationTokenManager.REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
> {code}
> The situation is the opposite in this class that
> {{hadoop.kms.authentication.delegation-token.renew-interval.sec}} is wrong
> but the other is correct...
> A patch should be like
> {code}
> 143c143
> < DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
> ---
> > DelegationTokenManager.RENEW_INTERVAL_DEFAULT) * 1000,
> {code}
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12469) distcp should not ignore the ignoreFailures option
[
https://issues.apache.org/jira/browse/HADOOP-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-12469:
---
Attachment: HADOOP-12469.002.patch
The v2 patch rebases from {{trunk}} branch to resolve trivial conflicts.
> distcp should not ignore the ignoreFailures option
> --
>
> Key: HADOOP-12469
> URL: https://issues.apache.org/jira/browse/HADOOP-12469
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Gera Shegalov
>Assignee: Mingliang Liu
>Priority: Critical
> Attachments: HADOOP-12469.000.patch, HADOOP-12469.001.patch,
> HADOOP-12469.002.patch
>
>
> {{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via
> # via {{RetriableCommand::execute}}
> # via {{CopyMapper#copyFileWithRetry}}
> before {{CopyMapper::handleFailure}} tests
> {code}
> if (ignoreFailures && exception.getCause() instanceof
> RetriableFileCopyCommand.CopyReadException
> {code}
> which is always false.
> Orthogonally, ignoring failures should be mutually exclusive with the atomic
> option otherwise an incomplete dir is eligible for commit defeating the
> purpose.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12659) Incorrect usage of config parameters in token manager of KMS
[
https://issues.apache.org/jira/browse/HADOOP-12659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111546#comment-15111546
]
Hudson commented on HADOOP-12659:
-
FAILURE: Integrated in Hadoop-trunk-Commit #9157 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9157/])
HADOOP-12659. Incorrect usage of config parameters in token manager of (xyao:
rev f3427d3766d7101d0d1c37d6281918551d221ebe)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenManager.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> Incorrect usage of config parameters in token manager of KMS
>
>
> Key: HADOOP-12659
> URL: https://issues.apache.org/jira/browse/HADOOP-12659
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.1, 2.6.2
>Reporter: Tianyin Xu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12659.000.patch
>
>
> Hi, the usage of the following configs of Key Management Server (KMS) are
> problematic:
> {{hadoop.kms.authentication.delegation-token.renew-interval.sec}}
> {{hadoop.kms.authentication.delegation-token.removal-scan-interval.sec}}
> The name indicates that the units are {{sec}}, and the online doc shows that
> the default values are {{86400}} and {{3600}}, respectively.
> https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
> which is also defined in
> {code:title=DelegationTokenManager.java|borderStyle=solid}
> 55 public static final String RENEW_INTERVAL = PREFIX +
> "renew-interval.sec";
> 56 public static final long RENEW_INTERVAL_DEFAULT = 24 * 60 * 60;
> ...
> 58 public static final String REMOVAL_SCAN_INTERVAL = PREFIX +
> 59 "removal-scan-interval.sec";
> 60 public static final long REMOVAL_SCAN_INTERVAL_DEFAULT = 60 * 60;
> {code}
> However, in {{DelegationTokenManager.java}} and
> {{ZKDelegationTokenSecretManager.java}}, these two parameters are used
> incorrectly.
> 1. *{{DelegationTokenManager.java}}*
> {code}
> 70 conf.getLong(RENEW_INTERVAL, RENEW_INTERVAL_DEFAULT) * 1000,
> 71 conf.getLong(REMOVAL_SCAN_INTERVAL,
> 72 REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
> {code}
> Apparently, at Line 72, {{REMOVAL_SCAN_INTERVAL}} should be used in the same
> way as {{RENEW_INTERVAL}}, like
> {code}
> 72c72
> < REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
> ---
> > REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
> {code}
> Currently, the unit of
> {{hadoop.kms.authentication.delegation-token.removal-scan-interval.sec}} is
> not {{sec}} but {{millisec}}.
> 2. *{{ZKDelegationTokenSecretManager.java}}*
> {code}
> 142 conf.getLong(DelegationTokenManager.RENEW_INTERVAL,
> 143 DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
> 144 conf.getLong(DelegationTokenManager.REMOVAL_SCAN_INTERVAL,
> 145 DelegationTokenManager.REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
> {code}
> The situation is the opposite in this class that
> {{hadoop.kms.authentication.delegation-token.renew-interval.sec}} is wrong
> but the other is correct...
> A patch should be like
> {code}
> 143c143
> < DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
> ---
> > DelegationTokenManager.RENEW_INTERVAL_DEFAULT) * 1000,
> {code}
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12651) Replace dev-support with wrappers to Yetus
[ https://issues.apache.org/jira/browse/HADOOP-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111372#comment-15111372 ] Hudson commented on HADOOP-12651: - FAILURE: Integrated in Hadoop-trunk-Commit #9156 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9156/]) HADOOP-12651. Replace dev-support with wrappers to Yetus (aw) (aw: rev 8cecad2d564d89d0f56e98de80a6f9b0bcb39d7a) * dev-support/smart-apply-patch.sh * hadoop-common-project/hadoop-common/pom.xml * BUILDING.txt * dev-support/test-patch.d/shellcheck.sh * dev-support/test-patch.sh * dev-support/test-patch.d/checkstyle.sh * dev-support/test-patch.d/whitespace.sh * dev-support/shelldocs.py * dev-support/releasedocmaker.py HADOOP-12651. Replace dev-support with wrappers to Yetus (aw) (aw: rev 2a867355dfce880bb82257508862ec26bc7f16b9) * dev-support/README.md * dev-support/bin/test-patch * dev-support/bin/releasedocmaker * dev-support/bin/shelldocs * dev-support/bin/yetus-wrapper * dev-support/bin/smart-apply-patch > Replace dev-support with wrappers to Yetus > -- > > Key: HADOOP-12651 > URL: https://issues.apache.org/jira/browse/HADOOP-12651 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 3.0.0 > > Attachments: HADOOP-12651.00.patch, HADOOP-12651.01.patch, > HADOOP-12651.02.patch, HADOOP-12651.03.patch, HADOOP-12651.04.patch > > > Now that Yetus has had a release, we should rip out the components that make > it up from dev-support and replace them with wrappers. The wrappers should: > * default to a sane version > * allow for version overrides via an env var > * download into patchprocess > * execute with the given parameters > Marking this as an incompatible change, since we should also remove the > filename extensions and move these into a bin directory for better > maintenance towards the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12659) Incorrect usage of config parameters in token manager of KMS
[
https://issues.apache.org/jira/browse/HADOOP-12659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111383#comment-15111383
]
Mingliang Liu commented on HADOOP-12659:
Thank you [~asuresh] and [~xyao] for the review.
> Incorrect usage of config parameters in token manager of KMS
>
>
> Key: HADOOP-12659
> URL: https://issues.apache.org/jira/browse/HADOOP-12659
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.1, 2.6.2
>Reporter: Tianyin Xu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-12659.000.patch
>
>
> Hi, the usage of the following configs of Key Management Server (KMS) are
> problematic:
> {{hadoop.kms.authentication.delegation-token.renew-interval.sec}}
> {{hadoop.kms.authentication.delegation-token.removal-scan-interval.sec}}
> The name indicates that the units are {{sec}}, and the online doc shows that
> the default values are {{86400}} and {{3600}}, respectively.
> https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
> which is also defined in
> {code:title=DelegationTokenManager.java|borderStyle=solid}
> 55 public static final String RENEW_INTERVAL = PREFIX +
> "renew-interval.sec";
> 56 public static final long RENEW_INTERVAL_DEFAULT = 24 * 60 * 60;
> ...
> 58 public static final String REMOVAL_SCAN_INTERVAL = PREFIX +
> 59 "removal-scan-interval.sec";
> 60 public static final long REMOVAL_SCAN_INTERVAL_DEFAULT = 60 * 60;
> {code}
> However, in {{DelegationTokenManager.java}} and
> {{ZKDelegationTokenSecretManager.java}}, these two parameters are used
> incorrectly.
> 1. *{{DelegationTokenManager.java}}*
> {code}
> 70 conf.getLong(RENEW_INTERVAL, RENEW_INTERVAL_DEFAULT) * 1000,
> 71 conf.getLong(REMOVAL_SCAN_INTERVAL,
> 72 REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
> {code}
> Apparently, at Line 72, {{REMOVAL_SCAN_INTERVAL}} should be used in the same
> way as {{RENEW_INTERVAL}}, like
> {code}
> 72c72
> < REMOVAL_SCAN_INTERVAL_DEFAULT * 1000));
> ---
> > REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
> {code}
> Currently, the unit of
> {{hadoop.kms.authentication.delegation-token.removal-scan-interval.sec}} is
> not {{sec}} but {{millisec}}.
> 2. *{{ZKDelegationTokenSecretManager.java}}*
> {code}
> 142 conf.getLong(DelegationTokenManager.RENEW_INTERVAL,
> 143 DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
> 144 conf.getLong(DelegationTokenManager.REMOVAL_SCAN_INTERVAL,
> 145 DelegationTokenManager.REMOVAL_SCAN_INTERVAL_DEFAULT) * 1000);
> {code}
> The situation is the opposite in this class that
> {{hadoop.kms.authentication.delegation-token.renew-interval.sec}} is wrong
> but the other is correct...
> A patch should be like
> {code}
> 143c143
> < DelegationTokenManager.RENEW_INTERVAL_DEFAULT * 1000),
> ---
> > DelegationTokenManager.RENEW_INTERVAL_DEFAULT) * 1000,
> {code}
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12469) distcp should not ignore the ignoreFailures option
[
https://issues.apache.org/jira/browse/HADOOP-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111677#comment-15111677
]
Hadoop QA commented on HADOOP-12469:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 19s
{color} | {color:green} hadoop-distcp in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 31s
{color} | {color:green} hadoop-distcp in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783685/HADOOP-12469.002.patch
|
| JIRA Issue | HADOOP-12469 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7c4655b03f00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f3427d3 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111838#comment-15111838
]
Xiao Chen commented on HADOOP-12699:
Zhe talked to me offline today regarding the sequence to reproduce the failure.
Below is the order of the events happened to result in an old key being
retrieved. Note that the race is only in the
{{EagerKeyGeneratorKeyProviderCryptoExtension}} in {{KMSWebApp}}, so steps are
only focused on the {{ValueQueue}} here.
# Caller {{generateEncryptedKey}}, resulting in the server side KMSProvider's
{{ValueQueue}} fired up an async filling task - a {{NamedRunnable}} is added
into the {{UniqueKeyBlockingQueue}}(queue).
# The {{NamedRunnable}} in step 1 is executed, filling in the local
{{retEdeks}}, but not yet {{keyQueue.addAll}}. Code
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/EagerKeyGeneratorKeyProviderCryptoExtension.java#L83].
# Caller {{rollNewVersion}}, {{ValueQueue}} is drained. That is, the queue of
the being rolled name in {{LoadingCache}}(keyQueues) is cleared.
# The {{NamedRunnable}} continues to run, adding the old keys from local
variable into {{keyQueue}}.
# Caller side {{generateEncryptedKey}}, poll from the keyQueue, get old key.
As a side note, simply not using the local variable wouldn't help, because even
if
> TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
>
>
> Key: HADOOP-12699
> URL: https://issues.apache.org/jira/browse/HADOOP-12699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch,
> HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch,
> HADOOP-12699.07.patch, HADOOP-12699.repro.2, HADOOP-12699.repro.patch
>
>
> I've seen several failures of testKMSProvider, all failed in the following
> snippet:
> {code}
> // test rollover draining
> KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
> createKeyProviderCryptoExtension(kp);
> .
> EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
> kpce.rollNewVersion("k6");
> EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
> Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
> ekv2.getEncryptionKeyVersionName());
> {code}
> with error message
> {quote}Values should be different. Actual: k6@0{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12588) Fix intermittent test failure of TestGangliaMetrics
[
https://issues.apache.org/jira/browse/HADOOP-12588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HADOOP-12588:
--
Status: Open (was: Patch Available)
> Fix intermittent test failure of TestGangliaMetrics
> ---
>
> Key: HADOOP-12588
> URL: https://issues.apache.org/jira/browse/HADOOP-12588
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Tsuyoshi Ozawa
>Assignee: Masatake Iwasaki
> Attachments: HADOOP-12588.001.patch, HADOOP-12588.addendum.02.patch,
> HADOOP-12588.addendum.patch
>
>
> Jenkins found this test failure on HADOOP-11149.
> {quote}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.773 sec <<<
> FAILURE! - in org.apache.hadoop.metrics2.impl.TestGangliaMetrics
> testGangliaMetrics2(org.apache.hadoop.metrics2.impl.TestGangliaMetrics) Time
> elapsed: 0.39 sec <<< FAILURE!
> java.lang.AssertionError: Missing metrics: test.s1rec.Xxx
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.checkMetrics(TestGangliaMetrics.java:159)
> at
> org.apache.hadoop.metrics2.impl.TestGangliaMetrics.testGangliaMetrics2(TestGangliaMetrics.java:137)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12668) Modify HDFS embeded jetty server logic in HttpServer2.java to exclude weak Ciphers through ssl-server.conf
[ https://issues.apache.org/jira/browse/HADOOP-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111702#comment-15111702 ] John Zhuge commented on HADOOP-12668: - +1 for the patch > Modify HDFS embeded jetty server logic in HttpServer2.java to exclude weak > Ciphers through ssl-server.conf > -- > > Key: HADOOP-12668 > URL: https://issues.apache.org/jira/browse/HADOOP-12668 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.7.1 >Reporter: Vijay Singh >Assignee: Vijay Singh >Priority: Critical > Labels: common, ha, hadoop, hdfs, security > Attachments: Hadoop-12668.006.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Currently Embeded jetty Server used across all hadoop services is configured > through ssl-server.xml file from their respective configuration section. > However, the SSL/TLS protocol being used for this jetty servers can be > downgraded to weak cipher suites. This code changes aims to add following > functionality: > 1) Add logic in hadoop common (HttpServer2.java and associated interfaces) to > spawn jetty servers with ability to exclude weak cipher suites. I propose we > make this though ssl-server.xml and hence each service can choose to disable > specific ciphers. > 2) Modify DFSUtil.java used by HDFS code to supply new parameter > ssl.server.exclude.cipher.list for hadoop-common code, so it can exclude the > ciphers supplied through this key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12725) RPC call benchmark and optimization in different SASL QOP level
[ https://issues.apache.org/jira/browse/HADOOP-12725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-12725: --- Summary: RPC call benchmark and optimization in different SASL QOP level (was: RPC call benchmark in different SASL QOP level) > RPC call benchmark and optimization in different SASL QOP level > --- > > Key: HADOOP-12725 > URL: https://issues.apache.org/jira/browse/HADOOP-12725 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Kai Zheng > > This would implement a benchmark tool to measure and compare the performance > of Hadoop IPC/RPC call when security is enabled and different SASL > QOP(Quality of Protection) is enforced. Given the data collected by this > benchmark, it would then be able to know if any performance concern when > considering to enforce privacy, integration, or authenticy protection level, > and do optimization accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-12563) Updated utility to create/modify token files
[
https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1574#comment-1574
]
Allen Wittenauer edited comment on HADOOP-12563 at 1/21/16 7:44 PM:
I'm still playing around a bit, but some feedback:
* Non-existent file handling should throw a message as well as give a failure
exit code:
{code}
$ rm foo
$ hadoop dtutil get foo
$ echo $?
0
{code}
* fetchdt needs to default to use the java-based serialization for backward
compatibility. Simple script that shows it:
{code}
#!/bin/bash
URL=hdfs://$(hostname):9000
hdfs fetchdt dt-fetchdt
hadoop dtutil get ${URL} dt-util-default
hadoop dtutil get ${URL} -format java dt-util-java
hadoop dtutil get ${URL} -format protobuf dt-util-proto
ls -l dt-*
{code}
was (Author: aw):
I'm still playing around a bit, but some feedback:
* Non-existent file handling should throw a message as well as give a failure
exit code:
{code}
$ rm foo
$ hadoop dtutil get foo
$ echo $?
0
{code}
* fetchdt needs to default to use the java-based serialization for backward
compatibility.
{code}
#!/bin/bash
URL=hdfs://$(hostname):9000
hdfs fetchdt dt-fetchdt
hadoop dtutil get ${URL} dt-util-default
hadoop dtutil get ${URL} -format java dt-util-java
hadoop dtutil get ${URL} -format protobuf dt-util-proto
ls -l dt-*
{code}
> Updated utility to create/modify token files
>
>
> Key: HADOOP-12563
> URL: https://issues.apache.org/jira/browse/HADOOP-12563
> Project: Hadoop Common
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Matthew Paduano
> Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch,
> HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch,
> HADOOP-12563.06.patch, example_dtutil_commands_and_output.txt,
> generalized_token_case.pdf
>
>
> hdfs fetchdt is missing some critical features and is geared almost
> exclusively towards HDFS operations. Additionally, the token files that are
> created use Java serializations which are hard/impossible to deal with in
> other languages. It should be replaced with a better utility in common that
> can read/write protobuf-based token files, has enough flexibility to be used
> with other services, and offers key functionality such as append and rename.
> The old version file format should still be supported for backward
> compatibility, but will be effectively deprecated.
> A follow-on JIRA will deprecrate fetchdt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
[
https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1574#comment-1574
]
Allen Wittenauer commented on HADOOP-12563:
---
I'm still playing around a bit, but some feedback:
* Non-existent file handling should throw a message as well as give a failure
exit code:
{code}
$ rm foo
$ hadoop dtutil get foo
$ echo $?
0
{code}
* fetchdt needs to default to use the java-based serialization for backward
compatibility.
{code}
#!/bin/bash
URL=hdfs://$(hostname):9000
hdfs fetchdt dt-fetchdt
hadoop dtutil get ${URL} dt-util-default
hadoop dtutil get ${URL} -format java dt-util-java
hadoop dtutil get ${URL} -format protobuf dt-util-proto
ls -l dt-*
{code}
> Updated utility to create/modify token files
>
>
> Key: HADOOP-12563
> URL: https://issues.apache.org/jira/browse/HADOOP-12563
> Project: Hadoop Common
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Matthew Paduano
> Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch,
> HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch,
> HADOOP-12563.06.patch, example_dtutil_commands_and_output.txt,
> generalized_token_case.pdf
>
>
> hdfs fetchdt is missing some critical features and is geared almost
> exclusively towards HDFS operations. Additionally, the token files that are
> created use Java serializations which are hard/impossible to deal with in
> other languages. It should be replaced with a better utility in common that
> can read/write protobuf-based token files, has enough flexibility to be used
> with other services, and offers key functionality such as append and rename.
> The old version file format should still be supported for backward
> compatibility, but will be effectively deprecated.
> A follow-on JIRA will deprecrate fetchdt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12728) "hadoop jar my.jar" should probably prioritize my.jar in the classpath by default
Ovidiu Gheorghioiu created HADOOP-12728: --- Summary: "hadoop jar my.jar" should probably prioritize my.jar in the classpath by default Key: HADOOP-12728 URL: https://issues.apache.org/jira/browse/HADOOP-12728 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.7.1 Reporter: Ovidiu Gheorghioiu Assignee: Allen Wittenauer Priority: Minor Found this surprising behavior when testing a dev version of a jar that was already in the hadoop classpath:"hadoop jar ./my.jar" used the system my.jar, which was an old version that did not contain my bug fix. Since "hadoop jar" is the rough equivalent of running an executable, it should use the version passed on the command line. Even worse than my case (which took a while to figure out with log messages) is when I'd be testing that the new version works the same as the old in some use case. Then I'd think it did, even though the new version was completely broken. Allen mentioned verbally that there are some tricky aspects to this, but to open this issue for tracking / brainstorming. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
