[jira] [Commented] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[
https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082654#comment-15082654
]
Kai Zheng commented on HADOOP-11828:
Thanks [~lirui] for the taking to fix the issue mentioned by Jack in
HADOOP-12685. Jack if you need the fix you might check the patch there.
By the way, do you have any update, Jack? Note there is some progress in
HDFS-9603 to apply {{ErasureCoder}} in HDFS side. With this done, we can try it
in HDFS to see the effect. Thanks.
> Implement the Hitchhiker erasure coding algorithm
> -
>
> Key: HADOOP-11828
> URL: https://issues.apache.org/jira/browse/HADOOP-11828
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: jack liuquan
> Attachments: 7715-hitchhikerXOR-v2-testcode.patch,
> 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch,
> HADOOP-11828-hitchhikerXOR-V4.patch, HADOOP-11828-hitchhikerXOR-V5.patch,
> HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch
>
>
> [Hitchhiker |
> http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is
> a new erasure coding algorithm developed as a research project at UC
> Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45%
> during data reconstruction while retaining the same storage capacity and
> failure tolerance capability as RS codes. This JIRA aims to introduce
> Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms.
> The existing implementation is based on HDFS-RAID.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12620) Advanced Hadoop Architecture (AHA) - Common
[
https://issues.apache.org/jira/browse/HADOOP-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082795#comment-15082795
]
Dinesh S. Atreya commented on HADOOP-12620:
---
[Steve Loughran |
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stevel%40apache.org],
Regarding
{quote}
-1 to moving my comment.
{quote}
I will make a final request to either move or even delete your comments since I
think your comments are premature and then leave it up to you. I had always
planned to create targeted JIRAs from the beginning. See the following
[comments |
https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046050=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046050
] above.
Also see one of the very first statements above *The next minimal enhancements
to core Hadoop include capability to do “updates-in-place” in HDFS*. I just did
not have time to get around to creating the JIRA [HDFS-9607 |
https://issues.apache.org/jira/browse/HDFS-9607] until this past weekend.
After reading your statement above,
{quote}
"let's use MVCC!".
{quote} I think you are misunderstanding me.
Let me be emphatic and clear. *There is no proposal at this time to add MVCC to
HDFS Layer!!*.
If we have *write/update-in-place* capability, it becomes easier to contemplate
native capabilities at higher layers for example
* JSON-ORC (native JSON support at ORC layer)
* MVCC-ORC (native MVCC support at ORC layer)
* RDF-ORC (native RDF support at ORC layer)
and so on.
> Advanced Hadoop Architecture (AHA) - Common
> ---
>
> Key: HADOOP-12620
> URL: https://issues.apache.org/jira/browse/HADOOP-12620
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Dinesh S. Atreya
>Assignee: Dinesh S. Atreya
>
> Advance Hadoop Architecture (AHA) / Advance Hadoop Adaptabilities (AHA):
> See
> https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046300=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046300
> for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9888) KerberosName static initialization gets default realm, which is unneeded in non-secure deployment.
[
https://issues.apache.org/jira/browse/HADOOP-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kabakchei updated HADOOP-9888:
-
Attachment: (was: HADOOP-9888-v2.7.2-patch.diff)
> KerberosName static initialization gets default realm, which is unneeded in
> non-secure deployment.
> --
>
> Key: HADOOP-9888
> URL: https://issues.apache.org/jira/browse/HADOOP-9888
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.1-beta, 3.0.0
>Reporter: Chris Nauroth
>Assignee: Dmytro Kabakchei
> Attachments: HADOOP-9888.001.patch
>
>
> {{KerberosName}} has a static initialization block that looks up the default
> realm. Running with Oracle JDK7, this code path triggers a DNS query. In
> some environments, we've seen this DNS query block and time out after 30
> seconds. This is part of static initialization, and the class is referenced
> from {{UserGroupInformation#initialize}}, so every daemon and every shell
> command experiences this delay. This occurs even for non-secure deployments,
> which don't need the default realm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-9888) KerberosName static initialization gets default realm, which is unneeded in non-secure deployment.
[
https://issues.apache.org/jira/browse/HADOOP-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083449#comment-15083449
]
Hadoop QA commented on HADOOP-9888:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 38s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 36s
{color} | {color:green} hadoop-auth in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 58s
{color} | {color:green} hadoop-auth in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 3s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12780572/HADOOP-9888.001.patch
|
| JIRA Issue | HADOOP-9888 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e9d104a9d05b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
|
[jira] [Updated] (HADOOP-9888) KerberosName static initialization gets default realm, which is unneeded in non-secure deployment.
[
https://issues.apache.org/jira/browse/HADOOP-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kabakchei updated HADOOP-9888:
-
Attachment: HADOOP-9888.001.patch
> KerberosName static initialization gets default realm, which is unneeded in
> non-secure deployment.
> --
>
> Key: HADOOP-9888
> URL: https://issues.apache.org/jira/browse/HADOOP-9888
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.1-beta, 3.0.0
>Reporter: Chris Nauroth
>Assignee: Dmytro Kabakchei
> Attachments: HADOOP-9888-v2.7.2-patch.diff, HADOOP-9888.001.patch
>
>
> {{KerberosName}} has a static initialization block that looks up the default
> realm. Running with Oracle JDK7, this code path triggers a DNS query. In
> some environments, we've seen this DNS query block and time out after 30
> seconds. This is part of static initialization, and the class is referenced
> from {{UserGroupInformation#initialize}}, so every daemon and every shell
> command experiences this delay. This occurs even for non-secure deployments,
> which don't need the default realm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-9888) KerberosName static initialization gets default realm, which is unneeded in non-secure deployment.
[
https://issues.apache.org/jira/browse/HADOOP-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kabakchei updated HADOOP-9888:
-
Status: Patch Available (was: Open)
The patch doesn't include tests because the only thing that was changed is a
time and way of private field initialization. So the patch doesn't change
expected behavior or any interface.
> KerberosName static initialization gets default realm, which is unneeded in
> non-secure deployment.
> --
>
> Key: HADOOP-9888
> URL: https://issues.apache.org/jira/browse/HADOOP-9888
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.1-beta, 3.0.0
>Reporter: Chris Nauroth
>Assignee: Dmytro Kabakchei
> Attachments: HADOOP-9888.001.patch
>
>
> {{KerberosName}} has a static initialization block that looks up the default
> realm. Running with Oracle JDK7, this code path triggers a DNS query. In
> some environments, we've seen this DNS query block and time out after 30
> seconds. This is part of static initialization, and the class is referenced
> from {{UserGroupInformation#initialize}}, so every daemon and every shell
> command experiences this delay. This occurs even for non-secure deployments,
> which don't need the default realm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-9888) KerberosName static initialization gets default realm, which is unneeded in non-secure deployment.
[
https://issues.apache.org/jira/browse/HADOOP-9888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083329#comment-15083329
]
Dmytro Kabakchei commented on HADOOP-9888:
--
Somebody, please, review.
> KerberosName static initialization gets default realm, which is unneeded in
> non-secure deployment.
> --
>
> Key: HADOOP-9888
> URL: https://issues.apache.org/jira/browse/HADOOP-9888
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.1.1-beta, 3.0.0
>Reporter: Chris Nauroth
>Assignee: Dmytro Kabakchei
> Attachments: HADOOP-9888.001.patch
>
>
> {{KerberosName}} has a static initialization block that looks up the default
> realm. Running with Oracle JDK7, this code path triggers a DNS query. In
> some environments, we've seen this DNS query block and time out after 30
> seconds. This is part of static initialization, and the class is referenced
> from {{UserGroupInformation#initialize}}, so every daemon and every shell
> command experiences this delay. This occurs even for non-secure deployments,
> which don't need the default realm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
[ https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083440#comment-15083440 ] Larry McCay commented on HADOOP-12563: -- I find this patch really interesting. It touches on some of the pain points that I have been thinking about for some time. I would like to see a bit more of the specific problems that are solved by this approach though. The attached generalized_token_usecase doc is a good start but I would like to see the addressed problems enumerated. I also wonder whether a token acquired through dtutil would be usable by services that can be configured to only accept this token as representation of the authentication event. Given some trust mechanism, such as SSL (even better 2 way SSL) we should be able to cryptographically verify and determine whether its issuer is from a trusted authority. I'm also curious about the choice of protobuf for the token rather than JWT. I'd like to understand the differences in portability that you see between the two. JWT has become a very popular format for such things. > Updated utility to create/modify token files > > > Key: HADOOP-12563 > URL: https://issues.apache.org/jira/browse/HADOOP-12563 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Matthew Paduano > Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, > HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch, > HADOOP-12563.06.patch, example_dtutil_commands_and_output.txt, > generalized_token_case.pdf > > > hdfs fetchdt is missing some critical features and is geared almost > exclusively towards HDFS operations. Additionally, the token files that are > created use Java serializations which are hard/impossible to deal with in > other languages. It should be replaced with a better utility in common that > can read/write protobuf-based token files, has enough flexibility to be used > with other services, and offers key functionality such as append and rename. > The old version file format should still be supported for backward > compatibility, but will be effectively deprecated. > A follow-on JIRA will deprecrate fetchdt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12690) Consolidate access of sun.misc.Unsafe
[ https://issues.apache.org/jira/browse/HADOOP-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-12690: Attachment: HADOOP-12690.patch Upload a patch to fix it. > Consolidate access of sun.misc.Unsafe > -- > > Key: HADOOP-12690 > URL: https://issues.apache.org/jira/browse/HADOOP-12690 > Project: Hadoop Common > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du > Attachments: HADOOP-12690.patch > > > Per discussion in Hadoop-12630 > (https://issues.apache.org/jira/browse/HADOOP-12630?focusedCommentId=15082142=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15082142), > we found the access of sun.misc.Unsafe could be problematic for some JVMs in > other platforms. Also, hints from other comments, it is better to consolidate > it as a helper/utility method to shared with several places > (FastByteComparisons, NativeIO, ShortCircuitShm). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12604) Exception may be swallowed in KMSClientProvider
[
https://issues.apache.org/jira/browse/HADOOP-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-12604:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Thanks [~ste...@apache.org] and [~zhz] much for the review. I committed to
trunk, branch-2 and 2.8 branches.
> Exception may be swallowed in KMSClientProvider
> ---
>
> Key: HADOOP-12604
> URL: https://issues.apache.org/jira/browse/HADOOP-12604
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HADOOP-12604.001.patch, HADOOP-12604.002.patch
>
>
> In KMSClientProvider# createConnection
> {code}
> try {
> is = conn.getInputStream();
> ret = mapper.readValue(is, klass);
> } catch (IOException ex) {
> if (is != null) {
> is.close(); <== close may throw exception
> }
> throw ex;
> } finally {
> if (is != null) {
> is.close();
> }
> }
> }
> {code}
> {{ex}} may be swallowed when {{close}} highlighted in the code throws
> exception. Thanks [~qwertymaniac] for pointing this out.
> BTW, I think we should be able to consolidate the two {{is.close()}} in the
> above code, so we don't close the same stream twice. The one in the {{finally
> block}} may be called after an exception is thrown or not, and it may throw
> exception too, we need to be careful not to swallow exception here too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12604) Exception may be swallowed in KMSClientProvider
[
https://issues.apache.org/jira/browse/HADOOP-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083607#comment-15083607
]
Hudson commented on HADOOP-12604:
-
FAILURE: Integrated in Hadoop-trunk-Commit #9050 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9050/])
HADOOP-12604. Exception may be swallowed in KMSClientProvider. (Yongjun
(yzhang: rev 28bd138018bea6fc9c3bfb94c7a4143420f02ced)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> Exception may be swallowed in KMSClientProvider
> ---
>
> Key: HADOOP-12604
> URL: https://issues.apache.org/jira/browse/HADOOP-12604
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: supportability
> Attachments: HADOOP-12604.001.patch, HADOOP-12604.002.patch
>
>
> In KMSClientProvider# createConnection
> {code}
> try {
> is = conn.getInputStream();
> ret = mapper.readValue(is, klass);
> } catch (IOException ex) {
> if (is != null) {
> is.close(); <== close may throw exception
> }
> throw ex;
> } finally {
> if (is != null) {
> is.close();
> }
> }
> }
> {code}
> {{ex}} may be swallowed when {{close}} highlighted in the code throws
> exception. Thanks [~qwertymaniac] for pointing this out.
> BTW, I think we should be able to consolidate the two {{is.close()}} in the
> above code, so we don't close the same stream twice. The one in the {{finally
> block}} may be called after an exception is thrown or not, and it may throw
> exception too, we need to be careful not to swallow exception here too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12690) Consolidate access of sun.misc.Unsafe
[ https://issues.apache.org/jira/browse/HADOOP-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-12690: Target Version/s: 2.8.0 Status: Patch Available (was: Open) > Consolidate access of sun.misc.Unsafe > -- > > Key: HADOOP-12690 > URL: https://issues.apache.org/jira/browse/HADOOP-12690 > Project: Hadoop Common > Issue Type: Bug >Reporter: Junping Du >Assignee: Junping Du > Attachments: HADOOP-12690.patch > > > Per discussion in Hadoop-12630 > (https://issues.apache.org/jira/browse/HADOOP-12630?focusedCommentId=15082142=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15082142), > we found the access of sun.misc.Unsafe could be problematic for some JVMs in > other platforms. Also, hints from other comments, it is better to consolidate > it as a helper/utility method to shared with several places > (FastByteComparisons, NativeIO, ShortCircuitShm). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12353) S3 Native filesystem does not retry all connection failures
[ https://issues.apache.org/jira/browse/HADOOP-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083710#comment-15083710 ] Ravi Prakash commented on HADOOP-12353: --- Hi Mariusz! Have you had a chance to look at this? > S3 Native filesystem does not retry all connection failures > --- > > Key: HADOOP-12353 > URL: https://issues.apache.org/jira/browse/HADOOP-12353 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.1 >Reporter: Mariusz Strzelecki >Assignee: Mariusz Strzelecki >Priority: Minor > Attachments: HADOOP-12353.001.patch, HADOOP-12353.002.patch > > > Current implementation of NativeS3FileSystem.java uses RetryProxy that > retries exceptions that may occur on network communication with S3 API, but > these exceptions must be exact instances of IOException: > https://github.com/apache/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java#L349 > Our tests show that HttpClient throws IOException subclasses which are not > handled by Proxy. > Additionally, not all methods that call S3 API are listed to be handled, i.e. > storeEmptyFile and retrieveMetadata are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12690) Consolidate access of sun.misc.Unsafe
[
https://issues.apache.org/jira/browse/HADOOP-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083674#comment-15083674
]
Hadoop QA commented on HADOOP-12690:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 30s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 51s
{color} | {color:red} root-jdk1.8.0_66 with JDK v1.8.0_66 generated 7 new
issues (was 730, now 728). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 23m 27s
{color} | {color:red} root-jdk1.7.0_91 with JDK v1.7.0_91 generated 11 new
issues (was 724, now 722). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 0s
{color} | {color:red} Patch generated 10 new checkstyle issues in root (total
was 153, now 161). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 5m 17s
{color} | {color:red} hadoop-common-project_hadoop-common-jdk1.7.0_91 with JDK
v1.7.0_91 generated 1 new issues (was 13, now 13). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 44s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 54s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green}
[jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
[ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083702#comment-15083702 ] Ravi Prakash commented on HADOOP-12689: --- Thanks for all your work with the FileSystem contracts, test and documentation Steve! Much appreciated. Thanks for pointing to the documentation. I was not aware of it. We are running with this patch inside Altiscale. I tested it also on my local laptop against a bucket in the "us-west-1" region (with a simple command). Matt also pointed me to http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/testing.html although he couldn't find tests for S3 (for S3n and S3a they exist). As an aside, I don't see the manual test results for HADOOP-10542 in that JIRA either. Were they run? > S3 filesystem operations stopped working correctly > -- > > Key: HADOOP-12689 > URL: https://issues.apache.org/jira/browse/HADOOP-12689 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.0 >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Labels: S3 > Attachments: HADOOP-12689.01.patch > > > HADOOP-10542 was resolved by replacing "return null;" with throwing > IOException. This causes several S3 filesystem operations to fail (possibly > more code is expecting that null return value; these are just the calls I > noticed): > S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException > but instead IOException) > FileSystem.exists() (which no longer returns false but instead raises > IOException) > S3FileSystem.create() (which no longer succeeds but instead raises > IOException) > Run command: > hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/ > Resulting stack trace: > 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test > doesn't exist > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170) > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source) > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > changing the "raise IOE..." to "return null" fixes all of the above code > sites and allows distcp to succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
[ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085150#comment-15085150 ] Ravi Prakash commented on HADOOP-12689: --- I've committed this change to trunk and branch-2. Thanks for your contribution Matt and for your review Steve! > S3 filesystem operations stopped working correctly > -- > > Key: HADOOP-12689 > URL: https://issues.apache.org/jira/browse/HADOOP-12689 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.0 >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Labels: S3 > Fix For: 2.8.0 > > Attachments: HADOOP-12689.01.patch > > > HADOOP-10542 was resolved by replacing "return null;" with throwing > IOException. This causes several S3 filesystem operations to fail (possibly > more code is expecting that null return value; these are just the calls I > noticed): > S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException > but instead IOException) > FileSystem.exists() (which no longer returns false but instead raises > IOException) > S3FileSystem.create() (which no longer succeeds but instead raises > IOException) > Run command: > hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/ > Resulting stack trace: > 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test > doesn't exist > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170) > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source) > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > changing the "raise IOE..." to "return null" fixes all of the above code > sites and allows distcp to succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12689) S3 filesystem operations stopped working correctly
[ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-12689: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > S3 filesystem operations stopped working correctly > -- > > Key: HADOOP-12689 > URL: https://issues.apache.org/jira/browse/HADOOP-12689 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.0 >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Labels: S3 > Fix For: 2.8.0 > > Attachments: HADOOP-12689.01.patch > > > HADOOP-10542 was resolved by replacing "return null;" with throwing > IOException. This causes several S3 filesystem operations to fail (possibly > more code is expecting that null return value; these are just the calls I > noticed): > S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException > but instead IOException) > FileSystem.exists() (which no longer returns false but instead raises > IOException) > S3FileSystem.create() (which no longer succeeds but instead raises > IOException) > Run command: > hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/ > Resulting stack trace: > 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test > doesn't exist > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170) > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source) > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > changing the "raise IOE..." to "return null" fixes all of the above code > sites and allows distcp to succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.001.patch Attaching first iteration for Append support in WASB. Testing: The patch contains new tests to verify the append support. Also changes have been tested against FileSystemContractLive tests for the both Block Blobs and Page Blobs. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
[ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085137#comment-15085137 ] Hudson commented on HADOOP-12689: - FAILURE: Integrated in Hadoop-trunk-Commit #9055 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9055/]) HADOOP-12689. S3 filesystem operations stopped working correctly (raviprak: rev 2d16f40dab291a29b3fc005221b12fd587615d4e) * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3/Jets3tFileSystemStore.java * hadoop-common-project/hadoop-common/CHANGES.txt > S3 filesystem operations stopped working correctly > -- > > Key: HADOOP-12689 > URL: https://issues.apache.org/jira/browse/HADOOP-12689 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.0 >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Labels: S3 > Attachments: HADOOP-12689.01.patch > > > HADOOP-10542 was resolved by replacing "return null;" with throwing > IOException. This causes several S3 filesystem operations to fail (possibly > more code is expecting that null return value; these are just the calls I > noticed): > S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException > but instead IOException) > FileSystem.exists() (which no longer returns false but instead raises > IOException) > S3FileSystem.create() (which no longer succeeds but instead raises > IOException) > Run command: > hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/ > Resulting stack trace: > 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test > doesn't exist > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170) > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source) > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > changing the "raise IOE..." to "return null" fixes all of the above code > sites and allows distcp to succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084623#comment-15084623
]
Hadoop QA commented on HADOOP-12041:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 45s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-common-project/hadoop-common (total was 9, now 11). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 45s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 41s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 9s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_91 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12780653/HADOOP-12041-v5.patch
|
| JIRA Issue | HADOOP-12041 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 591e53311eef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014
[jira] [Commented] (HADOOP-12557) Add information in BUILDING.txt about the need for the FINDBUGS_HOME environment variable for docs builds.
[
https://issues.apache.org/jira/browse/HADOOP-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084991#comment-15084991
]
John Zhuge commented on HADOOP-12557:
-
Typo in "HADOOP-12557.patch":
+need insrtall ...
Should be "need to install".
> Add information in BUILDING.txt about the need for the FINDBUGS_HOME
> environment variable for docs builds.
> --
>
> Key: HADOOP-12557
> URL: https://issues.apache.org/jira/browse/HADOOP-12557
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: huangzheng
>Priority: Minor
> Attachments: HADOOP-12557.patch
>
>
> BUILDING.txt mentions Findbugs 1.3.9 as a requirement, but it doesn't mention
> that the {{FINDBUGS_HOME}} environment variable must point to the base
> directory of the Findbugs installation when running builds with {{-Pdocs}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11661) Deprecate FileUtil#copyMerge
[ https://issues.apache.org/jira/browse/HADOOP-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085037#comment-15085037 ] Brahma Reddy Battula commented on HADOOP-11661: --- There is discussion about this in HDFS-8060..Once before raising pinging [~cnauroth] .. > Deprecate FileUtil#copyMerge > > > Key: HADOOP-11661 > URL: https://issues.apache.org/jira/browse/HADOOP-11661 > Project: Hadoop Common > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-11661-002.patch, HADOOP-11661-003-branch-2.patch, > HADOOP-11661.patch > > > FileUtil#copyMerge is currently unused in the Hadoop source tree. In > branch-1, it had been part of the implementation of the hadoop fs -getmerge > shell command. In branch-2, the code for that shell command was rewritten in > a way that no longer requires this method. > Please check more details here.. > https://issues.apache.org/jira/browse/HADOOP-11392?focusedCommentId=14339336=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14339336 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12353) S3 Native filesystem does not retry all connection failures
[
https://issues.apache.org/jira/browse/HADOOP-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083984#comment-15083984
]
Hadoop QA commented on HADOOP-12353:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 1s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 37s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 29s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 12s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 36s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.ipc.TestIPC |
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12752422/HADOOP-12353.002.patch
|
|
[jira] [Commented] (HADOOP-12546) Improve TestKMS
[ https://issues.apache.org/jira/browse/HADOOP-12546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083990#comment-15083990 ] Xiao Chen commented on HADOOP-12546: I've seen some intermittent failures of TestKMS (testKMSProvider specifically), and this JIRA looks to be helpful on exposing more details of it. I reviewed through Patch 03, looks good to me, +1 (non-binding). Thanks [~templedf] and [~ste...@apache.org]. > Improve TestKMS > --- > > Key: HADOOP-12546 > URL: https://issues.apache.org/jira/browse/HADOOP-12546 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: HADOOP-12546.001.patch, HADOOP-12546.002.patch, > HADOOP-12546.003.patch > > > The TestKMS class has some issues: > * It swallows some exceptions' stack traces > * It swallows some exceptions altogether > * Some of the tests aren't as tight as they could be > * Asserts lack messages > * Code style is a bit hodgepodge > This JIRA is to clean all that up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12634) Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
[
https://issues.apache.org/jira/browse/HADOOP-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083993#comment-15083993
]
Chris Nauroth commented on HADOOP-12634:
[~gouravk], OK, let's keep the test you already wrote, with a few changes:
# Please change the name of the test to
{{testLazyRenamePendingCanOverwriteExistingFile}}. I think this will help
clarify that the intent is to test an internal implementation detail. The
current name makes it sounds like we expect any {{FileSystem#rename}} operation
to be able to overwrite the destination, which would not be true.
# Please ensure that any stream returned from a {{FileSystem#create}} call gets
closed. try-with-resources or the {{IOUtils#cleanup}} utility method is
helpful for this.
# The {{LOG}} statements are unnecessary. Please remove them.
> Change Lazy Rename Pending Operation Completion of WASB to address case of
> potential data loss due to partial copy
> --
>
> Key: HADOOP-12634
> URL: https://issues.apache.org/jira/browse/HADOOP-12634
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Gaurav Kanade
>Assignee: Gaurav Kanade
>Priority: Critical
> Attachments: HADOOP-12634.01.patch
>
>
> HADOOP-12334 changed mode of Copy Operation of HBase WAL Archiving to bypass
> Azure Storage Throttling after retries. This was via client side copy.
> However a process crash when the copy is partially done would result in a
> scenario where the source and destination blobs will have different contents
> and lazy rename pending operation will not handle this thus causing data
> loss. We need to fix the lazy rename pending operation to address this issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083945#comment-15083945 ] Zhe Zhang commented on HADOOP-12685: Thanks Rui for the good fix. The patch LGTM. +1 and I'll commit the patch soon if Kai doesn't have further comments. > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12634) Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
[ https://issues.apache.org/jira/browse/HADOOP-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083652#comment-15083652 ] Gaurav Kanade commented on HADOOP-12634: [~cnauroth] - Thanks for the review. With regards to testing there are two aspects to consider here given the change we made to the behavior of lazy rename pending operation. 1. Post the change the lazy rename pending will behave in the case of src and dest both existing the same way it will behave in the case of src only existing. The new test tries to somewhat cover this case - i.e. if we call a rename on a file already existing does rename happen correctly? Thus this part of the change effect is covered. 2. The only other thing left to test is does the changed code path get executed in lazy rename pending - this as you see and as described is hard to capture in a single test given the various process crash and client side copy it requires to simulate. However the code change here is fairly straightforward; i.e. a simple change in condition; the fact that this condition change behaves appropriately is tested above. I am not clear yet as to have a manner of testing 2 cleanly; let me know your thoughts. I will also fix the braces issue with the next version of the patch > Change Lazy Rename Pending Operation Completion of WASB to address case of > potential data loss due to partial copy > -- > > Key: HADOOP-12634 > URL: https://issues.apache.org/jira/browse/HADOOP-12634 > Project: Hadoop Common > Issue Type: Bug >Reporter: Gaurav Kanade >Assignee: Gaurav Kanade >Priority: Critical > Attachments: HADOOP-12634.01.patch > > > HADOOP-12334 changed mode of Copy Operation of HBase WAL Archiving to bypass > Azure Storage Throttling after retries. This was via client side copy. > However a process crash when the copy is partially done would result in a > scenario where the source and destination blobs will have different contents > and lazy rename pending operation will not handle this thus causing data > loss. We need to fix the lazy rename pending operation to address this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12620) Advanced Hadoop Architecture (AHA) - Common
[ https://issues.apache.org/jira/browse/HADOOP-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082887#comment-15082887 ] Steve Loughran commented on HADOOP-12620: - # My comments are going to stay as they provide a historical log of the discussion. the only changes suggested by [~aw] was to move the summary from the JIRA description into a comment, so as to keep the body of emails on JIRA updates down. Sorry. # I'd be surprised if making HDFS R/W is "minimal" # If I confused your proposal with adding MVCC, my apologies. I found the proposal too long and confusing to understand. > Advanced Hadoop Architecture (AHA) - Common > --- > > Key: HADOOP-12620 > URL: https://issues.apache.org/jira/browse/HADOOP-12620 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Dinesh S. Atreya >Assignee: Dinesh S. Atreya > > Advance Hadoop Architecture (AHA) / Advance Hadoop Adaptabilities (AHA): > See > https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046300=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046300 > for more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
[ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082910#comment-15082910 ] Steve Loughran commented on HADOOP-12689: - If there's a regression that hasn't been caught, we need tests to verify that the regression is not only fixed, that it stays fixed. w.r.t applying/submitting patches against the object stores, it is critical that the tests are manually run with the credentials: https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure [~raviprakash] : I'm afraid that applies to you too. Please don't commit this without the test runs and the results logged here. > S3 filesystem operations stopped working correctly > -- > > Key: HADOOP-12689 > URL: https://issues.apache.org/jira/browse/HADOOP-12689 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.0 >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Labels: S3 > Attachments: HADOOP-12689.01.patch > > > HADOOP-10542 was resolved by replacing "return null;" with throwing > IOException. This causes several S3 filesystem operations to fail (possibly > more code is expecting that null return value; these are just the calls I > noticed): > S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException > but instead IOException) > FileSystem.exists() (which no longer returns false but instead raises > IOException) > S3FileSystem.create() (which no longer succeeds but instead raises > IOException) > Run command: > hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/ > Resulting stack trace: > 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test > doesn't exist > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170) > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source) > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > changing the "raise IOE..." to "return null" fixes all of the above code > sites and allows distcp to succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12620) Advanced Hadoop Architecture (AHA) - Common
[ https://issues.apache.org/jira/browse/HADOOP-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083784#comment-15083784 ] Haohui Mai commented on HADOOP-12620: - I agree that the capabilities can be quite powerful. The real issue how it can be done. There are some questions need to be answered: (1) What is the semantic of update-in-place precisely when there are failures? Is it atomic and transactional? What does the consistent model look like? What are the semantics and durability guarantee look like? For example, what happens if one of the DN in the pipeline is down? What will the reader see? (2) Once you define the semantic, is the semantic / specification meaningful and complete? Does it cover all the failure cases? How to evaluate and prove there is no corner cases? (3) How to implement the semantic in code? What is the approach you are taking? Is it MVCC, distributed transaction or an ad-hoc solution tailored to HDFS? So far we all agree that it is a useful capability. I don't think it require more communications to establish it enables a number new use cases. However, I don't see this is a complete solution without addressing Steve's questions and all the questions above. It would be beneficial to have a design doc and a working prototype to clarify the confusions. > Advanced Hadoop Architecture (AHA) - Common > --- > > Key: HADOOP-12620 > URL: https://issues.apache.org/jira/browse/HADOOP-12620 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Dinesh S. Atreya >Assignee: Dinesh S. Atreya > > Advance Hadoop Architecture (AHA) / Advance Hadoop Adaptabilities (AHA): > See > https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046300=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046300 > for more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
[ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083793#comment-15083793 ] Ravi Prakash commented on HADOOP-12689: --- In light of the above, I am still on track to commit this patch EOD. Matt, if you'd like, please feel free to create a new JIRA for adding the contract tests for S3 (although I fear it may be going the way of the dinosaur). I just don't want that effort blocking this patch. [~steve_l] Please let me know if you still have objections. > S3 filesystem operations stopped working correctly > -- > > Key: HADOOP-12689 > URL: https://issues.apache.org/jira/browse/HADOOP-12689 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.0 >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Labels: S3 > Attachments: HADOOP-12689.01.patch > > > HADOOP-10542 was resolved by replacing "return null;" with throwing > IOException. This causes several S3 filesystem operations to fail (possibly > more code is expecting that null return value; these are just the calls I > noticed): > S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException > but instead IOException) > FileSystem.exists() (which no longer returns false but instead raises > IOException) > S3FileSystem.create() (which no longer succeeds but instead raises > IOException) > Run command: > hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/ > Resulting stack trace: > 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: > attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test > doesn't exist > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170) > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source) > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > changing the "raise IOE..." to "return null" fixes all of the above code > sites and allows distcp to succeed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11661) Deprecate FileUtil#copyMerge
[ https://issues.apache.org/jira/browse/HADOOP-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083887#comment-15083887 ] Haohui Mai commented on HADOOP-11661: - Please file the following jira. It works best if these two jiras are reviewed and committed at the same time. > Deprecate FileUtil#copyMerge > > > Key: HADOOP-11661 > URL: https://issues.apache.org/jira/browse/HADOOP-11661 > Project: Hadoop Common > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-11661-002.patch, HADOOP-11661-003-branch-2.patch, > HADOOP-11661.patch > > > FileUtil#copyMerge is currently unused in the Hadoop source tree. In > branch-1, it had been part of the implementation of the hadoop fs -getmerge > shell command. In branch-2, the code for that shell command was rewritten in > a way that no longer requires this method. > Please check more details here.. > https://issues.apache.org/jira/browse/HADOOP-11392?focusedCommentId=14339336=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14339336 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
[
https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083775#comment-15083775
]
Ravi Prakash commented on HADOOP-12689:
---
Here's my log of what I ran
{code}
[raviprak@ravi trunk]$ hdfs dfs -get s3:///
-get: Fatal internal error
java.lang.NullPointerException
at
org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveBlock(Jets3tFileSystemStore.java:248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:255)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at com.sun.proxy.$Proxy4.retrieveBlock(Unknown Source)
at
org.apache.hadoop.fs.s3.S3InputStream.blockSeekTo(S3InputStream.java:170)
at org.apache.hadoop.fs.s3.S3InputStream.read(S3InputStream.java:129)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:88)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:62)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:122)
at
org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:482)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:404)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:339)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:274)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:259)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:321)
at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:293)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:254)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:275)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:259)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:225)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
at org.apache.hadoop.fs.shell.Command.run(Command.java:166)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:319)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:377)
{code}
After the patch
{code}
[raviprak@ravi trunk]$ hdfs dfs -get s3:///
get: Block missing from S3 store: block_-3447106738088433847
{code}
When the file is valid, I was successfully able to get the file
> S3 filesystem operations stopped working correctly
> --
>
> Key: HADOOP-12689
> URL: https://issues.apache.org/jira/browse/HADOOP-12689
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.7.0
>Reporter: Matthew Paduano
>Assignee: Matthew Paduano
> Labels: S3
> Attachments: HADOOP-12689.01.patch
>
>
> HADOOP-10542 was resolved by replacing "return null;" with throwing
> IOException. This causes several S3 filesystem operations to fail (possibly
> more code is expecting that null return value; these are just the calls I
> noticed):
> S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException
> but instead IOException)
> FileSystem.exists() (which no longer returns false but instead raises
> IOException)
> S3FileSystem.create() (which no longer succeeds but instead raises
> IOException)
> Run command:
> hadoop distcp hdfs://localhost:9000/test s3://xxx:y...@com.bar.foo/
> Resulting stack trace:
> 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1449826461866_0005_m_06_0 - exited : java.io.IOException: /test
> doesn't exist
> at
> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170)
> at
> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
[jira] [Commented] (HADOOP-12690) Consolidate access of sun.misc.Unsafe
[
https://issues.apache.org/jira/browse/HADOOP-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084463#comment-15084463
]
Kai Zheng commented on HADOOP-12690:
The patch looks nice. Thanks [~djp].
A few minor comments:
1. Maybe we could cache the {{unsafe}} instance in the new utility, so callers
won't do the cache thing themselves?
2. Better to have some comment to explain about the new approach, because
without looking at the discussion in the JIRAs, some one may be not easy to
catch why it goes like that.
3. I guess you added {{TestFastByteComparision}} by the way?
> Consolidate access of sun.misc.Unsafe
> --
>
> Key: HADOOP-12690
> URL: https://issues.apache.org/jira/browse/HADOOP-12690
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: HADOOP-12690.patch
>
>
> Per discussion in Hadoop-12630
> (https://issues.apache.org/jira/browse/HADOOP-12630?focusedCommentId=15082142=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15082142),
> we found the access of sun.misc.Unsafe could be problematic for some JVMs in
> other platforms. Also, hints from other comments, it is better to consolidate
> it as a helper/utility method to shared with several places
> (FastByteComparisons, NativeIO, ShortCircuitShm).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12634) Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
[
https://issues.apache.org/jira/browse/HADOOP-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084474#comment-15084474
]
Hadoop QA commented on HADOOP-12634:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 44s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12780647/HADOOP-12634.02.patch
|
| JIRA Issue | HADOOP-12634 |
| Optional Tests | asflicense |
| uname | Linux da0855d711bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c52b407 |
| modules | C: U: |
| Max memory used | 31MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/8344/console |
This message was automatically generated.
> Change Lazy Rename Pending Operation Completion of WASB to address case of
> potential data loss due to partial copy
> --
>
> Key: HADOOP-12634
> URL: https://issues.apache.org/jira/browse/HADOOP-12634
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Gaurav Kanade
>Assignee: Gaurav Kanade
>Priority: Critical
> Attachments: HADOOP-12634.01.patch, HADOOP-12634.02.patch
>
>
> HADOOP-12334 changed mode of Copy Operation of HBase WAL Archiving to bypass
> Azure Storage Throttling after retries. This was via client side copy.
> However a process crash when the copy is partially done would result in a
> scenario where the source and destination blobs will have different contents
> and lazy rename pending operation will not handle this thus causing data
> loss. We need to fix the lazy rename pending operation to address this issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12634) Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
[ https://issues.apache.org/jira/browse/HADOOP-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084494#comment-15084494 ] Gaurav Kanade commented on HADOOP-12634: [~cnauroth] Thanks. Please take a look at the latest patch. > Change Lazy Rename Pending Operation Completion of WASB to address case of > potential data loss due to partial copy > -- > > Key: HADOOP-12634 > URL: https://issues.apache.org/jira/browse/HADOOP-12634 > Project: Hadoop Common > Issue Type: Bug >Reporter: Gaurav Kanade >Assignee: Gaurav Kanade >Priority: Critical > Attachments: HADOOP-12634.01.patch, HADOOP-12634.02.patch > > > HADOOP-12334 changed mode of Copy Operation of HBase WAL Archiving to bypass > Azure Storage Throttling after retries. This was via client side copy. > However a process crash when the copy is partially done would result in a > scenario where the source and destination blobs will have different contents > and lazy rename pending operation will not handle this thus causing data > loss. We need to fix the lazy rename pending operation to address this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12634) Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
[
https://issues.apache.org/jira/browse/HADOOP-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084498#comment-15084498
]
Hadoop QA commented on HADOOP-12634:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 57s
{color} | {color:red} hadoop-tools_hadoop-azure-jdk1.8.0_66 with JDK v1.8.0_66
generated 12 new issues (was 26, now 26). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 24s
{color} | {color:red} hadoop-tools_hadoop-azure-jdk1.7.0_91 with JDK v1.7.0_91
generated 1 new issues (was 1, now 1). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-azure in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hadoop-azure in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12780647/HADOOP-12634.02.patch
|
| JIRA Issue | HADOOP-12634 |
| Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084503#comment-15084503
]
Kai Zheng commented on HADOOP-12041:
Hi [~walter.k.su],
I'm going to update this. Would you help with this question? In GF256.init(),
it's mainly to initialize {{theGfMulTab}}. What did you mean by "still possible
GF256 be inited twice"? Thanks.
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12678) Handle empty rename pending metadata file during atomic rename in redo path
[
https://issues.apache.org/jira/browse/HADOOP-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084107#comment-15084107
]
Chris Nauroth commented on HADOOP-12678:
[~madhuch-ms], thank you for the patch. I have a few notes.
# In {{NativeAzureFileSystem.FolderRenamePending#deleteRenamePendingFile}}, the
exception handling logic is quite complex. There is an unchecked downcast to
{{StorageException}}, which could cause a {{CastClassException}}. There is no
check that the cause is non-null. I don't see why there is a nested {{catch
(Exception e2)}}, because I don't see a possibility of any exception being
thrown in that block, unless this was put in place to mask the
{{CastClassException}}. There is no need to wrap the original {{IOException}}
in a whole new {{IOException}} before throwing it. This will only make the
stack trace longer without adding new information. The code was also
incorrectly indented, which made it difficult to read. I suggest simplifying
the {{catch (IOException e)}} block to this:
{code}
Throwable cause = e.getCause();
if (cause != null && cause instanceof StorageException &&
"BlobNotFound".equals(((StorageException)cause).getErrorCode())) {
LOG.warn("rename pending file " + redoFile + " is already deleted");
} else {
throw e;
}
{code}
# Is {{NativeAzureFileSystem.FolderRenamePending#deleteRenamePendingFile}}
marked {{public}} only so that the tests can call it? If so, then please make
it package-private (remove {{public}}) and apply the {{VisibleForTesting}}
annotation.
# Please ensure any lines that you are changing or adding are shorter than 80
characters.
> Handle empty rename pending metadata file during atomic rename in redo path
> ---
>
> Key: HADOOP-12678
> URL: https://issues.apache.org/jira/browse/HADOOP-12678
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: madhumita chakraborty
>Assignee: madhumita chakraborty
>Priority: Critical
> Attachments: HADOOP-12678.001.patch, HADOOP-12678.002.patch,
> HADOOP-12678.003.patch
>
>
> Handle empty rename pending metadata file during atomic rename in redo path
> During atomic rename we create metadata file for rename(-renamePending.json).
> We create that in 2 steps
> 1. We create an empty blob corresponding to the .json file in its real
> location
> 2. We create a scratch file to which we write the contents of the rename
> pending which is then copied over into the blob described in 1
> If process crash occurs after step 1 and before step 2 is complete - we will
> be left with a zero size blob corresponding to the pending rename metadata
> file.
> This kind of scenario can happen in the /hbase/.tmp folder because it is
> considered a candidate folder for atomic rename. Now when HMaster starts up
> it executes listStatus on the .tmp folder to clean up pending data. At this
> stage due to the lazy pending rename complete process we look for these json
> files. On seeing an empty file the process simply throws a fatal exception
> assuming something went wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12685: --- Affects Version/s: 3.0.0 > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12685: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) I just committed the patch to trunk. Thanks Rui for the work and Kai for reviewing! > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12685: --- Target Version/s: 3.0.0 > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12634) Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy
[ https://issues.apache.org/jira/browse/HADOOP-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Kanade updated HADOOP-12634: --- Attachment: HADOOP-12634.02.patch > Change Lazy Rename Pending Operation Completion of WASB to address case of > potential data loss due to partial copy > -- > > Key: HADOOP-12634 > URL: https://issues.apache.org/jira/browse/HADOOP-12634 > Project: Hadoop Common > Issue Type: Bug >Reporter: Gaurav Kanade >Assignee: Gaurav Kanade >Priority: Critical > Attachments: HADOOP-12634.01.patch, HADOOP-12634.02.patch > > > HADOOP-12334 changed mode of Copy Operation of HBase WAL Archiving to bypass > Azure Storage Throttling after retries. This was via client side copy. > However a process crash when the copy is partially done would result in a > scenario where the source and destination blobs will have different contents > and lazy rename pending operation will not handle this thus causing data > loss. We need to fix the lazy rename pending operation to address this issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084462#comment-15084462 ] Hudson commented on HADOOP-12685: - FAILURE: Integrated in Hadoop-trunk-Commit #9054 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9054/]) HADOOP-12685. Input buffer position after encode/decode not consistent (zhz: rev c52b407cbffc8693738b31c6cc4e71751efd70e8) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRawCoderBase.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureDecoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureDecoder.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/TestRSRawCoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RawErasureEncoder.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractRawErasureEncoder.java * hadoop-common-project/hadoop-common/CHANGES.txt > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12587) Hadoop AuthToken refuses to work without a maxinactive attribute in issued token
[ https://issues.apache.org/jira/browse/HADOOP-12587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084019#comment-15084019 ] Benoy Antony commented on HADOOP-12587: --- [~ste...@apache.org], Do you have some time to review the patch ? > Hadoop AuthToken refuses to work without a maxinactive attribute in issued > token > > > Key: HADOOP-12587 > URL: https://issues.apache.org/jira/browse/HADOOP-12587 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.8.0 > Environment: OSX heimdal kerberos client against Linux KDC -talking > to a Hadoop 2.6.0 cluster >Reporter: Steve Loughran >Assignee: Benoy Antony >Priority: Blocker > Attachments: HADOOP-12587-001.patch > > > If you don't have a max-inactive attribute in the auth token returned from > the web site, AuthToken will raise an exception. This stops callers without > this token being able to submit jobs to a secure Hadoop 2.6 YARN cluster with > timeline server enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084053#comment-15084053 ] Kai Zheng commented on HADOOP-12685: Zhe I don't have further comments. Thanks. > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12685) Input buffer position after encode/decode not consistent between different kinds of buffers
[ https://issues.apache.org/jira/browse/HADOOP-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084534#comment-15084534 ] Rui Li commented on HADOOP-12685: - Thanks guys for the review! > Input buffer position after encode/decode not consistent between different > kinds of buffers > --- > > Key: HADOOP-12685 > URL: https://issues.apache.org/jira/browse/HADOOP-12685 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HADOOP-12685.1.patch, HADOOP-12685.2.patch > > > As reported by [~jack_liuquan] in this > [comment|https://issues.apache.org/jira/browse/HADOOP-11828?focusedCommentId=15071796=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15071796], > the input buffer position change behavior is not consistent between direct > and non-direct buffers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12041:
---
Attachment: HADOOP-12041-v5.patch
Updated the patch:
* Added comments in many places.
* Rebased with the latest trunk.
* Removed unused codes here and will add them in other issues where needed.
[~walter.k.su], [~zhz] or someone else, would you help review one more time?
The new coder is extensively tested and works fine. The interoperable tests
will be added in the issue that implements ISA-L coder. The codes are mainly
new and won't break existing coders. Thanks.
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch, HADOOP-12041-v5.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
