[jira] [Comment Edited] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
[ https://issues.apache.org/jira/browse/HADOOP-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505426#comment-15505426 ] Genmao Yu edited comment on HADOOP-13624 at 9/20/16 5:55 AM: - [~drankye] updates: 1. typo fix: "TestAliyunOSSContractDispCp" => "TestAliyunOSSContractDistCp" 2. clean up start import except "import static org.apache.hadoop.fs.aliyun.oss.Constants.*", the same style with others. Is this OK ? was (Author: unclegen): [~drankye] updates: 1. typo fix: "TestAliyunOSSContractDispCp" => "TestAliyunOSSContractDistCp" 2. clean up start import except "import static org.apache.hadoop.fs.aliyun.oss.Constants.*". Is this OK ? > Rename TestAliyunOSSContractDispCp > -- > > Key: HADOOP-13624 > URL: https://issues.apache.org/jira/browse/HADOOP-13624 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Reporter: Kai Zheng >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13624-HADOOP-12756.001.patch > > > It should be TestAliyunOSSContractDistCp.java instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
[
https://issues.apache.org/jira/browse/HADOOP-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505488#comment-15505488
]
Hadoop QA commented on HADOOP-13624:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
49s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
51s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13624 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829319/HADOOP-13624-HADOOP-12756.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1a2d85231219 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-12756 / 08b3760 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10547/testReport/ |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10547/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Rename TestAliyunOSSContractDispCp
> --
>
> Key: HADOOP-13624
> URL: https://issues.apache.org/jira/browse/HADOOP-13624
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Kai Zheng
>Assignee: Genmao Yu
> Fix For: HADOOP-12756
>
> Attachments: HADOOP-13624-HADOOP-12756.001.patch
>
>
> It should be TestAliyunOSSContractDistCp.java instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
[ https://issues.apache.org/jira/browse/HADOOP-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13624: --- Status: Patch Available (was: In Progress) > Rename TestAliyunOSSContractDispCp > -- > > Key: HADOOP-13624 > URL: https://issues.apache.org/jira/browse/HADOOP-13624 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Reporter: Kai Zheng >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13624-HADOOP-12756.001.patch > > > It should be TestAliyunOSSContractDistCp.java instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
[ https://issues.apache.org/jira/browse/HADOOP-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13624: --- Attachment: HADOOP-13624-HADOOP-12756.001.patch [~drankye] updates: 1. typo fix: "TestAliyunOSSContractDispCp" => "TestAliyunOSSContractDistCp" 2. clean up start import except "import static org.apache.hadoop.fs.aliyun.oss.Constants.*". Is this OK ? > Rename TestAliyunOSSContractDispCp > -- > > Key: HADOOP-13624 > URL: https://issues.apache.org/jira/browse/HADOOP-13624 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Reporter: Kai Zheng >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13624-HADOOP-12756.001.patch > > > It should be TestAliyunOSSContractDistCp.java instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13573) S3Guard: create basic contract tests for MetadataStore implementations
[
https://issues.apache.org/jira/browse/HADOOP-13573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505228#comment-15505228
]
Hadoop QA commented on HADOOP-13573:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13573 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829306/HADOOP-13573-HADOOP-13345.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d5ba93213779 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 18f1f68 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10546/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10546/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: create basic contract tests for MetadataStore implementations
> --
>
> Key: HADOOP-13573
> URL: https://issues.apache.org/jira/browse/HADOOP-13573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13573-HADOOP-13345.002.patch,
> HADOOP-13573-HADOOP-13345.003.patch, HADOOP-13573.001.patch
>
>
> We should have some
[jira] [Commented] (HADOOP-13452) S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505221#comment-15505221
]
Aaron Fabbri commented on HADOOP-13452:
---
[~cnauroth] [~liuml07] I have a v2 patch that has roughly the same
LocalMetadataStore code, but includes the changes we discussed:
{{MetadataStore#initialize()}} takes a FileSystem, along with precondition
checks that paths are absolute.
I'm thinking of submitting this patch and creating a new sub-JIRA for adding
move() implementation and tests. My preference is for smaller commits and code
reviews, and move() could use its own.
Would you rather get this v2 patch in, or wait until move() implementation is
included? (I'm fine either way)
> S3Guard: Implement access policy for intra-client consistency with in-memory
> metadata store.
>
>
> Key: HADOOP-13452
> URL: https://issues.apache.org/jira/browse/HADOOP-13452
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13452.001.patch
>
>
> Implement an S3A access policy based on an in-memory metadata store. This
> can provide consistency within the same client without needing to integrate
> with an external system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13573) S3Guard: create basic contract tests for MetadataStore implementations
[ https://issues.apache.org/jira/browse/HADOOP-13573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13573: -- Attachment: HADOOP-13573-HADOOP-13345.003.patch v3 patch is same as v2, except I fixed the order of args to assertEquals() as requested. [~cnauroth] I think we should leave the TODOs for now. Better to get some tests integrated so folks can develop against them, IMO. TODOs are are either things to be addressed by coming changes to MetadataStore interface, pending move() implementation (which I'd like to be separate patch for review-ability), or suggestions on API changes we should discuss in followup JIRAs. > S3Guard: create basic contract tests for MetadataStore implementations > -- > > Key: HADOOP-13573 > URL: https://issues.apache.org/jira/browse/HADOOP-13573 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > Attachments: HADOOP-13573-HADOOP-13345.002.patch, > HADOOP-13573-HADOOP-13345.003.patch, HADOOP-13573.001.patch > > > We should have some contract-style unit tests for the MetadataStore interface > to validate that the different implementations provide correct semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13604) Abort retry loop when RPC has an unrecoverable Auth error
[ https://issues.apache.org/jira/browse/HADOOP-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505134#comment-15505134 ] Xiao Chen commented on HADOOP-13604: Thanks [~ste...@apache.org] for the pointer! I linked another TWMNBN related issue, HADOOP-13590, to HADOOP-12649 as well. :) I agree it would be difficult and non-optimal to filter Auth errors from current IOEs. Is there a specific jira for fixing this? > Abort retry loop when RPC has an unrecoverable Auth error > - > > Key: HADOOP-13604 > URL: https://issues.apache.org/jira/browse/HADOOP-13604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Henry Robinson >Assignee: Xiao Chen > > I've seen an issue where, after an RPC client hit an error obtaining a TGT > from Kerberos, the RPC client continues to retry even though there's no > chance of success (the no login window is set to 600s). > In this particular deployment, the client retries 15 times at 15s intervals, > leading to a delay of more than three minutes before the failure is bubbled > up to the client when the RPC ultimately fails. > Unrecoverable errors (like failures to login to Kerberos) should lead to fast > aborts of the retry loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12649) Improve Kerberos diagnostics and failure handling
[ https://issues.apache.org/jira/browse/HADOOP-12649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505123#comment-15505123 ] Xiao Chen commented on HADOOP-12649: Linking HADOOP-13590 here which does #4 of the description. Reviews appreciated. > Improve Kerberos diagnostics and failure handling > - > > Key: HADOOP-12649 > URL: https://issues.apache.org/jira/browse/HADOOP-12649 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.7.1 > Environment: Kerberos >Reporter: Steve Loughran > > Sometimes —apparently— some people cannot get kerberos to work. > The ability to diagnose problems here is hampered by some aspects of UGI > # the only way to turn on JAAS debug information is through an env var, not > within the JVM > # failures are potentially underlogged > # exceptions raised are generic IOEs, so can't be trapped and filtered > # failure handling on the TGT renewer thread is nonexistent > # the code is barely-readable, underdocumented mess. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13169) Randomize file list in SimpleCopyListing
[ https://issues.apache.org/jira/browse/HADOOP-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505078#comment-15505078 ] Hudson commented on HADOOP-13169: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10462 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10462/]) HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by (cnauroth: rev 98bdb5139769eb55893971b43b9c23da9513a784) * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java * (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java > Randomize file list in SimpleCopyListing > > > Key: HADOOP-13169 > URL: https://issues.apache.org/jira/browse/HADOOP-13169 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-13169-branch-2-001.patch, > HADOOP-13169-branch-2-002.patch, HADOOP-13169-branch-2-003.patch, > HADOOP-13169-branch-2-004.patch, HADOOP-13169-branch-2-005.patch, > HADOOP-13169-branch-2-006.patch, HADOOP-13169-branch-2-007.patch, > HADOOP-13169-branch-2-008.patch, HADOOP-13169-branch-2-009.patch, > HADOOP-13169-branch-2-010.patch > > > When copying files to S3, based on file listing some mappers can get into S3 > partition hotspots. This would be more visible, when data is copied from hive > warehouse with lots of partitions (e.g date partitions). In such cases, some > of the tasks would tend to be a lot more slower than others. It would be good > to randomize the file paths which are written out in SimpleCopyListing to > avoid this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13169) Randomize file list in SimpleCopyListing
[
https://issues.apache.org/jira/browse/HADOOP-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504883#comment-15504883
]
Chris Nauroth edited comment on HADOOP-13169 at 9/19/16 10:29 PM:
--
+1 for patch 010. I verified that {{TestAzureNativeContractDistCp}} and
{{ITestS3AContractDistCp}} are passing. Rajesh, thank you for the patch.
Steve, thank you for the code review.
I have committed this to trunk, branch-2 and branch-2.8.
was (Author: cnauroth):
+1 for patch 010. I verified that {{TestAzureNativeContractDistCp}} and
{{ITestS3AContractDistCp}} are passing. Rajesh, thank you for the patch.
Steve, thank you for the code review.
> Randomize file list in SimpleCopyListing
>
>
> Key: HADOOP-13169
> URL: https://issues.apache.org/jira/browse/HADOOP-13169
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13169-branch-2-001.patch,
> HADOOP-13169-branch-2-002.patch, HADOOP-13169-branch-2-003.patch,
> HADOOP-13169-branch-2-004.patch, HADOOP-13169-branch-2-005.patch,
> HADOOP-13169-branch-2-006.patch, HADOOP-13169-branch-2-007.patch,
> HADOOP-13169-branch-2-008.patch, HADOOP-13169-branch-2-009.patch,
> HADOOP-13169-branch-2-010.patch
>
>
> When copying files to S3, based on file listing some mappers can get into S3
> partition hotspots. This would be more visible, when data is copied from hive
> warehouse with lots of partitions (e.g date partitions). In such cases, some
> of the tasks would tend to be a lot more slower than others. It would be good
> to randomize the file paths which are written out in SimpleCopyListing to
> avoid this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13169) Randomize file list in SimpleCopyListing
[
https://issues.apache.org/jira/browse/HADOOP-13169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13169:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
+1 for patch 010. I verified that {{TestAzureNativeContractDistCp}} and
{{ITestS3AContractDistCp}} are passing. Rajesh, thank you for the patch.
Steve, thank you for the code review.
> Randomize file list in SimpleCopyListing
>
>
> Key: HADOOP-13169
> URL: https://issues.apache.org/jira/browse/HADOOP-13169
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HADOOP-13169-branch-2-001.patch,
> HADOOP-13169-branch-2-002.patch, HADOOP-13169-branch-2-003.patch,
> HADOOP-13169-branch-2-004.patch, HADOOP-13169-branch-2-005.patch,
> HADOOP-13169-branch-2-006.patch, HADOOP-13169-branch-2-007.patch,
> HADOOP-13169-branch-2-008.patch, HADOOP-13169-branch-2-009.patch,
> HADOOP-13169-branch-2-010.patch
>
>
> When copying files to S3, based on file listing some mappers can get into S3
> partition hotspots. This would be more visible, when data is copied from hive
> warehouse with lots of partitions (e.g date partitions). In such cases, some
> of the tasks would tend to be a lot more slower than others. It would be good
> to randomize the file paths which are written out in SimpleCopyListing to
> avoid this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
[
https://issues.apache.org/jira/browse/HADOOP-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504877#comment-15504877
]
Hadoop QA commented on HADOOP-13626:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hadoop-tools/hadoop-distcp: The patch generated 0
new + 36 unchanged - 3 fixed = 36 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} hadoop-tools_hadoop-distcp generated 0 new + 49
unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
15s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13626 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829275/HADOOP-13626.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 74dce1e90c5d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7558dbb |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10545/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10545/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Remove distcp dependency on FileStatus serialization
>
>
> Key: HADOOP-13626
> URL: https://issues.apache.org/jira/browse/HADOOP-13626
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Reporter: Chris Douglas
>Assignee: Chris Douglas
> Attachments: HADOOP-13626.001.patch, HADOOP-13626.002.patch,
>
[jira] [Updated] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
[
https://issues.apache.org/jira/browse/HADOOP-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-13626:
---
Attachment: HADOOP-13626.003.patch
> Remove distcp dependency on FileStatus serialization
>
>
> Key: HADOOP-13626
> URL: https://issues.apache.org/jira/browse/HADOOP-13626
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Reporter: Chris Douglas
>Assignee: Chris Douglas
> Attachments: HADOOP-13626.001.patch, HADOOP-13626.002.patch,
> HADOOP-13626.003.patch
>
>
> DistCp uses an internal struct {{CopyListingFileStatus}} to record metadata.
> Because this record extends {{FileStatus}}, it also relies on the
> {{Writable}} contract from that type. Because DistCp performs its checks on a
> subset of the fields (i.e., does not actually rely on {{FileStatus}} as a
> supertype), these types should be independent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
[
https://issues.apache.org/jira/browse/HADOOP-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504682#comment-15504682
]
Hadoop QA commented on HADOOP-13626:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hadoop-tools/hadoop-distcp: The patch generated 0
new + 36 unchanged - 3 fixed = 36 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} hadoop-tools_hadoop-distcp generated 0 new + 49
unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
32s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13626 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829267/HADOOP-13626.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4e0c5c0e087e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7558dbb |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10544/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10544/testReport/ |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10544/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Remove distcp dependency on FileStatus serialization
>
>
> Key: HADOOP-13626
> URL: https://issues.apache.org/jira/browse/HADOOP-13626
> Project: Hadoop Common
> Issue Type: Bug
>
[jira] [Commented] (HADOOP-13623) regression: ITestS3NContractRootDir
[ https://issues.apache.org/jira/browse/HADOOP-13623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504666#comment-15504666 ] Chris Nauroth commented on HADOOP-13623: I tried to repro this, but the test is passing fine for me on current trunk, branch-2 and branch-2.8. My test runs use a bucket in US-west-2. Steve, is there anything else unique about your test setup, aside from our difference in AWS region? > regression: ITestS3NContractRootDir > --- > > Key: HADOOP-13623 > URL: https://issues.apache.org/jira/browse/HADOOP-13623 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > I'm going to point the blame at HADOOP-13207; the new test is failing on S3n. > Somehow this wasn't showing up in tests until recently -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
[
https://issues.apache.org/jira/browse/HADOOP-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-13626:
---
Attachment: HADOOP-13626.002.patch
Missed some checkstyle complaints.
> Remove distcp dependency on FileStatus serialization
>
>
> Key: HADOOP-13626
> URL: https://issues.apache.org/jira/browse/HADOOP-13626
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Reporter: Chris Douglas
> Attachments: HADOOP-13626.001.patch, HADOOP-13626.002.patch
>
>
> DistCp uses an internal struct {{CopyListingFileStatus}} to record metadata.
> Because this record extends {{FileStatus}}, it also relies on the
> {{Writable}} contract from that type. Because DistCp performs its checks on a
> subset of the fields (i.e., does not actually rely on {{FileStatus}} as a
> supertype), these types should be independent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
[
https://issues.apache.org/jira/browse/HADOOP-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-13626:
---
Assignee: Chris Douglas
Status: Patch Available (was: Open)
> Remove distcp dependency on FileStatus serialization
>
>
> Key: HADOOP-13626
> URL: https://issues.apache.org/jira/browse/HADOOP-13626
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Reporter: Chris Douglas
>Assignee: Chris Douglas
> Attachments: HADOOP-13626.001.patch, HADOOP-13626.002.patch
>
>
> DistCp uses an internal struct {{CopyListingFileStatus}} to record metadata.
> Because this record extends {{FileStatus}}, it also relies on the
> {{Writable}} contract from that type. Because DistCp performs its checks on a
> subset of the fields (i.e., does not actually rely on {{FileStatus}} as a
> supertype), these types should be independent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
[
https://issues.apache.org/jira/browse/HADOOP-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-13626:
---
Attachment: HADOOP-13626.001.patch
> Remove distcp dependency on FileStatus serialization
>
>
> Key: HADOOP-13626
> URL: https://issues.apache.org/jira/browse/HADOOP-13626
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Reporter: Chris Douglas
> Attachments: HADOOP-13626.001.patch
>
>
> DistCp uses an internal struct {{CopyListingFileStatus}} to record metadata.
> Because this record extends {{FileStatus}}, it also relies on the
> {{Writable}} contract from that type. Because DistCp performs its checks on a
> subset of the fields (i.e., does not actually rely on {{FileStatus}} as a
> supertype), these types should be independent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13558) UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket
[
https://issues.apache.org/jira/browse/HADOOP-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504540#comment-15504540
]
Xiao Chen commented on HADOOP-13558:
FYI - I have backported this to branch-2.8 and branch-2.7 as suggested.
Compiled and ran the test locally before pushing. Thanks for the reminder, Zhe.
> UserGroupInformation created from a Subject incorrectly tries to renew the
> Kerberos ticket
> --
>
> Key: HADOOP-13558
> URL: https://issues.apache.org/jira/browse/HADOOP-13558
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.2, 2.6.4, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HADOOP-13558.01.patch, HADOOP-13558.02.patch,
> HADOOP-13558.branch-2.7.patch
>
>
> The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions
> and if they are met it invokes the {{reloginFromKeytab()}}. The
> {{reloginFromKeytab()}} method then fails with an {{IOException}}
> "loginUserFromKeyTab must be done first" because there is no keytab
> associated with the UGI.
> The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab
> ({{isKeytab}} UGI instance variable) associated with the UGI, if there is one
> it triggers a call to {{reloginFromKeytab()}}. The problem is that the
> {{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned
> {{IOException}}.
> The root of the problem seems to be when creating a UGI via the
> {{UGI.loginUserFromSubject(Subject)}} method, this method uses the
> {{UserGroupInformation(Subject)}} constructor, and this constructor does the
> following to determine if there is a keytab or not.
> {code}
> this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
> {code}
> If the {{Subject}} given had a keytab, then the UGI instance will have the
> {{isKeytab}} set to TRUE.
> It sets the UGI instance as it would have a keytab because the Subject has a
> keytab. This has 2 problems:
> First, it does not set the keytab file (and this, having the {{isKeytab}} set
> to TRUE and the {{keytabFile}} set to NULL) is what triggers the
> {{IOException}} in the method {{reloginFromKeytab()}}.
> Second (and even if the first problem is fixed, this still is a problem), it
> assumes that because the subject has a keytab it is up to UGI to do the
> relogin using the keytab. This is incorrect if the UGI was created using the
> {{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the
> Subject is not the UGI, but the caller, so the caller is responsible for
> renewing the Kerberos tickets and the UGI should not try to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13558) UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket
[
https://issues.apache.org/jira/browse/HADOOP-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13558:
---
Fix Version/s: 2.7.4
2.8.0
> UserGroupInformation created from a Subject incorrectly tries to renew the
> Kerberos ticket
> --
>
> Key: HADOOP-13558
> URL: https://issues.apache.org/jira/browse/HADOOP-13558
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.7.2, 2.6.4, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HADOOP-13558.01.patch, HADOOP-13558.02.patch,
> HADOOP-13558.branch-2.7.patch
>
>
> The UGI {{checkTGTAndReloginFromKeytab()}} method checks certain conditions
> and if they are met it invokes the {{reloginFromKeytab()}}. The
> {{reloginFromKeytab()}} method then fails with an {{IOException}}
> "loginUserFromKeyTab must be done first" because there is no keytab
> associated with the UGI.
> The {{checkTGTAndReloginFromKeytab()}} method checks if there is a keytab
> ({{isKeytab}} UGI instance variable) associated with the UGI, if there is one
> it triggers a call to {{reloginFromKeytab()}}. The problem is that the
> {{keytabFile}} UGI instance variable is NULL, and that triggers the mentioned
> {{IOException}}.
> The root of the problem seems to be when creating a UGI via the
> {{UGI.loginUserFromSubject(Subject)}} method, this method uses the
> {{UserGroupInformation(Subject)}} constructor, and this constructor does the
> following to determine if there is a keytab or not.
> {code}
> this.isKeytab = KerberosUtil.hasKerberosKeyTab(subject);
> {code}
> If the {{Subject}} given had a keytab, then the UGI instance will have the
> {{isKeytab}} set to TRUE.
> It sets the UGI instance as it would have a keytab because the Subject has a
> keytab. This has 2 problems:
> First, it does not set the keytab file (and this, having the {{isKeytab}} set
> to TRUE and the {{keytabFile}} set to NULL) is what triggers the
> {{IOException}} in the method {{reloginFromKeytab()}}.
> Second (and even if the first problem is fixed, this still is a problem), it
> assumes that because the subject has a keytab it is up to UGI to do the
> relogin using the keytab. This is incorrect if the UGI was created using the
> {{UGI.loginUserFromSubject(Subject)}} method. In such case, the owner of the
> Subject is not the UGI, but the caller, so the caller is responsible for
> renewing the Kerberos tickets and the UGI should not try to do so.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13548) Remove recursive dependencies of credential providers in LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13548: - Attachment: HADOOP-13548.002.patch v02: Fixed three bugs: (1) A TestLdapGroupsMapping test case used jceks://. Changed it to localjceks. (2) The ProviderUtils.excludeIncompatibleCredentialProviders had a bug identifying correct class corresponds to a scheme. (3) LdapGroupsMapping should also exclude user:// scheme. > Remove recursive dependencies of credential providers in LdapGroupsMapping > -- > > Key: HADOOP-13548 > URL: https://issues.apache.org/jira/browse/HADOOP-13548 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HADOOP-13548.001.patch, HADOOP-13548.002.patch > > > HADOOP-11934 discovered an infinite loop of dependencies in the use of > credential provider in LdapGroupsMapping. It added a new localjceks:// URI to > workaround the problem. The assumption is that the groups mapping is used > only in NameNode and that using a local credential file is not a problem. > However, there are cases where Hadoop clients, such as Sqoop, may use hdfs:// > based credential provider and use LdapGroupsMapping at the same time. We > should use HADOOP-12846 to exclude hdfs:// URI credential providers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13625) Document FileSystem actions that trigger update of modification time.
[ https://issues.apache.org/jira/browse/HADOOP-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504353#comment-15504353 ] Steve Loughran commented on HADOOP-13625: - This would be a good little part of the FS spec, something completely ignored by the docs as it is. Presumably rename(self, self) doesn' t update the modtime of the parent? > Document FileSystem actions that trigger update of modification time. > - > > Key: HADOOP-13625 > URL: https://issues.apache.org/jira/browse/HADOOP-13625 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Chris Nauroth > > Hadoop users and developers of Hadoop-compatible file systems have sometimes > asked questions about which file system actions trigger an update of the > path's modification time. This issue proposes to document which actions do > and do not update modification time, so that the information is easy to find > without reading HDFS code or manually testing individual cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13626) Remove distcp dependency on FileStatus serialization
Chris Douglas created HADOOP-13626:
--
Summary: Remove distcp dependency on FileStatus serialization
Key: HADOOP-13626
URL: https://issues.apache.org/jira/browse/HADOOP-13626
Project: Hadoop Common
Issue Type: Bug
Components: tools/distcp
Reporter: Chris Douglas
DistCp uses an internal struct {{CopyListingFileStatus}} to record metadata.
Because this record extends {{FileStatus}}, it also relies on the {{Writable}}
contract from that type. Because DistCp performs its checks on a subset of the
fields (i.e., does not actually rely on {{FileStatus}} as a supertype), these
types should be independent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13625) Document FileSystem actions that trigger update of modification time.
[
https://issues.apache.org/jira/browse/HADOOP-13625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504234#comment-15504234
]
Chris Nauroth commented on HADOOP-13625:
This information potentially could go into the file system spec or the end user
documentation (or both).
Here are the things that set mtime in HDFS. Some of these are clearly not
relevant to object stores (e.g. only HDFS implements snapshots and the symlink
feature is dormant/experimental/unused).
* {{FileSystem#concat}}
* {{FileSystem#create}}
* {{FileSystem#createSnapshot}}
* {{FileSystem#createSymlink}}
* {{FileSystem#rename}} updates mtime for the parent directories (but not the
renamed file/directory itself)
* {{FileSystem#setTimes}}
* {{FileSystem#truncate}}
* {{FSDataOutputStream#close}}
Here are some notable things that do not update mtime in HDFS.
* {{FileSystem#append}} (though the eventual FSDataOutputStream#close would do
it)
* {{FSDataOutputStream#hflush}} and {{hsync}}
* crossing a block boundary during {{FSDataOutputStream#write}}
> Document FileSystem actions that trigger update of modification time.
> -
>
> Key: HADOOP-13625
> URL: https://issues.apache.org/jira/browse/HADOOP-13625
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>
> Hadoop users and developers of Hadoop-compatible file systems have sometimes
> asked questions about which file system actions trigger an update of the
> path's modification time. This issue proposes to document which actions do
> and do not update modification time, so that the information is easy to find
> without reading HDFS code or manually testing individual cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13625) Document FileSystem actions that trigger update of modification time.
Chris Nauroth created HADOOP-13625: -- Summary: Document FileSystem actions that trigger update of modification time. Key: HADOOP-13625 URL: https://issues.apache.org/jira/browse/HADOOP-13625 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Chris Nauroth Hadoop users and developers of Hadoop-compatible file systems have sometimes asked questions about which file system actions trigger an update of the path's modification time. This issue proposes to document which actions do and do not update modification time, so that the information is easy to find without reading HDFS code or manually testing individual cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13548) Remove recursive dependencies of credential providers in LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13548: - Status: Open (was: Patch Available) Found a bug in the code, so cancel the patch. > Remove recursive dependencies of credential providers in LdapGroupsMapping > -- > > Key: HADOOP-13548 > URL: https://issues.apache.org/jira/browse/HADOOP-13548 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HADOOP-13548.001.patch > > > HADOOP-11934 discovered an infinite loop of dependencies in the use of > credential provider in LdapGroupsMapping. It added a new localjceks:// URI to > workaround the problem. The assumption is that the groups mapping is used > only in NameNode and that using a local credential file is not a problem. > However, there are cases where Hadoop clients, such as Sqoop, may use hdfs:// > based credential provider and use LdapGroupsMapping at the same time. We > should use HADOOP-12846 to exclude hdfs:// URI credential providers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13548) Remove recursive dependencies of credential providers in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504081#comment-15504081
]
Hadoop QA commented on HADOOP-13548:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 19s{color}
| {color:red} root generated 1 new + 709 unchanged - 0 fixed = 710 total (was
709) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 5s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestLdapGroupsMapping |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13548 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829231/HADOOP-13548.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9985652cc432 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b8a30f2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10543/artifact/patchprocess/diff-compile-javac-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10543/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10543/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10543/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Remove recursive dependencies of credential providers in LdapGroupsMapping
> --
>
> Key: HADOOP-13548
> URL:
[jira] [Commented] (HADOOP-13607) Specify and test contract for FileSystem#close.
[
https://issues.apache.org/jira/browse/HADOOP-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504021#comment-15504021
]
Chris Nauroth commented on HADOOP-13607:
I was thinking we'd cover both {{FSDataOutputStream#close}} (hence updates to
{{AbstractContractCreateTest}}) and {{FSDataInputStream#close}} (hence updates
to {{AbstractContractOpenTest}}).
> Specify and test contract for FileSystem#close.
> ---
>
> Key: HADOOP-13607
> URL: https://issues.apache.org/jira/browse/HADOOP-13607
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>
> This issue proposes to enhance the {{FileSystem}} specification by describing
> the expected semantics of {{FileSystem#close}} and adding corresponding
> contract tests. Notable aspects are that the method must be idempotent as
> dictated by {{java.io.Closeable}} and closing also interacts with the
> delete-on-exit feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13548) Remove recursive dependencies of credential providers in LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13548: - Status: Patch Available (was: Open) > Remove recursive dependencies of credential providers in LdapGroupsMapping > -- > > Key: HADOOP-13548 > URL: https://issues.apache.org/jira/browse/HADOOP-13548 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HADOOP-13548.001.patch > > > HADOOP-11934 discovered an infinite loop of dependencies in the use of > credential provider in LdapGroupsMapping. It added a new localjceks:// URI to > workaround the problem. The assumption is that the groups mapping is used > only in NameNode and that using a local credential file is not a problem. > However, there are cases where Hadoop clients, such as Sqoop, may use hdfs:// > based credential provider and use LdapGroupsMapping at the same time. We > should use HADOOP-12846 to exclude hdfs:// URI credential providers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13548) Remove recursive dependencies of credential providers in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-13548:
-
Attachment: HADOOP-13548.001.patch
v01: Turns out that LdapGroupsMapping must remove a provider URI if it's a
{{jceks://}}. So I expanded
{{ProviderUtils#excludeIncompatibleCredentialProviders}}. If the class type
passed in is a/subclass of {{FileSystem}}, the logic remains the same;
otherwise, if the class type is a/subclass of {{AbstractJavaKeyStoreProvider}},
use it to trim provider URI differently.
Note: this patch changes the method signature of
{{ProviderUtils#excludeIncompatibleCredentialProviders}} from {{Class}} to {{Class}}.
> Remove recursive dependencies of credential providers in LdapGroupsMapping
> --
>
> Key: HADOOP-13548
> URL: https://issues.apache.org/jira/browse/HADOOP-13548
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-13548.001.patch
>
>
> HADOOP-11934 discovered an infinite loop of dependencies in the use of
> credential provider in LdapGroupsMapping. It added a new localjceks:// URI to
> workaround the problem. The assumption is that the groups mapping is used
> only in NameNode and that using a local credential file is not a problem.
> However, there are cases where Hadoop clients, such as Sqoop, may use hdfs://
> based credential provider and use LdapGroupsMapping at the same time. We
> should use HADOOP-12846 to exclude hdfs:// URI credential providers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13548) Remove recursive dependencies of credential providers in LdapGroupsMapping
[ https://issues.apache.org/jira/browse/HADOOP-13548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HADOOP-13548: Assignee: Wei-Chiu Chuang > Remove recursive dependencies of credential providers in LdapGroupsMapping > -- > > Key: HADOOP-13548 > URL: https://issues.apache.org/jira/browse/HADOOP-13548 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > > HADOOP-11934 discovered an infinite loop of dependencies in the use of > credential provider in LdapGroupsMapping. It added a new localjceks:// URI to > workaround the problem. The assumption is that the groups mapping is used > only in NameNode and that using a local credential file is not a problem. > However, there are cases where Hadoop clients, such as Sqoop, may use hdfs:// > based credential provider and use LdapGroupsMapping at the same time. We > should use HADOOP-12846 to exclude hdfs:// URI credential providers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13604) Abort retry loop when RPC has an unrecoverable Auth error
[ https://issues.apache.org/jira/browse/HADOOP-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503842#comment-15503842 ] Steve Loughran commented on HADOOP-13604: - Renamed to make clear it's about auth: the problem is two fold: determining that there's an unrecoverable and then rejecting it. The retry handler is meant to retry on some things, ConnectionRefused, etc, but k-auth problems are unlikely to go away. That said, some network problems (especially DNS) tend not to resolve on their own. w.r.t Auth problems, the fact that UGI tends to generate IOEs rather than anything you can filter on is going to have to be fixed first > Abort retry loop when RPC has an unrecoverable Auth error > - > > Key: HADOOP-13604 > URL: https://issues.apache.org/jira/browse/HADOOP-13604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Henry Robinson >Assignee: Xiao Chen > > I've seen an issue where, after an RPC client hit an error obtaining a TGT > from Kerberos, the RPC client continues to retry even though there's no > chance of success (the no login window is set to 600s). > In this particular deployment, the client retries 15 times at 15s intervals, > leading to a delay of more than three minutes before the failure is bubbled > up to the client when the RPC ultimately fails. > Unrecoverable errors (like failures to login to Kerberos) should lead to fast > aborts of the retry loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13604) Abort retry loop when RPC has an unrecoverable Auth error
[ https://issues.apache.org/jira/browse/HADOOP-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13604: Summary: Abort retry loop when RPC has an unrecoverable Auth error (was: Abort retry loop when RPC has an unrecoverable error) > Abort retry loop when RPC has an unrecoverable Auth error > - > > Key: HADOOP-13604 > URL: https://issues.apache.org/jira/browse/HADOOP-13604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Henry Robinson >Assignee: Xiao Chen > > I've seen an issue where, after an RPC client hit an error obtaining a TGT > from Kerberos, the RPC client continues to retry even though there's no > chance of success (the no login window is set to 600s). > In this particular deployment, the client retries 15 times at 15s intervals, > leading to a delay of more than three minutes before the failure is bubbled > up to the client when the RPC ultimately fails. > Unrecoverable errors (like failures to login to Kerberos) should lead to fast > aborts of the retry loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13604) Abort retry loop when RPC has an unrecoverable error
[ https://issues.apache.org/jira/browse/HADOOP-13604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503824#comment-15503824 ] Steve Loughran commented on HADOOP-13604: - I'm supportive of this. Given it's reference to That Which Must Not be Named, linking to HADOOP-12649 > Abort retry loop when RPC has an unrecoverable error > > > Key: HADOOP-13604 > URL: https://issues.apache.org/jira/browse/HADOOP-13604 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Henry Robinson >Assignee: Xiao Chen > > I've seen an issue where, after an RPC client hit an error obtaining a TGT > from Kerberos, the RPC client continues to retry even though there's no > chance of success (the no login window is set to 600s). > In this particular deployment, the client retries 15 times at 15s intervals, > leading to a delay of more than three minutes before the failure is bubbled > up to the client when the RPC ultimately fails. > Unrecoverable errors (like failures to login to Kerberos) should lead to fast > aborts of the retry loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
[ https://issues.apache.org/jira/browse/HADOOP-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-13624 started by Genmao Yu. -- > Rename TestAliyunOSSContractDispCp > -- > > Key: HADOOP-13624 > URL: https://issues.apache.org/jira/browse/HADOOP-13624 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Reporter: Kai Zheng >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > > It should be TestAliyunOSSContractDistCp.java instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
[ https://issues.apache.org/jira/browse/HADOOP-13624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu reassigned HADOOP-13624: -- Assignee: Genmao Yu > Rename TestAliyunOSSContractDispCp > -- > > Key: HADOOP-13624 > URL: https://issues.apache.org/jira/browse/HADOOP-13624 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Reporter: Kai Zheng >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > > It should be TestAliyunOSSContractDistCp.java instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13061) Refactor erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13061: Attachment: HADOOP-13061.12.patch > Refactor erasure coders > --- > > Key: HADOOP-13061 > URL: https://issues.apache.org/jira/browse/HADOOP-13061 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Kai Sasaki > Attachments: HADOOP-13061.01.patch, HADOOP-13061.02.patch, > HADOOP-13061.03.patch, HADOOP-13061.04.patch, HADOOP-13061.05.patch, > HADOOP-13061.06.patch, HADOOP-13061.07.patch, HADOOP-13061.08.patch, > HADOOP-13061.09.patch, HADOOP-13061.10.patch, HADOOP-13061.11.patch, > HADOOP-13061.12.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
[
https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-13371 stopped by Steve Loughran.
---
> S3A globber to use bulk listObject call over recursive directory scan
> -
>
> Key: HADOOP-13371
> URL: https://issues.apache.org/jira/browse/HADOOP-13371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> HADOOP-13208 produces O(1) listing of directory trees in
> {{FileSystem.listStatus}} calls, but doesn't do anything for
> {{FileSystem.globStatus()}}, which uses a completely different codepath, one
> which does a selective recursive scan by pattern matching as it goes down,
> filtering out those patterns which don't match. Cost is
> O(matching-directories) + cost of examining the files.
> It should be possible to do the glob status listing in S3A not through the
> filtered treewalk, but through a list + filter operation. This would be an
> O(files) lookup *before any filtering took place*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13624) Rename TestAliyunOSSContractDispCp
Kai Zheng created HADOOP-13624: -- Summary: Rename TestAliyunOSSContractDispCp Key: HADOOP-13624 URL: https://issues.apache.org/jira/browse/HADOOP-13624 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng It should be TestAliyunOSSContractDistCp.java instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13607) Specify and test contract for FileSystem#close.
[
https://issues.apache.org/jira/browse/HADOOP-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503411#comment-15503411
]
Steve Loughran commented on HADOOP-13607:
-
I presume you mean {{DFSOutputStream.close()}}
be good to specify re-entrancy requirements, even if testing isn't something
that can be done.
> Specify and test contract for FileSystem#close.
> ---
>
> Key: HADOOP-13607
> URL: https://issues.apache.org/jira/browse/HADOOP-13607
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Reporter: Chris Nauroth
>
> This issue proposes to enhance the {{FileSystem}} specification by describing
> the expected semantics of {{FileSystem#close}} and adding corresponding
> contract tests. Notable aspects are that the method must be idempotent as
> dictated by {{java.io.Closeable}} and closing also interacts with the
> delete-on-exit feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13614) Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
[
https://issues.apache.org/jira/browse/HADOOP-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13614:
Status: Open (was: Patch Available)
> Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
> ---
>
> Key: HADOOP-13614
> URL: https://issues.apache.org/jira/browse/HADOOP-13614
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13614-branch-2-001.patch
>
>
> Some of the slow test cases contain tests that are now obsoleted by newer
> ones. For example, {{ITestS3ADeleteManyFiles}} has the test case
> {{testOpenCreate()}} which writes then reads files up 25 MB.
> Have a look at which of the s3a tests are taking time, review them to see if
> newer tests have superceded the slow ones; and cut them where appropriate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13623) regression: ITestS3NContractRootDir
[
https://issues.apache.org/jira/browse/HADOOP-13623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503280#comment-15503280
]
Steve Loughran commented on HADOOP-13623:
-
Note that this is potentially not a regression in s3n itself, more that a
change in the tests has broken things. Test endpoint: s3 ireland
{code}
testRecursiveRootListing(org.apache.hadoop.fs.contract.s3n.ITestS3NContractRootDir)
Time elapsed: 3.26 sec <<< ERROR!
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163)
at org.apache.hadoop.fs.Path.(Path.java:175)
at org.apache.hadoop.fs.Path.(Path.java:120)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:577)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1539)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1582)
at org.apache.hadoop.fs.FileSystem$4.(FileSystem.java:1745)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1744)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1727)
at org.apache.hadoop.fs.FileSystem$6.(FileSystem.java:1826)
at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:1822)
at
org.apache.hadoop.fs.contract.AbstractContractRootDirectoryTest.testRecursiveRootListing(AbstractContractRootDirectoryTest.java:171)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> regression: ITestS3NContractRootDir
> ---
>
> Key: HADOOP-13623
> URL: https://issues.apache.org/jira/browse/HADOOP-13623
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> I'm going to point the blame at HADOOP-13207; the new test is failing on S3n.
> Somehow this wasn't showing up in tests until recently
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13623) regression: ITestS3NContractRootDir
Steve Loughran created HADOOP-13623: --- Summary: regression: ITestS3NContractRootDir Key: HADOOP-13623 URL: https://issues.apache.org/jira/browse/HADOOP-13623 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.8.0 Reporter: Steve Loughran Assignee: Steve Loughran I'm going to point the blame at HADOOP-13207; the new test is failing on S3n. Somehow this wasn't showing up in tests until recently -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13584) Merge HADOOP-12756 branch to latest trunk
[
https://issues.apache.org/jira/browse/HADOOP-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503132#comment-15503132
]
Genmao Yu edited comment on HADOOP-13584 at 9/19/16 11:27 AM:
--
[~drankye] Ready for your review. There are some errors, but seems to has no
relationship with my patch.
{code}
Reason | Tests
Failed junit tests | hadoop.hdfs.TestErasureCodeBenchmarkThroughput
| hadoop.hdfs.TestDFSShell
|
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
|
hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!?
/testptch/hadoop/hadoop-build-tools/.externalToolBuilders/Maven_Ant_Builder.launch
!? /testptch/hadoop/hadoop-build-tools/maven-eclipse.xml
{code}
unit test results:
{code}
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop Aliyun OSS support 3.0.0-alpha2-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-aliyun ---
[INFO] Deleting /develop/github/hadoop/hadoop-tools/hadoop-aliyun/target
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-aliyun ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-aliyun
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-aliyun ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/src/main/resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-aliyun
---
[INFO] Compiling 8 source files to
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/classes
[INFO]
[INFO] --- maven-dependency-plugin:2.2:list (deplist) @ hadoop-aliyun ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-aliyun ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-aliyun ---
[INFO] Compiling 16 source files to
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-aliyun ---
[INFO] Surefire report directory:
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/surefire-reports
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.056 sec - in
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.429 sec - in
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.207 sec -
in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.276 sec - in
org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.656 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.229 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.53 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.17 sec - in
[jira] [Commented] (HADOOP-13061) Refactor erasure coders
[
https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503147#comment-15503147
]
Hadoop QA commented on HADOOP-13061:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 36s{color} | {color:orange} root: The patch generated 3 new + 188 unchanged
- 24 fixed = 191 total (was 212) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 58s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13061 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829162/HADOOP-13061.11.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a895b54cc307 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b8a30f2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10541/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Commented] (HADOOP-13584) Merge HADOOP-12756 branch to latest trunk
[
https://issues.apache.org/jira/browse/HADOOP-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503132#comment-15503132
]
Genmao Yu commented on HADOOP-13584:
[~drankye] Ready for your review. There are some errors, but seems to has no
relationship with my patch.
{code}
Reason | Tests
Failed junit tests | hadoop.hdfs.TestErasureCodeBenchmarkThroughput
| hadoop.hdfs.TestDFSShell
|
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
|
hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!?
/testptch/hadoop/hadoop-build-tools/.externalToolBuilders/Maven_Ant_Builder.launch
!? /testptch/hadoop/hadoop-build-tools/maven-eclipse.xml
{code}
> Merge HADOOP-12756 branch to latest trunk
> -
>
> Key: HADOOP-13584
> URL: https://issues.apache.org/jira/browse/HADOOP-13584
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: shimingfei
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13584.001.patch, HADOOP-13584.002.patch
>
>
> We have finished a round of improvement over Hadoop-12756 branch, which
> intends to incorporate Aliyun OSS support in Hadoop. This feature provides
> basic support for data access to Aliyun OSS from Hadoop applications.
> In the implementation, we follow the style of S3 support in Hadooop. Besides
> we also provide FileSystem contract test over real Aliyun OSS environment. By
> simple configuration, it can be enabled/disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13584) Merge HADOOP-12756 branch to latest trunk
[
https://issues.apache.org/jira/browse/HADOOP-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503116#comment-15503116
]
Hadoop QA commented on HADOOP-13584:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
9s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project . hadoop-tools hadoop-tools/hadoop-tools-dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 9s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.hdfs.TestDFSShell |
| |
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
|
| | hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13584 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829157/HADOOP-13584.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 6b5a59340513 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503029#comment-15503029 ] Steve Loughran commented on HADOOP-13622: - One challenge here is that we aren't going to easily tell if an FS is an object store, not unless HADOOP-9565 or equivalent gets in > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13619) missing data intermittently when reading avro file in Spark from Swift storage
[
https://issues.apache.org/jira/browse/HADOOP-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503018#comment-15503018
]
Steve Loughran commented on HADOOP-13619:
-
you aren't overwriting blobs with new ones of different sizes? that can cause
visible consistency problem.
An other possible cause is that these are clearly multipart files, which are
trouble in their own ways. Specifically, I've seen mismatches between the size
returned in FileSystem.getFileStatus and the actual file length as returned in
reads.
Can you replicate this locally? That is, repeatedly run getStatus() against the
file and verify that it always returns the length of file just written
Now the bad news: you are probably going to have to be the one to identify and,
if it is possible fix the problem. I would also recommend you work on on the
branch-2 branch, because that and trunk is where fixes will go, maybe
backported to 2.8 or 2.7.4.
I've been writing some spark & object store tests in SPARK-7481, albeit with a
focus on S3; In HADOOP-11694 we've been adding scale tests for s3 too.
One limitation we have there is the lack of large public datasets, and the
time/cost it takes to set up transient ones for a single test. Would you be
able to serve up the specific file causing problems here as a public object?
That way some of our integration tests could work with it direct, the way we do
for some AWS tests today.
> missing data intermittently when reading avro file in Spark from Swift storage
> --
>
> Key: HADOOP-13619
> URL: https://issues.apache.org/jira/browse/HADOOP-13619
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Priority: Blocker
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> We are loading avro files from Oracle Storage Service server (i.e., Swift
> server) into Spark DataFrame object through the Spark Data Source API. For
> example:
> return hiveCtx.read().format("com.databricks.spark.avro").load(objectName);
> The number of records is less than the actual record count in the avro file
> when reading the avro file from Storage Service server using OpenStack Swift
> API.
> If we run a SQL on top of the returned data frome like "select count(\*) as
> C1 from " we can see the record count is smaller when reading the
> same avro file from local file system.
> For a large avro file (awclassic.avro, 105M) the count is always wrong (42451
> records vs. 60855). From the log file we can see the reading os the file is
> splitted into 4:
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:100663296+10044747
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:33554432+33554432
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:0+33554432
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:67108864+33554432
> For a smaller avro file (wine.avro, 19M) the count sometimes is correct
> (57076 records) and sometimes wrong (26999 records). Run the same spark SQL
> 10 times back-to-back produces the following record count results:
> run 1: 26999
> run 2: 26999
> run 3: 57076
> run 4: 57056
> run 5: 57076
> run 6: 26999
> run 7: 57076
> run 8: 57076
> run 9: 57076
> run 10: 57076
> For this wine.avro test case there are two splits:
> 2016-08-31 17:42:32 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/wine.avro:9965269+9965270
> 2016-08-31 17:42:32 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/wine.avro:0+9965269
> I will attach a zip file containing the smaller avro file in question and the
> debugged log file section of reading wine.avro file - one with successful
> reading(C4.ok) and one with missing record reading(C5.miss).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13619) missing data intermittently when reading avro file in Spark from Swift storage
[
https://issues.apache.org/jira/browse/HADOOP-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13619:
Summary: missing data intermittently when reading avro file in Spark from
Swift storage (was: missing data intermittently when reading avro file)
> missing data intermittently when reading avro file in Spark from Swift storage
> --
>
> Key: HADOOP-13619
> URL: https://issues.apache.org/jira/browse/HADOOP-13619
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Priority: Blocker
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> We are loading avro files from Oracle Storage Service server (i.e., Swift
> server) into Spark DataFrame object through the Spark Data Source API. For
> example:
> return hiveCtx.read().format("com.databricks.spark.avro").load(objectName);
> The number of records is less than the actual record count in the avro file
> when reading the avro file from Storage Service server using OpenStack Swift
> API.
> If we run a SQL on top of the returned data frome like "select count(\*) as
> C1 from " we can see the record count is smaller when reading the
> same avro file from local file system.
> For a large avro file (awclassic.avro, 105M) the count is always wrong (42451
> records vs. 60855). From the log file we can see the reading os the file is
> splitted into 4:
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:100663296+10044747
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:33554432+33554432
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:0+33554432
> 2016-09-01 14:18:27 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/awclassic.avro:67108864+33554432
> For a smaller avro file (wine.avro, 19M) the count sometimes is correct
> (57076 records) and sometimes wrong (26999 records). Run the same spark SQL
> 10 times back-to-back produces the following record count results:
> run 1: 26999
> run 2: 26999
> run 3: 57076
> run 4: 57056
> run 5: 57076
> run 6: 26999
> run 7: 57076
> run 8: 57076
> run 9: 57076
> run 10: 57076
> For this wine.avro test case there are two splits:
> 2016-08-31 17:42:32 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/wine.avro:9965269+9965270
> 2016-08-31 17:42:32 INFO HadoopRDD:59 - Input split:
> swift://qaTestData.oracleswift/testAvro/wine.avro:0+9965269
> I will attach a zip file containing the smaller avro file in question and the
> debugged log file section of reading wine.avro file - one with successful
> reading(C4.ok) and one with missing record reading(C5.miss).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13617) Swift client retrying original request is using expired token after re-authentication
[
https://issues.apache.org/jira/browse/HADOOP-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502936#comment-15502936
]
Steve Loughran commented on HADOOP-13617:
-
you are probably the first person to notice this; on the special case of
failure during call. I suspect the retry code doesn't update the relevant
headers.
patches welcome, though there is a strict "declare your endpoint" policy for
object store submissions, as Yetus & jenkins don't test them
http://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure
> Swift client retrying original request is using expired token after
> re-authentication
> --
>
> Key: HADOOP-13617
> URL: https://issues.apache.org/jira/browse/HADOOP-13617
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
>Affects Versions: 2.6.0
> Environment: Linux EL6
>Reporter: Steve Yang
>Priority: Blocker
> Attachments: 2016_09_13.stderrout.log
>
>
> library used: org.apache.hadoop:hadoop-openstack:2.6.0
> For long running Swift read operation (e.g., reading a large container), the
> issued auth token has at most 30 minutes life span from Oracle Storage
> Service. If the token expired in the middle of the read operation the
> SwiftRestClient
> (https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/SwiftRestClient.java#L1701)
> re-authenticate and acquire a new auth token. However, in the retry request
> the old, expired token is still used, causing the whole operation to fail.
> Because of this bug any meaningful(i.e., long-running) Swift operation is not
> possible.
> Here is a summary of what happened with DEBUG logging turned on:
> ==
> 1. initial token acquired which will expire on 19:56:44(PDT; UTC-4):
> ---
> 2016-09-13 19:52:37 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk2dd9d639bbb992089dca008123c3046f',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@af28493,
> expires='2016-09-13T23:56:44Z'}
> 2. token expiration and re-authentication:
> --
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - GET
> https://em2.storage.oraclecloud.com/v1/Storage-paas132/allTaxi/?prefix=000182/=json=/
> X-Auth-Token: AUTH_tk2dd9d639bbb992089dca008123c3046f
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:44 WARN [pool-3-thread-1] HttpMethodDirector:697 - Unable
> to respond to any of these challenges: {token=Token}
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 401
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1698 -
> Reauthenticating
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1079 - started
> authentication
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1228 -
> Authenticating with Authenticate as tenant 'Storage-paas132' user
> 'radha.sriniva...@oracle.com' with password of length 9
> 2016-09-13 19:56:44 DEBUG [pool-3-thread-1] SwiftRestClient:1727 - POST
> https://em2.storage.oraclecloud.com/auth/v2.0/tokens
> User-Agent: Apache Hadoop Swift Client 2.6.0-cdh5.7.1 from
> ae44a8970a3f0da58d82e0fc65275fff8deabffd by jenkins source checksum
> 298b68dc3b308983f04cb37e8416f13
> .
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1731 - Status
> code = 200
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1149 - Catalog
> entry [swift: object-store];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1156 - Found
> swift catalog as swift => object-store
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1169 - Endpoint
> [US => https://em2.storage.oraclecloud.com/v1/Storage-paas132 / null];
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:268 - setAuth:
> endpoint=https://em2.storage.oraclecloud.com/v1/Storage-paas132;
> objectURI=https://em2.storage.oraclecloud.com/object_endpoint/null;
> token=AccessToken{id='AUTH_tk56bbb4d6fef57b7eeba7acae598f837c',
> tenant=org.apache.hadoop.fs.swift.auth.entities.Tenant@4f03838d,
> expires='2016-09-14T00:26:45Z'}
> 2016-09-13 19:56:45 DEBUG [pool-3-thread-1] SwiftRestClient:1216 -
> authenticated against https://em2.storage.oraclecloud.com/v1/Storage-paas132.
> 2016-09-13
[jira] [Commented] (HADOOP-13593) `hadoop distcp -atomic` invokes improper host check while copying data from HDFS to Swift
[
https://issues.apache.org/jira/browse/HADOOP-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502810#comment-15502810
]
Yuanbo Liu commented on HADOOP-13593:
-
[~ste...@apache.org] I close this jira and hope to get your thoughts in
HADOOP-13622. Thanks in advance.
> `hadoop distcp -atomic` invokes improper host check while copying data from
> HDFS to Swift
> -
>
> Key: HADOOP-13593
> URL: https://issues.apache.org/jira/browse/HADOOP-13593
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
> Attachments: HADOOP-13593.001.patch, HADOOP-13593.002.patch
>
>
> While copying data from HDFS to Swift by using `hadoop distcp -atomic`, for
> example:
> {code}
> hadoop distcp -atomic /tmp/100M swift://testhadoop.softlayer//tmp
> {code}
> it throws
> {code}
> java.lang.IllegalArgumentException: Work path
> swift://testhadoop.softlayer/._WIP_tmp546958075 and target path
> swift://testhadoop.softlayer/tmp are in different file system
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:351)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13593) `hadoop distcp -atomic` invokes improper host check while copying data from HDFS to Swift
[
https://issues.apache.org/jira/browse/HADOOP-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HADOOP-13593:
Resolution: Invalid
Status: Resolved (was: Patch Available)
> `hadoop distcp -atomic` invokes improper host check while copying data from
> HDFS to Swift
> -
>
> Key: HADOOP-13593
> URL: https://issues.apache.org/jira/browse/HADOOP-13593
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
> Attachments: HADOOP-13593.001.patch, HADOOP-13593.002.patch
>
>
> While copying data from HDFS to Swift by using `hadoop distcp -atomic`, for
> example:
> {code}
> hadoop distcp -atomic /tmp/100M swift://testhadoop.softlayer//tmp
> {code}
> it throws
> {code}
> java.lang.IllegalArgumentException: Work path
> swift://testhadoop.softlayer/._WIP_tmp546958075 and target path
> swift://testhadoop.softlayer/tmp are in different file system
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:351)
> .
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
Yuanbo Liu created HADOOP-13622: --- Summary: `-atomic` should not be supported while using `distcp` command in object file system Key: HADOOP-13622 URL: https://issues.apache.org/jira/browse/HADOOP-13622 Project: Hadoop Common Issue Type: Bug Reporter: Yuanbo Liu Assignee: Yuanbo Liu After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point that none of the object stores support atomic renames. So I file a new jira and ready to provide a patch to disable `distcp -atomic` in object file system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13061) Refactor erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-13061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13061: Attachment: HADOOP-13061.11.patch > Refactor erasure coders > --- > > Key: HADOOP-13061 > URL: https://issues.apache.org/jira/browse/HADOOP-13061 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Kai Sasaki > Attachments: HADOOP-13061.01.patch, HADOOP-13061.02.patch, > HADOOP-13061.03.patch, HADOOP-13061.04.patch, HADOOP-13061.05.patch, > HADOOP-13061.06.patch, HADOOP-13061.07.patch, HADOOP-13061.08.patch, > HADOOP-13061.09.patch, HADOOP-13061.10.patch, HADOOP-13061.11.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13584) Merge HADOOP-12756 branch to latest trunk
[ https://issues.apache.org/jira/browse/HADOOP-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu reassigned HADOOP-13584: -- Assignee: Genmao Yu > Merge HADOOP-12756 branch to latest trunk > - > > Key: HADOOP-13584 > URL: https://issues.apache.org/jira/browse/HADOOP-13584 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13584.001.patch, HADOOP-13584.002.patch > > > We have finished a round of improvement over Hadoop-12756 branch, which > intends to incorporate Aliyun OSS support in Hadoop. This feature provides > basic support for data access to Aliyun OSS from Hadoop applications. > In the implementation, we follow the style of S3 support in Hadooop. Besides > we also provide FileSystem contract test over real Aliyun OSS environment. By > simple configuration, it can be enabled/disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13584) Merge HADOOP-12756 branch to latest trunk
[ https://issues.apache.org/jira/browse/HADOOP-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13584: --- Attachment: HADOOP-13584.002.patch > Merge HADOOP-12756 branch to latest trunk > - > > Key: HADOOP-13584 > URL: https://issues.apache.org/jira/browse/HADOOP-13584 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13584.001.patch, HADOOP-13584.002.patch > > > We have finished a round of improvement over Hadoop-12756 branch, which > intends to incorporate Aliyun OSS support in Hadoop. This feature provides > basic support for data access to Aliyun OSS from Hadoop applications. > In the implementation, we follow the style of S3 support in Hadooop. Besides > we also provide FileSystem contract test over real Aliyun OSS environment. By > simple configuration, it can be enabled/disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[ https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-13591: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to the branch. Thanks [~uncleGen] for the contribution! > Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir > - > > Key: HADOOP-13591 > URL: https://issues.apache.org/jira/browse/HADOOP-13591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HADOOP-12756 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13591-HADOOP-12756.001.patch, > HADOOP-13591-HADOOP-12756.002.patch, HADOOP-13591-HADOOP-12756.003.patch, > HADOOP-13591-HADOOP-12756.004.patch, HADOOP-13591-HADOOP-12756.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[ https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502555#comment-15502555 ] Kai Zheng commented on HADOOP-13591: The latest patch LGTM and +1. > Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir > - > > Key: HADOOP-13591 > URL: https://issues.apache.org/jira/browse/HADOOP-13591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HADOOP-12756 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13591-HADOOP-12756.001.patch, > HADOOP-13591-HADOOP-12756.002.patch, HADOOP-13591-HADOOP-12756.003.patch, > HADOOP-13591-HADOOP-12756.004.patch, HADOOP-13591-HADOOP-12756.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[
https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502518#comment-15502518
]
Genmao Yu edited comment on HADOOP-13591 at 9/19/16 7:07 AM:
-
[~drankye] Ready for your review
{code}
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building Apache Hadoop Aliyun OSS support 3.0.0-alpha2-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-aliyun ---
[INFO] Deleting /develop/github/hadoop/hadoop-tools/hadoop-aliyun/target
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-aliyun ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-aliyun
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-aliyun ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/src/main/resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-aliyun
---
[INFO] Compiling 8 source files to
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/classes
[INFO]
[INFO] --- maven-dependency-plugin:2.2:list (deplist) @ hadoop-aliyun ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-aliyun ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-aliyun ---
[INFO] Compiling 16 source files to
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-aliyun ---
[INFO] Surefire report directory:
/develop/github/hadoop/hadoop-tools/hadoop-aliyun/target/surefire-reports
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.074 sec - in
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.226 sec - in
org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.113 sec -
in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
Running org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.892 sec - in
org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.819 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.009 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.913 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.255 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.506 sec - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete
Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.878 sec -
in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus
Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.668 sec - in
[jira] [Commented] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[ https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502518#comment-15502518 ] Genmao Yu commented on HADOOP-13591: [~drankye] Ready for your review > Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir > - > > Key: HADOOP-13591 > URL: https://issues.apache.org/jira/browse/HADOOP-13591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HADOOP-12756 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13591-HADOOP-12756.001.patch, > HADOOP-13591-HADOOP-12756.002.patch, HADOOP-13591-HADOOP-12756.003.patch, > HADOOP-13591-HADOOP-12756.004.patch, HADOOP-13591-HADOOP-12756.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[
https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502508#comment-15502508
]
Hadoop QA commented on HADOOP-13591:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-tools/hadoop-aliyun in HADOOP-12756 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} HADOOP-12756 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} hadoop-tools/hadoop-aliyun generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13591 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829150/HADOOP-13591-HADOOP-12756.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux e51ec7ac44c6 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-12756 / 9cd4760 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10539/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-aliyun-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10539/testReport/ |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10539/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
>
[jira] [Updated] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[ https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Genmao Yu updated HADOOP-13591: --- Attachment: HADOOP-13591-HADOOP-12756.005.patch > Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir > - > > Key: HADOOP-13591 > URL: https://issues.apache.org/jira/browse/HADOOP-13591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HADOOP-12756 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13591-HADOOP-12756.001.patch, > HADOOP-13591-HADOOP-12756.002.patch, HADOOP-13591-HADOOP-12756.003.patch, > HADOOP-13591-HADOOP-12756.004.patch, HADOOP-13591-HADOOP-12756.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13591) Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir
[ https://issues.apache.org/jira/browse/HADOOP-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502404#comment-15502404 ] Genmao Yu commented on HADOOP-13591: updating > Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir > - > > Key: HADOOP-13591 > URL: https://issues.apache.org/jira/browse/HADOOP-13591 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: HADOOP-12756 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: HADOOP-12756 > > Attachments: HADOOP-13591-HADOOP-12756.001.patch, > HADOOP-13591-HADOOP-12756.002.patch, HADOOP-13591-HADOOP-12756.003.patch, > HADOOP-13591-HADOOP-12756.004.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
