[jira] [Commented] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631590#comment-15631590
]
Hadoop QA commented on HADOOP-13720:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 74 unchanged - 2 fixed = 77 total (was 76) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
31s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
10s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Call to method of static java.text.DateFormat in
org.apache.hadoop.util.Time.formatTime(long) At Time.java:java.text.DateFormat
in org.apache.hadoop.util.Time.formatTime(long) At Time.java:[line 69] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13720 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836723/HADOOP-13720.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3dbbf0005cab 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7e521c5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10969/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10969/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10969/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-13787) Azure testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631583#comment-15631583
]
Hadoop QA commented on HADOOP-13787:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
36s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13787 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836724/HADOOP-13787.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a34db7a4b94a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7e521c5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10970/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10970/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Azure testGlobStatusThrowsExceptionForUnreadableDir fails
> -
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13787.001.patch
>
>
> Test
>
[jira] [Updated] (HADOOP-13787) Azure testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13787:
Attachment: HADOOP-13787.001.patch
Patch 001:
* Disable {{testGlobStatusThrowsExceptionForUnreadableDir}} in
{{TestNativeAzureFileSystemOperationsMocked}}, just as
{{testListStatusThrowsExceptionForUnreadableDir}}.
> Azure testGlobStatusThrowsExceptionForUnreadableDir fails
> -
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13787.001.patch
>
>
> Test
> {{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
> failed in trunk:
> {noformat}
> Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
> Time elapsed: 1.111 sec <<< FAILURE!
> java.lang.AssertionError: Should throw IOException
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13787) Azure testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13787:
Fix Version/s: 3.0.0-alpha2
Status: Patch Available (was: In Progress)
> Azure testGlobStatusThrowsExceptionForUnreadableDir fails
> -
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13787.001.patch
>
>
> Test
> {{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
> failed in trunk:
> {noformat}
> Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
> Time elapsed: 1.111 sec <<< FAILURE!
> java.lang.AssertionError: Should throw IOException
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631504#comment-15631504
]
Yongjun Zhang commented on HADOOP-13720:
Thanks [~ste...@apache.org], good comments! Uploaded rev3 to address that.
Would you please take a look?
Hi [~xiaochen], thanks again for your earlier review, I will appreciate it if
you could take a look at the new rev.
Thank you all!
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch,
> HADOOP-13720.003.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13787) Azure testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-13787 started by John Zhuge.
---
> Azure testGlobStatusThrowsExceptionForUnreadableDir fails
> -
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Test
> {{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
> failed in trunk:
> {noformat}
> Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
> Time elapsed: 1.111 sec <<< FAILURE!
> java.lang.AssertionError: Should throw IOException
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-13720:
---
Attachment: HADOOP-13720.003.patch
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch,
> HADOOP-13720.003.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13787) Azure testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631486#comment-15631486
]
John Zhuge commented on HADOOP-13787:
-
{{NativeAzureFileSystem}} does not support permission checking even though file
permission {{FileMetadata#permissionStatus}} is stored in the metadata store
{{AzureNativeFileSystemStore}}.
I will disable this test for NativeAzureFileSystem.
> Azure testGlobStatusThrowsExceptionForUnreadableDir fails
> -
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Test
> {{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
> failed in trunk:
> {noformat}
> Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
> Time elapsed: 1.111 sec <<< FAILURE!
> java.lang.AssertionError: Should throw IOException
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13768) AliyunOSS: handle deleteDirs reliably when too many objects to delete
[
https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631420#comment-15631420
]
Genmao Yu commented on HADOOP-13768:
get it
> AliyunOSS: handle deleteDirs reliably when too many objects to delete
> -
>
> Key: HADOOP-13768
> URL: https://issues.apache.org/jira/browse/HADOOP-13768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch
>
>
> Note in Aliyun OSS SDK, DeleteObjectsRequest has 1000 objects limit. This
> needs to improve {{deleteDirs}} operation to make it pass when more objects
> than the limit to delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13427) Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631329#comment-15631329
]
Hadoop QA commented on HADOOP-13427:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
27s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-applicationhistoryservice in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-server-sharedcachemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-mapreduce-client-core in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} hadoop-mapreduce-client-app in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-mapreduce-client-hs in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-mapreduce-examples in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-distcp in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-archives in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-archive-logs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-rumen in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hadoop-extras in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} hadoop-openstack in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 49s{color} | {color:orange} root: The patch generated 8 new + 871 unchanged
- 10 fixed = 879 total (was 881) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 5m
[jira] [Commented] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631288#comment-15631288
]
Hadoop QA commented on HADOOP-13720:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 74 unchanged - 2 fixed = 77 total (was 76) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red} hadoop-common-project/hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
46s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Invocation of toString on
org.apache.hadoop.security.token.Token.getIdentifier() in
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(Token,
String) At AbstractDelegationTokenSecretManager.java:in
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(Token,
String) At AbstractDelegationTokenSecretManager.java:[line 512] |
| | Call to method of static java.text.DateFormat in
org.apache.hadoop.util.Time.formatTime(long) At Time.java:java.text.DateFormat
in org.apache.hadoop.util.Time.formatTime(long) At Time.java:[line 69] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13720 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836701/HADOOP-13720.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7f07a5c06cb6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / dcc07ad |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-13768) AliyunOSS: handle deleteDirs reliably when too many objects to delete
[
https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631263#comment-15631263
]
Kai Zheng commented on HADOOP-13768:
Looking at the following codes:
{code}
+List deleteFailed = keysToDelete;
+while(CollectionUtils.isNotEmpty(deleteFailed)) {
+ List l = new ArrayList<>();
+ List smallerLists = Lists.partition(deleteFailed, 1000);
+ for (List smallerList : smallerLists) {
+DeleteObjectsRequest deleteRequest =
+new DeleteObjectsRequest(bucketName);
+deleteRequest.setKeys(smallerList);
+deleteRequest.setQuiet(true);
+DeleteObjectsResult result = ossClient.deleteObjects(deleteRequest);
+l.addAll(result.getDeletedObjects());
+ }
+ deleteFailed = l;
+ tries++;
+ if (tries == retry) {
+break;
+ }
+}
{code}
1. Please give {{l}} a more readable name.
2. Can you give some comments to explain some bit about the procedure? I
(probably others) wouldn't know why it's like that without querying the SDK's
manual. I know it now, there're 2 modes in the {{ossClient.deleteObjects}}
operation, one mode to return successfully deleted objects, and the other
returning the deleting-failed objects. You're using the latter, and use a loop
to try some times to delete and delete the failed-to-delete objects.
> AliyunOSS: handle deleteDirs reliably when too many objects to delete
> -
>
> Key: HADOOP-13768
> URL: https://issues.apache.org/jira/browse/HADOOP-13768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch
>
>
> Note in Aliyun OSS SDK, DeleteObjectsRequest has 1000 objects limit. This
> needs to improve {{deleteDirs}} operation to make it pass when more objects
> than the limit to delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13768) AliyunOSS: handle deleteDirs reliably when too many objects to delete
[
https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-13768:
---
Description: Note in Aliyun OSS SDK, DeleteObjectsRequest has 1000 objects
limit. This needs to improve {{deleteDirs}} operation to make it pass when more
objects than the limit to delete.
> AliyunOSS: handle deleteDirs reliably when too many objects to delete
> -
>
> Key: HADOOP-13768
> URL: https://issues.apache.org/jira/browse/HADOOP-13768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch
>
>
> Note in Aliyun OSS SDK, DeleteObjectsRequest has 1000 objects limit. This
> needs to improve {{deleteDirs}} operation to make it pass when more objects
> than the limit to delete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13768) AliyunOSS: handle deleteDirs reliably when too many objects to delete
[ https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-13768: --- Summary: AliyunOSS: handle deleteDirs reliably when too many objects to delete (was: AliyunOSS: DeleteObjectsRequest has 1000 objects limit ) > AliyunOSS: handle deleteDirs reliably when too many objects to delete > - > > Key: HADOOP-13768 > URL: https://issues.apache.org/jira/browse/HADOOP-13768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13784) Output javadoc inside the target directory
[
https://issues.apache.org/jira/browse/HADOOP-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631220#comment-15631220
]
Hudson commented on HADOOP-13784:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10759 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10759/])
HADOOP-13784. Output javadoc inside the target directory. (wang: rev
dcc07ad8c8abf9feea569283b4a4a096d893168a)
* (edit) hadoop-project-dist/pom.xml
> Output javadoc inside the target directory
> --
>
> Key: HADOOP-13784
> URL: https://issues.apache.org/jira/browse/HADOOP-13784
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13784.001.patch
>
>
> HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
> javadoc dirs being created outside target folders:
> {noformat}
> hadoop-common-project/hadoop-common/api/
> hadoop-hdfs-project/hadoop-hdfs-client/api/
> hadoop-hdfs-project/hadoop-hdfs/api/
> {noformat}
> Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13632) Daemonization does not check process liveness before renicing
[
https://issues.apache.org/jira/browse/HADOOP-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631195#comment-15631195
]
Hadoop QA commented on HADOOP-13632:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
58s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13632 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836699/HADOOP-13632.002.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 07f9b6505e51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / dcc07ad |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10967/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10967/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Daemonization does not check process liveness before renicing
> -
>
> Key: HADOOP-13632
> URL: https://issues.apache.org/jira/browse/HADOOP-13632
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13632.001.patch, HADOOP-13632.002.patch
>
>
> If you try to daemonize a process that is incorrectly configured, it will die
> quite quickly. However, the daemonization function will still try to renice
> it even if it's down, leading to something like this for my namenode:
> {noformat}
> -> % bin/hdfs --daemon start namenode
> ERROR: Cannot set priority of namenode process 12036
> {noformat}
> It'd be more user-friendly instead of this renice error, we said that the
> process couldn't be started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-13720:
---
Attachment: (was: HADOOP-13720.003.patch)
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-13720:
---
Attachment: HADOOP-13720.003.patch
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch,
> HADOOP-13720.003.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13632) Daemonization does not check process liveness before renicing
[
https://issues.apache.org/jira/browse/HADOOP-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13632:
-
Attachment: HADOOP-13632.002.patch
Good catch, changed that to daemonpidfile instead.
> Daemonization does not check process liveness before renicing
> -
>
> Key: HADOOP-13632
> URL: https://issues.apache.org/jira/browse/HADOOP-13632
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13632.001.patch, HADOOP-13632.002.patch
>
>
> If you try to daemonize a process that is incorrectly configured, it will die
> quite quickly. However, the daemonization function will still try to renice
> it even if it's down, leading to something like this for my namenode:
> {noformat}
> -> % bin/hdfs --daemon start namenode
> ERROR: Cannot set priority of namenode process 12036
> {noformat}
> It'd be more user-friendly instead of this renice error, we said that the
> process couldn't be started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13784) Output javadoc inside the target directory
[
https://issues.apache.org/jira/browse/HADOOP-13784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13784:
-
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Thanks for reviewing Xiao, I've committed this to trunk.
> Output javadoc inside the target directory
> --
>
> Key: HADOOP-13784
> URL: https://issues.apache.org/jira/browse/HADOOP-13784
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13784.001.patch
>
>
> HADOOP-8500 cleaned up the javadoc build, but as a result we now have a few
> javadoc dirs being created outside target folders:
> {noformat}
> hadoop-common-project/hadoop-common/api/
> hadoop-hdfs-project/hadoop-hdfs-client/api/
> hadoop-hdfs-project/hadoop-hdfs/api/
> {noformat}
> Thanks to [~aw] for finding this issue over on HADOOP-8500.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13787) testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13787:
Summary: testGlobStatusThrowsExceptionForUnreadableDir fails (was:
testGlobStatusThrowsExceptionForUnreadableDir failed)
> testGlobStatusThrowsExceptionForUnreadableDir fails
> ---
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Test
> {{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
> failed in trunk:
> {noformat}
> Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
> Time elapsed: 1.111 sec <<< FAILURE!
> java.lang.AssertionError: Should throw IOException
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13787) Azure testGlobStatusThrowsExceptionForUnreadableDir fails
[
https://issues.apache.org/jira/browse/HADOOP-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13787:
Summary: Azure testGlobStatusThrowsExceptionForUnreadableDir fails (was:
testGlobStatusThrowsExceptionForUnreadableDir fails)
> Azure testGlobStatusThrowsExceptionForUnreadableDir fails
> -
>
> Key: HADOOP-13787
> URL: https://issues.apache.org/jira/browse/HADOOP-13787
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Test
> {{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
> failed in trunk:
> {noformat}
> Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
> FAILURE! - in
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
> testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
> Time elapsed: 1.111 sec <<< FAILURE!
> java.lang.AssertionError: Should throw IOException
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13769) AliyunOSS: improve performance on close
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631047#comment-15631047 ] Kai Zheng commented on HADOOP-13769: [~uncleGen], I understand upgrading the OSS SDK may fix the issue. Could you also consider [~ste...@apache.org]'s suggestion as below and add some similar tests to guard against this? Thanks! bq. The scale/seek tests in S3A are designed to catch that —maybe look at something similar. > AliyunOSS: improve performance on close > --- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13769) AliyunOSS: improve performance on close
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-13769: --- Summary: AliyunOSS: improve performance on close (was: AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.) > AliyunOSS: improve performance on close > --- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13769) AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded.
[ https://issues.apache.org/jira/browse/HADOOP-13769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-13769: --- Description: AliyunOSS object inputstream.close() will read the remaining bytes of the OSS object, potentially transferring a lot of bytes from OSS that are discarded. > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. > -- > > Key: HADOOP-13769 > URL: https://issues.apache.org/jira/browse/HADOOP-13769 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/oss >Affects Versions: 3.0.0-alpha2 >Reporter: Genmao Yu >Assignee: Genmao Yu > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13769.001.patch > > > AliyunOSS object inputstream.close() will read the remaining bytes of the OSS > object, potentially transferring a lot of bytes from OSS that are discarded. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13778) Unit test failed in TestAliyunOSSContractCreate
[
https://issues.apache.org/jira/browse/HADOOP-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631028#comment-15631028
]
Kai Zheng commented on HADOOP-13778:
I thought the test did against the AliyunOSS infrastructure, [~uncleGen] please
clarify if this isn't the case.
I noticed in HADOOP-13502 two new flags were introduced or added to the cloud
backed file systems as below. Only one was considered here, do we need the
other? I mean {{fs.contract.create-overwrites-directory}}.
{code}
+fs.contract.create-overwrites-directory
+true
+
+
+
+fs.contract.create-visibility-delayed
+true
+
+
+
{code}
> Unit test failed in TestAliyunOSSContractCreate
> ---
>
> Key: HADOOP-13778
> URL: https://issues.apache.org/jira/browse/HADOOP-13778
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Attachments: HADOOP-13778.001.patch
>
>
> {{fs.contract.is-blobstore}} is splitted into more descriptive flags in
> HADOOP-13502, {{fs.contract.create-visibility-delayed}} need also to be added
> into {{aliyun-oss.xml}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-7352) FileSystem#listStatus should throw IOE upon access error
[
https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631019#comment-15631019
]
John Zhuge commented on HADOOP-7352:
Thanks [~liuml07] for reporting the issue. Filed HADOOP-13787. Probably Azure
{{listStatus}} does not follow the new contract introduced in this JIRA.
> FileSystem#listStatus should throw IOE upon access error
>
>
> Key: HADOOP-7352
> URL: https://issues.apache.org/jira/browse/HADOOP-7352
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.6.0
>Reporter: Matt Foley
>Assignee: John Zhuge
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-7352.001.patch, HADOOP-7352.002.patch,
> HADOOP-7352.003.patch, HADOOP-7352.004.patch, HADOOP-7352.005.patch
>
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should
> throw FileNotFoundException instead of returning null, when the target
> directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus
> still may return null, when the target directory exists but does not grant
> read permission. This causes NPE in many callers, for all the reasons cited
> in HADOOP-6201 and HDFS-538. See HADOOP-7327 and its linked issues for
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13787) testGlobStatusThrowsExceptionForUnreadableDir failed
John Zhuge created HADOOP-13787:
---
Summary: testGlobStatusThrowsExceptionForUnreadableDir failed
Key: HADOOP-13787
URL: https://issues.apache.org/jira/browse/HADOOP-13787
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Affects Versions: 3.0.0-alpha2
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
Test
{{TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir}}
failed in trunk:
{noformat}
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
FAILURE! - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
Time elapsed: 1.111 sec <<< FAILURE!
java.lang.AssertionError: Should throw IOException
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Results :
Failed tests:
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
Should throw IOException
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-7352) FileSystem#listStatus should throw IOE upon access error
[
https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630868#comment-15630868
]
Mingliang Liu commented on HADOOP-7352:
---
Is {{testGlobStatusThrowsExceptionForUnreadableDir}} failing in {{trunk}}?
Thanks,
I got:
{quote}
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.182 sec <<<
FAILURE! - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
testGlobStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
Time elapsed: 1.111 sec <<< FAILURE!
java.lang.AssertionError: Should throw IOException
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir(FSMainOperationsBaseTest.java:643)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Results :
Failed tests:
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
Should throw IOException
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
{quote}
> FileSystem#listStatus should throw IOE upon access error
>
>
> Key: HADOOP-7352
> URL: https://issues.apache.org/jira/browse/HADOOP-7352
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.6.0
>Reporter: Matt Foley
>Assignee: John Zhuge
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-7352.001.patch, HADOOP-7352.002.patch,
> HADOOP-7352.003.patch, HADOOP-7352.004.patch, HADOOP-7352.005.patch
>
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should
> throw FileNotFoundException instead of returning null, when the target
> directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus
> still may return null, when the target directory exists but does not grant
> read permission. This causes NPE in many callers, for all the reasons cited
> in HADOOP-6201 and HDFS-538. See HADOOP-7327 and its linked issues for
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13427) Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13427:
---
Attachment: HADOOP-13427.004.patch
So we'll see less test failures with v4 patch?
{{org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked}} is
failing without this patch. I posted comment in [HADOOP-7352].
Findbugs warnings are interesting. Seems existing behavior; will confirm this.
> Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
> -
>
> Key: HADOOP-13427
> URL: https://issues.apache.org/jira/browse/HADOOP-13427
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch,
> HADOOP-13427.003.patch, HADOOP-13427.004.patch
>
>
> We're cleaning up Hive and Spark's use of FileSystem.exists, because it is
> often the case we see code of exists+open, exists+delete, when the exists
> probe is needless. Against object stores, expensive needless.
> Hadoop can set an example here by stripping them out. It will also show where
> there are opportunities to optimise things better and/or improve reporting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630773#comment-15630773
]
Hadoop QA commented on HADOOP-13781:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836672/HADOOP-13781.6.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux de882b0c0aa8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f2800f5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10965/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10965/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch,
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Attachment: HADOOP-13781.6.patch
Fixed new generated checkstyle issues
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha2
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch, HADOOP-13781.5.patch, HADOOP-13781.6.patch,
> HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Affects Version/s: (was: 3.0.0-alpha1)
(was: 2.8.0)
3.0.0-alpha2
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 3.0.0-alpha2
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch, HADOOP-13781.5.patch, HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630483#comment-15630483
]
Hudson commented on HADOOP-13717:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10755 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10755/])
Revert "HADOOP-13717. Normalize daemonization behavior of the (wang: rev
f2800f523d4a4f9ff9c3286f9a839d081dcb5b96)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630448#comment-15630448
]
Anu Engineer commented on HADOOP-13717:
---
[~andrew.wang]Thanks for reverting it.
[~aw] Would this be a good time for us to see if we should fix how to discover
which commands are daemons and which are normal commands in hdfs command set ?
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang reopened HADOOP-13717:
--
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HADOOP-13717.
--
Resolution: Invalid
Fix Version/s: (was: 3.0.0-alpha2)
Reverted and reclosing, thanks again Anu for catching this.
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13578) Add Codec for ZStandard Compression
[
https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630357#comment-15630357
]
Hadoop QA commented on HADOOP-13578:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-project-dist . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 37s{color} | {color:orange} root: The patch generated 56 new + 91 unchanged
- 1 fixed = 147 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
14s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 132 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-project-dist . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 33s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
31s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630254#comment-15630254
]
Hadoop QA commented on HADOOP-13781:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 4 new + 50 unchanged - 0 fixed = 54 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
0s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836642/HADOOP-13781.5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f7ee8784b68c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0dc2a6a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10963/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10963/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10963/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
>
[jira] [Commented] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630241#comment-15630241
]
Hadoop QA commented on HADOOP-13720:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
0s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
29s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Invocation of toString on
org.apache.hadoop.security.token.Token.getIdentifier() in
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(Token,
String) At AbstractDelegationTokenSecretManager.java:in
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(Token,
String) At AbstractDelegationTokenSecretManager.java:[line 512] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13720 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836643/HADOOP-13720.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ea3b344a599e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0dc2a6a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10964/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
|
| Test Results |
[jira] [Commented] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630136#comment-15630136
]
Steve Loughran commented on HADOOP-13720:
-
looks pretty good. I think the bit where {{" currentTime: " + now}} is
included in an error message needs to convert {{now}} from a long into a human
readable time though.
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12804) Read Proxy Password from Credential Providers in S3 FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630129#comment-15630129
]
Steve Loughran commented on HADOOP-12804:
-
right. There's one little detail that still needs to be worked on; this patch
won't raise an exception if the username is defined but the password isn't,
because the check for a username being valid only happens if the password is
defined.
{code}
141 if (proxyPass != null) {
142 proxyPassword = new String(proxyPass).trim();
143 if ((proxyUsername == null) || (proxyPassword == null)) {
// proxyPassword is never null here
144 String msg = "Proxy error: " + PROXY_USERNAME + " or " +
145 PROXY_PASSWORD + " set without the other.";
146 LOG.error(msg);
147 throw new IllegalArgumentException(msg);
148 }
{code}
Does that matter? It's good form to provide useful diagnostics here.
> Read Proxy Password from Credential Providers in S3 FileSystem
> --
>
> Key: HADOOP-12804
> URL: https://issues.apache.org/jira/browse/HADOOP-12804
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Larry McCay
>Assignee: Larry McCay
>Priority: Minor
> Attachments: HADOOP-12804-001.patch, HADOOP-12804-003.patch,
> HADOOP-12804-004.patch, HADOOP-12804-005.patch,
> HADOOP-12804-branch-2-002.patch, HADOOP-12804-branch-2-003.patch
>
>
> HADOOP-12548 added credential provider support for the AWS credentials to
> S3FileSystem. This JIRA is for considering the use of the credential
> providers for the proxy password as well.
> Instead of adding the proxy password to the config file directly and in clear
> text, we could provision it in addition to the AWS credentials into a
> credential provider and keep it out of clear text.
> In terms of usage, it could be added to the same credential store as the AWS
> credentials or potentially to a more universally available path - since it is
> the same for everyone. This would however require multiple providers to be
> configured in the provider.path property and more open file permissions on
> the store itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630107#comment-15630107
]
Yongjun Zhang commented on HADOOP-13720:
Thanks [~xiaochen] much for the review, Very good comment!
I just uploaded a new rev, with a lot more other similar changes, and we can
change the summary of this jira to "Add more information to
AbstractDelegationTokenSecretManager for better supportability"
Would you please take a look again?
+ [~ste...@apache.org] too.
Thank you guys.
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Attachment: HADOOP-13781.5.patch
Added test ensuring that {{ActiveStandbyElector}} constructor throws the
expected exception when trying to connect with {{failFast = false}} and do not
fail silently ensuring {{ZKFailoverController#initZK}} works properly. Also the
log now records the number of attempts and not just the exception thrown
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch, HADOOP-13781.5.patch, HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13720) Add more info to "token ... is expired" message
[
https://issues.apache.org/jira/browse/HADOOP-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-13720:
---
Attachment: HADOOP-13720.002.patch
> Add more info to "token ... is expired" message
> ---
>
> Key: HADOOP-13720
> URL: https://issues.apache.org/jira/browse/HADOOP-13720
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HADOOP-13720.001.patch, HADOOP-13720.002.patch
>
>
> Currently AbstractDelegationTokenSecretManager$checkToken does
> {code}
> protected DelegationTokenInformation checkToken(TokenIdent identifier)
> throws InvalidToken {
> assert Thread.holdsLock(this);
> DelegationTokenInformation info = getTokenInfo(identifier);
> if (info == null) {
> throw new InvalidToken("token (" + identifier.toString()
> + ") can't be found in cache");
> }
> if (info.getRenewDate() < Time.now()) {
> throw new InvalidToken("token (" + identifier.toString() + ") is
> expired");
> }
> return info;
> }
> {code}
> When a token is expried, we throw the above exception without printing out
> the {{info.getRenewDate()}} in the message. If we print it out, we could know
> for how long the token has not been renewed. This will help us investigate
> certain issues.
> Create this jira as a request to add that part.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630005#comment-15630005 ] Steve Loughran commented on HADOOP-13345: - +1 for rebasing; I know the S3AFS class is a moving target, but regular resyncs will stop things divering/breaking each other > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630005#comment-15630005 ] Steve Loughran edited comment on HADOOP-13345 at 11/2/16 6:40 PM: -- +1 for merging; I know the S3AFS class is a moving target, but regular resyncs will stop things divering/breaking each other was (Author: ste...@apache.org): +1 for rebasing; I know the S3AFS class is a moving target, but regular resyncs will stop things divering/breaking each other > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629966#comment-15629966 ] Lei (Eddy) Xu commented on HADOOP-13345: Thanks, [~fabbri]. It works for me. Looking forward to it. > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13578) Add Codec for ZStandard Compression
[ https://issues.apache.org/jira/browse/HADOOP-13578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HADOOP-13578: Attachment: HADOOP-13578.v2.patch Here is the streaming version of the patch. [~jlowe] it passed all scenarios which failed for you using the block based approach. Thank you for the review. > Add Codec for ZStandard Compression > --- > > Key: HADOOP-13578 > URL: https://issues.apache.org/jira/browse/HADOOP-13578 > Project: Hadoop Common > Issue Type: New Feature >Reporter: churro morales >Assignee: churro morales > Attachments: HADOOP-13578.patch, HADOOP-13578.v1.patch, > HADOOP-13578.v2.patch > > > ZStandard: https://github.com/facebook/zstd has been used in production for 6 > months by facebook now. v1.0 was recently released. Create a codec for this > library. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13712) S3A open to avoid needless HEAD on the successful execution path
[
https://issues.apache.org/jira/browse/HADOOP-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-13712.
-
Resolution: Won't Fix
> S3A open to avoid needless HEAD on the successful execution path
>
>
> Key: HADOOP-13712
> URL: https://issues.apache.org/jira/browse/HADOOP-13712
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>
> S3A's open() operation does a {{getFileStatus()}} check to see if a file is
> not a directory before opening with a GET. That initial check will take up at
> least one HEAD request if the file is present, more if it isn't.
> As the GET itself performs the existence check, it is needless. A successful
> GET of a path which doesn't end in "/" means a file was there. The only
> reason a getFileStatus call is needed is to choose which error message to
> display if the path isn't there: is it an FNFE or is it path-is-directory.
> Proposed: reorder the code to do the GET; only if that fails fallback to
> getFileStatus()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13712) S3A open to avoid needless HEAD on the successful execution path
[
https://issues.apache.org/jira/browse/HADOOP-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629959#comment-15629959
]
Steve Loughran commented on HADOOP-13712:
-
We need to know the content length when opening the file. And with lazy seek,
there's no GET request initiated in the open() call —only on the first actual
read(). Which means the requirement "open("nonexistent file) MUST raise FNFE"
won't hold.
closing as a WONTFIX
> S3A open to avoid needless HEAD on the successful execution path
>
>
> Key: HADOOP-13712
> URL: https://issues.apache.org/jira/browse/HADOOP-13712
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>
> S3A's open() operation does a {{getFileStatus()}} check to see if a file is
> not a directory before opening with a GET. That initial check will take up at
> least one HEAD request if the file is present, more if it isn't.
> As the GET itself performs the existence check, it is needless. A successful
> GET of a path which doesn't end in "/" means a file was there. The only
> reason a getFileStatus call is needed is to choose which error message to
> display if the path isn't there: is it an FNFE or is it path-is-directory.
> Proposed: reorder the code to do the GET; only if that fails fallback to
> getFileStatus()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629922#comment-15629922 ] Mingliang Liu commented on HADOOP-13345: That sounds perfect. I think Steve also has some plan about this? Perhaps you can sync with him before rebasing. Thanks, > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629911#comment-15629911 ] Aaron Fabbri commented on HADOOP-13345: --- FYI [~eddyxu], [~liuml07], [~ste...@apache.org], I'd like to rebase the feature branch on latest trunk tomorrow morning. Let me know if this works for you. > S3Guard: Improved Consistency for S3A > - > > Key: HADOOP-13345 > URL: https://issues.apache.org/jira/browse/HADOOP-13345 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13345.prototype1.patch, > S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, > S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch > > > This issue proposes S3Guard, a new feature of S3A, to provide an option for a > stronger consistency model than what is currently offered. The solution > coordinates with a strongly consistent external store to resolve > inconsistencies caused by the S3 eventual consistency model. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13786) add output committer which uses s3guard for consistent O(1) commits to S3
Steve Loughran created HADOOP-13786: --- Summary: add output committer which uses s3guard for consistent O(1) commits to S3 Key: HADOOP-13786 URL: https://issues.apache.org/jira/browse/HADOOP-13786 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0-alpha2 Reporter: Steve Loughran Assignee: Steve Loughran A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly). I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629852#comment-15629852 ] Lei (Eddy) Xu commented on HADOOP-12928: Hi, [~ozawa] Sure. Will check against ZK-FC. > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13781) ZKFailoverController#initZK should use the ActiveStanbyElector constructor with failFast as false
[
https://issues.apache.org/jira/browse/HADOOP-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres Perez updated HADOOP-13781:
--
Affects Version/s: 2.8.0
> ZKFailoverController#initZK should use the ActiveStanbyElector constructor
> with failFast as false
> -
>
> Key: HADOOP-13781
> URL: https://issues.apache.org/jira/browse/HADOOP-13781
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Andres Perez
>Priority: Minor
> Attachments: HADOOP-13781.2.patch, HADOOP-13781.3.patch,
> HADOOP-13781.4.patch, HADOOP-13781.patch
>
>
> YARN-4243 introduced the logic that lets retry establishing the connection
> when initializing the {{ActiveStandbyElector}} adding the parameter
> {{failFast}}.
> {{ZKFailoverController#initZK}} should use this constructor with {{failFast}}
> set to false, to let the ZFKC wait longer for the Zookeeper server be in a
> ready state when first initializing it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13717) Normalize daemonization behavior of the diskbalancer with balancer and mover
[
https://issues.apache.org/jira/browse/HADOOP-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629744#comment-15629744
]
Anu Engineer commented on HADOOP-13717:
---
[~andrew.wang] [~aw] Sorry to comment on this late. Just saw it when I was
looking at the git log. The DiskBalancerCLI runs but for a brief moment when
planning and writes a plan file out. Then when you submit plan it reads the
plan file and sends it datanode via RPC. Both of those AFAIK does not qualify
as a daemon behaviour. So why classify this as a daemon ?
I agree with the balancer comments, it does run for a long time. All long
running activity for diskbalancer is confined within the datanode, Perhaps I am
missing something here ?
> Normalize daemonization behavior of the diskbalancer with balancer and mover
>
>
> Key: HADOOP-13717
> URL: https://issues.apache.org/jira/browse/HADOOP-13717
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13717.001.patch
>
>
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which
> specifies the log location isn't even used here, since the command is being
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629488#comment-15629488
]
Xiao Chen commented on HADOOP-12453:
bq. The TOKEN_KIND/TOKEN_KIND_STR in KMSCP have been used by upstream projects
...
It's marked as {{@InterfaceAudience.Private}} so from hadoop's POV we should be
free to change. Best way I think is to mark the new KMSDT class public and let
downstream use that.
All that said, I'm okay with the current patch since that's not within hadoop's
control. +1 pending checkstyle fixes. Thanks Xiaoyu.
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch, HADOOP-12453.01.patch,
> HADOOP-12453.02.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629312#comment-15629312 ] Kihwal Lee commented on HADOOP-13742: - There still is a race. > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12705) Upgrade Jackson 2.2.3 to 2.5.3 or later
[ https://issues.apache.org/jira/browse/HADOOP-12705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629174#comment-15629174 ] PJ Fanning commented on HADOOP-12705: - I think only 2.7.6 and 2.8.x have the XEE fix. > Upgrade Jackson 2.2.3 to 2.5.3 or later > --- > > Key: HADOOP-12705 > URL: https://issues.apache.org/jira/browse/HADOOP-12705 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran > Attachments: HADOOP-12705.002.patch, HADOOP-12705.01.patch, > HADOOP-13050-001.patch > > > There's no rush to do this; this is just the JIRA to track versions. However, > without the upgrade, things written for Jackson 2.4.4 can break ( SPARK-12807) > being Jackson, this is a potentially dangerous update. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13742) Expose "NumOpenConnectionsPerUser" as a metric
[ https://issues.apache.org/jira/browse/HADOOP-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629172#comment-15629172 ] Brahma Reddy Battula commented on HADOOP-13742: --- [~kihwal] can you please check once..? > Expose "NumOpenConnectionsPerUser" as a metric > -- > > Key: HADOOP-13742 > URL: https://issues.apache.org/jira/browse/HADOOP-13742 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HADOOP-13742-002.patch, HADOOP-13742.patch > > > To track user level connections( How many connections for each user) in busy > cluster where so many connections to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13667) Fix typing mistake of inline document in hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629108#comment-15629108
]
Hudson commented on HADOOP-13667:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10753 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10753/])
HADOOP-13667. Fix typing mistake of inline document in (templedf: rev
0dc2a6a3a50add18e959475032264b04aff18fc4)
* (edit)
hadoop-common-project/hadoop-common/src/main/conf/hadoop-metrics2.properties
> Fix typing mistake of inline document in hadoop-metrics2.properties
> ---
>
> Key: HADOOP-13667
> URL: https://issues.apache.org/jira/browse/HADOOP-13667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Rui Gao
>Assignee: Rui Gao
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13667.2.patch, HADOOP-13667.patch
>
>
> Fix typing mistake of inline document in hadoop-metrics2.properties.
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcesName{code} should be
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcessName{code}
> And also could add examples into the inline document for easier understanding
> of metrics tag related configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12705) Upgrade Jackson 2.2.3 to 2.5.3 or later
[ https://issues.apache.org/jira/browse/HADOOP-12705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629064#comment-15629064 ] Steve Loughran commented on HADOOP-12705: - This upgrade broke rumen? So it's probably going to break other code down below. Is there a version of Jackson which (a) fixes the XEE bug and (b) remains compatible at compile/run time with existing work. We know 2.6.6 does (b) > Upgrade Jackson 2.2.3 to 2.5.3 or later > --- > > Key: HADOOP-12705 > URL: https://issues.apache.org/jira/browse/HADOOP-12705 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran > Attachments: HADOOP-12705.002.patch, HADOOP-12705.01.patch, > HADOOP-13050-001.patch > > > There's no rush to do this; this is just the JIRA to track versions. However, > without the upgrade, things written for Jackson 2.4.4 can break ( SPARK-12807) > being Jackson, this is a potentially dangerous update. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13050) Upgrade to AWS SDK 10.10+ for Java 8u60+
[ https://issues.apache.org/jira/browse/HADOOP-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629065#comment-15629065 ] Steve Loughran commented on HADOOP-13050: - Jackson update should go under HADOOP-12705; see discussion there on versions > Upgrade to AWS SDK 10.10+ for Java 8u60+ > > > Key: HADOOP-13050 > URL: https://issues.apache.org/jira/browse/HADOOP-13050 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: HADOOP-13050-001.patch, HADOOP-13050-branch-2.002.patch, > HADOOP-13050-branch-2.003.patch > > > HADOOP-13044 highlights that AWS SDK 10.6 —shipping in Hadoop 2.7+, doesn't > work on open jdk >= 8u60, because a change in the JDK broke the version of > Joda time that AWS uses. > Fix, update the JDK. Though, that implies updating http components: > HADOOP-12767. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13667) Fix typing mistake of inline document in hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated HADOOP-13667:
--
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Thanks for the fix, [~demongaorui]. Committed to trunk.
> Fix typing mistake of inline document in hadoop-metrics2.properties
> ---
>
> Key: HADOOP-13667
> URL: https://issues.apache.org/jira/browse/HADOOP-13667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Rui Gao
>Assignee: Rui Gao
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13667.2.patch, HADOOP-13667.patch
>
>
> Fix typing mistake of inline document in hadoop-metrics2.properties.
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcesName{code} should be
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcessName{code}
> And also could add examples into the inline document for easier understanding
> of metrics tag related configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629032#comment-15629032
]
Steve Loughran commented on HADOOP-13650:
-
Overall, like the code, but I do want the (proposed) s3 entry point to cover
all things, including s3guard. One port of call for s3 operations. This patch
could go in as an interim measure, with a separate JIRA "migrate to s3cmd"
being another subtask, one dependent on s3cmd existing.
h3. S3GuardTool
# L90: why not just switch on the string, have the default: clause raise the
exception. Same for Line 438
# Isn't the S3aFS going to create/init its store? So no need to do that again
in this class, just ask
for the (inited) one in S3AFS
# importDir should use listFiles(path, recursive=true) for scalability;
recursive directory walks die on S3.
There's one catch: listFiles only returns files, not directories. This could be
the time to implement one that returns the full tree, which we can start off in
s3a, then pull up into the full FS API later.
h3. MetadataStoreTestBase
Can the {{assertTrue() calls have meaningful strings to throw on failures?}}
h3. TestS3GuardCommand
* {{ContractTestUtils}} has util functions to create/touch files, etc. Not
worth importing for creating 1-byte files though, unless you want to use the
touch() call.
* {{testDiffCommand}} should include {{out.toString()}} in is assertion errors,
so that the failing logs get printed.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13667) Fix typing mistake of inline document in hadoop-metrics2.properties
[
https://issues.apache.org/jira/browse/HADOOP-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629033#comment-15629033
]
Daniel Templeton commented on HADOOP-13667:
---
Alrighty then. +1 from me.
> Fix typing mistake of inline document in hadoop-metrics2.properties
> ---
>
> Key: HADOOP-13667
> URL: https://issues.apache.org/jira/browse/HADOOP-13667
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Rui Gao
>Assignee: Rui Gao
> Attachments: HADOOP-13667.2.patch, HADOOP-13667.patch
>
>
> Fix typing mistake of inline document in hadoop-metrics2.properties.
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcesName{code} should be
> {code}#*.sink.ganglia.tagsForPrefix.jvm=ProcessName{code}
> And also could add examples into the inline document for easier understanding
> of metrics tag related configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13427) Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628982#comment-15628982
]
Steve Loughran commented on HADOOP-13427:
-
both these cases are pushing into the exception handler what was was going on
already; essentially: better diagnostics
> Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
> -
>
> Key: HADOOP-13427
> URL: https://issues.apache.org/jira/browse/HADOOP-13427
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch,
> HADOOP-13427.003.patch
>
>
> We're cleaning up Hive and Spark's use of FileSystem.exists, because it is
> often the case we see code of exists+open, exists+delete, when the exists
> probe is needless. Against object stores, expensive needless.
> Hadoop can set an example here by stripping them out. It will also show where
> there are opportunities to optimise things better and/or improve reporting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13427) Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
[
https://issues.apache.org/jira/browse/HADOOP-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628981#comment-15628981
]
Steve Loughran commented on HADOOP-13427:
-
we can do a joint vote on the final patch, perhaps.
tests are still failing though; we'll have to look at that
> Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}
> -
>
> Key: HADOOP-13427
> URL: https://issues.apache.org/jira/browse/HADOOP-13427
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13427-001.patch, HADOOP-13427-002.patch,
> HADOOP-13427.003.patch
>
>
> We're cleaning up Hive and Spark's use of FileSystem.exists, because it is
> often the case we see code of exists+open, exists+delete, when the exists
> probe is needless. Against object stores, expensive needless.
> Hadoop can set an example here by stripping them out. It will also show where
> there are opportunities to optimise things better and/or improve reporting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13785) Clean up tests that conditionally handle IBM_JAVA
[
https://issues.apache.org/jira/browse/HADOOP-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628825#comment-15628825
]
Steve Loughran commented on HADOOP-13785:
-
Note here that it's often setting up security things for tests, and because
most people don't use IBM java, we not only don't see the problems, we don't
catch regressions. having it in one place should aid maintenance
> Clean up tests that conditionally handle IBM_JAVA
> -
>
> Key: HADOOP-13785
> URL: https://issues.apache.org/jira/browse/HADOOP-13785
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.8.0
>Reporter: Xiao Chen
>Priority: Minor
>
> Initially from a [review
> comment|https://issues.apache.org/jira/browse/HADOOP-13590?focusedCommentId=15625141=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15625141]
> by [~ste...@apache.org].
> In some tests we see code like below:
> {code}
> if (PlatformName.IBM_JAVA) {
> options.put("useKeytab", keytab.getPath());
> options.put("credsType", "both");
> } else {
> options.put("keyTab", keytab.getPath());
> options.put("useKeyTab", "true");
> options.put("storeKey", "true");
> options.put("doNotPrompt", "true");
> options.put("useTicketCache", "true");
> options.put("renewTGT", "true");
> options.put("isInitiator", Boolean.toString(true));
> }
> {code}
> We should extract a util function to some degree, and clean them up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13783) Improve efficiency of WASB over page blobs
[
https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628817#comment-15628817
]
Steve Loughran commented on HADOOP-13783:
-
1. telemetry is always good; maybe make a separate JIRA. FWIW, S3a will print
out its metrics on a toString() call of an FS instance or output stream, so
letting downstream projects log the stats on a call without making changes to
their code which only works on 2.8+ to use the storage stats code. Worth
copying.
There's a class in the Swift FS client.
{{org.apache.hadoop.fs.swift.util.DurationStats}} which is used to track stats
on HTTP calls, including min/max/variance; I used this to track down throttling
of DELETE requests against one swift service. That class could be moved into
Hadoop-common and used in WASB as well as other blobstores, where it can
isolate problems at the HTTPS level
2. Interesting to hear that azure has blob placement issues on key name too; so
does S3 —the traditional layout of hive-friendly datasets appears to be
suboptimal for object stores. It'd be good to help work out a layout policy
which Hive can use that lays out datasets better
> Improve efficiency of WASB over page blobs
> --
>
> Key: HADOOP-13783
> URL: https://issues.apache.org/jira/browse/HADOOP-13783
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>
> 1)Add telemetry to WASB driver. WASB driver is lack of any log or
> telemetry which makes trouble shoot very difficult. For example, we don’t
> know where is high e2e latency between HBase and Azure storage came from when
> Azure storage server latency was very low. Also we don’t know why WASB can
> only do 166 IOPs which is way below azure storage 500 IOPs. And we had
> several incidents before related to storage latency, because of lacking logs,
> we couldn’t find the ownership of the incident quickly.
> 2)Resolving the hot spotting issue when WASB driver partition the azure
> page blobs by changing the key. Current key design is causing the hot
> spotting on azure storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13075: Parent Issue: HADOOP-11694 (was: HADOOP-13204) > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628670#comment-15628670
]
ASF GitHub Bot commented on HADOOP-13075:
-
Github user steveloughran commented on the issue:
https://github.com/apache/hadoop/pull/113
@elad best to discuss that on the JIRA itself; FWIW, EMR's S3 connector is
Amazon's codebase, so what we do in the ASF doesn't have any impact there. We
are striving to be better :-)
Status: we've gone and broken it in Hadoop branch-2 by replacing
{{S3AFastOutputStream}} with {{S3ABlockOutputStream}};
@fedecz —could you bring this up to sync with branch-2 and I'll add
reviewing it to my todo list. thanks
> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Federico Czerwinski
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5].
> With native support in aws-java-sdk already available it should be fairly
> straightforward [6],[7] to support the other two types of SSE with some
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-12928: Target Version/s: 3.0.0-alpha1, 2.9.0 Status: Open (was: Patch Available) > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628096#comment-15628096 ] Tsuyoshi Ozawa edited comment on HADOOP-12928 at 11/2/16 7:49 AM: -- [~eddyxu] Thanks for your patch, my review comment is as follows: 1. Let's update zookeeper.version to 3.4.9 to sync with netty's version. It's compatible change, according to changelog of ZK. https://zookeeper.apache.org/doc/r3.4.9/releasenotes.html 2. Let's check ZK-FC and RM-HA work well after the dependency update, including the zk dependency upgrade as I mentioned 1. I will check RM-HA's behavior, so could you check ZK-FC change? was (Author: ozawa): [~eddyxu] 1. Let's update zookeeper.version to 3.4.9 to sync with netty's version. It's compatible change, according to changelog of ZK. https://zookeeper.apache.org/doc/r3.4.9/releasenotes.html 2. Let's check ZK-FC and RM-HA work well after upgrading. I will check RM-HA's behavior, so could you check ZK-FC change with your patch + the zk dependency upgrade? > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628096#comment-15628096 ] Tsuyoshi Ozawa commented on HADOOP-12928: - [~eddyxu] 1. Let's update zookeeper.version to 3.4.9 to sync with netty's version. It's compatible change, according to changelog of ZK. https://zookeeper.apache.org/doc/r3.4.9/releasenotes.html 2. Let's check ZK-FC and RM-HA work well after upgrading. I will check RM-HA's behavior, so could you check ZK-FC change with your patch + the zk dependency upgrade? > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628057#comment-15628057
]
Hadoop QA commented on HADOOP-12453:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-common-project: The patch generated 3 new
+ 132 unchanged - 0 fixed = 135 total (was 132) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
19s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
7s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-12453 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836514/HADOOP-12453.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0687035d3048 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cb5cc0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10961/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10961/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10961/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT
[jira] [Commented] (HADOOP-12928) Update netty to 3.10.5.Final to sync with zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628034#comment-15628034 ] Tsuyoshi Ozawa commented on HADOOP-12928: - [~andrew.wang] Thanks for your ping! [~eddyxu] OK, I will check it. > Update netty to 3.10.5.Final to sync with zookeeper > --- > > Key: HADOOP-12928 > URL: https://issues.apache.org/jira/browse/HADOOP-12928 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.7.2 >Reporter: Hendy Irawan >Assignee: Lei (Eddy) Xu > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12928-branch-2.00.patch, > HADOOP-12928-branch-2.01.patch, HADOOP-12928.01.patch, HADOOP-12928.02.patch, > HDFS-12928.00.patch > > > Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper > 3.4.6 which depends on netty 3.7.x. Related to HADOOP-12927 > Pull request: https://github.com/apache/hadoop/pull/85 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12453) Support decoding KMS Delegation Token with its own Identifier
[
https://issues.apache.org/jira/browse/HADOOP-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-12453:
Attachment: HADOOP-12453.02.patch
Thanks [~xiaochen] for the review. Update the patch based on the feedback. The
TOKEN_KIND/TOKEN_KIND_STR in KMSCP have been used by upstream projects such as
Ranger KMS and other KMS server implementations. Keep it as-is to make the
change backward compatible.
> Support decoding KMS Delegation Token with its own Identifier
> -
>
> Key: HADOOP-12453
> URL: https://issues.apache.org/jira/browse/HADOOP-12453
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, security
>Affects Versions: 2.4.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-12453.00.patch, HADOOP-12453.01.patch,
> HADOOP-12453.02.patch
>
>
> kms-dt currently does not have its own token identifier class to decode it
> properly as shown in the HDFS logs below. This JIRA is opened to add support
> for that.
> {code}
> 2015-09-30
> 22:36:14,379|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 WARN token.Token: Cannot find class for token kind kms-dt
> 2015-09-30
> 22:36:14,380|beaver.machine|INFO|5619|140004068153152|MainThread|15/09/30
> 22:36:14 INFO security.TokenCache: Got dt for
> hdfs://tde-hdfs-3.novalocal:8020; Kind: kms-dt, Service: 172.22.64.179:9292,
> Ident: 00 06 68 72 74 5f 71 61 02 72 6d 00 8a 01 50 20 66 1c a3 8a 01 50 44
> 72 a0 a3 0f 03
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13590) Retry until TGT expires even if the UGI renewal thread encountered exception
[
https://issues.apache.org/jira/browse/HADOOP-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627915#comment-15627915
]
Hadoop QA commented on HADOOP-13590:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 5 new + 145 unchanged - 0 fixed = 150 total (was 145) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
56s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12836504/HADOOP-13590.09.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 043e4538fbfa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cb5cc0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10960/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10960/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10960/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Retry until TGT expires even if the UGI renewal thread encountered exception
>
>
> Key: HADOOP-13590
> URL: https://issues.apache.org/jira/browse/HADOOP-13590
> Project: Hadoop Common
> Issue Type: Improvement
>
[jira] [Commented] (HADOOP-13037) Azure Data Lake Client: Support Azure data lake as a file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627882#comment-15627882
]
Chris Douglas commented on HADOOP-13037:
Thanks for working through the build issues, and following up on the review
feedback [~vishwajeet.dusane].
This may not require a branch. While the patch is large, much of it is removing
code (to sever the WebHDFS dependency) and adding new tests. The former can be
committed directly to trunk, and the latter we can commit in one or more
followup issues (as subtasks of HADOOP-13257?). Any objections to this approach?
A few minor comments from skimming the latest patch:
* {{AccessTokenProviderCache}} uses an unusual singleton pattern. Does it need
to create a private instance of itself, rather than initializing a static map
directly?
* {{AdlPermission#equals}} and {{#hashcode}} implementations are identical to
what they override. Instead of reading from the config to report {{getAclBit}}
(which is expensive, and unstable) it could either store the boolean or return
basic {{FsPermission}} instances when disabled (and conversely,
{{AdlPermission}} can always return true when the config knob is set). That
this is a configurable boolean is surprising, and could be explained in this
class's (empty) javadoc.
* {{TokenProviderType::fromString}} is unused
* In initialize, {{accountFQDN}} init can move up, so the
{{accessTokenProvider}} init can be contiguous
* Both the subclasses of {{AzureADTokenProvider}} throw exceptions for all
abstract methods save init, and appear not to use the private instance they
configure. The init pattern here is pretty complicated, with {{initialize}}
creating an instance and adding it to a global cache, and not using the
reference it retains. Can this be simplified? If the instance were returned,
{{AccessTokenProviderCache}} additions can be moved to {{AdlFileSystem}}, so
all the get/add calls are in one class.
* The {{TokenProviderCache}} type seems unnecessary, if {{AzureADTP}} instances
can return null to prevent caching (instead of the instanceof/cast)
* It's a matter of taste, but importing all the static members of
{{AdlConfKeys}} in {{AdlFileSystem}} (or none of them) is easier to read.
* The {{}} tags are unnecessary in javadocs. Some are also oddly formatted.
* Some of the test javadocs still refer to {{BufferManager.java}} and related
classes that have been removed
> Azure Data Lake Client: Support Azure data lake as a file system in Hadoop
> --
>
> Key: HADOOP-13037
> URL: https://issues.apache.org/jira/browse/HADOOP-13037
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/azure, tools
>Reporter: Shrikant Naidu
>Assignee: Vishwajeet Dusane
> Fix For: 2.9.0
>
> Attachments: HADOOP-13037 Proposal.pdf, HADOOP-13037-001.patch,
> HADOOP-13037-002.patch, HADOOP-13037-003.patch
>
>
> The jira proposes an improvement over HADOOP-12666 to remove webhdfs
> dependencies from the ADL file system client and build out a standalone
> client. At a high level, this approach would extend the Hadoop file system
> class to provide an implementation for accessing Azure Data Lake. The scheme
> used for accessing the file system will continue to be
> adl://.azuredatalake.net/path/to/file.
> The Azure Data Lake Cloud Store will continue to provide a webHDFS rest
> interface. The client will access the ADLS store using WebHDFS Rest APIs
> provided by the ADLS store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12483) Maintain wrapped SASL ordering for postponed IPC responses
[ https://issues.apache.org/jira/browse/HADOOP-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12483: --- Fix Version/s: 2.7.4 I backported this bug fix to branch-2.7 since I just backported HADOOP-10300. > Maintain wrapped SASL ordering for postponed IPC responses > -- > > Key: HADOOP-12483 > URL: https://issues.apache.org/jira/browse/HADOOP-12483 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: HADOOP-12483.patch > > > A SASL encryption algorithm (wrapping) may have a required ordering for > encrypted responses. The IPC layer encrypts when the response is set based > on the assumption it is being immediately sent. Postponed responses violate > that assumption. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
