[jira] [Commented] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[
https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695605#comment-16695605
]
Hadoop QA commented on HADOOP-15602:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-15602 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15602 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949147/HADOOP-15602.03.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15551/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support SASL Rpc request handling in separate Handlers
> ---
>
> Key: HADOOP-15602
> URL: https://issues.apache.org/jira/browse/HADOOP-15602
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HADOOP-15602.01.patch, HADOOP-15602.02.patch,
> HADOOP-15602.03.patch
>
>
> Right now, during RPC Connection establishment, all SASL requests are
> considered as OutOfBand requests and handled within the same Reader thread.
> SASL handling involves authentication with Kerberos and SecretManagers(for
> Token validation). During this time, Reader thread would be blocked, hence
> blocking all the incoming RPC requests on other established connections. Some
> secretManager impls require to communicate to external systems (ex: ZK) for
> verification.
> SASL RPC handling in separate dedicated handlers, would enable Reader threads
> to read RPC requests from established connections without blocking.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[ https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695594#comment-16695594 ] Vinayakumar B commented on HADOOP-15602: Fixed checkstyle errors. > Support SASL Rpc request handling in separate Handlers > --- > > Key: HADOOP-15602 > URL: https://issues.apache.org/jira/browse/HADOOP-15602 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-15602.01.patch, HADOOP-15602.02.patch, > HADOOP-15602.03.patch > > > Right now, during RPC Connection establishment, all SASL requests are > considered as OutOfBand requests and handled within the same Reader thread. > SASL handling involves authentication with Kerberos and SecretManagers(for > Token validation). During this time, Reader thread would be blocked, hence > blocking all the incoming RPC requests on other established connections. Some > secretManager impls require to communicate to external systems (ex: ZK) for > verification. > SASL RPC handling in separate dedicated handlers, would enable Reader threads > to read RPC requests from established connections without blocking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[ https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-15602: --- Attachment: HADOOP-15602.03.patch > Support SASL Rpc request handling in separate Handlers > --- > > Key: HADOOP-15602 > URL: https://issues.apache.org/jira/browse/HADOOP-15602 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-15602.01.patch, HADOOP-15602.02.patch, > HADOOP-15602.03.patch > > > Right now, during RPC Connection establishment, all SASL requests are > considered as OutOfBand requests and handled within the same Reader thread. > SASL handling involves authentication with Kerberos and SecretManagers(for > Token validation). During this time, Reader thread would be blocked, hence > blocking all the incoming RPC requests on other established connections. Some > secretManager impls require to communicate to external systems (ex: ZK) for > verification. > SASL RPC handling in separate dedicated handlers, would enable Reader threads > to read RPC requests from established connections without blocking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695205#comment-16695205
]
Ewan Higgs commented on HADOOP-14693:
-
In various tests (e.g. TestECSchema) we have {{@Rule Timeout globalTimeout =
new Timeout(3000);}} which means the entire test and all child tests have a
timeout set. Junit5 doesn't have this (yet). Discussion is here:
https://github.com/junit-team/junit5/issues/80
I'm not sure if this is a blocker for anyone. If the tests behave well enough
then maybe we can remove the timeouts?
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15229: Status: Open (was: Patch Available) > Add FileSystem builder-based openFile() API to match createFile() > - > > Key: HADOOP-15229 > URL: https://issues.apache.org/jira/browse/HADOOP-15229 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15229-001.patch, HADOOP-15229-002.patch > > > Replicate HDFS-1170 and HADOOP-14365 with an API to open files. > A key requirement of this is not HDFS, it's to put in the fadvise policy for > working with object stores, where getting the decision to do a full GET and > TCP abort on seek vs smaller GETs is fundamentally different: the wrong > option can cost you minutes. S3A and Azure both have adaptive policies now > (first backward seek), but they still don't do it that well. > Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" > "random" as an option when they open files; I can imagine other options too. > The Builder model of [~eddyxu] is the one to mimic, method for method. > Ideally with as much code reuse as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15370) S3A log message on rm s3a://bucket/ not intuitive
[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695062#comment-16695062
]
Hadoop QA commented on HADOOP-15370:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
30s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949076/HADOOP-15370.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6b7cfeba495c 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 34b6fa7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15550/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15550/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> S3A log message on rm s3a://bucket/ not
[jira] [Work started] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-15798 started by Gabor Bota.
---
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694973#comment-16694973
]
Gabor Bota commented on HADOOP-15798:
-
Thanks! I think that's maybe another issue, and not strongly related to this.
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15370) S3A log message on rm s3a://bucket/ not intuitive
[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15370:
Status: Patch Available (was: Open)
> S3A log message on rm s3a://bucket/ not intuitive
> -
>
> Key: HADOOP-15370
> URL: https://issues.apache.org/jira/browse/HADOOP-15370
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Trivial
> Attachments: HADOOP-15370.001.patch
>
>
> when you try to delete the root of a bucket from command line, e.g. {{hadoop
> fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful
> {code}
> 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem
> (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the
> hwdev-steve-new root directory of true
> rm: `s3a://hwdev-steve-new/': Input/output error
> 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
> {code}
> the single log message doesn't parse, and the error message raised is lost by
> the FS -rm CLI command (why?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15370) S3A log message on rm s3a://bucket/ not intuitive
[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694952#comment-16694952
]
Gabor Bota edited comment on HADOOP-15370 at 11/21/18 5:07 PM:
---
Created an initial patch based on the idea that
{{org.apache.hadoop.fs.s3a.S3AFileSystem#innerDelete}} returns what returned
from {{org.apache.hadoop.fs.s3a.S3AFileSystem#rejectRootDirectoryDelete}}, and
the same is returned to {{org.apache.hadoop.fs.s3a.S3AFileSystem#delete}} which
ultimately returned to {{org.apache.hadoop.fs.shell.Delete.Rm#processPath}} and
we have the {{PathIOException}} which outputs the {{Input/output error}}.
Based on that, I found that the best places to add the messages are the ones I
used.
was (Author: gabor.bota):
Created an initial patch based on the idea that
org.apache.hadoop.fs.s3a.S3AFileSystem#innerDelete returns what returned from
org.apache.hadoop.fs.s3a.S3AFileSystem#rejectRootDirectoryDelete, and the same
is returned to org.apache.hadoop.fs.s3a.S3AFileSystem#delete which ultimately
returned to org.apache.hadoop.fs.shell.Delete.Rm#processPath and we have the
{{PathIOException}} which outputs the Input/output error.
Based on that, I found that the best places to add the messages are the ones I
used.
> S3A log message on rm s3a://bucket/ not intuitive
> -
>
> Key: HADOOP-15370
> URL: https://issues.apache.org/jira/browse/HADOOP-15370
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Trivial
> Attachments: HADOOP-15370.001.patch
>
>
> when you try to delete the root of a bucket from command line, e.g. {{hadoop
> fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful
> {code}
> 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem
> (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the
> hwdev-steve-new root directory of true
> rm: `s3a://hwdev-steve-new/': Input/output error
> 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
> {code}
> the single log message doesn't parse, and the error message raised is lost by
> the FS -rm CLI command (why?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15370) S3A log message on rm s3a://bucket/ not intuitive
[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694952#comment-16694952
]
Gabor Bota commented on HADOOP-15370:
-
Created an initial patch based on the idea that
org.apache.hadoop.fs.s3a.S3AFileSystem#innerDelete returns what returned from
org.apache.hadoop.fs.s3a.S3AFileSystem#rejectRootDirectoryDelete, and the same
is returned to org.apache.hadoop.fs.s3a.S3AFileSystem#delete which ultimately
returned to org.apache.hadoop.fs.shell.Delete.Rm#processPath and we have the
{{PathIOException}} which outputs the Input/output error.
Based on that, I found that the best places to add the messages are the ones I
used.
> S3A log message on rm s3a://bucket/ not intuitive
> -
>
> Key: HADOOP-15370
> URL: https://issues.apache.org/jira/browse/HADOOP-15370
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Trivial
> Attachments: HADOOP-15370.001.patch
>
>
> when you try to delete the root of a bucket from command line, e.g. {{hadoop
> fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful
> {code}
> 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem
> (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the
> hwdev-steve-new root directory of true
> rm: `s3a://hwdev-steve-new/': Input/output error
> 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
> {code}
> the single log message doesn't parse, and the error message raised is lost by
> the FS -rm CLI command (why?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15370) S3A log message on rm s3a://bucket/ not intuitive
[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15370:
Attachment: HADOOP-15370.001.patch
> S3A log message on rm s3a://bucket/ not intuitive
> -
>
> Key: HADOOP-15370
> URL: https://issues.apache.org/jira/browse/HADOOP-15370
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Trivial
> Attachments: HADOOP-15370.001.patch
>
>
> when you try to delete the root of a bucket from command line, e.g. {{hadoop
> fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful
> {code}
> 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem
> (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the
> hwdev-steve-new root directory of true
> rm: `s3a://hwdev-steve-new/': Input/output error
> 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
> {code}
> the single log message doesn't parse, and the error message raised is lost by
> the FS -rm CLI command (why?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694773#comment-16694773
]
Steve Loughran commented on HADOOP-15870:
-
Now, code review
{{InputStream.available()}} declares the #of bytes which can be read *without
blocking*. This is not what is currently being calculated for S3A input stream
-it's broken. And this patch, while making it more accurate, isn't doing it
either.
We should just be doing something like
{code}
public synchronized int available() throws IOException {
checkNotClosed();
return wrappedStream != null? wrappedStream.available() : 0;
}
{code}
All the new tests will validate this behaviour
> S3AInputStream.remainingInFile should use nextReadPos
> -
>
> Key: HADOOP-15870
> URL: https://issues.apache.org/jira/browse/HADOOP-15870
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.4, 3.1.1
>Reporter: Shixiong Zhu
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15870-002.patch
>
>
> Otherwise `remainingInFile` will not change after `seek`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15920) get patch for S3a nextReadPos(), through Yetus
[ https://issues.apache.org/jira/browse/HADOOP-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694687#comment-16694687 ] Steve Loughran commented on HADOOP-15920: - OK, the local test FS is failing. I'm going to submit this to HDFS too to see what it says. HDFS and posix are the reference points for what we want the object stores to do, so their test outcomes have to be considered normative unless we can show that they are broken and, for HDFS, fix in a way which doesn't break anyone's code > get patch for S3a nextReadPos(), through Yetus > -- > > Key: HADOOP-15920 > URL: https://issues.apache.org/jira/browse/HADOOP-15920 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.1 >Reporter: Steve Loughran >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-001.diff, HADOOP-15870-002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694685#comment-16694685
]
Steve Loughran commented on HADOOP-15872:
-
Is the object store ready for this patch now?
If so, this patch is nearly ready. One little checkstyle
{code}
./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:92:import
static
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME;:15:
Unused import -
org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_AUTH_TYPE_PROPERTY_NAME.
[UnusedImports]
{code}
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
> URL: https://issues.apache.org/jira/browse/HADOOP-15872
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Thomas Marquardt
>Assignee: junhua gu
>Priority: Major
> Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch,
> HADOOP-15872-003.patch, HADOOP-15872-004.patch
>
>
> This update to the latest REST version (2018-11-09) will make the following
> changes to the ABFS driver:
> 1) The ABFS implementation of getFileStatus currently requires read
> permission. According to HDFS permissions guide, it should only require
> execute on the parent folders (traversal access). A new REST API has been
> introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem.
> 2) The new "2018-11-09" REST version introduces support to i) automatically
> translate UPNs to OIDs when setting the owner, owning group, or ACL and ii)
> optionally translate OIDs to UPNs in the responses when getting the owner,
> owning group, or ACL. Configuration will be introduced to optionally
> translate OIDs to UPNs in the responses. Since translation has a performance
> impact, the default will be to perform no translation and return the OIDs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[
https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694546#comment-16694546
]
Hadoop QA commented on HADOOP-15602:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 26 new + 361 unchanged - 10 fixed = 387 total (was 371) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestDiskCheckerWithDiskIo |
| | hadoop.util.TestReadWriteDiskValidator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15602 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949009/HADOOP-15602.02.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 96195cf1e20d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c8b3dfa |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15549/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15549/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Commented] (HADOOP-14604) Fix the bug of aggregation log have one replication in HDFS Federation environment
[
https://issues.apache.org/jira/browse/HADOOP-14604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694510#comment-16694510
]
Elvis commented on HADOOP-14604:
This patch may have some problems. Before applying this patch, the time between
LOCALIZING and LOCALIZED is very fast. But after applying this patch, the time
between LOCALIZING and LOCALIZED becomes more than 10s
> Fix the bug of aggregation log have one replication in HDFS Federation
> environment
> --
>
> Key: HADOOP-14604
> URL: https://issues.apache.org/jira/browse/HADOOP-14604
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, viewfs
>Affects Versions: 2.6.0
> Environment: Hadoop 2.6.0 with HDFS Federation
>Reporter: dingguotao
>Assignee: Doris Gu
>Priority: Major
> Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14604.patch
>
>
> With HDFS Federation configured, the aggregation log have a default
> replication count of 1, rather than the value of dfs.replication in
> hdfs-site.xml configuration file.
> Before:
> {code:java}
> [op@NM-304-SA5212M4-BIGDATA-640 ~]$ hdfs dfs -ls
> /yarn/apps/op/logs/application_1498544648312_0001
> Found 3 items
> -rw-r- 1 op hadoop 8270 2017-06-27 14:25
> /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-645_8041
> -rw-r- 1 op hadoop 54469 2017-06-27 14:25
> /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-646_8041
> -rw-r- 1 op hadoop 6537 2017-06-27 14:25
> /yarn/apps/op/logs/application_1498544648312_0001/NM-304-SA5212M4-BIGDATA-672_8041
> {code}
> The aggregation log have only 1 replication in default. But the value of
> dfs.replication in hdfs-site.xml configuration file is 3
> After apply this patch:
> {code:java}
> [op@NM-304-SA5212M4-BIGDATA-640 ~]$ hdfs dfs -ls
> /yarn/apps/op/logs/application_1498635214020_0001
> Found 3 items
> -rw-r- 3 op hadoop 5729 2017-06-28 15:34
> /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-645_8041
> -rw-r- 3 op hadoop 38439 2017-06-28 15:34
> /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-646_8041
> -rw-r- 3 op hadoop 8270 2017-06-28 15:34
> /yarn/apps/op/logs/application_1498635214020_0001/NM-304-SA5212M4-BIGDATA-671_8041
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[ https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-15602: --- Attachment: HADOOP-15602.02.patch > Support SASL Rpc request handling in separate Handlers > --- > > Key: HADOOP-15602 > URL: https://issues.apache.org/jira/browse/HADOOP-15602 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-15602.01.patch, HADOOP-15602.02.patch > > > Right now, during RPC Connection establishment, all SASL requests are > considered as OutOfBand requests and handled within the same Reader thread. > SASL handling involves authentication with Kerberos and SecretManagers(for > Token validation). During this time, Reader thread would be blocked, hence > blocking all the incoming RPC requests on other established connections. Some > secretManager impls require to communicate to external systems (ex: ZK) for > verification. > SASL RPC handling in separate dedicated handlers, would enable Reader threads > to read RPC requests from established connections without blocking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers
[ https://issues.apache.org/jira/browse/HADOOP-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694402#comment-16694402 ] Vinayakumar B commented on HADOOP-15602: Rebased and Handled more error cases. For SIMPLE authentications, SASL messages will be processed in the same Reader thread to avoid races. > Support SASL Rpc request handling in separate Handlers > --- > > Key: HADOOP-15602 > URL: https://issues.apache.org/jira/browse/HADOOP-15602 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Attachments: HADOOP-15602.01.patch, HADOOP-15602.02.patch > > > Right now, during RPC Connection establishment, all SASL requests are > considered as OutOfBand requests and handled within the same Reader thread. > SASL handling involves authentication with Kerberos and SecretManagers(for > Token validation). During this time, Reader thread would be blocked, hence > blocking all the incoming RPC requests on other established connections. Some > secretManager impls require to communicate to external systems (ex: ZK) for > verification. > SASL RPC handling in separate dedicated handlers, would enable Reader threads > to read RPC requests from established connections without blocking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
