[jira] [Updated] (HADOOP-16933) Backport HADOOP-16890- "ABFS: Change in expiry calculation for MSI token provider" & HADOOP-16825 "ITestAzureBlobFileSystemCheckAccess failing" to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16933: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Backport HADOOP-16890- "ABFS: Change in expiry calculation for MSI token > provider" & HADOOP-16825 "ITestAzureBlobFileSystemCheckAccess failing" to > branch-2 > --- > > Key: HADOOP-16933 > URL: https://issues.apache.org/jira/browse/HADOOP-16933 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Bilahari T H >Priority: Minor > Fix For: 2.11.0 > > > Backport "ABFS: Change in expiry calculation for MSI token provider" to > branch-2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16922) ABFS: Change in User-Agent header
[ https://issues.apache.org/jira/browse/HADOOP-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16922: -- Description: * Add more inforrmation to the User-Agent header like cluster name, cluster type, java vendor etc. * Add APN/1.0 in the begining was: Add more inforrmation to the User-Agent header like cluster name, cluster type, java vendor etc. Add APN/1.0 in the begining > ABFS: Change in User-Agent header > - > > Key: HADOOP-16922 > URL: https://issues.apache.org/jira/browse/HADOOP-16922 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Fix For: 3.3.1 > > > * Add more inforrmation to the User-Agent header like cluster name, cluster > type, java vendor etc. > * Add APN/1.0 in the begining -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16922) ABFS: Change in User-Agent header
[ https://issues.apache.org/jira/browse/HADOOP-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilahari T H updated HADOOP-16922: -- Description: Add more inforrmation to the User-Agent header like cluster name, cluster type, java vendor etc. Add APN/1.0 in the begining was:Move the configured prefix from the end of the User-Agent value to right after the driver version. > ABFS: Change in User-Agent header > - > > Key: HADOOP-16922 > URL: https://issues.apache.org/jira/browse/HADOOP-16922 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Fix For: 3.3.1 > > > Add more inforrmation to the User-Agent header like cluster name, cluster > type, java vendor etc. > Add APN/1.0 in the begining -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ctest updated HADOOP-16958:
---
Attachment: HADOOP-16958.001.patch
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch, HADOOP-16958.001.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077820#comment-17077820
]
Ctest commented on HADOOP-16958:
Add HADOOP-16958.001.patch to fix as many *checkstyle* problems as possible.
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch, HADOOP-16958.001.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1945: HDFS-15269. NameNode should check the authorization API version only …
hadoop-yetus commented on issue #1945: HDFS-15269. NameNode should check the authorization API version only … URL: https://github.com/apache/hadoop/pull/1945#issuecomment-610740611 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 20s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 23m 43s | trunk passed | | +1 :green_heart: | compile | 1m 16s | trunk passed | | +1 :green_heart: | checkstyle | 0m 49s | trunk passed | | +1 :green_heart: | mvnsite | 1m 23s | trunk passed | | +1 :green_heart: | shadedclient | 18m 7s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 47s | trunk passed | | +0 :ok: | spotbugs | 3m 21s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 18s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 18s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | -0 :warning: | checkstyle | 0m 43s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 74 unchanged - 0 fixed = 75 total (was 74) | | +1 :green_heart: | mvnsite | 1m 16s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 16m 0s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | | -1 :x: | findbugs | 3m 27s | hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 123m 17s | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 50s | The patch does not generate ASF License warnings. | | | | 200m 40s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs | | | Null passed for non-null parameter of FSDirectory.setINodeAttributeProvider(INodeAttributeProvider) in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopCommonServices() At FSNamesystem.java:of FSDirectory.setINodeAttributeProvider(INodeAttributeProvider) in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopCommonServices() At FSNamesystem.java:[line 1297] | | Failed junit tests | hadoop.hdfs.server.namenode.TestINodeAttributeProvider | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1945/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1945 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux b2bb8221c257 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 20eec95 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1945/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1945/1/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1945/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1945/1/testReport/ | | Max. process+thread count | 3349 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1945/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at:
[jira] [Commented] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077804#comment-17077804
]
Hadoop QA commented on HADOOP-16958:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 41s{color} | {color:orange} root: The patch generated 12 new + 50 unchanged
- 0 fixed = 62 total (was 50) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
46s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
34s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}231m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
| JIRA Issue | HADOOP-16958 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12999277/HADOOP-16958.000.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fc7eb4792fb9 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 20eec95 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[GitHub] [hadoop] jojochuang opened a new pull request #1945: HDFS-15269. NameNode should check the authorization API version only …
jojochuang opened a new pull request #1945: HDFS-15269. NameNode should check the authorization API version only … URL: https://github.com/apache/hadoop/pull/1945 …once during initialization. Change-Id: I83b2146164b996c14c685cf9a7c62d2f48ec2315 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077667#comment-17077667
]
Ctest commented on HADOOP-16958:
[~ayushtkn] Thx for the quick reply, I submitted a patch with a UT.
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ctest updated HADOOP-16958:
---
Attachment: (was: HADOOP-16958.000.patch)
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ctest updated HADOOP-16958:
---
Attachment: HADOOP-16958.000.patch
Status: Patch Available (was: Open)
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ctest updated HADOOP-16958:
---
Flags: Patch
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16958) NullPointerException(NPE) when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL
[
https://issues.apache.org/jira/browse/HADOOP-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ctest updated HADOOP-16958:
---
Attachment: HADOOP-16958.000.patch
> NullPointerException(NPE) when hadoop.security.authorization is enabled but
> the input PolicyProvider for ZKFCRpcServer is NULL
> --
>
> Key: HADOOP-16958
> URL: https://issues.apache.org/jira/browse/HADOOP-16958
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, ha
>Affects Versions: 3.2.1
>Reporter: Ctest
>Priority: Critical
> Attachments: HADOOP-16958.000.patch
>
>
> During initialization, ZKFCRpcServer refreshes the service authorization ACL
> for the service handled by this server if config
> hadoop.security.authorization is enabled, by calling refreshServiceAcl with
> the input PolicyProvider and Configuration.
> {code:java}
> ZKFCRpcServer(Configuration conf,
> InetSocketAddress bindAddr,
> ZKFailoverController zkfc,
> PolicyProvider policy) throws IOException {
> this.server = ...
>
> // set service-level authorization security policy
> if (conf.getBoolean(
> CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION, false)) {
> server.refreshServiceAcl(conf, policy);
> }
> }{code}
> refreshServiceAcl calls
> ServiceAuthorizationManager#refreshWithLoadedConfiguration which directly
> gets services from the provider with provider.getServices(). When the
> provider is NULL, the code throws NPE without an informative message. In
> addition, the default value of config
> `hadoop.security.authorization.policyprovider` (which controls PolicyProvider
> here) is NULL and the only usage of ZKFCRpcServer initializer provides only
> an abstract method getPolicyProvider which does not enforce that
> PolicyProvider should not be NULL.
> The suggestion here is to either add a guard check or exception handling with
> an informative logging message on ZKFCRpcServer to handle input
> PolicyProvider being NULL.
>
> I am very happy to provide a patch for it if the issue is confirmed :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
hadoop-yetus commented on issue #1899: HADOOP-16914 Adding Output Stream Counters in ABFS URL: https://github.com/apache/hadoop/pull/1899#issuecomment-610641314 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 12s | trunk passed | | +1 :green_heart: | compile | 0m 30s | trunk passed | | +1 :green_heart: | checkstyle | 0m 24s | trunk passed | | +1 :green_heart: | mvnsite | 0m 35s | trunk passed | | +1 :green_heart: | shadedclient | 15m 13s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 26s | trunk passed | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 49s | trunk passed | | -0 :warning: | patch | 1m 8s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | checkstyle | 0m 14s | the patch passed | | +1 :green_heart: | mvnsite | 0m 24s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 5s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | | -1 :x: | findbugs | 0m 57s | hadoop-tools/hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | -1 :x: | unit | 1m 48s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 58m 21s | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-tools/hadoop-azure | | | Increment of volatile field org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl.queueShrunkOps in org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl.queueShrunk() At AbfsOutputStreamStatisticsImpl.java:in org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl.queueShrunk() At AbfsOutputStreamStatisticsImpl.java:[line 118] | | | Increment of volatile field org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl.writeCurrentBufferOperations in org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl.writeCurrentBuffer() At AbfsOutputStreamStatisticsImpl.java:in org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl.writeCurrentBuffer() At AbfsOutputStreamStatisticsImpl.java:[line 129] | | Failed junit tests | hadoop.fs.azurebfs.services.TestAbfsClientThrottlingAnalyzer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1899 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a734d6832366 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 20eec95 | | Default Java | 1.8.0_242 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/5/artifact/out/new-findbugs-hadoop-tools_hadoop-azure.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/5/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/5/testReport/ | | Max. process+thread count | 414 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1899/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405109621
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
+ for (int i = 0; i < largeValue; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+}
+ }
+
+ /**
+ * Tests to check time spent on waiting for tasks to be complete on a
+ * blocking queue in {@link
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405106456
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStreamStatistics.java
##
@@ -0,0 +1,296 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStreamStatistics
+extends AbstractAbfsIntegrationTest {
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ public ITestAbfsOutputStreamStatistics() throws Exception {
+ }
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = path(getMethodName());
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatisticsForUploadBytes =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for zero bytes To upload.
+ assertValues("bytes to upload", 0,
+ abfsOutputStreamStatisticsForUploadBytes.getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ abfsOutputStreamStatisticsForUploadBytes =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload.
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatisticsForUploadBytes.getBytesToUpload());
+
+ //Test for successful bytes uploaded.
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatisticsForUploadBytes.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed.
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatisticsForUploadBytes.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload.
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatisticsForUploadBytes.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ for (int i = 0; i < LARGE_OPERATIONS; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload.
+ assertValues("bytes to upload",
+ LARGE_OPERATIONS * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded.
+ assertValues("successful bytes uploaded",
+ LARGE_OPERATIONS * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed.
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload.
+ assertValues("bytes failed to upload", randomBytesFailed,
+
[jira] [Commented] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[
https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077532#comment-17077532
]
Hadoop QA commented on HADOOP-16960:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} hadoop-tools_hadoop-azure-datalake generated 0 new +
5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
| JIRA Issue | HADOOP-16960 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12999262/HADOOP-16960.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 1653b9b68f0f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 20eec95 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16824/testReport/ |
| Max. process+thread count | 312 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
hadoop-tools/hadoop-azure-datalake |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16824/console |
| Powered by | Apache Yetus
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405059806
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -133,58 +133,60 @@ public void testAbfsOutputStreamTimeSpendOnWaitTask()
throws IOException {
public void testAbfsOutputStreamQueueShrink() throws IOException {
describe("Testing Queue Shrink calls in AbfsOutputStream");
final AzureBlobFileSystem fs = getFileSystem();
-Path TEST_PATH = new Path("AbfsOutputStreamStatsPath");
+Path queueShrinkFilePath = new Path("AbfsOutputStreamStatsPath");
AzureBlobFileSystemStore abfss = fs.getAbfsStore();
abfss.getAbfsConfiguration().setDisableOutputStreamFlush(false);
FileSystem.Statistics statistics = fs.getFsStatistics();
String testQueueShrink = "testQueue";
-
AbfsOutputStream outForOneOp = null;
try {
- outForOneOp = (AbfsOutputStream) abfss.createFile(TEST_PATH, statistics,
-true,
- FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ outForOneOp =
+ (AbfsOutputStream) abfss.createFile(queueShrinkFilePath, statistics,
+ true,
+ FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
//Test for shrinking Queue zero time
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 0,
+ assertValues("number of queueShrink() Calls", 0,
outForOneOp.getOutputStreamStatistics().queueShrink);
outForOneOp.write(testQueueShrink.getBytes());
// Queue is shrunk 2 times when outStream is flushed
outForOneOp.flush();
//Test for shrinking Queue 2 times
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 2,
+ assertValues("number of queueShrink() Calls", 2,
outForOneOp.getOutputStreamStatistics().queueShrink);
} finally {
- if(outForOneOp != null){
+ if (outForOneOp != null) {
outForOneOp.close();
}
}
AbfsOutputStream outForLargeOps = null;
try {
- outForLargeOps = (AbfsOutputStream) abfss.createFile(TEST_PATH,
+ outForLargeOps = (AbfsOutputStream) abfss.createFile(queueShrinkFilePath,
statistics, true,
FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ int largeValue = 1000;
//QueueShrink is called 2 times in 1 flush(), hence 1000 flushes must
// give 2000 QueueShrink calls
- for (int i = 0; i < 1000; i++) {
+ for (int i = 0; i < largeValue; i++) {
outForLargeOps.write(testQueueShrink.getBytes());
//Flush is quite expensive so 1000 calls only which takes 1 min+
outForLargeOps.flush();
Review comment:
No I can't, basically calling flush after I write means the write task is
done. This would trigger the shrinkWriteOperationQueue() method and we need to
do it after each write to get 10 calls to shrinkWriteOperationQueue() .
If I flush after the loop, it would take all the write calls as 1 write
operation and only 1 time the shrinkWriteOperationQueue() method is triggered.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405061359
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -384,6 +398,7 @@ private synchronized void
flushWrittenBytesToServiceInternal(final long offset,
* operation FIFO queue.
*/
private synchronized void shrinkWriteOperationQueue() throws IOException {
+outputStreamStatistics.queueShrinked();
Review comment:
It is an expensive method. So, I thought it would benefit in knowing how
many times it's being called after some write operations.
Sorry for the spelling mistake.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405059806
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -133,58 +133,60 @@ public void testAbfsOutputStreamTimeSpendOnWaitTask()
throws IOException {
public void testAbfsOutputStreamQueueShrink() throws IOException {
describe("Testing Queue Shrink calls in AbfsOutputStream");
final AzureBlobFileSystem fs = getFileSystem();
-Path TEST_PATH = new Path("AbfsOutputStreamStatsPath");
+Path queueShrinkFilePath = new Path("AbfsOutputStreamStatsPath");
AzureBlobFileSystemStore abfss = fs.getAbfsStore();
abfss.getAbfsConfiguration().setDisableOutputStreamFlush(false);
FileSystem.Statistics statistics = fs.getFsStatistics();
String testQueueShrink = "testQueue";
-
AbfsOutputStream outForOneOp = null;
try {
- outForOneOp = (AbfsOutputStream) abfss.createFile(TEST_PATH, statistics,
-true,
- FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ outForOneOp =
+ (AbfsOutputStream) abfss.createFile(queueShrinkFilePath, statistics,
+ true,
+ FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
//Test for shrinking Queue zero time
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 0,
+ assertValues("number of queueShrink() Calls", 0,
outForOneOp.getOutputStreamStatistics().queueShrink);
outForOneOp.write(testQueueShrink.getBytes());
// Queue is shrunk 2 times when outStream is flushed
outForOneOp.flush();
//Test for shrinking Queue 2 times
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 2,
+ assertValues("number of queueShrink() Calls", 2,
outForOneOp.getOutputStreamStatistics().queueShrink);
} finally {
- if(outForOneOp != null){
+ if (outForOneOp != null) {
outForOneOp.close();
}
}
AbfsOutputStream outForLargeOps = null;
try {
- outForLargeOps = (AbfsOutputStream) abfss.createFile(TEST_PATH,
+ outForLargeOps = (AbfsOutputStream) abfss.createFile(queueShrinkFilePath,
statistics, true,
FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ int largeValue = 1000;
//QueueShrink is called 2 times in 1 flush(), hence 1000 flushes must
// give 2000 QueueShrink calls
- for (int i = 0; i < 1000; i++) {
+ for (int i = 0; i < largeValue; i++) {
outForLargeOps.write(testQueueShrink.getBytes());
//Flush is quite expensive so 1000 calls only which takes 1 min+
outForLargeOps.flush();
Review comment:
No I can't, basically calling flush after I write means the write task is
done. This would trigger the shrinkWriteOperationQueue() method and we need to
do it after each write to get 20 operations.
If I flush after the loop, it would take all the write calls as 1 write
operation and only 1 time the shrinkWriteOperationQueue() method is triggered.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405059806
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -133,58 +133,60 @@ public void testAbfsOutputStreamTimeSpendOnWaitTask()
throws IOException {
public void testAbfsOutputStreamQueueShrink() throws IOException {
describe("Testing Queue Shrink calls in AbfsOutputStream");
final AzureBlobFileSystem fs = getFileSystem();
-Path TEST_PATH = new Path("AbfsOutputStreamStatsPath");
+Path queueShrinkFilePath = new Path("AbfsOutputStreamStatsPath");
AzureBlobFileSystemStore abfss = fs.getAbfsStore();
abfss.getAbfsConfiguration().setDisableOutputStreamFlush(false);
FileSystem.Statistics statistics = fs.getFsStatistics();
String testQueueShrink = "testQueue";
-
AbfsOutputStream outForOneOp = null;
try {
- outForOneOp = (AbfsOutputStream) abfss.createFile(TEST_PATH, statistics,
-true,
- FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ outForOneOp =
+ (AbfsOutputStream) abfss.createFile(queueShrinkFilePath, statistics,
+ true,
+ FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
//Test for shrinking Queue zero time
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 0,
+ assertValues("number of queueShrink() Calls", 0,
outForOneOp.getOutputStreamStatistics().queueShrink);
outForOneOp.write(testQueueShrink.getBytes());
// Queue is shrunk 2 times when outStream is flushed
outForOneOp.flush();
//Test for shrinking Queue 2 times
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 2,
+ assertValues("number of queueShrink() Calls", 2,
outForOneOp.getOutputStreamStatistics().queueShrink);
} finally {
- if(outForOneOp != null){
+ if (outForOneOp != null) {
outForOneOp.close();
}
}
AbfsOutputStream outForLargeOps = null;
try {
- outForLargeOps = (AbfsOutputStream) abfss.createFile(TEST_PATH,
+ outForLargeOps = (AbfsOutputStream) abfss.createFile(queueShrinkFilePath,
statistics, true,
FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ int largeValue = 1000;
//QueueShrink is called 2 times in 1 flush(), hence 1000 flushes must
// give 2000 QueueShrink calls
- for (int i = 0; i < 1000; i++) {
+ for (int i = 0; i < largeValue; i++) {
outForLargeOps.write(testQueueShrink.getBytes());
//Flush is quite expensive so 1000 calls only which takes 1 min+
outForLargeOps.flush();
Review comment:
No I can't, basically calling flush after I write means the write task is
done. This would trigger the shrinkWriteOperationQueue() method and we need to
do it after each read to get 20 operations.
If I flush after the loop, it would take all the write calls as 1 write
operation and only 1 time the shrinkWriteOperationQueue() method is triggered.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report
read-ahead error back
URL: https://github.com/apache/hadoop/pull/1898#discussion_r405032853
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -289,6 +298,27 @@ private ReadBuffer getFromList(final
Collection list, final AbfsInpu
return null;
}
+ /**
+ * Returns buffers that failed or passed from completed queue
+ * @param stream
+ * @param requestedOffset
+ * @return
+ */
+ private ReadBuffer getBufferFromCompletedQueue(final AbfsInputStream stream,
final long requestedOffset) {
+for (ReadBuffer buffer : completedReadList) {
+ // Buffer is returned if the requestedOffset is at or above buffer's
+ // offset but less than buffer's length or the actual requestedLength
+ if ((buffer.getStream() == stream)
+ && (requestedOffset >= buffer.getOffset())
+ && ((requestedOffset < buffer.getOffset() + buffer.getLength())
Review comment:
Should always be for requested length? That is: we don't care about the
total length of the buffer, only the amount which was requested from the far
end -which may be less?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report
read-ahead error back
URL: https://github.com/apache/hadoop/pull/1898#discussion_r405030875
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -0,0 +1,433 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH;
+
+/**
+ * Unit test AbfsInputStream.
+ */
+public class TestAbfsInputStream extends
+AbstractAbfsIntegrationTest {
+
+ private static final int KILOBYTE = 1024;
+
+ private AbfsRestOperation getMockRestOp() {
+AbfsRestOperation op = mock(AbfsRestOperation.class);
+AbfsHttpOperation httpOp = mock(AbfsHttpOperation.class);
+when(httpOp.getBytesReceived()).thenReturn(1024L);
+when(op.getResult()).thenReturn(httpOp);
+return op;
+ }
+
+ private AbfsClient getMockAbfsClient() {
+// Mock failure for client.read()
+AbfsClient client = mock(AbfsClient.class);
+AbfsPerfTracker tracker = new AbfsPerfTracker(
+"test",
+this.getAccountName(),
+this.getConfiguration());
+when(client.getAbfsPerfTracker()).thenReturn(tracker);
+
+return client;
+ }
+
+ private AbfsInputStream getAbfsInputStream(AbfsClient mockAbfsClient, String
fileName) {
+// Create AbfsInputStream with the client instance
+AbfsInputStream inputStream = new AbfsInputStream(
+mockAbfsClient,
+null,
+FORWARD_SLASH + fileName,
+3 * KILOBYTE,
+1 * KILOBYTE, // Setting read ahead buffer size of 1 KB
+this.getConfiguration().getReadAheadQueueDepth(),
+this.getConfiguration().getTolerateOobAppends(),
+"eTag");
+
+return inputStream;
+ }
+
+ private void queueReadAheads(AbfsInputStream inputStream) {
+// Mimic AbfsInputStream readAhead queue requests
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 0, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 1 * KILOBYTE, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 2 * KILOBYTE, 1 * KILOBYTE);
+ }
+
+ private void verifyReadCallCount(AbfsClient client, int count) throws
+ AzureBlobFileSystemException, InterruptedException {
+// ReadAhead threads are triggered asynchronously.
+// Wait a second before verifying the number of total calls.
+Thread.sleep(1000);
+verify(client, times(count)).read(any(String.class), any(Long.class),
+any(byte[].class), any(Integer.class), any(Integer.class),
+any(String.class));
+ }
+
+ private void checkEvictedStatus(AbfsInputStream inputStream, int position,
boolean expectedToThrowException)
+ throws Exception {
+// Sleep for the eviction threshold time
+
Thread.sleep(ReadBufferManager.getBufferManager().getThresholdAgeMilliseconds()
+ 1000);
+
+// Eviction is done only when AbfsInputStream tries to queue new items.
+// 1 tryEvict will remove 1 eligible item. To ensure that the current test
buffer
+// will get evicted (considering there could be other tests running in
parallel),
+// call tryEvict for the number of items that are there in
completedReadList.
+int numOfCompletedReadListItems =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
+while
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report
read-ahead error back
URL: https://github.com/apache/hadoop/pull/1898#discussion_r405033372
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java
##
@@ -299,11 +327,32 @@ private void clearFromReadAheadQueue(final
AbfsInputStream stream, final long re
}
private int getBlockFromCompletedQueue(final AbfsInputStream stream, final
long position, final int length,
- final byte[] buffer) {
-ReadBuffer buf = getFromList(completedReadList, stream, position);
-if (buf == null || position >= buf.getOffset() + buf.getLength()) {
+ final byte[] buffer) throws
IOException {
+ReadBuffer buf = getBufferFromCompletedQueue(stream, position);
+
+if (buf == null) {
return 0;
}
+
+if (buf.getStatus() == ReadBufferStatus.READ_FAILED) {
+ // Eviction of a read buffer is triggered only when a queue request
comes in
+ // and each eviction attempt tries to find one eligible buffer.
+ // Hence there are chances that an old read-ahead buffer with exception
is still
+ // available. To prevent new read requests to fail due to such old
buffers,
+ // return exception only from buffers that failed within last
THRESHOLD_AGE_MILLISECONDS
+ if ((currentTimeMillis() - (buf.getTimeStamp()) <
THRESHOLD_AGE_MILLISECONDS)) {
Review comment:
why 30s? Any way to tune this for test runs?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report
read-ahead error back
URL: https://github.com/apache/hadoop/pull/1898#discussion_r405028733
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -0,0 +1,433 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH;
+
+/**
+ * Unit test AbfsInputStream.
+ */
+public class TestAbfsInputStream extends
+AbstractAbfsIntegrationTest {
+
+ private static final int KILOBYTE = 1024;
+
+ private AbfsRestOperation getMockRestOp() {
+AbfsRestOperation op = mock(AbfsRestOperation.class);
+AbfsHttpOperation httpOp = mock(AbfsHttpOperation.class);
+when(httpOp.getBytesReceived()).thenReturn(1024L);
+when(op.getResult()).thenReturn(httpOp);
+return op;
+ }
+
+ private AbfsClient getMockAbfsClient() {
+// Mock failure for client.read()
+AbfsClient client = mock(AbfsClient.class);
+AbfsPerfTracker tracker = new AbfsPerfTracker(
+"test",
+this.getAccountName(),
+this.getConfiguration());
+when(client.getAbfsPerfTracker()).thenReturn(tracker);
+
+return client;
+ }
+
+ private AbfsInputStream getAbfsInputStream(AbfsClient mockAbfsClient, String
fileName) {
+// Create AbfsInputStream with the client instance
+AbfsInputStream inputStream = new AbfsInputStream(
+mockAbfsClient,
+null,
+FORWARD_SLASH + fileName,
+3 * KILOBYTE,
+1 * KILOBYTE, // Setting read ahead buffer size of 1 KB
+this.getConfiguration().getReadAheadQueueDepth(),
+this.getConfiguration().getTolerateOobAppends(),
+"eTag");
+
+return inputStream;
+ }
+
+ private void queueReadAheads(AbfsInputStream inputStream) {
+// Mimic AbfsInputStream readAhead queue requests
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 0, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 1 * KILOBYTE, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 2 * KILOBYTE, 1 * KILOBYTE);
+ }
+
+ private void verifyReadCallCount(AbfsClient client, int count) throws
+ AzureBlobFileSystemException, InterruptedException {
+// ReadAhead threads are triggered asynchronously.
+// Wait a second before verifying the number of total calls.
+Thread.sleep(1000);
+verify(client, times(count)).read(any(String.class), any(Long.class),
+any(byte[].class), any(Integer.class), any(Integer.class),
+any(String.class));
+ }
+
+ private void checkEvictedStatus(AbfsInputStream inputStream, int position,
boolean expectedToThrowException)
+ throws Exception {
+// Sleep for the eviction threshold time
+
Thread.sleep(ReadBufferManager.getBufferManager().getThresholdAgeMilliseconds()
+ 1000);
+
+// Eviction is done only when AbfsInputStream tries to queue new items.
+// 1 tryEvict will remove 1 eligible item. To ensure that the current test
buffer
+// will get evicted (considering there could be other tests running in
parallel),
+// call tryEvict for the number of items that are there in
completedReadList.
+int numOfCompletedReadListItems =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
+while
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back URL: https://github.com/apache/hadoop/pull/1898#discussion_r405031623 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java ## @@ -289,6 +298,27 @@ private ReadBuffer getFromList(final Collection list, final AbfsInpu return null; } + /** + * Returns buffers that failed or passed from completed queue Review comment: add trailing . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report
read-ahead error back
URL: https://github.com/apache/hadoop/pull/1898#discussion_r405028966
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -0,0 +1,433 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH;
+
+/**
+ * Unit test AbfsInputStream.
+ */
+public class TestAbfsInputStream extends
+AbstractAbfsIntegrationTest {
+
+ private static final int KILOBYTE = 1024;
+
+ private AbfsRestOperation getMockRestOp() {
+AbfsRestOperation op = mock(AbfsRestOperation.class);
+AbfsHttpOperation httpOp = mock(AbfsHttpOperation.class);
+when(httpOp.getBytesReceived()).thenReturn(1024L);
+when(op.getResult()).thenReturn(httpOp);
+return op;
+ }
+
+ private AbfsClient getMockAbfsClient() {
+// Mock failure for client.read()
+AbfsClient client = mock(AbfsClient.class);
+AbfsPerfTracker tracker = new AbfsPerfTracker(
+"test",
+this.getAccountName(),
+this.getConfiguration());
+when(client.getAbfsPerfTracker()).thenReturn(tracker);
+
+return client;
+ }
+
+ private AbfsInputStream getAbfsInputStream(AbfsClient mockAbfsClient, String
fileName) {
+// Create AbfsInputStream with the client instance
+AbfsInputStream inputStream = new AbfsInputStream(
+mockAbfsClient,
+null,
+FORWARD_SLASH + fileName,
+3 * KILOBYTE,
+1 * KILOBYTE, // Setting read ahead buffer size of 1 KB
+this.getConfiguration().getReadAheadQueueDepth(),
+this.getConfiguration().getTolerateOobAppends(),
+"eTag");
+
+return inputStream;
+ }
+
+ private void queueReadAheads(AbfsInputStream inputStream) {
+// Mimic AbfsInputStream readAhead queue requests
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 0, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 1 * KILOBYTE, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 2 * KILOBYTE, 1 * KILOBYTE);
+ }
+
+ private void verifyReadCallCount(AbfsClient client, int count) throws
+ AzureBlobFileSystemException, InterruptedException {
+// ReadAhead threads are triggered asynchronously.
+// Wait a second before verifying the number of total calls.
+Thread.sleep(1000);
+verify(client, times(count)).read(any(String.class), any(Long.class),
+any(byte[].class), any(Integer.class), any(Integer.class),
+any(String.class));
+ }
+
+ private void checkEvictedStatus(AbfsInputStream inputStream, int position,
boolean expectedToThrowException)
+ throws Exception {
+// Sleep for the eviction threshold time
+
Thread.sleep(ReadBufferManager.getBufferManager().getThresholdAgeMilliseconds()
+ 1000);
+
+// Eviction is done only when AbfsInputStream tries to queue new items.
+// 1 tryEvict will remove 1 eligible item. To ensure that the current test
buffer
+// will get evicted (considering there could be other tests running in
parallel),
+// call tryEvict for the number of items that are there in
completedReadList.
+int numOfCompletedReadListItems =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
+while
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report
read-ahead error back
URL: https://github.com/apache/hadoop/pull/1898#discussion_r405027481
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java
##
@@ -0,0 +1,433 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+import static
org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH;
+
+/**
+ * Unit test AbfsInputStream.
+ */
+public class TestAbfsInputStream extends
+AbstractAbfsIntegrationTest {
+
+ private static final int KILOBYTE = 1024;
+
+ private AbfsRestOperation getMockRestOp() {
+AbfsRestOperation op = mock(AbfsRestOperation.class);
+AbfsHttpOperation httpOp = mock(AbfsHttpOperation.class);
+when(httpOp.getBytesReceived()).thenReturn(1024L);
+when(op.getResult()).thenReturn(httpOp);
+return op;
+ }
+
+ private AbfsClient getMockAbfsClient() {
+// Mock failure for client.read()
+AbfsClient client = mock(AbfsClient.class);
+AbfsPerfTracker tracker = new AbfsPerfTracker(
+"test",
+this.getAccountName(),
+this.getConfiguration());
+when(client.getAbfsPerfTracker()).thenReturn(tracker);
+
+return client;
+ }
+
+ private AbfsInputStream getAbfsInputStream(AbfsClient mockAbfsClient, String
fileName) {
+// Create AbfsInputStream with the client instance
+AbfsInputStream inputStream = new AbfsInputStream(
+mockAbfsClient,
+null,
+FORWARD_SLASH + fileName,
+3 * KILOBYTE,
+1 * KILOBYTE, // Setting read ahead buffer size of 1 KB
+this.getConfiguration().getReadAheadQueueDepth(),
+this.getConfiguration().getTolerateOobAppends(),
+"eTag");
+
+return inputStream;
+ }
+
+ private void queueReadAheads(AbfsInputStream inputStream) {
+// Mimic AbfsInputStream readAhead queue requests
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 0, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 1 * KILOBYTE, 1 * KILOBYTE);
+ReadBufferManager.getBufferManager()
+.queueReadAhead(inputStream, 2 * KILOBYTE, 1 * KILOBYTE);
+ }
+
+ private void verifyReadCallCount(AbfsClient client, int count) throws
+ AzureBlobFileSystemException, InterruptedException {
+// ReadAhead threads are triggered asynchronously.
+// Wait a second before verifying the number of total calls.
+Thread.sleep(1000);
+verify(client, times(count)).read(any(String.class), any(Long.class),
+any(byte[].class), any(Integer.class), any(Integer.class),
+any(String.class));
+ }
+
+ private void checkEvictedStatus(AbfsInputStream inputStream, int position,
boolean expectedToThrowException)
+ throws Exception {
+// Sleep for the eviction threshold time
+
Thread.sleep(ReadBufferManager.getBufferManager().getThresholdAgeMilliseconds()
+ 1000);
+
+// Eviction is done only when AbfsInputStream tries to queue new items.
+// 1 tryEvict will remove 1 eligible item. To ensure that the current test
buffer
+// will get evicted (considering there could be other tests running in
parallel),
+// call tryEvict for the number of items that are there in
completedReadList.
+int numOfCompletedReadListItems =
ReadBufferManager.getBufferManager().getCompletedReadListSize();
+while
[GitHub] [hadoop] mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610556018 @steveloughran @bgaborg @mehakmeet . Not sure why github is not letting me add anybody as reviewer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back URL: https://github.com/apache/hadoop/pull/1898#discussion_r405026217 ## File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java ## @@ -0,0 +1,438 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.fs.azurebfs.services; + +import java.io.IOException; +import java.util.UUID; + +import org.junit.Assert; +import org.junit.Test; + +import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest; +import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException; +import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException; + +import static java.util.UUID.randomUUID; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; +import static org.apache.hadoop.test.LambdaTestUtils.intercept; +import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH; + +/** + * Unit test AbfsInputStream. + */ +public class TestAbfsInputStream extends Review comment: that's always been trouble in the past; major source of maintenance pain in the wasb tests. Better use the assertion This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back
steveloughran commented on a change in pull request #1898: HADOOP-16852: Report read-ahead error back URL: https://github.com/apache/hadoop/pull/1898#discussion_r405023982 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBuffer.java ## @@ -17,11 +17,13 @@ */ package org.apache.hadoop.fs.azurebfs.services; - +import java.io.IOException; Review comment: add space above this line This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
steveloughran commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405023294
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -384,6 +398,7 @@ private synchronized void
flushWrittenBytesToServiceInternal(final long offset,
* operation FIFO queue.
*/
private synchronized void shrinkWriteOperationQueue() throws IOException {
+outputStreamStatistics.queueShrinked();
Review comment:
it would also be "queueShrunk" as a name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
steveloughran commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405022845
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -294,19 +299,23 @@ private synchronized void flushInternalAsync() throws
IOException {
}
private synchronized void writeCurrentBufferToService() throws IOException {
+outputStreamStatistics.writeCurrentBuffer();
Review comment:
yes, let's do that
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
steveloughran commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r405022610
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -279,6 +283,7 @@ public synchronized void close() throws IOException {
threadExecutor.shutdownNow();
}
}
+LOG.debug("Closing AbfsOutputStream ", toString());
Review comment:
let's guard this with a LOG.isDebugEnabled(), because that toString()
operation is doing enough work. Or, use `this` as the argument and have SLF4J
Call this.toString() only if it is printing the log entry
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Status: Open (was: Patch Available) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch, HADOOP-16960.002.patch, > HADOOP-16960.003.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Attachment: HADOOP-16960.003.patch Status: Patch Available (was: Open) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch, HADOOP-16960.002.patch, > HADOOP-16960.003.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1938: HADOOP-16922. abfs: Changing User-Agent header
bilaharith commented on a change in pull request #1938: HADOOP-16922. abfs:
Changing User-Agent header
URL: https://github.com/apache/hadoop/pull/1938#discussion_r405015031
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -679,32 +679,60 @@ public AuthType getAuthType() {
@VisibleForTesting
String initializeUserAgent(final AbfsConfiguration abfsConfiguration,
- final String sslProviderName) {
+ final String sslProviderName) {
+
StringBuilder sb = new StringBuilder();
-sb.append("(JavaJRE ");
+
Review comment:
CLIENT_VERSION includes org.apache.hadoop.util.VersionInfo,getVersion()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1938: HADOOP-16922. abfs: Changing User-Agent header
bilaharith commented on a change in pull request #1938: HADOOP-16922. abfs:
Changing User-Agent header
URL: https://github.com/apache/hadoop/pull/1938#discussion_r405013532
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -679,32 +679,60 @@ public AuthType getAuthType() {
@VisibleForTesting
String initializeUserAgent(final AbfsConfiguration abfsConfiguration,
- final String sslProviderName) {
+ final String sslProviderName) {
Review comment:
There are test cases written to validate this as this method is package
protected and with the @VisibleForTesting.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-11616) Remove workaround for Curator's ChildReaper requiring Guava 15+
[
https://issues.apache.org/jira/browse/HADOOP-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077477#comment-17077477
]
Wei-Chiu Chuang edited comment on HADOOP-11616 at 4/7/20, 6:02 PM:
---
We should do this now that Hadoop is on guava 27 and curator 4.2. Add a target
version so I don't forget.
was (Author: jojochuang):
We should do this now that Hadoop is on guava 27 and curator 4.2.
> Remove workaround for Curator's ChildReaper requiring Guava 15+
> ---
>
> Key: HADOOP-11616
> URL: https://issues.apache.org/jira/browse/HADOOP-11616
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Robert Kanter
>Priority: Major
>
> HADOOP-11612 adds a copy of Curator 2.7.1's {{ChildReaper}} and
> {{TestChildReaper}} with minor modifications to work with Guava 11.0.2. We
> should remove these classes and update any usages to point to Curator itself
> once we update Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11616) Remove workaround for Curator's ChildReaper requiring Guava 15+
[
https://issues.apache.org/jira/browse/HADOOP-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-11616:
-
Target Version/s: 3.4.0 (was: )
> Remove workaround for Curator's ChildReaper requiring Guava 15+
> ---
>
> Key: HADOOP-11616
> URL: https://issues.apache.org/jira/browse/HADOOP-11616
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Robert Kanter
>Priority: Major
>
> HADOOP-11612 adds a copy of Curator 2.7.1's {{ChildReaper}} and
> {{TestChildReaper}} with minor modifications to work with Guava 11.0.2. We
> should remove these classes and update any usages to point to Curator itself
> once we update Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11616) Remove workaround for Curator's ChildReaper requiring Guava 15+
[
https://issues.apache.org/jira/browse/HADOOP-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077477#comment-17077477
]
Wei-Chiu Chuang commented on HADOOP-11616:
--
We should do this now that Hadoop is on guava 27 and curator 4.2.
> Remove workaround for Curator's ChildReaper requiring Guava 15+
> ---
>
> Key: HADOOP-11616
> URL: https://issues.apache.org/jira/browse/HADOOP-11616
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Robert Kanter
>Priority: Major
>
> HADOOP-11612 adds a copy of Curator 2.7.1's {{ChildReaper}} and
> {{TestChildReaper}} with minor modifications to work with Guava 11.0.2. We
> should remove these classes and update any usages to point to Curator itself
> once we update Guava.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[
https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077472#comment-17077472
]
Hadoop QA commented on HADOOP-16960:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} hadoop-tools_hadoop-azure-datalake generated 0 new +
5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange} hadoop-tools/hadoop-azure-datalake: The patch
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
| JIRA Issue | HADOOP-16960 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12999253/HADOOP-16960.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 2b05e928fa2c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 20eec95 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16823/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16823/testReport/ |
| Max. process+thread count |
[jira] [Updated] (HADOOP-16932) distcp copy calls getFileStatus() needlessly and can fail against S3
[ https://issues.apache.org/jira/browse/HADOOP-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16932: Summary: distcp copy calls getFileStatus() needlessly and can fail against S3 (was: Distcp - Error: java.io.FileNotFoundException: No such file or directory) > distcp copy calls getFileStatus() needlessly and can fail against S3 > > > Key: HADOOP-16932 > URL: https://issues.apache.org/jira/browse/HADOOP-16932 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, tools/distcp >Affects Versions: 3.0.0, 3.1.2, 3.2.1 > Environment: Hadoop CDH 6.3 >Reporter: Thangamani Murugasamy >Priority: Minor > > Distcp to AWS s3 was working fine on CDH 5.16 with distcp 2.6.0. but after > upgrade to CDH 6.3 which comes with distcp-3.0 JAR which is through error as > below. > The same error with repeats on Hadoop-distcp-3.2.1.jar as well. Tried with > -direct option in 3.2.1, still same error. > > Error: java.io.FileNotFoundException: No such file or directory: > s3a://X/part-00012-baa6a706-3816-4dfa-ba07-0fb56fd38178-c000.snappy.parquet > at > org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088) > at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:203) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang closed pull request #1941: HADOOP-16924. Update guava version to 28.2-jre DO NOT MERGE
jojochuang closed pull request #1941: HADOOP-16924. Update guava version to 28.2-jre DO NOT MERGE URL: https://github.com/apache/hadoop/pull/1941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16932) Distcp - Error: java.io.FileNotFoundException: No such file or directory
[ https://issues.apache.org/jira/browse/HADOOP-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077428#comment-17077428 ] Hudson commented on HADOOP-16932: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18125 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18125/]) HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail (github: rev 20eec958674a9c343a80c9fccd1383ef7c1b57f5) * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java * (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java > Distcp - Error: java.io.FileNotFoundException: No such file or directory > > > Key: HADOOP-16932 > URL: https://issues.apache.org/jira/browse/HADOOP-16932 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, tools/distcp >Affects Versions: 3.0.0, 3.1.2, 3.2.1 > Environment: Hadoop CDH 6.3 >Reporter: Thangamani Murugasamy >Priority: Minor > > Distcp to AWS s3 was working fine on CDH 5.16 with distcp 2.6.0. but after > upgrade to CDH 6.3 which comes with distcp-3.0 JAR which is through error as > below. > The same error with repeats on Hadoop-distcp-3.2.1.jar as well. Tried with > -direct option in 3.2.1, still same error. > > Error: java.io.FileNotFoundException: No such file or directory: > s3a://X/part-00012-baa6a706-3816-4dfa-ba07-0fb56fd38178-c000.snappy.parquet > at > org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088) > at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:203) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16932) Distcp - Error: java.io.FileNotFoundException: No such file or directory
[ https://issues.apache.org/jira/browse/HADOOP-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077194#comment-17077194 ] Steve Loughran edited comment on HADOOP-16932 at 4/7/20, 4:58 PM: -- merged to trunk and now verifying backports to 3.3.x was (Author: ste...@apache.org): one more thing for the customer: recommend they use -pr on the command line, so as to ensure incremental updates don't copy up unchanged files. The regression which caused this problem has more consequences than just this JIRA > Distcp - Error: java.io.FileNotFoundException: No such file or directory > > > Key: HADOOP-16932 > URL: https://issues.apache.org/jira/browse/HADOOP-16932 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, tools/distcp >Affects Versions: 3.0.0, 3.1.2, 3.2.1 > Environment: Hadoop CDH 6.3 >Reporter: Thangamani Murugasamy >Priority: Minor > > Distcp to AWS s3 was working fine on CDH 5.16 with distcp 2.6.0. but after > upgrade to CDH 6.3 which comes with distcp-3.0 JAR which is through error as > below. > The same error with repeats on Hadoop-distcp-3.2.1.jar as well. Tried with > -direct option in 3.2.1, still same error. > > Error: java.io.FileNotFoundException: No such file or directory: > s3a://X/part-00012-baa6a706-3816-4dfa-ba07-0fb56fd38178-c000.snappy.parquet > at > org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088) > at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:203) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Attachment: HADOOP-16960.002.patch Status: Patch Available (was: Open) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch, HADOOP-16960.002.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Status: Open (was: Patch Available) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch, HADOOP-16960.002.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
steveloughran merged pull request #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 URL: https://github.com/apache/hadoop/pull/1936 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem
steveloughran commented on issue #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem URL: https://github.com/apache/hadoop/pull/1944#issuecomment-610500898 test failures * #1919 * timeout in one of the MR tests, ITestS3ACommitterMRJob.test_200_execute:290 » , when running @ threads = 8. Seen that elsewhere. I'm thinking about how best to test the cleanup logic. * createFile() with block size == 1 (will also need to tweak create() to actually listen to block size) * write 10K worth of blocks & expect failure * verify that there are no pending uploads under the path * and that metrics for abort ops > 0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13873) log DNS addresses on s3a init
[ https://issues.apache.org/jira/browse/HADOOP-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-13873: --- Assignee: Mukund Thakur > log DNS addresses on s3a init > - > > Key: HADOOP-13873 > URL: https://issues.apache.org/jira/browse/HADOOP-13873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Mukund Thakur >Priority: Minor > > HADOOP-13871 has shown that network problems can kill perf, and that it's v. > hard to track down, even if you turn up the logging in hadoop.fs.s3a and > com.amazon layers to debug. > we could maybe improve things by printing out the IPAddress of the s3 > endpoint, as that could help with the network tracing. Printing from within > hadoop shows the one given to S3a, not a different one returned by any load > balancer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem
hadoop-yetus commented on issue #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem URL: https://github.com/apache/hadoop/pull/1944#issuecomment-610477231 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 34s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 1s | trunk passed | | +1 :green_heart: | compile | 0m 35s | trunk passed | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | trunk passed | | +1 :green_heart: | shadedclient | 15m 4s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | trunk passed | | +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 57s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 33s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | -0 :warning: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 4s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 15s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 58m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1944/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1944 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 11a582320463 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / bffb43b | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1944/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1944/1/testReport/ | | Max. process+thread count | 438 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1944/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[
https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077362#comment-17077362
]
Hadoop QA commented on HADOOP-16960:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} hadoop-tools_hadoop-azure-datalake generated 0 new +
5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-azure-datalake: The patch
generated 31 new + 0 unchanged - 0 fixed = 31 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
| JIRA Issue | HADOOP-16960 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12999246/HADOOP-16960.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 0d0a3307d598 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 45362a9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16822/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure-datalake.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16822/testReport/ |
| asflicense |
[GitHub] [hadoop] steveloughran opened a new pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem
steveloughran opened a new pull request #1944: HADOOP-16900. Very large files can be truncated when written through S3AFileSystem URL: https://github.com/apache/hadoop/pull/1944 Contributed by Steve Loughran. WriteOperationsHelper now raises a PathIOException when the number of parts to write is too high. This is cached during block uploads in S3ABlockOutputStream, so that once one has been raised, there's no attempt to upload other blocks, or to complete() the upload as if it was considered a success. S3ABlockOutputStream also caches any failure raised during block write, so things will fail earlier (currently it would only surface in close()), and will abort() if anything went wrong during the upload. This is intended to ensure that * if too many parts are uploaded, the operation fails * if anything causes a block upload to fail, the entire write is considered a failure * after a failure, the upload is never completed * and abort() is always closed to remove any pending data Change-Id: I7ee6cd1defcc4c7ccb1f95b933af2959cd14ddc2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
mukund-thakur commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610433133 2 of my new tests are failing on command line but succeeding in IDE. `[ERROR] ITestS3AFileOperationCost.testCostOfListLocatedStatusOnEmptyDir:141->verifyOperationCount:190->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=0 current=1 diff=1: object_list_requests expected:<0> but was:<1> [ERROR] ITestS3AFileOperationCost.testCostOfListLocatedStatusOnNonEmptyDir:159->verifyOperationCount:190->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Count of object_list_requests starting=2 current=3 diff=1: object_list_requests expected:<0> but was:<1>` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on issue #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc...
tasanuma commented on issue #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... URL: https://github.com/apache/hadoop/pull/1829#issuecomment-610425722 @jojochuang No, I see the logs from namenode in a cluster that doesn't have Routers. I also confirmed it on my local laptop. https://gist.github.com/tasanuma/c066b0d3cadf3be38b2b6d921d4ae28f When FileSystem API is executed, it seems to generate the logs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Attachment: HADOOP-16960.001.patch Status: Patch Available (was: Open) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Attachment: (was: HADOOP-16960.001.patch) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Attachment: HADOOP-16960.001.patch > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1829: HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc.
jojochuang commented on issue #1829: HDFS-14743. Enhance
INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support
Authorization of mkdir, rm, rmdir, copy, move etc...
URL: https://github.com/apache/hadoop/pull/1829#issuecomment-610406367
Hi @tasanuma do you have RBF?
I don't see this message in my test cluster (we don't have RBF) and wonder
if it comes from RBF.
Not familiar with RBF but if each RPC call results in one permission checker
object I can see why there are many messages.
` /**
* Get a new permission checker used for making mount table access
* control. This method will be invoked during each RPC call in router
* admin server.
*
* @return Router permission checker.
* @throws AccessControlException If the user is not authorized.
*/
public static RouterPermissionChecker getPermissionChecker()
throws AccessControlException {
if (!isPermissionEnabled) {
return null;
}
try {
return new RouterPermissionChecker(routerOwner, superGroup,
NameNode.getRemoteUser());
} catch (IOException e) {
throw new AccessControlException(e);
}
}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Release Note: Enabled setting clientId and tenantId for ADL MsiTokenProvider. This allows MSI authentication on VMs with attached multiple MSIs. > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
Vasil Vasilev created HADOOP-16960: -- Summary: Enable setting clientId and tenantId for ADL MsiTokenProvider Key: HADOOP-16960 URL: https://issues.apache.org/jira/browse/HADOOP-16960 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Reporter: Vasil Vasilev AdlFileSystem is [currently using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] a [deprecated constructor for MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], only passing a port setting which is not used anymore. The purpose of this improvement is to update AdlFileSystem to pass optional clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
hadoop-yetus commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 URL: https://github.com/apache/hadoop/pull/1936#issuecomment-610398485 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 21s | trunk passed | | +1 :green_heart: | compile | 0m 28s | trunk passed | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 31s | trunk passed | | +1 :green_heart: | shadedclient | 14m 50s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | trunk passed | | +0 :ok: | spotbugs | 0m 44s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 41s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 26s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | checkstyle | 0m 19s | hadoop-tools/hadoop-distcp: The patch generated 0 new + 159 unchanged - 12 fixed = 159 total (was 171) | | +1 :green_heart: | mvnsite | 0m 24s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 14m 0s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 21s | the patch passed | | +1 :green_heart: | findbugs | 0m 45s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 11m 20s | hadoop-distcp in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 67m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1936 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 050e7d2a70ac 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c12ddbd | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/4/testReport/ | | Max. process+thread count | 414 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1936/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mehakmeet commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404807005
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
+ for (int i = 0; i < largeValue; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+}
+ }
+
+ /**
+ * Tests to check time spent on waiting for tasks to be complete on a
+ * blocking queue in {@link
[GitHub] [hadoop] steveloughran commented on issue #1929: HADOOP-16855. Changing wildfly dependency scope in hadoop-azure to compile
steveloughran commented on issue #1929: HADOOP-16855. Changing wildfly dependency scope in hadoop-azure to compile URL: https://github.com/apache/hadoop/pull/1929#issuecomment-610381121 It's not mandatory for hadoop-azure though, is it? if you want it in the hadoop-dist packaging, let's add it there This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13430) Optimize getFileStatus in S3A
[ https://issues.apache.org/jira/browse/HADOOP-13430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077221#comment-17077221 ] Steve Loughran commented on HADOOP-13430: - Looking at this in HADOOP-13230. * isDirectory/isFile will only do the explicit probes they need * going to add a hint inside about whether a file or dir is expected (e.g mkdirs will look for dirs), and so order the probes appropriately > Optimize getFileStatus in S3A > - > > Key: HADOOP-13430 > URL: https://issues.apache.org/jira/browse/HADOOP-13430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steven K. Wong >Assignee: Steven K. Wong >Priority: Minor > Attachments: HADOOP-13430.001.WIP.patch > > > Currently, S3AFileSystem.getFileStatus(Path f) sends up to 3 requests to S3 > when pathToKey(f) = key = "foo/bar" is a directory: > 1. HEAD key=foo/bar \[continue if not found] > 2. HEAD key=foo/bar/ \[continue if not found] > 3. LIST prefix=foo/bar/ delimiter=/ max-keys=1 > My experience (and generally true, I reckon) is that almost all directories > are nonempty directories without a "fake directory" file (e.g. "foo/bar/"). > Under this condition, request #2 is mostly unhelpful; it only slows down > getFileStatus. Therefore, I propose swapping the order of requests #2 and #3. > The swapped HEAD request will be skipped in practically all cases. > Furthermore, when key = "foo/bar" is a nonempty directory that contains a > "fake directory" file (in addition to actual files), getFileStatus currently > returns an S3AFileStatus with isEmptyDirectory=true, which is wrong. Swapping > will fix this. The swapped LIST request will use max-keys=2 to determine > isEmptyDirectory correctly. (Removing the delimiter from the LIST request > should make the logic a little simpler than otherwise.) > Note that key = "foo/bar/" has the same problem with isEmptyDirectory. To fix > it, I propose skipping request #1 when key ends with "/". The price is this > will, for an empty directory, replace a HEAD request with a LIST request > that's generally more taxing on S3. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16846) add experimental optimization of s3a directory marker handling
[ https://issues.apache.org/jira/browse/HADOOP-16846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16846. - Resolution: Duplicate > add experimental optimization of s3a directory marker handling > -- > > Key: HADOOP-16846 > URL: https://issues.apache.org/jira/browse/HADOOP-16846 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Priority: Major > > Add support for doing "less" about directory markers. Make this an > experimental option for now, to see what breaks. > Rationale > * reduce throttling from deletes > * reduce tombstone markers in versioned buckets -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13873) log DNS addresses on s3a init
[ https://issues.apache.org/jira/browse/HADOOP-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13873: Target Version/s: 3.4.0 (was: 3.3.0) > log DNS addresses on s3a init > - > > Key: HADOOP-13873 > URL: https://issues.apache.org/jira/browse/HADOOP-13873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13871 has shown that network problems can kill perf, and that it's v. > hard to track down, even if you turn up the logging in hadoop.fs.s3a and > com.amazon layers to debug. > we could maybe improve things by printing out the IPAddress of the s3 > endpoint, as that could help with the network tracing. Printing from within > hadoop shows the one given to S3a, not a different one returned by any load > balancer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13873) log DNS addresses on s3a init
[ https://issues.apache.org/jira/browse/HADOOP-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077216#comment-17077216 ] Steve Loughran commented on HADOOP-13873: - do in bucket existence check or async -DNS delays shouldn't hold up init > log DNS addresses on s3a init > - > > Key: HADOOP-13873 > URL: https://issues.apache.org/jira/browse/HADOOP-13873 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13871 has shown that network problems can kill perf, and that it's v. > hard to track down, even if you turn up the logging in hadoop.fs.s3a and > com.amazon layers to debug. > we could maybe improve things by printing out the IPAddress of the s3 > endpoint, as that could help with the network tracing. Printing from within > hadoop shows the one given to S3a, not a different one returned by any load > balancer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16482) S3A doesn't actually verify paths have the correct authority
[
https://issues.apache.org/jira/browse/HADOOP-16482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16482.
-
Fix Version/s: 3.3.0
Assignee: Steve Loughran
Resolution: Duplicate
HADOOP-15430 covers this
> S3A doesn't actually verify paths have the correct authority
>
>
> Key: HADOOP-16482
> URL: https://issues.apache.org/jira/browse/HADOOP-16482
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> Probably been around a *long* time, but we've never noticed, assuming that
> {{Path.makeQualified(uri, workingDir)}} did the right thing.
> You can provide any s3a URI to an S3 command and it'll get mapped to the
> current bucket without any validation that the authorities are equal. Oops.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14564) s3a test can hang in teardown with network problems
[ https://issues.apache.org/jira/browse/HADOOP-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14564. - Resolution: Won't Fix > s3a test can hang in teardown with network problems > --- > > Key: HADOOP-14564 > URL: https://issues.apache.org/jira/browse/HADOOP-14564 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Priority: Minor > > If you've a transient network test and things fail, then the directory > cleanup in teardown can block a test so that it times out entirely. > Proposed: shorten socket timeouts for s3 connections, assuming this is > possible (stack trace implies it was in a read(), not a connect()). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16932) Distcp - Error: java.io.FileNotFoundException: No such file or directory
[ https://issues.apache.org/jira/browse/HADOOP-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077194#comment-17077194 ] Steve Loughran commented on HADOOP-16932: - one more thing for the customer: recommend they use -pr on the command line, so as to ensure incremental updates don't copy up unchanged files. The regression which caused this problem has more consequences than just this JIRA > Distcp - Error: java.io.FileNotFoundException: No such file or directory > > > Key: HADOOP-16932 > URL: https://issues.apache.org/jira/browse/HADOOP-16932 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, tools/distcp >Affects Versions: 3.0.0, 3.1.2, 3.2.1 > Environment: Hadoop CDH 6.3 >Reporter: Thangamani Murugasamy >Priority: Minor > > Distcp to AWS s3 was working fine on CDH 5.16 with distcp 2.6.0. but after > upgrade to CDH 6.3 which comes with distcp-3.0 JAR which is through error as > below. > The same error with repeats on Hadoop-distcp-3.2.1.jar as well. Tried with > -direct option in 3.2.1, still same error. > > Error: java.io.FileNotFoundException: No such file or directory: > s3a://X/part-00012-baa6a706-3816-4dfa-ba07-0fb56fd38178-c000.snappy.parquet > at > org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2255) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2149) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088) > at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:203) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) > at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16947) Expired SampleStat should ignore when generating SlowPeersReport
[
https://issues.apache.org/jira/browse/HADOOP-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077164#comment-17077164
]
Haibin Huang commented on HADOOP-16947:
---
[~elgoiri], dou you have time to look at this? Thanks :)
> Expired SampleStat should ignore when generating SlowPeersReport
>
>
> Key: HADOOP-16947
> URL: https://issues.apache.org/jira/browse/HADOOP-16947
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Haibin Huang
>Assignee: Haibin Huang
>Priority: Major
> Attachments: HADOOP-16947-001.patch, HADOOP-16947-002.patch,
> HADOOP-16947-003.patch, HADOOP-16947-004.patch, HADOOP-16947-005.patch,
> HDFS-14783, HDFS-14783-001.patch, HDFS-14783-002.patch, HDFS-14783-003.patch,
> HDFS-14783-004.patch, HDFS-14783-005.patch
>
>
> SlowPeersReport is generated by the SampleStat between tow dn, so it can
> present on nn's jmx like this:
> {code:java}
> "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
> {code}
> In each period, MutableRollingAverages will do a rollOverAvgs(), it will
> generate a SumAndCount object which is based on SampleStat, and store it in a
> LinkedBlockingDeque, the deque will be used to generate
> SlowPeersReport. And the old member of deque won't be removed until the queue
> is full. However, if dn1 don't send any packet to dn2 in the last of
> 36*300_000 ms, the deque will be filled with an old member, because the
> number of last SampleStat never change.I think these old SampleStats should
> be considered as expired message and ignore them when generating a new
> SlowPeersReport.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation
hadoop-yetus commented on issue #1943: Hadoop 16465 listlocatedstatus optimisation URL: https://github.com/apache/hadoop/pull/1943#issuecomment-610336818 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 6s | trunk passed | | +1 :green_heart: | compile | 0m 45s | trunk passed | | +1 :green_heart: | checkstyle | 0m 29s | trunk passed | | +1 :green_heart: | mvnsite | 0m 48s | trunk passed | | +1 :green_heart: | shadedclient | 19m 33s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 34s | trunk passed | | +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 9s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 41s | the patch passed | | +1 :green_heart: | compile | 0m 33s | the patch passed | | +1 :green_heart: | javac | 0m 33s | the patch passed | | -0 :warning: | checkstyle | 0m 21s | hadoop-tools/hadoop-aws: The patch generated 4 new + 10 unchanged - 0 fixed = 14 total (was 10) | | +1 :green_heart: | mvnsite | 0m 39s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 18m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed | | +1 :green_heart: | findbugs | 1m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 24s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 73m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1943 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8a4843288838 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c12ddbd | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/testReport/ | | Max. process+thread count | 428 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1943/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16937) Multiple Tests failure in hadoop-azure componnt
[
https://issues.apache.org/jira/browse/HADOOP-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077127#comment-17077127
]
Hadoop QA commented on HADOOP-16937:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
3s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1937/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1937 |
| JIRA Issue | HADOOP-16937 |
| Optional Tests | dupname asflicense xml compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle markdownlint |
| uname | Linux a8ad6bcdb77e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c12ddbd |
| Default Java | 1.8.0_242 |
| Test Results |
[GitHub] [hadoop] hadoop-yetus commented on issue #1937: [HADOOP-16937] ABFS: revert combined append+flush calls. with default config update to disable append+flush calls.
hadoop-yetus commented on issue #1937: [HADOOP-16937] ABFS: revert combined append+flush calls. with default config update to disable append+flush calls. URL: https://github.com/apache/hadoop/pull/1937#issuecomment-610324927 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 26m 31s | trunk passed | | +1 :green_heart: | compile | 0m 37s | trunk passed | | +1 :green_heart: | checkstyle | 0m 30s | trunk passed | | +1 :green_heart: | mvnsite | 0m 45s | trunk passed | | +1 :green_heart: | shadedclient | 18m 10s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | trunk passed | | +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 59s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 36s | the patch passed | | +1 :green_heart: | compile | 0m 27s | the patch passed | | +1 :green_heart: | javac | 0m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 20s | the patch passed | | +1 :green_heart: | mvnsite | 0m 37s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 17m 38s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 28s | the patch passed | | +1 :green_heart: | findbugs | 1m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 33s | The patch does not generate ASF License warnings. | | | | 73m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1937/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1937 | | JIRA Issue | HADOOP-16937 | | Optional Tests | dupname asflicense xml compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux a8ad6bcdb77e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c12ddbd | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1937/3/testReport/ | | Max. process+thread count | 340 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1937/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16859) ABFS: Add unbuffer support to AbfsInputStream
[
https://issues.apache.org/jira/browse/HADOOP-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16859:
Fix Version/s: (was: 3.4.0)
3.3.0
> ABFS: Add unbuffer support to AbfsInputStream
> -
>
> Key: HADOOP-16859
> URL: https://issues.apache.org/jira/browse/HADOOP-16859
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.3.0
>
>
> Added unbuffer support to {{AbfsInputStream}} so that apps can cache ABFS
> file handles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16859) ABFS: Add unbuffer support to AbfsInputStream
[
https://issues.apache.org/jira/browse/HADOOP-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-16859:
Component/s: fs/azure
> ABFS: Add unbuffer support to AbfsInputStream
> -
>
> Key: HADOOP-16859
> URL: https://issues.apache.org/jira/browse/HADOOP-16859
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.3.0
>
>
> Added unbuffer support to {{AbfsInputStream}} so that apps can cache ABFS
> file handles.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16959) resolve hadoop-cos dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16959: Summary: resolve hadoop-cos dependency conflict (was: resolve the dependency conflict) > resolve hadoop-cos dependency conflict > -- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16956) Tencent Cloud COS: Package it into hadoop-dist
[ https://issues.apache.org/jira/browse/HADOOP-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077120#comment-17077120 ] Steve Loughran commented on HADOOP-16956: - it should end up in share/hadoop/tools/lib/ ; add it to hadoop-tools/hadoop-tools-dist/pom.xml downstream releases of hadoop often pull cloud modules up from there to hadoop common lib as needed (e.g HDI does this for azure) You should also add it as a dependency of hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml That pom is intended to export all dependencies for cloud storage *excluding anything which comes from hadoop-common*; it is picked up by applications like spark, hive, druid so ends up in their packages/builds (usually some extra build options to get the cloud stuff in; look at the poms for the projects individually) > Tencent Cloud COS: Package it into hadoop-dist > -- > > Key: HADOOP-16956 > URL: https://issues.apache.org/jira/browse/HADOOP-16956 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/cos >Affects Versions: 3.3.0 >Reporter: YangY >Assignee: YangY >Priority: Major > > [~ste...@apache.org] Hi, Steve, the Hadoop-COS has been submitted to the > trunk branch last year. (Related jira: > https://issues.apache.org/jira/browse/HADOOP-15616). > I am the contributor and the maintainer of this patch. Considering that you > are the expert on the object storage support in Hadoop, I have some questions > for you. > # The Hadoop-cos patch is placed in the *hadoop-cloud-storage-project > directory. How to package it into the hadoop-dist directory as the > distribution package?* > # How to manage the dependencies of the Hadoop-cos, such as cos-java-sdk? I > use the maven to compile and package it, but only the > hadoop-cos-3.4.0-SNAPSHOT.jar is generated. I compare the pom.xml between the > hadoop-cos and hadoop-oss, and find the dependency scope is '*compile*' in > their pom.xml. The hadoop-oss has oss-java-sdk.jar in dist directory, but the > hadoop-cos does not. > Can you guide me how to solve these two problems? > > Thanks for your time and attention. :) > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16959) resolve the dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077114#comment-17077114 ] Steve Loughran commented on HADOOP-16959: - The best way to prevent this is not to declare versions of dependencies in the -cos subdir, but in hadoop-project/pom.xml and pick them up there. if they are coming in from other dependencies, its still good to declare those external ones in hadoop-project and exclude the transient ones causing trouble thanks > resolve the dependency conflict > --- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: YangY >Assignee: YangY >Priority: Major > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16959) resolve the dependency conflict
[ https://issues.apache.org/jira/browse/HADOOP-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16959: Component/s: fs/cos build > resolve the dependency conflict > --- > > Key: HADOOP-16959 > URL: https://issues.apache.org/jira/browse/HADOOP-16959 > Project: Hadoop Common > Issue Type: Sub-task > Components: build, fs/cos >Reporter: YangY >Assignee: YangY >Priority: Major > > There are some dependency conflicts between the Hadoop-common and Hadoop-cos. > For example, joda time lib, HTTP client lib and etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3
steveloughran commented on issue #1936: HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 URL: https://github.com/apache/hadoop/pull/1936#issuecomment-610313950 thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur opened a new pull request #1943: Hadoop 16465 listlocatedstatus optimisation
mukund-thakur opened a new pull request #1943: Hadoop 16465 listlocatedstatus optimisation URL: https://github.com/apache/hadoop/pull/1943 Optimize S3AFileSystem.listLocatedStatus() to perform list operations directly and then fallback to head checks for file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ishaniahuja commented on issue #1937: [HADOOP-16937] ABFS: revert combined append+flush calls. with default config update to disable append+flush calls.
ishaniahuja commented on issue #1937: [HADOOP-16937] ABFS: revert combined append+flush calls. with default config update to disable append+flush calls. URL: https://github.com/apache/hadoop/pull/1937#issuecomment-610285402 [result.txt](https://github.com/apache/hadoop/files/4443273/result.txt) [result_ns.txt](https://github.com/apache/hadoop/files/4443274/result_ns.txt) results with the latest commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers
steveloughran commented on issue #1861: HADOOP-13230. Optionally retain directory markers URL: https://github.com/apache/hadoop/pull/1861#issuecomment-610280135 Operation cost tests failiing; I'm trying to (a) cut down things to a minimum and (b) understand why every head/list call takes place, rather than change the assertions to match what's happening. Hadn't realised until now, for example, that rename(file, dest) has 3 HEAD calls, for example ``` [INFO] [ERROR] Failures: [ERROR] ITestS3AFileOperationCost.testFakeDirectoryDeletion:598->verifyMetrics:490->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 after rename(srcFilePath, destFilePath) [object_delete_requests starting=5 current=8 diff=3, directories_created starting=4 current=5 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_deleted starting=31 current=46 diff=15, object_list_requests starting=20 current=23 diff=3, object_metadata_requests starting=18 current=21 diff=3] dest dir depth=8: directories_created expected:<0> but was:<1> [ERROR] ITestS3AFileOperationCost.testFakeDirectoryDeletion:621->verifyMetrics:490->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 after rename(s3a://stevel-london/test/testFakeDirectoryDeletion%5Braw-keep-markers%5D/1/2/3/4/5/6/source2.txt, s3a://stevel-london/test/testFakeDirectoryDeletion[raw-keep-markers]/1/2/3/4/5/6/source.txt) [object_delete_requests starting=1 current=2 diff=1, directories_created starting=4 current=5 diff=1, directories_deleted starting=0 current=0 diff=0, fake_directories_deleted starting=0 current=0 diff=0, object_list_requests starting=25 current=28 diff=3, object_metadata_requests starting=25 current=28 diff=3] dest dir depth=8: directories_created expected:<0> but was:<1> [INFO] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404618650
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
+ for (int i = 0; i < largeValue; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+}
+ }
+
+ /**
+ * Tests to check time spent on waiting for tasks to be complete on a
+ * blocking queue in {@link
[GitHub] [hadoop] hadoop-yetus commented on issue #1942: MAPREDUCE-7270. TestHistoryViewerPrinter could be failed when the locale isn't English.
hadoop-yetus commented on issue #1942: MAPREDUCE-7270. TestHistoryViewerPrinter could be failed when the locale isn't English. URL: https://github.com/apache/hadoop/pull/1942#issuecomment-610242188 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 42s | trunk passed | | +1 :green_heart: | compile | 0m 42s | trunk passed | | +1 :green_heart: | checkstyle | 0m 37s | trunk passed | | +1 :green_heart: | mvnsite | 0m 49s | trunk passed | | +1 :green_heart: | shadedclient | 19m 26s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 20s | trunk passed | | +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 1m 21s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 26s | the patch passed | | +1 :green_heart: | javac | 0m 26s | the patch passed | | -0 :warning: | checkstyle | 0m 21s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105) | | +1 :green_heart: | mvnsite | 0m 30s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 29s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 15s | the patch passed | | +1 :green_heart: | findbugs | 1m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 5m 25s | hadoop-mapreduce-client-core in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 72m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1942/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1942 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c58d79a68b29 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c12ddbd | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1942/1/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1942/1/testReport/ | | Max. process+thread count | 1079 (vs. ulimit of 5500) | | modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1942/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404611696
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -133,58 +133,60 @@ public void testAbfsOutputStreamTimeSpendOnWaitTask()
throws IOException {
public void testAbfsOutputStreamQueueShrink() throws IOException {
describe("Testing Queue Shrink calls in AbfsOutputStream");
final AzureBlobFileSystem fs = getFileSystem();
-Path TEST_PATH = new Path("AbfsOutputStreamStatsPath");
+Path queueShrinkFilePath = new Path("AbfsOutputStreamStatsPath");
AzureBlobFileSystemStore abfss = fs.getAbfsStore();
abfss.getAbfsConfiguration().setDisableOutputStreamFlush(false);
FileSystem.Statistics statistics = fs.getFsStatistics();
String testQueueShrink = "testQueue";
-
AbfsOutputStream outForOneOp = null;
try {
- outForOneOp = (AbfsOutputStream) abfss.createFile(TEST_PATH, statistics,
-true,
- FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ outForOneOp =
+ (AbfsOutputStream) abfss.createFile(queueShrinkFilePath, statistics,
+ true,
+ FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
//Test for shrinking Queue zero time
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 0,
+ assertValues("number of queueShrink() Calls", 0,
outForOneOp.getOutputStreamStatistics().queueShrink);
outForOneOp.write(testQueueShrink.getBytes());
// Queue is shrunk 2 times when outStream is flushed
outForOneOp.flush();
//Test for shrinking Queue 2 times
- Assert.assertEquals("Mismatch in number of queueShrink() Calls", 2,
+ assertValues("number of queueShrink() Calls", 2,
outForOneOp.getOutputStreamStatistics().queueShrink);
} finally {
- if(outForOneOp != null){
+ if (outForOneOp != null) {
outForOneOp.close();
}
}
AbfsOutputStream outForLargeOps = null;
try {
- outForLargeOps = (AbfsOutputStream) abfss.createFile(TEST_PATH,
+ outForLargeOps = (AbfsOutputStream) abfss.createFile(queueShrinkFilePath,
statistics, true,
FsPermission.getDefault(), FsPermission.getUMask(fs.getConf()));
+ int largeValue = 1000;
//QueueShrink is called 2 times in 1 flush(), hence 1000 flushes must
// give 2000 QueueShrink calls
- for (int i = 0; i < 1000; i++) {
+ for (int i = 0; i < largeValue; i++) {
outForLargeOps.write(testQueueShrink.getBytes());
//Flush is quite expensive so 1000 calls only which takes 1 min+
outForLargeOps.flush();
Review comment:
You can call flush outside the for loop?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404608299
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
+ for (int i = 0; i < largeValue; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+}
+ }
+
+ /**
+ * Tests to check time spent on waiting for tasks to be complete on a
+ * blocking queue in {@link
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404606007
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
Review comment:
Create a new protected method in the base clase to get methode name using
methodName.getMethodName()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404605628
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
+ for (int i = 0; i < largeValue; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+}
+ }
+
+ /**
+ * Tests to check time spent on waiting for tasks to be complete on a
+ * blocking queue in {@link
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404604989
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
Review comment:
Use LARGE_OPERATIONS directly. Why to create a new variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404601085
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -384,6 +398,7 @@ private synchronized void
flushWrittenBytesToServiceInternal(final long offset,
* operation FIFO queue.
*/
private synchronized void shrinkWriteOperationQueue() throws IOException {
+outputStreamStatistics.queueShrinked();
Review comment:
I don't understand the benifits of capturing this metric?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404600592
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -294,19 +299,23 @@ private synchronized void flushInternalAsync() throws
IOException {
}
private synchronized void writeCurrentBufferToService() throws IOException {
+outputStreamStatistics.writeCurrentBuffer();
Review comment:
Should go down after if clause?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404591370
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ * timeSpendOnTaskWait - Time spend on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times
+ * {@code writeCurrentBufferToService()} calls were made.
+ */
+public class AbfsOutputStreamStatisticsImpl
+implements AbfsOutputStreamStatistics {
+ private volatile long bytesToUpload;
+ private volatile long bytesUploadSuccessful;
+ private volatile long bytesUploadFailed;
+ private volatile long timeSpendOnTaskWait;
+ private volatile long queueShrink;
+ private volatile long writeCurrentBufferOperations;
+
+ /**
+ * Number of bytes uploaded.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void bytesToUpload(long bytes) {
+if (bytes > 0) {
+ bytesToUpload += bytes;
+}
+ }
+
+ /**
+ * Upload successful with the number of bytes.
+ * @param bytes number of bytes that were successfully uploaded
+ */
+ @Override
+ public void uploadSuccessful(long bytes) {
+if (bytes > 0) {
+ bytesUploadSuccessful += bytes;
+}
+ }
+
+ /**
+ * Upload failed and the number of bytes.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void uploadFailed(long bytes) {
+if (bytes > 0) {
+ bytesUploadFailed += bytes;
+}
+ }
+
+ /**
+ * Time spent for waiting a task to be completed.
+ *
+ * @param startTime on calling {@code waitForTaskToComplete()}
+ * @param endTime on method completing
+ */
+ @Override
+ public void timeSpentTaskWait(long startTime, long endTime) {
+timeSpendOnTaskWait += endTime - startTime;
+ }
+
+ /**
+ * Number of calls to {@code shrinkWriteOperationQueue()}.
+ */
+ @Override
+ public void queueShrinked() {
Review comment:
How is this metrics important??
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404591627
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ * timeSpendOnTaskWait - Time spend on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times
+ * {@code writeCurrentBufferToService()} calls were made.
+ */
+public class AbfsOutputStreamStatisticsImpl
+implements AbfsOutputStreamStatistics {
+ private volatile long bytesToUpload;
+ private volatile long bytesUploadSuccessful;
+ private volatile long bytesUploadFailed;
+ private volatile long timeSpendOnTaskWait;
+ private volatile long queueShrink;
+ private volatile long writeCurrentBufferOperations;
+
+ /**
+ * Number of bytes uploaded.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void bytesToUpload(long bytes) {
+if (bytes > 0) {
+ bytesToUpload += bytes;
+}
+ }
+
+ /**
+ * Upload successful with the number of bytes.
+ * @param bytes number of bytes that were successfully uploaded
+ */
+ @Override
+ public void uploadSuccessful(long bytes) {
+if (bytes > 0) {
+ bytesUploadSuccessful += bytes;
+}
+ }
+
+ /**
+ * Upload failed and the number of bytes.
+ *
+ * @param bytes negative values are ignored
+ */
+ @Override
+ public void uploadFailed(long bytes) {
+if (bytes > 0) {
+ bytesUploadFailed += bytes;
+}
+ }
+
+ /**
+ * Time spent for waiting a task to be completed.
+ *
+ * @param startTime on calling {@code waitForTaskToComplete()}
+ * @param endTime on method completing
+ */
+ @Override
+ public void timeSpentTaskWait(long startTime, long endTime) {
+timeSpendOnTaskWait += endTime - startTime;
+ }
+
+ /**
+ * Number of calls to {@code shrinkWriteOperationQueue()}.
+ */
+ @Override
+ public void queueShrinked() {
Review comment:
typo in times, small t in try.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404589778
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStreamStatisticsImpl.java
##
@@ -0,0 +1,171 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+/**
+ * OutputStream Statistics Implementation for Abfs.
+ */
+public class AbfsOutputStreamStatisticsImpl
+implements AbfsOutputStreamStatistics {
+
+ /**
+ * timeSpentOnTaskWait - Time spent on waiting for tasks to be complete on
+ * Blocking Queue in AbfsOutputStream.
+ *
+ * queueShrink - Number of times Blocking Queue was shrunk after writing
+ * data.
+ *
+ * WriteCurrentBufferOperations - Number of times the current buffer which
+ * was written has been forwarded to the service.
+ */
Review comment:
Move the java doc to corresponding variables.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404589011
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
+ outForSomeBytes.getOutputStreamStatistics().getBytesToUpload());
+
+ outForSomeBytes.write(testBytesToUpload.getBytes());
+ outForSomeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForSomeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload", testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ testBytesToUpload.getBytes().length,
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random value for bytesFailed
+ int randomBytesFailed = new Random().nextInt(LOW_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("number fo bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+
+}
+
+try (AbfsOutputStream outForLargeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)) {
+
+ int largeValue = LARGE_OPERATIONS;
+ for (int i = 0; i < largeValue; i++) {
+outForLargeBytes.write(testBytesToUpload.getBytes());
+ }
+ outForLargeBytes.flush();
+ AbfsOutputStreamStatisticsImpl abfsOutputStreamStatistics =
+ outForLargeBytes.getOutputStreamStatistics();
+
+ //Test for bytes to upload
+ assertValues("bytes to upload",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesToUpload());
+
+ //Test for successful bytes uploaded
+ assertValues("successful bytes uploaded",
+ largeValue * (testBytesToUpload.getBytes().length),
+ abfsOutputStreamStatistics.getBytesUploadSuccessful());
+
+ //Populating random values for bytesFailed
+ int randomBytesFailed = new
Random().nextInt(HIGH_RANGE_FOR_RANDOM_VALUE);
+ abfsOutputStreamStatistics.uploadFailed(randomBytesFailed);
+ //Test for bytes failed to upload
+ assertValues("bytes failed to upload", randomBytesFailed,
+ abfsOutputStreamStatistics.getBytesUploadFailed());
+}
+ }
+
+ /**
+ * Tests to check time spent on waiting for tasks to be complete on a
+ * blocking queue in {@link
[GitHub] [hadoop] mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding Output Stream Counters in ABFS
mukund-thakur commented on a change in pull request #1899: HADOOP-16914 Adding
Output Stream Counters in ABFS
URL: https://github.com/apache/hadoop/pull/1899#discussion_r404588417
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsOutputStream.java
##
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.io.IOException;
+import java.util.Random;
+
+import org.junit.Test;
+
+import org.apache.hadoop.fs.Path;
+import
org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream;
+import org.apache.hadoop.fs.azurebfs.services.AbfsOutputStreamStatisticsImpl;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * Test AbfsOutputStream statistics.
+ */
+public class ITestAbfsOutputStream extends AbstractAbfsIntegrationTest {
+ public ITestAbfsOutputStream() throws Exception {
+ }
+
+ private static final int LARGE_OPERATIONS = 10;
+ private static final int LOW_RANGE_FOR_RANDOM_VALUE = 49;
+ private static final int HIGH_RANGE_FOR_RANDOM_VALUE = ;
+
+ /**
+ * Tests to check bytes Uploading in {@link AbfsOutputStream}.
+ *
+ * @throws IOException
+ */
+ @Test
+ public void testAbfsOutputStreamUploadingBytes() throws IOException {
+describe("Testing Bytes uploaded in AbfsOutputSteam");
+final AzureBlobFileSystem fs = getFileSystem();
+Path uploadBytesFilePath = new Path("AbfsOutputStreamStatsPath");
+String testBytesToUpload = "bytes";
+
+try (AbfsOutputStream outForSomeBytes = createAbfsOutputStream(fs,
+uploadBytesFilePath)
+) {
+
+ //Test for zero bytes To upload
+ assertValues("bytes to upload", 0,
Review comment:
extract in variable utForSomeBytes.getOutputStreamStatistics(). like it is
done at L65.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
