[jira] [Commented] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196524#comment-16196524 ] John Zhuge commented on HADOOP-14184: - +1 LGTM TestZKFailoverController and TestKDiag failures are unrelated. > Remove service loader config entry for ftp fs > - > > Key: HADOOP-14184 > URL: https://issues.apache.org/jira/browse/HADOOP-14184 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.7.3 >Reporter: John Zhuge >Assignee: Sen Zhao >Priority: Minor > Labels: newbie > Attachments: HADOOP-14184.001.patch, HADOOP-14184.002.patch > > > Per discussion in HADOOP-14132. Remove line > {{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config > file > hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem > and add property {{fs.ftp.impl}} to {{core-default.xml}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196495#comment-16196495 ] Hadoop QA commented on HADOOP-14184: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 26s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 28s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 43s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ha.TestZKFailoverController | | | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14184 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12891003/HADOOP-14184.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 7c6bfe9df39b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2856eb2 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13473/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13473/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output |
[jira] [Commented] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196485#comment-16196485 ] Varada Hemeswari commented on HADOOP-14913: --- [~ste...@apache.org], Can you please review the patch attached? > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14937) initial part uploads seem to block unnecessarily in S3ABlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Rand updated HADOOP-14937: - Attachment: yjp_threads.png > initial part uploads seem to block unnecessarily in S3ABlockOutputStream > > > Key: HADOOP-14937 > URL: https://issues.apache.org/jira/browse/HADOOP-14937 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steven Rand > Attachments: yjp_threads.png > > > From looking at a YourKit snapshot of an FsShell process running a {{hadoop > fs -put file:///... s3a://...}}, it seems that the first part in the > multipart upload doesn't begin to upload until n of the > {{s3a-transfer-shared-pool}} threads are able to start uploading, where n is > the value of {{fs.s3a.fast.upload.active.blocks}}. > To hopefully clarify a bit, the series of events that I expected to see with > {{fs.s3a.fast.upload.active.blocks}} set to 4 is: > 1. An amount of data equal to {{fs.s3a.multipart.size}} is buffered into > off-heap memory (I have {{fs.s3a.fast.upload.buffer = bytebuffer}}). > 2. As soon as that happens, a thread begins to upload that part. Meanwhile, > the main thread continues to buffer data into off-heap memory. > 3. Once another part has been buffered into off-heap memory, a separate > thread uploads that part, and so on. > Whereas what I think the YK snapshot shows happening is: > 1. An amount of data equal to {{fs.s3a.multipart.size}} * 4 is buffered into > off-heap memory. > 2. Four threads start to upload one part each at the same time. > I've attached a picture of the "Threads" tab to show what I mean. Basically > the times at which the first four {{s3a-transfer-shared-pool}} threads start > to upload are roughly the same, whereas I would've expected them to be more > staggered. > I'm actually not sure whether this is the expected behavior or not, so feel > free to close if this doesn't come as a surprise to anyone. > For some context, I've been trying to get a sense for roughly which values of > {{fs.s3a.multipart.size}} perform the best at different file sizes. One thing > that I found confusing is that a part size of 5 MB seems to outperform a part > size of 64 MB up until files that are upwards of about 500 MB in size. This > seems odd, since each {{uploadPart}} call is its own HTTP request, and I > would've expected the overhead of those to become costly at small part sizes. > My suspicion is that with 4 concurrent part uploads and 64 MB blocks, we have > to wait until 256 MB are buffered before we can start uploading, while with 5 > MB blocks we can start uploading as soon as we buffer 20 MB, and that's what > gives the smaller parts the advantage for smaller files. > I'm happy to submit a patch if this is in fact a problem, but wanted to check > to make sure I'm not just misunderstanding something. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14937) initial part uploads seem to block unnecessarily in S3ABlockOutputStream
Steven Rand created HADOOP-14937: Summary: initial part uploads seem to block unnecessarily in S3ABlockOutputStream Key: HADOOP-14937 URL: https://issues.apache.org/jira/browse/HADOOP-14937 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.0.0-beta1 Reporter: Steven Rand >From looking at a YourKit snapshot of an FsShell process running a {{hadoop fs >-put file:///... s3a://...}}, it seems that the first part in the multipart >upload doesn't begin to upload until n of the {{s3a-transfer-shared-pool}} >threads are able to start uploading, where n is the value of >{{fs.s3a.fast.upload.active.blocks}}. To hopefully clarify a bit, the series of events that I expected to see with {{fs.s3a.fast.upload.active.blocks}} set to 4 is: 1. An amount of data equal to {{fs.s3a.multipart.size}} is buffered into off-heap memory (I have {{fs.s3a.fast.upload.buffer = bytebuffer}}). 2. As soon as that happens, a thread begins to upload that part. Meanwhile, the main thread continues to buffer data into off-heap memory. 3. Once another part has been buffered into off-heap memory, a separate thread uploads that part, and so on. Whereas what I think the YK snapshot shows happening is: 1. An amount of data equal to {{fs.s3a.multipart.size}} * 4 is buffered into off-heap memory. 2. Four threads start to upload one part each at the same time. I've attached a picture of the "Threads" tab to show what I mean. Basically the times at which the first four {{s3a-transfer-shared-pool}} threads start to upload are roughly the same, whereas I would've expected them to be more staggered. I'm actually not sure whether this is the expected behavior or not, so feel free to close if this doesn't come as a surprise to anyone. For some context, I've been trying to get a sense for roughly which values of {{fs.s3a.multipart.size}} perform the best at different file sizes. One thing that I found confusing is that a part size of 5 MB seems to outperform a part size of 64 MB up until files that are upwards of about 500 MB in size. This seems odd, since each {{uploadPart}} call is its own HTTP request, and I would've expected the overhead of those to become costly at small part sizes. My suspicion is that with 4 concurrent part uploads and 64 MB blocks, we have to wait until 256 MB are buffered before we can start uploading, while with 5 MB blocks we can start uploading as soon as we buffer 20 MB, and that's what gives the smaller parts the advantage for smaller files. I'm happy to submit a patch if this is in fact a problem, but wanted to check to make sure I'm not just misunderstanding something. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Zhao updated HADOOP-14184: -- Attachment: HADOOP-14184.002.patch To fix the junit tests, add ftp entries to TestCommonConfigurationFields. > Remove service loader config entry for ftp fs > - > > Key: HADOOP-14184 > URL: https://issues.apache.org/jira/browse/HADOOP-14184 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.7.3 >Reporter: John Zhuge >Assignee: Sen Zhao >Priority: Minor > Labels: newbie > Attachments: HADOOP-14184.001.patch, HADOOP-14184.002.patch > > > Per discussion in HADOOP-14132. Remove line > {{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config > file > hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem > and add property {{fs.ftp.impl}} to {{core-default.xml}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196440#comment-16196440 ] Hadoop QA commented on HADOOP-14184: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 37m 29s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 8s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 85m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | | | hadoop.conf.TestCommonConfigurationFields | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14184 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890994/HADOOP-14184.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux eb5a9210fb07 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2856eb2 | | Default Java | 1.8.0_144 | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13472/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13472/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/13472/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Remove service loader config entry for ftp fs > - > > Key: HADOOP-14184 > URL: https://issues.apache.org/jira/browse/HADOOP-14184 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >
[jira] [Commented] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196415#comment-16196415 ] KUMAR NISHANT commented on HADOOP-14184: Hi Sen, Gr8..Looks like you fixed the bug. I assume that you are also new in contributing to open source. Can you help me with information that how are you testing/verifying your fix? I was trying to do this setup/installation in Windows but was seeing challenges/issue in setup. Thanks Nishant > Remove service loader config entry for ftp fs > - > > Key: HADOOP-14184 > URL: https://issues.apache.org/jira/browse/HADOOP-14184 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.7.3 >Reporter: John Zhuge >Assignee: Sen Zhao >Priority: Minor > Labels: newbie > Attachments: HADOOP-14184.001.patch > > > Per discussion in HADOOP-14132. Remove line > {{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config > file > hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem > and add property {{fs.ftp.impl}} to {{core-default.xml}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Zhao updated HADOOP-14184: -- Status: Patch Available (was: Open) > Remove service loader config entry for ftp fs > - > > Key: HADOOP-14184 > URL: https://issues.apache.org/jira/browse/HADOOP-14184 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.7.3 >Reporter: John Zhuge >Assignee: Sen Zhao >Priority: Minor > Labels: newbie > Attachments: HADOOP-14184.001.patch > > > Per discussion in HADOOP-14132. Remove line > {{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config > file > hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem > and add property {{fs.ftp.impl}} to {{core-default.xml}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14184) Remove service loader config entry for ftp fs
[ https://issues.apache.org/jira/browse/HADOOP-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Zhao updated HADOOP-14184: -- Attachment: HADOOP-14184.001.patch > Remove service loader config entry for ftp fs > - > > Key: HADOOP-14184 > URL: https://issues.apache.org/jira/browse/HADOOP-14184 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.7.3 >Reporter: John Zhuge >Assignee: Sen Zhao >Priority: Minor > Labels: newbie > Attachments: HADOOP-14184.001.patch > > > Per discussion in HADOOP-14132. Remove line > {{org.apache.hadoop.fs.ftp.FTPFileSystem}} from the service loader config > file > hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem > and add property {{fs.ftp.impl}} to {{core-default.xml}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org