[jira] [Commented] (HADOOP-14549) Use GenericTestUtils.setLogLevel when available
[
https://issues.apache.org/jira/browse/HADOOP-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056999#comment-16056999
]
Akira Ajisaka commented on HADOOP-14549:
Thanks [~vincent he] for taking this issue!
bq. Which one do you intend to do? Replacing all
org.apache.log4j.Logger.setLevel where GenericTestUtils.setLogLevel available
in WHOLE hadoop project or in some certain module like hadoop-tool?
Replacing in whole hadoop project seems big change, I prefer doing it by
package/module
bq. GenericTestUtils.setLogLevel is still using org.apache.log4j.Level as
parameter which is log4j1 api. In log4j2, this changed to
org.apache.logging.log4j.Level. Is this OK for us to migration to log4j2?
This is okay. When migrating to log4j2, we need to update GenericTestUtils and
that's all. If we don't do this, we need to update all the usage of
{{org.apache.log4j.Logger.setLoglevel}} when upgrading to log4j2.
> Use GenericTestUtils.setLogLevel when available
> ---
>
> Key: HADOOP-14549
> URL: https://issues.apache.org/jira/browse/HADOOP-14549
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>
> Based on Brahma's
> [comment|https://issues.apache.org/jira/browse/HADOOP-14296?focusedCommentId=16054390=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16054390]
> in HADOOP-14296, it's better to use GenericTestUtils.setLogLevel as possible
> to make the migration easier.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14543) Should use getAversion() while setting the zkacl
[
https://issues.apache.org/jira/browse/HADOOP-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056994#comment-16056994
]
Brahma Reddy Battula commented on HADOOP-14543:
---
[~arpitagarwal] and [~hanishakoneru] thanks for taking a look. Please see
following example
{noformat}
[zk: host-110:24002,host-155:24002,host-243:24002(CONNECTED) 1] create /testAcl
Created /testAcl
[zk: host-110:24002,host-155:24002,host-243:24002(CONNECTED) 2] stat /testAcl
cZxid = 0xc8c7c
ctime = Wed Jun 21 12:49:38 CST 2017
mZxid = 0xc8c7c
mtime = Wed Jun 21 12:49:38 CST 2017
pZxid = 0xc8c7c
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: host-10-110:24002,host-155:24002,host-243:24002(CONNECTED) 3] setAcl -v 0
/testAcl "world:anyone:cdwra"
[zk: host-110:24002,host-155:24002,host-243:24002(CONNECTED) 4] stat /testAcl
cZxid = 0xc8c7c
ctime = Wed Jun 21 12:49:38 CST 2017
mZxid = 0xc8c7c
mtime = Wed Jun 21 12:49:38 CST 2017
pZxid = 0xc8c7c
cversion = 0
dataVersion = 0
aclVersion = 1
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: host-110:24002,host-155:24002,host-243:24002(CONNECTED) 5] setAcl -v 0
/testAcl "world:anyone:cdwra"
version No is not valid : /testAcl
{noformat}
{{SetAcl}} will be success when it's both version number's correct(when we pass
the version) and after setting the ACL, it will increment the {{aclversion}} so
next time it will fail.
Please correct me if I am wrong.
> Should use getAversion() while setting the zkacl
>
>
> Key: HADOOP-14543
> URL: https://issues.apache.org/jira/browse/HADOOP-14543
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-14543.patch
>
>
> while setting the zkacl we used {color:red}{{getVersion()}}{color} which is
> dataVersion,Ideally we should use {{{color:#14892c}getAversion{color}()}}. If
> there is any acl changes( i.e relam change/..) ,we set the ACL with
> dataversion which will cause {color:#d04437}BADVersion {color}and
> {color:#d04437}*process will not start*{color}. See
> [here|https://issues.apache.org/jira/browse/HDFS-11403?focusedCommentId=16051804=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051804]
> {{zkClient.setACL(path, zkAcl, stat.getVersion());}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14549) Use GenericTestUtils.setLogLevel when available
[
https://issues.apache.org/jira/browse/HADOOP-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056926#comment-16056926
]
wenxin he commented on HADOOP-14549:
hi [~ajisakaa], I'll take this issue.
Before that I want to make sure:
1. Which one do you intend to do? Replacing all
org.apache.log4j.Logger.setLevel where GenericTestUtils.setLogLevel available
in *WHOLE* hadoop project or in some certain module like hadoop-tool?
2. GenericTestUtils.setLogLevel is still using org.apache.log4j.Level as
parameter which is log4j1 api. In log4j2, this changed to
org.apache.{color:red}logging{color}.log4j.Level. Is this OK for us to
migration to log4j2?
> Use GenericTestUtils.setLogLevel when available
> ---
>
> Key: HADOOP-14549
> URL: https://issues.apache.org/jira/browse/HADOOP-14549
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>
> Based on Brahma's
> [comment|https://issues.apache.org/jira/browse/HADOOP-14296?focusedCommentId=16054390=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16054390]
> in HADOOP-14296, it's better to use GenericTestUtils.setLogLevel as possible
> to make the migration easier.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056919#comment-16056919
]
Hongyuan Li commented on HADOOP-14469:
--
will resubmit the patch till HADOOP-14559 is resolved to avoid code conflict.
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch, HADOOP-14469-007.patch, HADOOP-14469-008.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14546) [WASB] Concurrent I/O does not work when secure.mode is enabled.
[
https://issues.apache.org/jira/browse/HADOOP-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056906#comment-16056906
]
Hadoop QA commented on HADOOP-14546:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
32s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated
14 new + 59 unchanged - 0 fixed = 73 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873784/HADOOP-14546.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 66590ee04b35 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1a59847 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [WASB] Concurrent I/O does not work when secure.mode is enabled.
>
>
> Key: HADOOP-14546
> URL: https://issues.apache.org/jira/browse/HADOOP-14546
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Thomas
>Assignee: Thomas
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14546.001.patch, HADOOP-14546.002.patch
>
>
> This change allows the concurrent I/O feature
> (fs.azure.io.read.tolerate.concurrent.append = true) to work
[jira] [Commented] (HADOOP-14542) Add IOUtils.cleanup or something that accepts slf4j logger API
[
https://issues.apache.org/jira/browse/HADOOP-14542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056864#comment-16056864
]
Akira Ajisaka commented on HADOOP-14542:
Thanks [~vagarychen] for the update. Would you add null check for the
{{logger}}?
> Add IOUtils.cleanup or something that accepts slf4j logger API
> --
>
> Key: HADOOP-14542
> URL: https://issues.apache.org/jira/browse/HADOOP-14542
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Chen Liang
> Attachments: HADOOP-14542.001.patch, HADOOP-14542.002.patch
>
>
> Split from HADOOP-14539.
> Now IOUtils.cleanup only accepts commons-logging logger API. Now we are
> migrating the APIs to slf4j, slf4j logger API should be accepted as well.
> Adding {{IOUtils.cleanup(Logger, Closeable...)}} causes
> {{IOUtils.cleanup(null, Closeable...)}} to fail (incompatible change), so
> it's better to change the method name to avoid the conflict.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14546) [WASB] Concurrent I/O does not work when secure.mode is enabled.
[ https://issues.apache.org/jira/browse/HADOOP-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas updated HADOOP-14546: Attachment: HADOOP-14546.002.patch Attaching new patch (HADOOP-14546.002.patch) with fix for javadoc issue. > [WASB] Concurrent I/O does not work when secure.mode is enabled. > > > Key: HADOOP-14546 > URL: https://issues.apache.org/jira/browse/HADOOP-14546 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.0-alpha4 >Reporter: Thomas >Assignee: Thomas > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-14546.001.patch, HADOOP-14546.002.patch > > > This change allows the concurrent I/O feature > (fs.azure.io.read.tolerate.concurrent.append = true) to work when secure mode > is enabled (fs.azure.secure.mode = true). > > While running the test TestAzureConcurrentOutOfBandIo.testReadOOBWrites, I > discovered that it fails when fs.azure.secure.mode = true with the error > below: > > com.microsoft.azure.storage.StorageException: The condition specified using > HTTP conditional header(s) is not met. > at > com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:733) > at > com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:264) > at > com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448) > at > com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at java.io.DataInputStream.read(DataInputStream.java:149) > at > org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo.testReadOOBWrites(TestAzureConcurrentOutOfBandIo.java:167) > > There were a couple problems causing this failure: > > 1) AzureNativeFileSystemStore.connectToAzureStorageInSecureMode was disabling > concurrent I/O by setting fs.azure.io.read.tolerate.concurrent.append to > false. > 2) SendRequestIntercept was unnecessarily updating the SAS for the request. > Since this intercept only sets the request header "If-Match: *" to override > the existing recondition, it is not necessary to update the SAS. > > The above issues have been fixed and a new test case has been added so that > testReadOOBWrites now runs both with and without secure mode enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14495) Add set options interface to FSDataOutputStreamBuilder
[
https://issues.apache.org/jira/browse/HADOOP-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056833#comment-16056833
]
Hadoop QA commented on HADOOP-14495:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 17
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 99 unchanged - 0 fixed = 100 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
33s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14495 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873762/HADOOP-14495.00.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b729bf265afd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5157f6c |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12581/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12581/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12581/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12581/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add set options interface to FSDataOutputStreamBuilder
> ---
>
> Key: HADOOP-14495
> URL: https://issues.apache.org/jira/browse/HADOOP-14495
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>
[jira] [Commented] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056831#comment-16056831
]
Hadoop QA commented on HADOOP-14469:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-14469 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14469 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873776/HADOOP-14469-008.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12583/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch, HADOOP-14469-007.patch, HADOOP-14469-008.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14550) remove unused parameter of javadoc
[
https://issues.apache.org/jira/browse/HADOOP-14550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056829#comment-16056829
]
Hadoop QA commented on HADOOP-14550:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
58s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
36s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 17
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
50s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14550 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873761/HADOOP-14550.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f99ce2cf327f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5157f6c |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12580/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12580/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12580/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> remove unused parameter of javadoc
> --
>
> Key: HADOOP-14550
> URL: https://issues.apache.org/jira/browse/HADOOP-14550
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Trivial
> Labels: documentation
> Attachments: HADOOP-14550.001.patch, HADOOP-14550.002.patch
>
>
> the param 'subpath' is unused in GenericTestUtils.getRandomizedTempPath. so
> the javadoc need to be modified.
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Attachment: HADOOP-14469-008.patch
fix code style check warning and close the FTPFileSystem used by
{{TestFTPFileSystem}}#{{testListStatus}}
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch, HADOOP-14469-007.patch, HADOOP-14469-008.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056807#comment-16056807 ] Yonger commented on HADOOP-14475: - @Deng, each task should have an individual s3a file system that i agree with you. But i seen the s3a file system had been initialized multiple times with the same path in driver application, you can also check in my last uploaded logs. BTW, the new patch that fix this issue totally is in the process of verifying, i don't know when it available to upload since i have no aws s3 test environment, i am running on Ceph now. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14559) The used FTPFileSystem in TestFTPFileSystem should be closed in each test case.
[ https://issues.apache.org/jira/browse/HADOOP-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongyuan Li updated HADOOP-14559: - Component/s: test fs > The used FTPFileSystem in TestFTPFileSystem should be closed in each test > case. > --- > > Key: HADOOP-14559 > URL: https://issues.apache.org/jira/browse/HADOOP-14559 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, test >Reporter: Hongyuan Li >Assignee: Hongyuan Li > > The used FTPFileSystem in TestFTPFileSystem should be closed in each test > case as an improvement. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14559) The used FTPFileSystem in TestFTPFileSystem should be closed in each test case.
[ https://issues.apache.org/jira/browse/HADOOP-14559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongyuan Li reassigned HADOOP-14559: Assignee: Hongyuan Li > The used FTPFileSystem in TestFTPFileSystem should be closed in each test > case. > --- > > Key: HADOOP-14559 > URL: https://issues.apache.org/jira/browse/HADOOP-14559 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hongyuan Li >Assignee: Hongyuan Li > > The used FTPFileSystem in TestFTPFileSystem should be closed in each test > case as an improvement. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14558) RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
[
https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056771#comment-16056771
]
Mostafa Mokhtar commented on HADOOP-14558:
--
[~hanm]
Yes, looks like a duplicate.
> RPC requests on a secure cluster are 10x slower due to expensive encryption
> and decryption
> ---
>
> Key: HADOOP-14558
> URL: https://issues.apache.org/jira/browse/HADOOP-14558
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Mostafa Mokhtar
>Priority: Critical
> Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure
> clusters I noticed that metadata loading operations are 10x slower on a
> cluster with Kerberos+SSL enabled.
> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit
> for metadata loading.
> The majority of the slowdown is coming from the two threads below.
> {code}
> Stack Trace Sample CountPercentage(%)
> org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
>org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203
> 46.505
> java.io.DataInputStream.readInt() 5,039 45.039
> java.io.BufferedInputStream.read() 5,038 45.03
> java.io.BufferedInputStream.fill()5,038 45.03
>
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
> int) 5,036 45.013
> java.io.FilterInputStream.read(byte[], int, int)5,036
> 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int,
> int) 5,036 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
>5,035 45.004
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775
> 42.68
> sun.security.jgss.GSSContextImpl.unwrap(byte[],
> int, int, MessageProp) 4,775 42.68
>
> sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)
> 4,768 42.617
>
> sun.security.jgss.krb5.WrapToken.getData()4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714
> 42.134
>
> sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int,
> byte[], int)3,083 27.556
>
> sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int,
> int, byte[], int) 3,078 27.512
>
> sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
>
> sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
> {code}
> And
> {code}
> Stack Trace Sample CountPercentage(%)
> java.lang.Thread.run()3,379 30.202
>java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> 3,379 30.202
> java.util.concurrent.FutureTask.run()3,367 30.095
> java.util.concurrent.Executors$RunnableAdapter.call() 3,367
> 30.095
>org.apache.hadoop.ipc.Client$Connection$3.run()3,367
> 30.095
> java.io.DataOutputStream.flush()3,367 30.095
> java.io.BufferedOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flushBuffer()3,367
> 30.095
>
> org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
> int, int) 3,367 30.095
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int)3,281
> 29.326
>
> sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281
> 29.326
>
> sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)
> 3,280 29.317
>
> sun.security.jgss.krb5.WrapToken.(Krb5Context, MessageProp, byte[],
> int, int)1,872 16.732
>
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056767#comment-16056767
]
Hongyuan Li commented on HADOOP-14429:
--
Thanks [~yzhangal], for your review and commit. Thanks [~brahmareddy] for your
review. I file HADOOP-14559 as an improvement to close the ftp.
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch,
> HADOOP-14429-006.patch, HADOOP-14429-007.patch, HADOOP-14429-008.patch,
> HADOOP-14429-009.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14548: -- Priority: Minor (was: Major) > S3Guard: issues running parallel tests w/ S3N > -- > > Key: HADOOP-14548 > URL: https://issues.apache.org/jira/browse/HADOOP-14548 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Minor > > In general, running S3Guard and parallel tests with S3A and S3N contract > tests enabled is asking for trouble: S3Guard code assumes there are not > other non-S3Guard clients modifying the bucket. > Goal of this JIRA is to: > - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard > -Ddynamo` with S3A and S3N contract tests configured. > - Identify any failures here that are worth looking into. > - Document (or enforce) that people should not do this, or should expect > failures if they do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14559) The used FTPFileSystem in TestFTPFileSystem should be closed in each test case.
Hongyuan Li created HADOOP-14559: Summary: The used FTPFileSystem in TestFTPFileSystem should be closed in each test case. Key: HADOOP-14559 URL: https://issues.apache.org/jira/browse/HADOOP-14559 Project: Hadoop Common Issue Type: Improvement Reporter: Hongyuan Li The used FTPFileSystem in TestFTPFileSystem should be closed in each test case as an improvement. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N
[
https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056763#comment-16056763
]
Aaron Fabbri commented on HADOOP-14548:
---
For the record, here are the failures on trunk (no s3guard) if I try to run
s3a/s3n integration tests in parallel, using a single bucket
{noformat}
Failed tests:
(2)
ITestJets3tNativeS3FileSystemContract>NativeS3FileSystemContractBaseTest.testListStatusForRoot:73
Root directory is not empty; expected:<0> but was:<2>
(3)
ITestJets3tNativeS3FileSystemContract>FileSystemContractBaseTest.testListStatus:321
expected:<1> but was:<12>
Tests in error:
(4)
ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:116
» PathIO
(6)
ITestJets3tNativeS3FileSystemContract>FileSystemContractBaseTest.testLSRootDir:833->FileSystemContractBaseTest.assertListFilesFinds:851
» FileNotFound
{noformat}
If I point s3a/s3n contract tests at different buckets, I still get the first
three (but not #6).
> S3Guard: issues running parallel tests w/ S3N
> --
>
> Key: HADOOP-14548
> URL: https://issues.apache.org/jira/browse/HADOOP-14548
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>
> In general, running S3Guard and parallel tests with S3A and S3N contract
> tests enabled is asking for trouble: S3Guard code assumes there are not
> other non-S3Guard clients modifying the bucket.
> Goal of this JIRA is to:
> - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard
> -Ddynamo` with S3A and S3N contract tests configured.
> - Identify any failures here that are worth looking into.
> - Document (or enforce) that people should not do this, or should expect
> failures if they do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14495) Add set options interface to FSDataOutputStreamBuilder
[
https://issues.apache.org/jira/browse/HADOOP-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-14495:
---
Attachment: HADOOP-14495.00.patch
Hey, [~ste...@apache.org] [~andrew.wang]
I upload a draft that only provides API of {{opt(...)}} and {{must(...)}}
calls, and very basic tests to check the existence of keys to get the interface
right first.
Could you take a look? Thanks!
> Add set options interface to FSDataOutputStreamBuilder
> ---
>
> Key: HADOOP-14495
> URL: https://issues.apache.org/jira/browse/HADOOP-14495
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-14495.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14495) Add set options interface to FSDataOutputStreamBuilder
[ https://issues.apache.org/jira/browse/HADOOP-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-14495: --- Status: Patch Available (was: Open) > Add set options interface to FSDataOutputStreamBuilder > --- > > Key: HADOOP-14495 > URL: https://issues.apache.org/jira/browse/HADOOP-14495 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HADOOP-14495.00.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14520) Block compaction for WASB
[ https://issues.apache.org/jira/browse/HADOOP-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14520: - Status: In Progress (was: Patch Available) > Block compaction for WASB > - > > Key: HADOOP-14520 > URL: https://issues.apache.org/jira/browse/HADOOP-14520 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov > Attachments: HADOOP-14520-01.patch, HADOOP-14520-01-test.txt > > > Block Compaction for WASB allows uploading new blocks for every hflush/hsync > call. When the number of blocks is above a predefined, configurable value, > next hflush/hsync triggers the block compaction process. Block compaction > replaces a sequence of blocks with one block. From all the sequences with > total length less than 4M, compaction chooses the longest one. It is a greedy > algorithm that preserve all potential candidates for the next round. Block > Compaction for WASB increases data durability and allows using block blobs > instead of page blobs. By default, block compaction is disabled. Similar to > the configuration for page blobs, the client needs to specify HDFS folders > where block compaction over block blobs is enabled. > Results for HADOOP-14520-01.patch > Tests run: 704, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14550) remove unused parameter of javadoc
[ https://issues.apache.org/jira/browse/HADOOP-14550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056725#comment-16056725 ] hu xiaodong commented on HADOOP-14550: -- Hello [~vagarychen], thank you very much for your comment. I have modified as you said. see 002.patch. thanks. > remove unused parameter of javadoc > -- > > Key: HADOOP-14550 > URL: https://issues.apache.org/jira/browse/HADOOP-14550 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Trivial > Labels: documentation > Attachments: HADOOP-14550.001.patch, HADOOP-14550.002.patch > > > the param 'subpath' is unused in GenericTestUtils.getRandomizedTempPath. so > the javadoc need to be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14550) remove unused parameter of javadoc
[ https://issues.apache.org/jira/browse/HADOOP-14550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hu xiaodong updated HADOOP-14550: - Attachment: HADOOP-14550.002.patch > remove unused parameter of javadoc > -- > > Key: HADOOP-14550 > URL: https://issues.apache.org/jira/browse/HADOOP-14550 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Trivial > Labels: documentation > Attachments: HADOOP-14550.001.patch, HADOOP-14550.002.patch > > > the param 'subpath' is unused in GenericTestUtils.getRandomizedTempPath. so > the javadoc need to be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056724#comment-16056724
]
Yongjun Zhang commented on HADOOP-14429:
Committed to trunk and branch-2.
Thanks [~Hongyuan Li] for the contribution, [~brahmareddy] for the review.
BTW, found that there is a pre-existing issue in the test code, the {{ftp}} is
never closed. Suggest to solve it as a follow-up jira.
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch,
> HADOOP-14429-006.patch, HADOOP-14429-007.patch, HADOOP-14429-008.patch,
> HADOOP-14429-009.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-14429:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha4
2.9.0
Status: Resolved (was: Patch Available)
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch,
> HADOOP-14429-006.patch, HADOOP-14429-007.patch, HADOOP-14429-008.patch,
> HADOOP-14429-009.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056723#comment-16056723
]
Hudson commented on HADOOP-14429:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11898 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11898/])
HADOOP-14429. FTPFileSystem#getFsAction always returns FsAction.NONE. (yzhang:
rev 5157f6c46ec342fb650b3c5853061ed1e4a182b6)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/ftp/TestFTPFileSystem.java
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch,
> HADOOP-14429-006.patch, HADOOP-14429-007.patch, HADOOP-14429-008.patch,
> HADOOP-14429-009.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files
[
https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056703#comment-16056703
]
Aaron Fabbri commented on HADOOP-14457:
---
[~mackrorysd] and I just chatted about this and I think we came up with a
better option. Instead of adding a "recursive" put to MetadataStore, I
suggested a batched put.
The main thing I disliked about adding a new method
{{MetadataStore#putRecursive(PathMetadata p)}} is that it is not explicit as to
which ancestors are created. It is this sort of sloppy method "I'm creating
this thing, make sure the ancestors exist" instead of an explicit "here are the
paths that I'm creating". The former makes it hard to use MetadataStore
interface for precise metadata logging.
One of the problems with requiring the FS client to put() each path it creates
is that, since dynamo has to create ancestors internally for its schema,
DynamoDB ends up having to do extra round trips (both FS and dynamo making sure
ancestors are created). [~mackrorysd] rightfully pushed back on this. Even
with the capability check I suggested (which should be correct but not trivial
to reason about), Dynamo would be affected by extra round trips once we added
authoritative listing support (HADOOP-14154).
We both seemed to agree that adding a batched
{{MetadataStore#put(Collection
pathsToCreate)}} seemed like a better option than the capability check.
The dynamo implementation can use existing logic from its move() implementation
to pre-process the list to include ancestors in the batched put list. Dynamo
gets optimal round trips, now and in the future, and MetadataStore is still
able to express explicit logging of metadata changes.
Thanks for great discussion on this [~mackrorysd], please add anything I missed.
> create() does not notify metadataStore of parent directories or ensure
> they're not existing files
> -
>
> Key: HADOOP-14457
> URL: https://issues.apache.org/jira/browse/HADOOP-14457
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14457-HADOOP-13345.001.patch,
> HADOOP-14457-HADOOP-13345.002.patch, HADOOP-14457-HADOOP-13345.003.patch,
> HADOOP-14457-HADOOP-13345.004.patch, HADOOP-14457-HADOOP-13345.005.patch,
> HADOOP-14457-HADOOP-13345.006.patch, HADOOP-14457-HADOOP-13345.007.patch,
> HADOOP-14457-HADOOP-13345.008.patch, HADOOP-14457-HADOOP-13345.009.patch
>
>
> Not a great test yet, but it at least reliably demonstrates the issue.
> LocalMetadataStore will sometimes erroneously report that a directory is
> empty with isAuthoritative = true when it *definitely* has children the
> metadatastore should know about. It doesn't appear to happen if the children
> are just directory. The fact that it's returning an empty listing is
> concerning, but the fact that it says it's authoritative *might* be a second
> bug.
> {code}
> diff --git
> a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
>
> b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> index 78b3970..1821d19 100644
> ---
> a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> +++
> b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
> @@ -965,7 +965,7 @@ public boolean hasMetadataStore() {
>}
>
>@VisibleForTesting
> - MetadataStore getMetadataStore() {
> + public MetadataStore getMetadataStore() {
> return metadataStore;
>}
>
> diff --git
> a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
>
> b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
> index 4339649..881bdc9 100644
> ---
> a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
> +++
> b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
> @@ -23,6 +23,11 @@
> import org.apache.hadoop.fs.contract.AbstractFSContract;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> +import org.apache.hadoop.fs.s3a.S3AFileSystem;
> +import org.apache.hadoop.fs.s3a.Tristate;
> +import org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata;
> +import org.apache.hadoop.fs.s3a.s3guard.MetadataStore;
> +import org.junit.Test;
>
> import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
> import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
> @@ -72,4 +77,24 @@ public void testRenameDirIntoExistingDir() throws
> Throwable {
> boolean rename =
[jira] [Commented] (HADOOP-14558) RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
[
https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056696#comment-16056696
]
Michael Han commented on HADOOP-14558:
--
This could be a known issue described in HADOOP-10768
> RPC requests on a secure cluster are 10x slower due to expensive encryption
> and decryption
> ---
>
> Key: HADOOP-14558
> URL: https://issues.apache.org/jira/browse/HADOOP-14558
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Mostafa Mokhtar
>Priority: Critical
> Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure
> clusters I noticed that metadata loading operations are 10x slower on a
> cluster with Kerberos+SSL enabled.
> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit
> for metadata loading.
> The majority of the slowdown is coming from the two threads below.
> {code}
> Stack Trace Sample CountPercentage(%)
> org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
>org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203
> 46.505
> java.io.DataInputStream.readInt() 5,039 45.039
> java.io.BufferedInputStream.read() 5,038 45.03
> java.io.BufferedInputStream.fill()5,038 45.03
>
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int,
> int) 5,036 45.013
> java.io.FilterInputStream.read(byte[], int, int)5,036
> 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int,
> int) 5,036 45.013
>
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
>5,035 45.004
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775
> 42.68
> sun.security.jgss.GSSContextImpl.unwrap(byte[],
> int, int, MessageProp) 4,775 42.68
>
> sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)
> 4,768 42.617
>
> sun.security.jgss.krb5.WrapToken.getData()4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
>
> sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714
> 42.134
>
> sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int,
> byte[], int)3,083 27.556
>
> sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int,
> int, byte[], int) 3,078 27.512
>
> sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
>
> sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[],
> byte[], int, int) 3,076 27.494
> {code}
> And
> {code}
> Stack Trace Sample CountPercentage(%)
> java.lang.Thread.run()3,379 30.202
>java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
> 3,379 30.202
> java.util.concurrent.FutureTask.run()3,367 30.095
> java.util.concurrent.Executors$RunnableAdapter.call() 3,367
> 30.095
>org.apache.hadoop.ipc.Client$Connection$3.run()3,367
> 30.095
> java.io.DataOutputStream.flush()3,367 30.095
> java.io.BufferedOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flushBuffer()3,367
> 30.095
>
> org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[],
> int, int) 3,367 30.095
>
> com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int)3,281
> 29.326
>
> sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281
> 29.326
>
> sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)
> 3,280 29.317
>
> sun.security.jgss.krb5.WrapToken.(Krb5Context, MessageProp, byte[],
> int, int)1,872 16.732
>
[jira] [Commented] (HADOOP-14429) FTPFileSystem#getFsAction always returns FsAction.NONE
[
https://issues.apache.org/jira/browse/HADOOP-14429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056670#comment-16056670
]
Yongjun Zhang commented on HADOOP-14429:
Thanks [~brahmareddy] for the review and [~Hongyuan Li] for the updated patch.
+1 on rev9. will commit shortly.
> FTPFileSystem#getFsAction always returns FsAction.NONE
> ---
>
> Key: HADOOP-14429
> URL: https://issues.apache.org/jira/browse/HADOOP-14429
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0-alpha2
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
> Attachments: HADOOP-14429-001.patch, HADOOP-14429-002.patch,
> HADOOP-14429-003.patch, HADOOP-14429-004.patch, HADOOP-14429-005.patch,
> HADOOP-14429-006.patch, HADOOP-14429-007.patch, HADOOP-14429-008.patch,
> HADOOP-14429-009.patch
>
>
>
> {code}
> private FsAction getFsAction(int accessGroup, FTPFile ftpFile) {
> FsAction action = FsAction.NONE;
> if (ftpFile.hasPermission(accessGroup, FTPFile.READ_PERMISSION)) {
> action.or(FsAction.READ);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.WRITE_PERMISSION)) {
> action.or(FsAction.WRITE);
> }
> if (ftpFile.hasPermission(accessGroup, FTPFile.EXECUTE_PERMISSION)) {
> action.or(FsAction.EXECUTE);
> }
> return action;
> }
> {code}
> from code above, we can see that the getFsAction method doesnot modify the
> action generated by FsAction action = FsAction.NONE,which means it return
> FsAction.NONE all the time;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14558) RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
Mostafa Mokhtar created HADOOP-14558:
Summary: RPC requests on a secure cluster are 10x slower due to
expensive encryption and decryption
Key: HADOOP-14558
URL: https://issues.apache.org/jira/browse/HADOOP-14558
Project: Hadoop Common
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.6.0
Reporter: Mostafa Mokhtar
Priority: Critical
While running a performance tests for Impala comparing secure and un-secure
clusters I noticed that metadata loading operations are 10x slower on a cluster
with Kerberos+SSL enabled.
hadoop.rpc.protection is set to privacy
Any recommendations on how this can be mitigated? 10x slowdown is a big hit for
metadata loading.
The majority of the slowdown is coming from the two threads below.
{code}
Stack Trace Sample CountPercentage(%)
org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203 46.505
java.io.DataInputStream.readInt() 5,039 45.039
java.io.BufferedInputStream.read() 5,038 45.03
java.io.BufferedInputStream.fill() 5,038 45.03
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int)
5,036 45.013
java.io.FilterInputStream.read(byte[], int, int) 5,036
45.013
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int,
int) 5,036 45.013
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
5,035 45.004
com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775
42.68
sun.security.jgss.GSSContextImpl.unwrap(byte[],
int, int, MessageProp)4,775 42.68
sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)
4,768 42.617
sun.security.jgss.krb5.WrapToken.getData()
4,714 42.134
sun.security.jgss.krb5.WrapToken.getData(byte[], int)4,714 42.134
sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714
42.134
sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int,
byte[], int) 3,083 27.556
sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int,
byte[], int) 3,078 27.512
sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[],
int, int) 3,076 27.494
sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[],
byte[], int, int) 3,076 27.494
{code}
And
{code}
Stack Trace Sample CountPercentage(%)
java.lang.Thread.run() 3,379 30.202
java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
3,379 30.202
java.util.concurrent.FutureTask.run() 3,367 30.095
java.util.concurrent.Executors$RunnableAdapter.call() 3,367
30.095
org.apache.hadoop.ipc.Client$Connection$3.run() 3,367 30.095
java.io.DataOutputStream.flush() 3,367 30.095
java.io.BufferedOutputStream.flush() 3,367 30.095
java.io.BufferedOutputStream.flushBuffer() 3,367
30.095
org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int,
int) 3,367 30.095
com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281 29.326
sun.security.jgss.GSSContextImpl.wrap(byte[],
int, int, MessageProp) 3,281 29.326
sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)
3,280 29.317
sun.security.jgss.krb5.WrapToken.(Krb5Context, MessageProp, byte[], int,
int) 1,872 16.732
sun.security.jgss.krb5.WrapToken.encode()1,407 12.576
{code}
This is the Impala Catalog thread which initiates the NameNode request
{code}
Stack Trace Sample CountPercentage(%)
org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)
2,378 21.255
org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)
2,378 21.255
[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056591#comment-16056591
]
Hadoop QA commented on HADOOP-14499:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} hadoop-tools/hadoop-aws in HADOOP-13345 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} hadoop-tools/hadoop-aws generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14499 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873329/HADOOP-14499-HADOOP-13345.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5456467e15e3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / 0db7176 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12579/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-aws-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12579/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12579/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056578#comment-16056578 ] Mithun Radhakrishnan commented on HADOOP-8143: -- bq. What you proposed, sounds to be that -pb becomes a deprecated option because block size is always preserved. Ah, yes. I see. I stand corrected. :] Your phrasing is more accurate. Thank you. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056553#comment-16056553 ] Aaron Fabbri commented on HADOOP-14548: --- Yeah, I will add something in the testing.md docs to that effect, at least. > S3Guard: issues running parallel tests w/ S3N > -- > > Key: HADOOP-14548 > URL: https://issues.apache.org/jira/browse/HADOOP-14548 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > > In general, running S3Guard and parallel tests with S3A and S3N contract > tests enabled is asking for trouble: S3Guard code assumes there are not > other non-S3Guard clients modifying the bucket. > Goal of this JIRA is to: > - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard > -Ddynamo` with S3A and S3N contract tests configured. > - Identify any failures here that are worth looking into. > - Document (or enforce) that people should not do this, or should expect > failures if they do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056547#comment-16056547 ] Steve Loughran commented on HADOOP-14548: - Could skip all s3n tests if s3guard is turned on? > S3Guard: issues running parallel tests w/ S3N > -- > > Key: HADOOP-14548 > URL: https://issues.apache.org/jira/browse/HADOOP-14548 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > > In general, running S3Guard and parallel tests with S3A and S3N contract > tests enabled is asking for trouble: S3Guard code assumes there are not > other non-S3Guard clients modifying the bucket. > Goal of this JIRA is to: > - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard > -Ddynamo` with S3A and S3N contract tests configured. > - Identify any failures here that are worth looking into. > - Document (or enforce) that people should not do this, or should expect > failures if they do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056548#comment-16056548 ] Steve Loughran commented on HADOOP-14548: - No, that's silly. Aaron: use a different test filesystem for s3n > S3Guard: issues running parallel tests w/ S3N > -- > > Key: HADOOP-14548 > URL: https://issues.apache.org/jira/browse/HADOOP-14548 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > > In general, running S3Guard and parallel tests with S3A and S3N contract > tests enabled is asking for trouble: S3Guard code assumes there are not > other non-S3Guard clients modifying the bucket. > Goal of this JIRA is to: > - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard > -Ddynamo` with S3A and S3N contract tests configured. > - Identify any failures here that are worth looking into. > - Document (or enforce) that people should not do this, or should expect > failures if they do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14547) [WASB] the configured retry policy is not used for all storage operations.
[ https://issues.apache.org/jira/browse/HADOOP-14547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14547: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14552 > [WASB] the configured retry policy is not used for all storage operations. > -- > > Key: HADOOP-14547 > URL: https://issues.apache.org/jira/browse/HADOOP-14547 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0, 3.0.0-alpha4 >Reporter: Thomas >Assignee: Thomas > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-14547.001.patch > > > There are a few places where the WASB retry policy was not used, and instead > the Azure Storage SDK default retry policy was used. These places include > some calls to blob.exist(), container.exists(), blob.delete(), and operations > with secure mode enabled (fs.azure.secure.mode = true). > You can set a break point on > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry in the > Azure Storage SDK and see that the WASB configured retry policy > (DEFAULT_MAX_RETRY_ATTEMPTS, etc.) is not used. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056541#comment-16056541 ] Steve Loughran commented on HADOOP-14518: - test failures were unrelated: HADOOP-14538 > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.id > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14461) Azure: handle failure gracefully in case of missing account access key
[
https://issues.apache.org/jira/browse/HADOOP-14461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056535#comment-16056535
]
Steve Loughran commented on HADOOP-14461:
-
I like the code now.
One thing I do see (having looked @ the azure test code) is that the code in
{{AzureBlobStorageTestAccount}} is returning null & letting the caller test
suites deal with it. We can/should just have it call {{assume()}} with a
message, so that when the test is skipped the text actually appears.
We could do that here or as part of a broader review of the tests
> Azure: handle failure gracefully in case of missing account access key
> --
>
> Key: HADOOP-14461
> URL: https://issues.apache.org/jira/browse/HADOOP-14461
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14461.000.patch, HADOOP-14461.001.patch,
> HADOOP-14461.002.patch, HADOOP-14461.003.patch
>
>
> Currently if the {{fs.azure.account.key.youraccount}} is missing, we will get
> error stack like this:
> {code}
> java.lang.IllegalArgumentException: The String is not a valid Base64-encoded
> string.
> at com.microsoft.azure.storage.core.Base64.decode(Base64.java:63)
> at
> com.microsoft.azure.storage.StorageCredentialsAccountAndKey.(StorageCredentialsAccountAndKey.java:81)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createStorageAccount(AzureBlobStorageTestAccount.java:464)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.createTestAccount(AzureBlobStorageTestAccount.java:501)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:522)
> at
> org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.create(AzureBlobStorageTestAccount.java:451)
> at
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization.createTestAccount(TestNativeAzureFileSystemAuthorization.java:50)
> at
> org.apache.hadoop.fs.azure.AbstractWasbTestBase.setUp(AbstractWasbTestBase.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
> Actually, this error message is not very helpful. I'd admire you if you can
> immediately find the root cause of this failure.
> Currently if the test *account* is missing, we simply skip the test with
> meaningful error message. Here if the *account key* is missing, we should do
> the same thing: skip the test, and tell the user why we skip it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056522#comment-16056522 ] Ai Deng commented on HADOOP-14475: -- Hello [~iyonger], [~ste...@apache.org] I found this ticket when I tried to collect some S3a metrics as well. I think when Hadoop commit each tasks, it need create the FileSystem with the specific path for that task (need call rename and delete operation in that task), so it is normal that s3a fs has been initialized multi time in one Hadoop job? Any issue if we have multi instances of "S3AFileSystem" in one JVM? > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14499) Findbugs warning in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14499:
Status: Patch Available (was: Open)
> Findbugs warning in LocalMetadataStore
> --
>
> Key: HADOOP-14499
> URL: https://issues.apache.org/jira/browse/HADOOP-14499
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HADOOP-14499-HADOOP-13345.001.patch,
> HADOOP-14499-HADOOP-13345.002.patch
>
>
> First saw this raised by Yetus on HADOOP-14433:
> {code}
> Bug type UC_USELESS_OBJECT (click for details)
> In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
> In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
> Value ancestors
> Type java.util.LinkedList
> At LocalMetadataStore.java:[line 300]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056487#comment-16056487 ] Wei-Chiu Chuang commented on HADOOP-8143: - Hi Mithun, My understanding of the patch, is that if -p is not specified, block size is preserved; however, if say distcp runs with -pr (preserve replication factor), then block size is not preserved. What you proposed, sounds to be that -pb becomes a deprecated option because block size is _always_ preserved, which I don't think is the purpose of this jira. Please correct me if I am wrong. Thanks. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[
https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056478#comment-16056478
]
Mithun Radhakrishnan commented on HADOOP-8143:
--
bq. If -p option of distcp command is unspecified, block size is preserved.
That looks good. What about:
{noformat}
Block-size is preserved, even if the "-p" option of distcp command is
unspecified.
{noformat}
?
> Change distcp to have -pb on by default
> ---
>
> Key: HADOOP-8143
> URL: https://issues.apache.org/jira/browse/HADOOP-8143
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Dave Thompson
>Assignee: Mithun Radhakrishnan
>Priority: Minor
> Fix For: 3.0.0-alpha4
>
> Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch,
> HADOOP-8143.3.patch
>
>
> We should have the preserve blocksize (-pb) on in distcp by default.
> checksum which is on by default will always fail if blocksize is not the same.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14502) Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps
[
https://issues.apache.org/jira/browse/HADOOP-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HADOOP-14502:
---
Hadoop Flags: Incompatible change
> Confusion/name conflict between NameNodeActivity#BlockReportNumOps and
> RpcDetailedActivity#BlockReportNumOps
>
>
> Key: HADOOP-14502
> URL: https://issues.apache.org/jira/browse/HADOOP-14502
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
> Labels: Incompatible
> Attachments: HADOOP-14502.000.patch, HADOOP-14502.001.patch,
> HADOOP-14502.002.patch
>
>
> Currently the {{BlockReport(NumOps|AvgTime)}} metrics emitted under the
> {{RpcDetailedActivity}} context and those emitted under the
> {{NameNodeActivity}} context are actually reporting different things despite
> having the same name. {{NameNodeActivity}} reports the count/time of _per
> storage_ block reports, whereas {{RpcDetailedActivity}} reports the
> count/time of _per datanode_ block reports. This makes for a confusing
> experience with two metrics having the same name reporting different values.
> We already have the {{StorageBlockReportsOps}} metric under
> {{NameNodeActivity}}. Can we make {{StorageBlockReport}} a {{MutableRate}}
> metric and remove {{NameNodeActivity#BlockReport}} metric? Open to other
> suggestions about how to address this as well. The 3.0 release seems a good
> time to make this incompatible change.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-14536) Update azure-storage sdk to version 5.3.0
[ https://issues.apache.org/jira/browse/HADOOP-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Georgi Chalakov updated HADOOP-14536: - Comment: was deleted (was: It was tested against: fs.contract.test.fs.wasb wasb://testh...@hdfs4.blob.core.windows.net ) > Update azure-storage sdk to version 5.3.0 > - > > Key: HADOOP-14536 > URL: https://issues.apache.org/jira/browse/HADOOP-14536 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Mingliang Liu >Assignee: Georgi Chalakov > Attachments: HADOOP-14536-01.patch > > > Update WASB driver to use the latest version (5.3.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.3: > * Fixed a bug where the transactional MD5 check would fail when downloading > a range of blob or file and the recovery action is performed on a subsection > of the range. > * Fixed leaking connections for table requests. > * Fixed a bug where retries happened immediately when experiencing a network > exception uploading data or getting the response. > * Fixed a bug where the response stream was not being closed on nonretryable > exceptions. > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14536) Update azure-storage sdk to version 5.3.0
[ https://issues.apache.org/jira/browse/HADOOP-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056368#comment-16056368 ] Georgi Chalakov commented on HADOOP-14536: -- The patch changes the version of Azure Client SDK. The new SDK offers fixes and new functionality, but it should not change current behavior, so it doesn't require modification of existing tests or new tests. All tests pass when the change was tested against wasb://testh...@hdfs4.blob.core.windows.net > Update azure-storage sdk to version 5.3.0 > - > > Key: HADOOP-14536 > URL: https://issues.apache.org/jira/browse/HADOOP-14536 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Mingliang Liu >Assignee: Georgi Chalakov > Attachments: HADOOP-14536-01.patch > > > Update WASB driver to use the latest version (5.3.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.3: > * Fixed a bug where the transactional MD5 check would fail when downloading > a range of blob or file and the recovery action is performed on a subsection > of the range. > * Fixed leaking connections for table requests. > * Fixed a bug where retries happened immediately when experiencing a network > exception uploading data or getting the response. > * Fixed a bug where the response stream was not being closed on nonretryable > exceptions. > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14536) Update azure-storage sdk to version 5.3.0
[ https://issues.apache.org/jira/browse/HADOOP-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056362#comment-16056362 ] Georgi Chalakov commented on HADOOP-14536: -- It was tested against: fs.contract.test.fs.wasb wasb://testh...@hdfs4.blob.core.windows.net > Update azure-storage sdk to version 5.3.0 > - > > Key: HADOOP-14536 > URL: https://issues.apache.org/jira/browse/HADOOP-14536 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Mingliang Liu >Assignee: Georgi Chalakov > Attachments: HADOOP-14536-01.patch > > > Update WASB driver to use the latest version (5.3.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.3: > * Fixed a bug where the transactional MD5 check would fail when downloading > a range of blob or file and the recovery action is performed on a subsection > of the range. > * Fixed leaking connections for table requests. > * Fixed a bug where retries happened immediately when experiencing a network > exception uploading data or getting the response. > * Fixed a bug where the response stream was not being closed on nonretryable > exceptions. > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14543) Should use getAversion() while setting the zkacl
[
https://issues.apache.org/jira/browse/HADOOP-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056271#comment-16056271
]
Arpit Agarwal commented on HADOOP-14543:
Hi Brahma, I agree with Hanisha's comment - the setVersion parameters look
correct to me too, after looking at the implementation of ZooKeeper#setVersion.
Have you run into this error?
> Should use getAversion() while setting the zkacl
>
>
> Key: HADOOP-14543
> URL: https://issues.apache.org/jira/browse/HADOOP-14543
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-14543.patch
>
>
> while setting the zkacl we used {color:red}{{getVersion()}}{color} which is
> dataVersion,Ideally we should use {{{color:#14892c}getAversion{color}()}}. If
> there is any acl changes( i.e relam change/..) ,we set the ACL with
> dataversion which will cause {color:#d04437}BADVersion {color}and
> {color:#d04437}*process will not start*{color}. See
> [here|https://issues.apache.org/jira/browse/HDFS-11403?focusedCommentId=16051804=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051804]
> {{zkClient.setACL(path, zkAcl, stat.getVersion());}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14542) Add IOUtils.cleanup or something that accepts slf4j logger API
[
https://issues.apache.org/jira/browse/HADOOP-14542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056220#comment-16056220
]
Hadoop QA commented on HADOOP-14542:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
1s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 18
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14542 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873714/HADOOP-14542.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 143bb641700b 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2b654a4 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12578/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12578/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12578/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add IOUtils.cleanup or something that accepts slf4j logger API
> --
>
> Key: HADOOP-14542
> URL: https://issues.apache.org/jira/browse/HADOOP-14542
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Chen Liang
> Attachments: HADOOP-14542.001.patch, HADOOP-14542.002.patch
>
>
> Split from HADOOP-14539.
> Now IOUtils.cleanup only accepts commons-logging
[jira] [Commented] (HADOOP-14479) Erasurecode testcase failures with ISA-L
[ https://issues.apache.org/jira/browse/HADOOP-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056176#comment-16056176 ] Andrew Wang commented on HADOOP-14479: -- SGTM, thanks Kai! The concern on HADOOP-13342 was getting a stable reference to a single version, which is something we should address when bringing this back. > Erasurecode testcase failures with ISA-L > - > > Key: HADOOP-14479 > URL: https://issues.apache.org/jira/browse/HADOOP-14479 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.0.0-alpha3 > Environment: x86_64 Ubuntu 16.04.02 LTS >Reporter: Ayappan >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > I built hadoop with ISA-L support. I took the ISA-L code from > https://github.com/01org/isa-l (tag v2.18.0) and built it. While running the > UTs , following three testcases are failing > 1)TestHHXORErasureCoder > Tests run: 7, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 1.106 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder > testCodingDirectBuffer_10x4_erasing_p1(org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder) > Time elapsed: 0.029 sec <<< FAILURE! > java.lang.AssertionError: Decoding and comparing failed. > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.io.erasurecode.TestCoderBase.compareAndVerify(TestCoderBase.java:170) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.compareAndVerify(TestErasureCoderBase.java:141) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.performTestCoding(TestErasureCoderBase.java:98) > at > org.apache.hadoop.io.erasurecode.coder.TestErasureCoderBase.testCoding(TestErasureCoderBase.java:69) > at > org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder.testCodingDirectBuffer_10x4_erasing_p1(TestHHXORErasureCoder.java:64) > 2)TestRSErasureCoder > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - > in org.apache.hadoop.io.erasurecode.coder.TestXORCoder > Running org.apache.hadoop.io.erasurecode.coder.TestRSErasureCoder > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x7f486a28a6e4, pid=8970, tid=0x7f4850927700 > # > # JRE version: OpenJDK Runtime Environment (8.0_121-b13) (build > 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13) > # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 > compressed oops) > # Problematic frame: > # C [libc.so.6+0x8e6e4] > # > # Failed to write core dump. Core dumps have been disabled. To enable core > dumping, try "ulimit -c unlimited" before starting Java again > # > # An error report file with more information is saved as: > # /home/ayappan/hadoop/hadoop-common-project/hadoop-common/hs_err_pid8970.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > # The crash happened outside the Java Virtual Machine in native code. > # See problematic frame for where to report the bug. > # > 3)TestCodecRawCoderMapping > Running org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.559 sec <<< > FAILURE! - in org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping > testRSDefaultRawCoder(org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping) > Time elapsed: 0.015 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.io.erasurecode.TestCodecRawCoderMapping.testRSDefaultRawCoder(TestCodecRawCoderMapping.java:58) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14543) Should use getAversion() while setting the zkacl
[
https://issues.apache.org/jira/browse/HADOOP-14543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056174#comment-16056174
]
Hanisha Koneru commented on HADOOP-14543:
-
[~brahmareddy], did you see any error/ exception for this?
The _Zookeeper#setACL()_ method sets the version number of the SetACLRequest.
So ideally, we should be setting the version only and not the aversion.
{code}
request.setAcl(acl);
request.setVersion(version);
{code}
If the client doesn't get the correct version number of the data from the ACL,
the update/ delete operation would fail.
> Should use getAversion() while setting the zkacl
>
>
> Key: HADOOP-14543
> URL: https://issues.apache.org/jira/browse/HADOOP-14543
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-14543.patch
>
>
> while setting the zkacl we used {color:red}{{getVersion()}}{color} which is
> dataVersion,Ideally we should use {{{color:#14892c}getAversion{color}()}}. If
> there is any acl changes( i.e relam change/..) ,we set the ACL with
> dataversion which will cause {color:#d04437}BADVersion {color}and
> {color:#d04437}*process will not start*{color}. See
> [here|https://issues.apache.org/jira/browse/HDFS-11403?focusedCommentId=16051804=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051804]
> {{zkClient.setACL(path, zkAcl, stat.getVersion());}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14550) remove unused parameter of javadoc
[
https://issues.apache.org/jira/browse/HADOOP-14550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056128#comment-16056128
]
Chen Liang commented on HADOOP-14550:
-
Thanks [~xiaodong.hu] for the contribution, v001 patch LGTM. One minor thing I
noticed is that {{getTestDir(String subdir)}} of the same file also misses
{{@param}} in its doc, additionally, {{getFilesDiff}} also has incomplete
javadoc for its {{@param}} fields. It's up to you, but do you mind fixing these
as well?
> remove unused parameter of javadoc
> --
>
> Key: HADOOP-14550
> URL: https://issues.apache.org/jira/browse/HADOOP-14550
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: hu xiaodong
>Assignee: hu xiaodong
>Priority: Trivial
> Labels: documentation
> Attachments: HADOOP-14550.001.patch
>
>
> the param 'subpath' is unused in GenericTestUtils.getRandomizedTempPath. so
> the javadoc need to be modified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13728) S3A can support short user-friendly aliases for configuration of credential providers.
[ https://issues.apache.org/jira/browse/HADOOP-13728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056104#comment-16056104 ] Steve Loughran commented on HADOOP-13728: - You started this yet? > S3A can support short user-friendly aliases for configuration of credential > providers. > -- > > Key: HADOOP-13728 > URL: https://issues.apache.org/jira/browse/HADOOP-13728 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Priority: Minor > > This issue proposes to support configuration of the S3A credential provider > chain using short aliases to refer to the common credential providers in > addition to allowing full class names. Supporting short aliases would > provide a simpler operations experience for the most common cases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14542) Add IOUtils.cleanup or something that accepts slf4j logger API
[
https://issues.apache.org/jira/browse/HADOOP-14542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HADOOP-14542:
Attachment: HADOOP-14542.002.patch
Thanks [~ajisakaa] for the comments! Update v002 patch as you suggested.
> Add IOUtils.cleanup or something that accepts slf4j logger API
> --
>
> Key: HADOOP-14542
> URL: https://issues.apache.org/jira/browse/HADOOP-14542
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Akira Ajisaka
>Assignee: Chen Liang
> Attachments: HADOOP-14542.001.patch, HADOOP-14542.002.patch
>
>
> Split from HADOOP-14539.
> Now IOUtils.cleanup only accepts commons-logging logger API. Now we are
> migrating the APIs to slf4j, slf4j logger API should be accepted as well.
> Adding {{IOUtils.cleanup(Logger, Closeable...)}} causes
> {{IOUtils.cleanup(null, Closeable...)}} to fail (incompatible change), so
> it's better to change the method name to avoid the conflict.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14557) Document HADOOP-8143 (Change distcp to have -pb on by default)
Wei-Chiu Chuang created HADOOP-14557: Summary: Document HADOOP-8143 (Change distcp to have -pb on by default) Key: HADOOP-14557 URL: https://issues.apache.org/jira/browse/HADOOP-14557 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0-alpha4 Reporter: Wei-Chiu Chuang Priority: Trivial HADOOP-8143 is an incompatible change. I think it deserves an update in the distcp doc. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14556:
Description:
S3A to support delegation tokens where
* an authenticated client can request a token via
{{FileSystem.getDelegationToken()}}
* Amazon's token service is used to request short-lived session secret & id;
these will be saved in the token and marshalled with jobs
* A new authentication provider will look for a token for the current user and
authenticate the user if found
This will not support renewals; the lifespan of a token will be limited to the
initial duration. Also, as you can't request an STS token from a temporary
session, IAM instances won't be able to issue tokens.
was:
S3A to support delegation tokens where
* authenticated client can request a token via
{{FileSystem.getDelegationToken()}}
* STS is used to request short-lived session secret & id; these will be saved
in the token and marshalled with jobs
* new authentication provider to look for a token for the current user.
This will not support renewals; the lifespan of a token will be limited to the
initial duration. Also, as you can't request an STS token from a temporary
session, IAM instances won't be able to issue tokens.
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14556) S3A to support Delegation Tokens
Steve Loughran created HADOOP-14556:
---
Summary: S3A to support Delegation Tokens
Key: HADOOP-14556
URL: https://issues.apache.org/jira/browse/HADOOP-14556
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Steve Loughran
Assignee: Steve Loughran
S3A to support delegation tokens where
* authenticated client can request a token via
{{FileSystem.getDelegationToken()}}
* STS is used to request short-lived session secret & id; these will be saved
in the token and marshalled with jobs
* new authentication provider to look for a token for the current user.
This will not support renewals; the lifespan of a token will be limited to the
initial duration. Also, as you can't request an STS token from a temporary
session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056048#comment-16056048
]
Hadoop QA commented on HADOOP-14469:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 18
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
34s{color} | {color:green} root generated 0 new + 821 unchanged - 2 fixed = 821
total (was 823) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
28s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14469 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873694/HADOOP-14469-007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dad1e2a6ad2b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / dd65eea |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12577/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12577/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12577/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12577/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
>
[jira] [Updated] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-8143: Release Note: If -p option of distcp command is unspecified, block size is preserved. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056032#comment-16056032 ] Wei-Chiu Chuang commented on HADOOP-8143: - I am sorry I just noticed this now. This incompatible deserves a release note. Would anyone up for a short description? I would also recommend updating the DistCp documentation to note this change. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056016#comment-16056016 ] Hudson commented on HADOOP-8143: SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11895 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11895/]) HADOOP-8143. Change distcp to have -pb on by default. Contributed by (jlowe: rev dd65eea74b1f9dde858ff34df8111e5340115511) * (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055963#comment-16055963
]
Xiao Chen edited comment on HADOOP-14521 at 6/20/17 3:43 PM:
-
bq. pre-commit
I tried to ran {{TestAclsEndToEnd}} before and after applying this patch on
branch-2.8. It was passing before, failing after. Could you try the same?
Perhaps something odd happening on my local. (I didn't look inside except for
running the tests)
bq. I don't fully understand your comment.
Your patch is cool, just the tool was confused due to the naming.
{{smart-apply-patch}} uses the wiki's naming convention to choose patch for
trunk, branch-2, branch-2.8 etc. I was saying running
{{dev-support/bin/smart-apply-patch HADOOP-14521}} was trying to apply the
branch-2.8 patch when running on trunk:
{quote}
$ ./dev-support/bin/smart-apply-patch HADOOP-14521
Processing: HADOOP-14521
HADOOP-14521 patch is being downloaded at Tue Jun 20 08:31:28 PDT 2017 from
https://issues.apache.org/jira/secure/attachment/12873350/HDFS-11804-branch-2.8.patch
-> Downloaded
ERROR: Aborting! HADOOP-14521 cannot be verified.
{quote}
Maybe we can improve yetus on that since as you said, pre-commit ran fine. This
is NBD for this jira as I said, since I can download and apply the patch
manually.
was (Author: xiaochen):
bq. pre-commit
I tried to ran {{TestAclsEndToEnd}} before and after applying this patch on
branch-2.8. It was passing before, failing after. Could you try the same?
Perhaps something odd happening on my local. (I didn't look inside except for
running the tests)
bq. I don't fully understand your comment.
Your patch is cool, just the tool was confused due to the naming.
{{smart-apply-patch}} uses the wiki's naming convention to choose patch for
trunk, branch-2, branch-2.8 etc. I was saying running
{{dev-support/bin/smart-apply-patch HADOOP-14521}} was trying to apply the
branch-2.8 patch when running on trunk:
{quote}
$ ./dev-support/bin/smart-apply-patch HADOOP-14521
Processing: HADOOP-14521
HADOOP-14521 patch is being downloaded at Tue Jun 20 08:31:28 PDT 2017 from
https://issues.apache.org/jira/secure/attachment/12873350/HDFS-11804-branch-2.8.patch
-> Downloaded
ERROR: Aborting! HADOOP-14521 cannot be verified.
{quote}
Maybe we can improve yetus on that since as you said, pre-commit ran fine.
> KMS client needs retry logic
>
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14521.09.patch, HDFS-11804-branch-2.8.patch,
> HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch,
> HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch,
> HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055963#comment-16055963
]
Xiao Chen commented on HADOOP-14521:
bq. pre-commit
I tried to ran {{TestAclsEndToEnd}} before and after applying this patch on
branch-2.8. It was passing before, failing after. Could you try the same?
Perhaps something odd happening on my local. (I didn't look inside except for
running the tests)
bq. I don't fully understand your comment.
Your patch is cool, just the tool was confused due to the naming.
{{smart-apply-patch}} uses the wiki's naming convention to choose patch for
trunk, branch-2, branch-2.8 etc. I was saying running
{{dev-support/bin/smart-apply-patch HADOOP-14521}} was trying to apply the
branch-2.8 patch when running on trunk:
{quote}
$ ./dev-support/bin/smart-apply-patch HADOOP-14521
Processing: HADOOP-14521
HADOOP-14521 patch is being downloaded at Tue Jun 20 08:31:28 PDT 2017 from
https://issues.apache.org/jira/secure/attachment/12873350/HDFS-11804-branch-2.8.patch
-> Downloaded
ERROR: Aborting! HADOOP-14521 cannot be verified.
{quote}
Maybe we can improve yetus on that since as you said, pre-commit ran fine.
> KMS client needs retry logic
>
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14521.09.patch, HDFS-11804-branch-2.8.patch,
> HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch,
> HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch,
> HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-8143: --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Fix Version/s: 3.0.0-alpha4 Status: Resolved (was: Patch Available) Thanks to [~mithun] for the contribution and to [~aw] for additional review! I committed this to trunk. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055869#comment-16055869
]
Hongyuan Li edited comment on HADOOP-14469 at 6/20/17 2:56 PM:
---
resubmit a new patch to correct the following things:
1、fix the related findbug warn
2、make the test units more gracefully and more obvious
*Update/Correction*
3、this bug affects the {{FTPFileSystem}} since version 2.6.0
*Update*
the latest patch correct the spelling error
was (Author: hongyuan li):
resubmit a new patch to correct the following things:
1、fix the related findbug warn
2、make the test units more gracefully and more obvious
*Update/Correction*
3、this bug affects the {{FTPFileSystem}} since version 2.6.0
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch, HADOOP-14469-007.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Attachment: HADOOP-14469-007.patch
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch, HADOOP-14469-007.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8143) Change distcp to have -pb on by default
[ https://issues.apache.org/jira/browse/HADOOP-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055881#comment-16055881 ] Jason Lowe commented on HADOOP-8143: +1 for the latest patch. Committing this. > Change distcp to have -pb on by default > --- > > Key: HADOOP-8143 > URL: https://issues.apache.org/jira/browse/HADOOP-8143 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Dave Thompson >Assignee: Mithun Radhakrishnan >Priority: Minor > Attachments: HADOOP-8143.1.patch, HADOOP-8143.2.patch, > HADOOP-8143.3.patch > > > We should have the preserve blocksize (-pb) on in distcp by default. > checksum which is on by default will always fail if blocksize is not the same. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055869#comment-16055869
]
Hongyuan Li edited comment on HADOOP-14469 at 6/20/17 2:51 PM:
---
resubmit a new patch to correct the following things:
1、fix the related findbug warn
2、make the test units more gracefully and more obvious
*Update/Correction*
3、this bug affects the {{FTPFileSystem}} since version 2.6.0
was (Author: hongyuan li):
resubmit a new patch to correct the following things:
1、fix the related findbug warn
2、make the test units more gracefully and more obvious
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Affects Version/s: (was: 3.0.0-alpha3)
2.6.0
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 2.6.0
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055869#comment-16055869
]
Hongyuan Li commented on HADOOP-14469:
--
resubmit a new patch to correct the following things:
1、fix the related findbug warn
2、make the test units more gracefully and more obvious
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Attachment: HADOOP-14469-006.patch
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Attachment: (was: HADOOP-14469-005.patch)
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-006.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14469) FTPFileSystem#listStatus get currentPath and parentPath at the same time, causing recursively list action endless
[
https://issues.apache.org/jira/browse/HADOOP-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongyuan Li updated HADOOP-14469:
-
Attachment: HADOOP-14469-005.patch
> FTPFileSystem#listStatus get currentPath and parentPath at the same time,
> causing recursively list action endless
> -
>
> Key: HADOOP-14469
> URL: https://issues.apache.org/jira/browse/HADOOP-14469
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, tools/distcp
>Affects Versions: 3.0.0-alpha3
> Environment: ftp build by windows7 + Serv-U_64 12.1.0.8
> code runs any os
>Reporter: Hongyuan Li
>Assignee: Hongyuan Li
>Priority: Critical
> Attachments: HADOOP-14469-001.patch, HADOOP-14469-002.patch,
> HADOOP-14469-003.patch, HADOOP-14469-004.patch, HADOOP-14469-005.patch,
> HADOOP-14469-005.patch
>
>
> for some ftpsystems, liststatus method will return new Path(".") and new
> Path(".."), thus causing list op looping.for example, Serv-U
> We can see the logic in code below:
> {code}
> private FileStatus[] listStatus(FTPClient client, Path file)
> throws IOException {
> ……
> FileStatus[] fileStats = new FileStatus[ftpFiles.length];
> for (int i = 0; i < ftpFiles.length; i++) {
> fileStats[i] = getFileStatus(ftpFiles[i], absolute);
> }
> return fileStats;
> }
> {code}
> {code}
> public void test() throws Exception{
> FTPFileSystem ftpFileSystem = new FTPFileSystem();
> ftpFileSystem.initialize(new
> Path("ftp://test:123456@192.168.44.1/;).toUri(),
> new Configuration());
> FileStatus[] fileStatus = ftpFileSystem.listStatus(new Path("/new"));
> for(FileStatus fileStatus1 : fileStatus)
> System.out.println(fileStatus1);
> }
> {code}
> using test code below, the test results list below
> {code}
> FileStatus{path=ftp://test:123456@192.168.44.1/new; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop; isDirectory=true;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14431-002.patch;
> isDirectory=false; length=2036; replication=1; blocksize=4096;
> modification_time=149579778; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/HADOOP-14486-001.patch;
> isDirectory=false; length=1322; replication=1; blocksize=4096;
> modification_time=149671698; access_time=0; owner=user; group=group;
> permission=-; isSymlink=false}
> FileStatus{path=ftp://test:123456@192.168.44.1/new/hadoop-main;
> isDirectory=true; modification_time=149579712; access_time=0; owner=user;
> group=group; permission=-; isSymlink=false}
> {code}
> In results above, {{FileStatus{path=ftp://test:123456@192.168.44.1/new; ……}}
> is obviously the current Path, and
> {{FileStatus{path=ftp://test:123456@192.168.44.1/;……}} is obviously the
> parent Path.
> So, if we want to walk the directory recursively, it will stuck.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-9189) "mvn package -Pdist" generates the docs even without the -Pdocs option
[
https://issues.apache.org/jira/browse/HADOOP-9189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor resolved HADOOP-9189.
--
Resolution: Duplicate
> "mvn package -Pdist" generates the docs even without the -Pdocs option
> --
>
> Key: HADOOP-9189
> URL: https://issues.apache.org/jira/browse/HADOOP-9189
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.2-alpha
>Reporter: Karthik Kambatla
>Assignee: Andras Bokor
>
> {{mvn package}} shouldn't generate docs without {{-Pdocs}} option being
> specified.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055823#comment-16055823 ] Rushabh S Shah commented on HADOOP-14521: - bq. In general it'd be nice to confirm the failed tests are unrelated. >From [this comment | >https://issues.apache.org/jira/browse/HADOOP-14521?focusedCommentId=16050827=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16050827], > I did verify all the failed test were unrelated. > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HADOOP-14521.09.patch, HDFS-11804-branch-2.8.patch, > HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch, > HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch, > HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[
https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055821#comment-16055821
]
Rushabh S Shah commented on HADOOP-14521:
-
bq. TestAclsEndToEnd failure seems to be caused by this change.
This test failed in [HDFS-11885 Precommit
|https://issues.apache.org/jira/browse/HDFS-11885?focusedCommentId=16040117=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16040117]
also.
I have also created jira HDFS-11944 and will work on that soon.
Last 2 patches were just log messages change and addressing checkstyle issues
that's why I didn't run all the failed tests again.
bq. So please follow wiki's patch naming convention so the branch-2.8 patch
can be applied automatically
I did create the branch-2.8 patch. The last precommit build that ran was for
branch-2.8 {{git revision -- branch-2.8 / e47ec94}}.
I don't fully understand your comment.
> KMS client needs retry logic
>
>
> Key: HADOOP-14521
> URL: https://issues.apache.org/jira/browse/HADOOP-14521
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HADOOP-14521.09.patch, HDFS-11804-branch-2.8.patch,
> HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch,
> HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch,
> HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch
>
>
> The kms client appears to have no retry logic – at all. It's completely
> decoupled from the ipc retry logic. This has major impacts if the KMS is
> unreachable for any reason, including but not limited to network connection
> issues, timeouts, the +restart during an upgrade+.
> This has some major ramifications:
> # Jobs may fail to submit, although oozie resubmit logic should mask it
> # Non-oozie launchers may experience higher rates if they do not already have
> retry logic.
> # Tasks reading EZ files will fail, probably be masked by framework reattempts
> # EZ file creation fails after creating a 0-length file – client receives
> EDEK in the create response, then fails when decrypting the EDEK
> # Bulk hadoop fs copies, and maybe distcp, will prematurely fail
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14533) Size of args cannot be less than zero in TraceAdmin#run as its linkedlist
[
https://issues.apache.org/jira/browse/HADOOP-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055726#comment-16055726
]
Hudson commented on HADOOP-14533:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11893 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11893/])
HADOOP-14533. Size of args cannot be less than zero in TraceAdmin#run as
(brahma: rev 2e9daa2e27536484846d9c5f020c6afaf27932ce)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/TraceAdmin.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTraceAdmin.java
> Size of args cannot be less than zero in TraceAdmin#run as its linkedlist
> -
>
> Key: HADOOP-14533
> URL: https://issues.apache.org/jira/browse/HADOOP-14533
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, tracing
>Affects Versions: 2.6.0
>Reporter: Weisen Han
>Assignee: Weisen Han
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14533-001.patch, HADOOP-14533-002.patch
>
>
> {code}
> @Override
> public int run(String argv[]) throws Exception {
> LinkedList args = new LinkedList();
> ……
> if (args.size() < 0) {
> System.err.println("You must specify an operation.");
> return 1;
> }
> ……
> }
> {code}
> From the code above, the {{args}} is a linklist obejct, so it cannot be less
> than zero.meaning that code below is wrong
> {code}
> if (args.size() < 0) {
> System.err.println("You must specify an operation.");
> return 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14533) Size of args cannot be less than zero in TraceAdmin#run as its linkedlist
[
https://issues.apache.org/jira/browse/HADOOP-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-14533:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.2
3.0.0-alpha4
2.9.0
Status: Resolved (was: Patch Available)
Committed to {{trunk}},{{branch-2}} and {{branch-2.8}}. [~Weisen Han] thanks
for reporting and contributing this.
thanks to [~jojochuang] and [~vagarychen] for additional review.
> Size of args cannot be less than zero in TraceAdmin#run as its linkedlist
> -
>
> Key: HADOOP-14533
> URL: https://issues.apache.org/jira/browse/HADOOP-14533
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, tracing
>Affects Versions: 2.6.0
>Reporter: Weisen Han
>Assignee: Weisen Han
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: HADOOP-14533-001.patch, HADOOP-14533-002.patch
>
>
> {code}
> @Override
> public int run(String argv[]) throws Exception {
> LinkedList args = new LinkedList();
> ……
> if (args.size() < 0) {
> System.err.println("You must specify an operation.");
> return 1;
> }
> ……
> }
> {code}
> From the code above, the {{args}} is a linklist obejct, so it cannot be less
> than zero.meaning that code below is wrong
> {code}
> if (args.size() < 0) {
> System.err.println("You must specify an operation.");
> return 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14533) Size of args cannot be less than zero in TraceAdmin#run as its linkedlist
[
https://issues.apache.org/jira/browse/HADOOP-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-14533:
--
Summary: Size of args cannot be less than zero in TraceAdmin#run as its
linkedlist (was: TraceAdmin#run, the size of args cannot be less than
zero,which is a linklist)
> Size of args cannot be less than zero in TraceAdmin#run as its linkedlist
> -
>
> Key: HADOOP-14533
> URL: https://issues.apache.org/jira/browse/HADOOP-14533
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, tracing
>Affects Versions: 2.6.0
>Reporter: Weisen Han
>Assignee: Weisen Han
>Priority: Trivial
> Attachments: HADOOP-14533-001.patch, HADOOP-14533-002.patch
>
>
> {code}
> @Override
> public int run(String argv[]) throws Exception {
> LinkedList args = new LinkedList();
> ……
> if (args.size() < 0) {
> System.err.println("You must specify an operation.");
> return 1;
> }
> ……
> }
> {code}
> From the code above, the {{args}} is a linklist obejct, so it cannot be less
> than zero.meaning that code below is wrong
> {code}
> if (args.size() < 0) {
> System.err.println("You must specify an operation.");
> return 1;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14555) document how to run wasb tests in azure docs site/testing.md
[
https://issues.apache.org/jira/browse/HADOOP-14555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055568#comment-16055568
]
Steve Loughran commented on HADOOP-14555:
-
+include policy: no patch review without submitters declaration of endpoint
tested against.
The hadoop-aws testing.md can act as the basis for this doc
> document how to run wasb tests in azure docs site/testing.md
>
>
> Key: HADOOP-14555
> URL: https://issues.apache.org/jira/browse/HADOOP-14555
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> There's no single (current) documentation on running the azure tests
> * There's some in site/index.md, but iit looks potentially out of date
> (refers to an older azure SDK version)
> * There's a file
> {{src/test/org/apache/hadoop/fs/azure/RunningLiveWasbTests.txt}} which
> refers to a nonexistent doc {{hadoop-tools/hadoop-azure/README.txt }} for
> instructions.
> Proposed:
> # move testing docs out of main azure doc page, with link from there.
> # bring up to date with SDK, move of tests to ITests.
> # purge all other references, including bits of test javadocs which are no
> longer correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14555) document how to run wasb tests in azure docs site/testing.md
Steve Loughran created HADOOP-14555:
---
Summary: document how to run wasb tests in azure docs
site/testing.md
Key: HADOOP-14555
URL: https://issues.apache.org/jira/browse/HADOOP-14555
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 2.8.1
Reporter: Steve Loughran
There's no single (current) documentation on running the azure tests
* There's some in site/index.md, but iit looks potentially out of date (refers
to an older azure SDK version)
* There's a file
{{src/test/org/apache/hadoop/fs/azure/RunningLiveWasbTests.txt}} which refers
to a nonexistent doc {{hadoop-tools/hadoop-azure/README.txt }} for instructions.
Proposed:
# move testing docs out of main azure doc page, with link from there.
# bring up to date with SDK, move of tests to ITests.
# purge all other references, including bits of test javadocs which are no
longer correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14554) TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest
[
https://issues.apache.org/jira/browse/HADOOP-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14554:
Description:
the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
{{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
you run this test, it reruns all those base test cases.
was:
the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
{{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
you run this test, it reruns all those base test cases.
Proposed: have an abstract base class for azure FS tests which extends contract
tests, move all tests underneath that in a hierarchy which stops tests being
rerun when not needed
> TestFileSystemOperationExceptionMessage to not rerun all of
> NativeAzureFileSystemBaseTest
> -
>
> Key: HADOOP-14554
> URL: https://issues.apache.org/jira/browse/HADOOP-14554
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
> {{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
> you run this test, it reruns all those base test cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14554) TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest
[
https://issues.apache.org/jira/browse/HADOOP-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055548#comment-16055548
]
Steve Loughran commented on HADOOP-14554:
-
Moving under {{AbstractWasbTestBase}} should fix
> TestFileSystemOperationExceptionMessage to not rerun all of
> NativeAzureFileSystemBaseTest
> -
>
> Key: HADOOP-14554
> URL: https://issues.apache.org/jira/browse/HADOOP-14554
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>
> the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
> {{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
> you run this test, it reruns all those base test cases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14554) TestFileSystemOperationExceptionMessage to not rerun all of NativeAzureFileSystemBaseTest
Steve Loughran created HADOOP-14554:
---
Summary: TestFileSystemOperationExceptionMessage to not rerun all
of NativeAzureFileSystemBaseTest
Key: HADOOP-14554
URL: https://issues.apache.org/jira/browse/HADOOP-14554
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 2.8.1
Reporter: Steve Loughran
the Wasb test {{TestFileSystemOperationExceptionMessage}} subclasses
{{NativeAzureFileSystemBaseTest}} to test a new situation. As a result, when
you run this test, it reruns all those base test cases.
Proposed: have an abstract base class for azure FS tests which extends contract
tests, move all tests underneath that in a hierarchy which stops tests being
rerun when not needed
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14553) Move wasb tests to Integration tests, parallelize
Steve Loughran created HADOOP-14553:
---
Summary: Move wasb tests to Integration tests, parallelize
Key: HADOOP-14553
URL: https://issues.apache.org/jira/browse/HADOOP-14553
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 2.8.1
Reporter: Steve Loughran
The Azure tests are slow to run as they are serialized, as they are all called
Test* there's no clear differentiation from unit tests which Jenkins can run,
and integration tests which it can't.
Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
(which includes having separate paths for every test suite). The code in
hadoop-aws's POM show what to do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13754) Hadoop-Azure Update WASB URI format to support SAS token in it.
[ https://issues.apache.org/jira/browse/HADOOP-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13754: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14552 > Hadoop-Azure Update WASB URI format to support SAS token in it. > --- > > Key: HADOOP-13754 > URL: https://issues.apache.org/jira/browse/HADOOP-13754 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Sumit Dubey >Assignee: Sumit Dubey > Attachments: HADOOP-13754-branch-2.7.3.patch > > Original Estimate: 3h > Remaining Estimate: 3h > > Currently Azure WASB adapter code supports wasb url in this format > wasb://[containername@]youraccount.blob.core.windows.net/testDir with the > credentials retrieved from configuration and scoped to a container. > With this change we want > 1) change the url to contain file level sas token in the url > wasb://[containername[:]]@youraccount.blob.core.windows.net/testDir > 2) Scope access to a blob/file level. > 3) Tests to test the new url format -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13134) WASB's file delete still throwing Blob not found exception
[ https://issues.apache.org/jira/browse/HADOOP-13134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13134: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14552 > WASB's file delete still throwing Blob not found exception > -- > > Key: HADOOP-13134 > URL: https://issues.apache.org/jira/browse/HADOOP-13134 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.1 >Reporter: Lin Chan >Assignee: Dushyanth > > WASB is still throwing blob not found exception as shown in the following > stack. Need to catch that and convert to Boolean return code in WASB delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11196) Implement isFileClosed in WASB to make Flume happy
[ https://issues.apache.org/jira/browse/HADOOP-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11196: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14552 > Implement isFileClosed in WASB to make Flume happy > -- > > Key: HADOOP-11196 > URL: https://issues.apache.org/jira/browse/HADOOP-11196 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.5.1 >Reporter: Mostafa Elhemali >Priority: Minor > > Flume uses reflection to call isFileClosed() which is a method implemented by > DFS but not part of the FileSystem abstract base class. It still works if the > function is not implemented, but issues a lot of warnings and is distracting. > It would help make it work smoother if we added the same method in the > NativeAzureFileSystem class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14183:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-14552
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Priority: Minor
> Labels: newbie
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13733) Support WASB connections through an HTTP proxy server.
[ https://issues.apache.org/jira/browse/HADOOP-13733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13733: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14552 > Support WASB connections through an HTTP proxy server. > -- > > Key: HADOOP-13733 > URL: https://issues.apache.org/jira/browse/HADOOP-13733 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Chris Nauroth > > WASB currently does not support use of an HTTP proxy server to connect to the > Azure Storage back-end. The Azure Storage SDK does support use of a proxy, > so we can enhance WASB to read proxy settings from configuration and pass > them along in the Azure Storage SDK calls. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13783) Improve efficiency of WASB over page blobs
[ https://issues.apache.org/jira/browse/HADOOP-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13783: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14552 > Improve efficiency of WASB over page blobs > -- > > Key: HADOOP-13783 > URL: https://issues.apache.org/jira/browse/HADOOP-13783 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > > 1)Add telemetry to WASB driver. WASB driver is lack of any log or > telemetry which makes trouble shoot very difficult. For example, we don’t > know where is high e2e latency between HBase and Azure storage came from when > Azure storage server latency was very low. Also we don’t know why WASB can > only do 166 IOPs which is way below azure storage 500 IOPs. And we had > several incidents before related to storage latency, because of lacking logs, > we couldn’t find the ownership of the incident quickly. > 2)Resolving the hot spotting issue when WASB driver partition the azure > page blobs by changing the key. Current key design is causing the hot > spotting on azure storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14552) Über-jira: WASB client phase II: performance and testing
[ https://issues.apache.org/jira/browse/HADOOP-14552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14552: Summary: Über-jira: WASB client phase II: performance and testing (was: WASB client phase II: performance and testing) > Über-jira: WASB client phase II: performance and testing > > > Key: HADOOP-14552 > URL: https://issues.apache.org/jira/browse/HADOOP-14552 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.8.1 >Reporter: Steve Loughran > > Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9. > I think the focus is on performance, but looking at the tests there are some > things we need there first: parallel test execution, move over to IT tests, > other tuning. > All patches must come with declarations of which azure endpoint they were > tested against. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14490) Upgrade azure-storage sdk version >5.2.0
[
https://issues.apache.org/jira/browse/HADOOP-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14490:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-14552
> Upgrade azure-storage sdk version >5.2.0
>
>
> Key: HADOOP-14490
> URL: https://issues.apache.org/jira/browse/HADOOP-14490
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Mingliang Liu
>Assignee: Rajesh Balamohan
>
> As required by [HADOOP-14478], we're expecting the {{BlobInputStream}} to
> support advanced {{readFully()}} by taking hints of mark. This can only be
> done by means of sdk version bump.
> cc: [~rajesh.balamohan].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14536) Update azure-storage sdk to version 5.3.0
[ https://issues.apache.org/jira/browse/HADOOP-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055526#comment-16055526 ] Steve Loughran commented on HADOOP-14536: - Now I'm confused. Why not just submit this patch for HADOOP-14490 (we'll just change its title to the exact JAR thats in)? Also, need declaration of azure endpoint you ran all the tests against > Update azure-storage sdk to version 5.3.0 > - > > Key: HADOOP-14536 > URL: https://issues.apache.org/jira/browse/HADOOP-14536 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Mingliang Liu >Assignee: Georgi Chalakov > Attachments: HADOOP-14536-01.patch > > > Update WASB driver to use the latest version (5.3.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.3: > * Fixed a bug where the transactional MD5 check would fail when downloading > a range of blob or file and the recovery action is performed on a subsection > of the range. > * Fixed leaking connections for table requests. > * Fixed a bug where retries happened immediately when experiencing a network > exception uploading data or getting the response. > * Fixed a bug where the response stream was not being closed on nonretryable > exceptions. > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14536) Update azure-storage sdk to version 5.3.0
[ https://issues.apache.org/jira/browse/HADOOP-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055526#comment-16055526 ] Steve Loughran edited comment on HADOOP-14536 at 6/20/17 10:29 AM: --- Now I'm confused. Why not just submit this patch for HADOOP-14490 (we'll just change its title to the exact JAR thats in)? Also, need declaration of the specific azure endpoint you ran all the tests against was (Author: ste...@apache.org): Now I'm confused. Why not just submit this patch for HADOOP-14490 (we'll just change its title to the exact JAR thats in)? Also, need declaration of azure endpoint you ran all the tests against > Update azure-storage sdk to version 5.3.0 > - > > Key: HADOOP-14536 > URL: https://issues.apache.org/jira/browse/HADOOP-14536 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Mingliang Liu >Assignee: Georgi Chalakov > Attachments: HADOOP-14536-01.patch > > > Update WASB driver to use the latest version (5.3.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.3: > * Fixed a bug where the transactional MD5 check would fail when downloading > a range of blob or file and the recovery action is performed on a subsection > of the range. > * Fixed leaking connections for table requests. > * Fixed a bug where retries happened immediately when experiencing a network > exception uploading data or getting the response. > * Fixed a bug where the response stream was not being closed on nonretryable > exceptions. > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14516) Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients
[ https://issues.apache.org/jira/browse/HADOOP-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14516: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14552 > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients > --- > > Key: HADOOP-14516 > URL: https://issues.apache.org/jira/browse/HADOOP-14516 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov > > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.2: > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14552) WASB client phase II: performance and testing
Steve Loughran created HADOOP-14552: --- Summary: WASB client phase II: performance and testing Key: HADOOP-14552 URL: https://issues.apache.org/jira/browse/HADOOP-14552 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 2.8.1 Reporter: Steve Loughran Uber-JIRA for wasb:// phase II: the things we want for Hadoop 2.9. I think the focus is on performance, but looking at the tests there are some things we need there first: parallel test execution, move over to IT tests, other tuning. All patches must come with declarations of which azure endpoint they were tested against. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[ https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14518: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14552 > Customize User-Agent header sent in HTTP/HTTPS requests by WASB. > > > Key: HADOOP-14518 > URL: https://issues.apache.org/jira/browse/HADOOP-14518 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov >Assignee: Georgi Chalakov >Priority: Minor > Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt, > HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch > > > WASB passes a User-Agent header to the Azure back-end. Right now, it uses the > default value set by the Azure Client SDK, so Hadoop traffic doesn't appear > any different from general Blob traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by Azure > service. > The following configuration > > fs.azure.user.agent.id > MSFT > > set the user agent to > User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0 > (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3) > Test Results : > Tests run: 703, Failures: 0, Errors: 0, Skipped: 119 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
[
https://issues.apache.org/jira/browse/HADOOP-14518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055516#comment-16055516
]
Steve Loughran commented on HADOOP-14518:
-
Test failures appear unrelated; I'll have to see if I can replicate them & work
out what changed. I can't see how it has broken from your patch, certainly.
Regarding this patch, the production code looks OK.
# I'd like to see a test which sets the property to something other than the
default, just to make sure that codepath doesn't have surprises. You can just
add it to some existing test suite, like {{TestWasbUriAndConfiguration}} or
similar: just set the property one time an agent is created. Maybe also unset
it entirely to see what happens.
* I'll require a declaration from you as to which azure endpoint you tested
against. I'm confident you have the ability to do this, I just want the
rigorousness. Yetus/Jenkins can't test the azure client (no credentials), so we
absolutely require that due diligence from the submitter. (we committers
generally do a test run too, but I don't want to invest the time until its
ready to play with)
> Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
>
>
> Key: HADOOP-14518
> URL: https://issues.apache.org/jira/browse/HADOOP-14518
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.0.0-alpha3
>Reporter: Georgi Chalakov
>Assignee: Georgi Chalakov
>Priority: Minor
> Attachments: HADOOP-14518-01.patch, HADOOP-14518-01-test.txt,
> HADOOP-14518-02.patch, HADOOP-14518-03.patch, HADOOP-14518-04.patch
>
>
> WASB passes a User-Agent header to the Azure back-end. Right now, it uses the
> default value set by the Azure Client SDK, so Hadoop traffic doesn't appear
> any different from general Blob traffic. If we customize the User-Agent
> header, then it will enable better troubleshooting and analysis by Azure
> service.
> The following configuration
>
> fs.azure.user.agent.id
> MSFT
>
> set the user agent to
> User-Agent: WASB/3.0.0-alpha4-SNAPSHOT (MSFT) Azure-Storage/4.2.0
> (JavaJRE 1.8.0_131; WindowsServer2012R2 6.3)
> Test Results :
> Tests run: 703, Failures: 0, Errors: 0, Skipped: 119
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14551) F3A init hangs if you try to connect while the system is offline
[
https://issues.apache.org/jira/browse/HADOOP-14551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055488#comment-16055488
]
Steve Loughran commented on HADOOP-14551:
-
OS gives up after 20s. (Notable: it could have failed faster. mDNS trying to
ask around?)
{code}
time nslookup s3.amaazon.com
;; connection timed out; no servers could be reached
18.09 real 0.04 user 0.02 sys
{code}
No active NICs.
{code}
> ifconfig
lo0: flags=8049 mtu 16384
options=1203
inet 127.0.0.1 netmask 0xff00
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201
gif0: flags=8010 mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8823 mtu 1500
ether f4:5c:89:a3:82:99
nd6 options=201
media: autoselect ()
status: inactive
en1: flags=963 mtu 1500
options=60
ether 6a:00:01:c6:4d:00
media: autoselect
status: inactive
en2: flags=963 mtu 1500
options=60
ether 6a:00:01:c6:4d:01
media: autoselect
status: inactive
p2p0: flags=8802 mtu 2304
ether 06:5c:89:a3:82:99
media: autoselect
status: inactive
bridge0: flags=8863 mtu 1500
options=63
ether 6a:00:01:c6:4d:00
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x2
member: en1 flags=3
ifmaxaddr 0 port 5 priority 0 path cost 0
member: en2 flags=3
ifmaxaddr 0 port 6 priority 0 path cost 0
nd6 options=201
media:
status: inactive
awdl0: flags=8902 mtu 1484
ether ba:d6:00:b9:42:05
nd6 options=201
media: autoselect
status: inactive
utun0: flags=8051 mtu 2000
inet6 fe80::bffe:5ddf:7506:15ca%utun0 prefixlen 64 scopeid 0xa
nd6 options=201
{code}
meanwhile, the S3 client retries, without seeming to fail the test. I got bored
eventually
{code}
at java.lang.Thread.run(Thread.java:745)
"Thread-0" #10 prio=5 os_prio=31 tid=0x7fc783b79800 nid=0x5703 waiting on
condition [0x7ca4b000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doPauseBeforeRetry(AmazonHttpClient.java:1656)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.pauseBeforeRetry(AmazonHttpClient.java:1630)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1143)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1306)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1263)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:305)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:260)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3258)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3307)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3275)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at
