[jira] [Commented] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696457#comment-16696457
]
wujinhu commented on HADOOP-15943:
--
Thanks [~cheersyang] :)
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1, 2.9.3
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696444#comment-16696444
]
Weiwei Yang commented on HADOOP-15943:
--
I've committed the patch to trunk, branch-3.2, branch-3.1, branch-3.0, branch-2
and branch-2.9. Thanks for the contribution [~wujinhu].
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1, 2.9.3
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-15943:
-
Fix Version/s: 2.9.3
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1, 2.9.3
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-15943:
-
Fix Version/s: 2.10.0
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-15943:
-
Fix Version/s: 3.0.4
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-15943:
-
Fix Version/s: 3.1.2
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696415#comment-16696415
]
Hudson commented on HADOOP-15943:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15498 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15498/])
HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss (wwei:
rev 5ff0cf86a940fd83f1425794921cc075b19f1108)
* (add)
hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/OSSFileStatus.java
* (edit)
hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystemStore.java
* (edit)
hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java
* (edit)
hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-15943:
-
Fix Version/s: 3.2.1
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-15943:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15798) LocalMetadataStore put() does not retain isDeleted in parent listing
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696283#comment-16696283
]
Hadoop QA commented on HADOOP-15798:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
31s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15798 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949227/HADOOP-15798.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0b130c33d68a 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e223a79 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15554/testReport/ |
| Max. process+thread count | 310 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15554/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> LocalMetadataStore put() does not retain isDeleted in parent listing
>
>
> Key:
[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()
[
https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696276#comment-16696276
]
Hadoop QA commented on HADOOP-15229:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 21s{color} | {color:orange} root: The patch generated 24 new + 116 unchanged
- 0 fixed = 140 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 55 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
1s{color} | {color:red} hadoop-tools/hadoop-aws generated 3 new + 0 unchanged -
0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hadoop-tools_hadoop-aws generated 3 new + 1 unchanged
- 0 fixed = 4 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 57s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
40s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | org.apache.hadoop.fs.s3a.select.SelectConstants.SELECT_OPTIONS isn't
final but should be At SelectConstants.java:be At SelectConstants.java:[line
102] |
| | instanceof will always return true for all non-null values in
org.apache.hadoop.fs.s3a.select.SelectInputStream.abort(), since all
com.amazonaws.services.s3.model.SelectRecordsInputStream are instances of
[jira] [Comment Edited] (HADOOP-15798) LocalMetadataStore put() does not retain isDeleted in parent listing
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696271#comment-16696271
]
Gabor Bota edited comment on HADOOP-15798 at 11/22/18 9:40 PM:
---
Made some sense into
{{org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore#put(org.apache.hadoop.fs.s3a.s3guard.PathMetadata)}}
Update cached parent dir part while fixing the issue.
mvn verify ran against eu-west-1. Two test error:
testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)
testWithMiniCluster(org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster)
testDestroyNoBucket is known, but testWithMiniCluster is not, and seems
unrelated.
was (Author: gabor.bota):
Made some sense into
{{org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore#put(org.apache.hadoop.fs.s3a.s3guard.PathMetadata)}}
Update cached parent dir part while fixing the issue.
> LocalMetadataStore put() does not retain isDeleted in parent listing
>
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15798.001.patch
>
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15798) LocalMetadataStore put() does not retain isDeleted in parent listing
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696271#comment-16696271
]
Gabor Bota commented on HADOOP-15798:
-
Made some sense into
{{org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore#put(org.apache.hadoop.fs.s3a.s3guard.PathMetadata)}}
Update cached parent dir part while fixing the issue.
> LocalMetadataStore put() does not retain isDeleted in parent listing
>
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15798.001.patch
>
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15798) LocalMetadataStore put() does not retain isDeleted in parent listing
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15798:
Status: Patch Available (was: In Progress)
> LocalMetadataStore put() does not retain isDeleted in parent listing
>
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15798.001.patch
>
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15798) LocalMetadataStore put() does not retain isDeleted in parent listing
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15798:
Summary: LocalMetadataStore put() does not retain isDeleted in parent
listing (was: ITestS3GuardListConsistency#testConsistentListAfterDelete
failing with LocalMetadataStore)
> LocalMetadataStore put() does not retain isDeleted in parent listing
>
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15798.001.patch
>
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15798:
Attachment: HADOOP-15798.001.patch
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15798.001.patch
>
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[
https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696254#comment-16696254
]
Hadoop QA commented on HADOOP-15872:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-15872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949222/HADOOP-15872-005.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4bc28bf221c0 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e223a79 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15552/testReport/ |
| Max. process+thread count | 310 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15552/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
> -
>
> Key: HADOOP-15872
>
[jira] [Updated] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15229: Status: Patch Available (was: Open) > Add FileSystem builder-based openFile() API to match createFile() > - > > Key: HADOOP-15229 > URL: https://issues.apache.org/jira/browse/HADOOP-15229 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15229-001.patch, HADOOP-15229-002.patch, > HADOOP-15229-003.patch > > > Replicate HDFS-1170 and HADOOP-14365 with an API to open files. > A key requirement of this is not HDFS, it's to put in the fadvise policy for > working with object stores, where getting the decision to do a full GET and > TCP abort on seek vs smaller GETs is fundamentally different: the wrong > option can cost you minutes. S3A and Azure both have adaptive policies now > (first backward seek), but they still don't do it that well. > Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" > "random" as an option when they open files; I can imagine other options too. > The Builder model of [~eddyxu] is the one to mimic, method for method. > Ideally with as much code reuse as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()
[
https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696237#comment-16696237
]
Steve Loughran commented on HADOOP-15229:
-
Patch 003
This incorporates HADOOP-15364: S3 select, as a validation of this design
beyond just "we can do this"
A key change since before is that the return of the openFile().build() call is
actually a CompletableFuture; you have to call that to get the stream
* Base implementation actually evaluates in the build(), but hides the outcome
in the future, to line people up for lazy failures.
* FS Specs says "lazy eval", as does the test of when FNFEs surface
* S3A impl submits classic open and new select into its worker pool: the
GET/POST requests are all asynchronous.
* Tests in AbstractContractOpenTest for the openFile(path) sequence (but not
PathHandle, yet)
If you can line things up for async code execution then this will hide the
overhead of initiating an HTTP request, and the even longer delays of the
select. Otherwise, well, IOExceptions get wrapped more than they need to be.
The more you do with lambda expressions, the more checked exceptions suck.
In use the API Will look like
{code}
FileSystem.FSDataInputStreamBuilder builder =
filesystem.openFile("s3a://bucket/path-to-file.csv")
.must("s3a:fs.s3a.select.sql",
"SELECT * FROM S3OBJECT s WHERE s.\"odd\" = `TRUE`")
.must("s3a:fs.s3a.select.compression", "NONE")
.must("s3a:fs.s3a.select.csv.header", "use");
CompletableFuture future = builder.build();
try (FSDataInputStream select = future.get()) {
// process the output
stream.read();
}
{code}
> Add FileSystem builder-based openFile() API to match createFile()
> -
>
> Key: HADOOP-15229
> URL: https://issues.apache.org/jira/browse/HADOOP-15229
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15229-001.patch, HADOOP-15229-002.patch,
> HADOOP-15229-003.patch
>
>
> Replicate HDFS-1170 and HADOOP-14365 with an API to open files.
> A key requirement of this is not HDFS, it's to put in the fadvise policy for
> working with object stores, where getting the decision to do a full GET and
> TCP abort on seek vs smaller GETs is fundamentally different: the wrong
> option can cost you minutes. S3A and Azure both have adaptive policies now
> (first backward seek), but they still don't do it that well.
> Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise"
> "random" as an option when they open files; I can imagine other options too.
> The Builder model of [~eddyxu] is the one to mimic, method for method.
> Ideally with as much code reuse as possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15229: Attachment: HADOOP-15229-003.patch > Add FileSystem builder-based openFile() API to match createFile() > - > > Key: HADOOP-15229 > URL: https://issues.apache.org/jira/browse/HADOOP-15229 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15229-001.patch, HADOOP-15229-002.patch, > HADOOP-15229-003.patch > > > Replicate HDFS-1170 and HADOOP-14365 with an API to open files. > A key requirement of this is not HDFS, it's to put in the fadvise policy for > working with object stores, where getting the decision to do a full GET and > TCP abort on seek vs smaller GETs is fundamentally different: the wrong > option can cost you minutes. S3A and Azure both have adaptive policies now > (first backward seek), but they still don't do it that well. > Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" > "random" as an option when they open files; I can imagine other options too. > The Builder model of [~eddyxu] is the one to mimic, method for method. > Ideally with as much code reuse as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696216#comment-16696216 ] Da Zhou commented on HADOOP-15872: -- Attached 005 patch, fixed the checkstyle issue. Yes it is ready. *HNS account, using SharedKey* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 19 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 *HNS account, using OAuth* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 20 Tests run: 165, Failures: 0, Errors: 0, Skipped: 21 *Non-HNS account, using SharedKey:* Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 Tests run: 316, Failures: 0, Errors: 0, Skipped: 205 Tests run: 165, Failures: 0, Errors: 0, Skipped: 15 > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch, HADOOP-15872-005.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15872) ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
[ https://issues.apache.org/jira/browse/HADOOP-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-15872: - Attachment: HADOOP-15872-005.patch > ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2 > - > > Key: HADOOP-15872 > URL: https://issues.apache.org/jira/browse/HADOOP-15872 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Thomas Marquardt >Assignee: junhua gu >Priority: Major > Attachments: HADOOP-15872-001.patch, HADOOP-15872-002.patch, > HADOOP-15872-003.patch, HADOOP-15872-004.patch, HADOOP-15872-005.patch > > > This update to the latest REST version (2018-11-09) will make the following > changes to the ABFS driver: > 1) The ABFS implementation of getFileStatus currently requires read > permission. According to HDFS permissions guide, it should only require > execute on the parent folders (traversal access). A new REST API has been > introduced in REST version "2018-11-09" of ADLS Gen 2 to fix this problem. > 2) The new "2018-11-09" REST version introduces support to i) automatically > translate UPNs to OIDs when setting the owner, owning group, or ACL and ii) > optionally translate OIDs to UPNs in the responses when getting the owner, > owning group, or ACL. Configuration will be introduced to optionally > translate OIDs to UPNs in the responses. Since translation has a performance > impact, the default will be to perform no translation and return the OIDs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15798:
Comment: was deleted
(was: The fast and dirty fix would be to add
{code:java}
if(meta.isDeleted()) {
parentMeta.getDirListingMeta().markDeleted(path);
}
{code}
to {{LocalMetadataStore.java:281}} else branch, but I'd prefer a solution which
is more versatile: add a new {{put}} method to {{DirListingMetadata}} with
{{PathMetadata}} parameter type. This way we could add PathMetadata instead of
just a FileStatus, so additional parameters like isDeleted could be set
automatically and we could avoid problems like this. Currently running the
tests, will report back soon.)
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696146#comment-16696146
]
Gabor Bota edited comment on HADOOP-15798 at 11/22/18 5:32 PM:
---
The fast and dirty fix would be to add
{code:java}
if(meta.isDeleted()) {
parentMeta.getDirListingMeta().markDeleted(path);
}
{code}
to {{LocalMetadataStore.java:281}} else branch, but I'd prefer a solution which
is more versatile: add a new {{put}} method to {{DirListingMetadata}} with
{{PathMetadata}} parameter type. This way we could add PathMetadata instead of
just a FileStatus, so additional parameters like isDeleted could be set
automatically and we could avoid problems like this. Currently running the
tests, will report back soon.
was (Author: gabor.bota):
The fast and dirty fix would be to add
{code:java}
if(meta.isDeleted()) {
parentMeta.getDirListingMeta().markDeleted(path);
}
{code}
to {{LocalMetadataStore.java:281}} else branch, but I'd prefer a solution which
is more versatile: add a new {{put}} method to {{DirListingMetadata}} with
{{PathMetadata}} parameter type. This way we could add PathMetadata instead of
just a FileStatus, so additional parameters like isDeleted could be set and we
could avoid problems like this. Currently running the tests, will report back
soon.
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696146#comment-16696146
]
Gabor Bota commented on HADOOP-15798:
-
The fast and dirty fix would be to add
{code:java}
if(meta.isDeleted()) {
parentMeta.getDirListingMeta().markDeleted(path);
}
{code}
to {{LocalMetadataStore.java:281}} else branch, but I'd prefer a solution which
is more versatile: add a new {{put}} method to {{DirListingMetadata}} with
{{PathMetadata}} parameter type. This way we could add PathMetadata instead of
just a FileStatus, so additional parameters like isDeleted could be set and we
could avoid problems like this. Currently running the tests, will report back
soon.
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15798) ITestS3GuardListConsistency#testConsistentListAfterDelete failing with LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696050#comment-16696050
]
Gabor Bota commented on HADOOP-15798:
-
It's an implementation bug in
{{org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore#put(org.apache.hadoop.fs.s3a.s3guard.PathMetadata)}}.
{{parentDirMeta.put(status);}} and
{{parentMeta.getDirListingMeta().put(status);}} creates the element in the
DirListingMetadata for the child element even if its deleted (tombstone).
> ITestS3GuardListConsistency#testConsistentListAfterDelete failing with
> LocalMetadataStore
> -
>
> Key: HADOOP-15798
> URL: https://issues.apache.org/jira/browse/HADOOP-15798
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Test fails constantly when running with LocalMetadataStore.
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListAfterDelete(ITestS3GuardListConsistency.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696028#comment-16696028 ] lqjacklee commented on HADOOP-15870: [~ste...@apache.org] How can I upload the attachment of the patch? > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14693) Upgrade JUnit from 4 to 5
[
https://issues.apache.org/jira/browse/HADOOP-14693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695934#comment-16695934
]
Steve Loughran commented on HADOOP-14693:
-
Global timeout control is a critical part of long-lived tests: we want to stop
jenkins runs hanging. I put a lot of work into the S3A scale tests to make
those timeouts configurable from the CLI, so a 4GB test run could have a longer
timeout than a 32MB dataset, and {{org.apache.hadoop.test.HadoopTestBase}}
tries to provide the same for all its subclasses.
Looking at the github issue, it looks like we just need to write and ship with
our own timing extension, which we can sneak into hadoop-common test for all
our tests to pick up
> Upgrade JUnit from 4 to 5
> -
>
> Key: HADOOP-14693
> URL: https://issues.apache.org/jira/browse/HADOOP-14693
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Priority: Major
>
> JUnit 4 does not support Java 9. We need to upgrade this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15370) S3A log message on rm s3a://bucket/ not intuitive
[
https://issues.apache.org/jira/browse/HADOOP-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695892#comment-16695892
]
Steve Loughran commented on HADOOP-15370:
-
LGTM. As usual, where did you test it, what s3guard settings, etc?
> S3A log message on rm s3a://bucket/ not intuitive
> -
>
> Key: HADOOP-15370
> URL: https://issues.apache.org/jira/browse/HADOOP-15370
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Trivial
> Attachments: HADOOP-15370.001.patch
>
>
> when you try to delete the root of a bucket from command line, e.g. {{hadoop
> fs -rm -r -skipTrash s3a://hwdev-steve-new/}}, the output isn't that useful
> {code}
> 2018-04-06 16:35:23,048 [main] INFO s3a.S3AFileSystem
> (S3AFileSystem.java:rejectRootDirectoryDelete(1837)) - s3a delete the
> hwdev-steve-new root directory of true
> rm: `s3a://hwdev-steve-new/': Input/output error
> 2018-04-06 16:35:23,050 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
> {code}
> the single log message doesn't parse, and the error message raised is lost by
> the FS -rm CLI command (why?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15943) AliyunOSS: add missing owner & group attributes for oss FileStatus
[
https://issues.apache.org/jira/browse/HADOOP-15943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695851#comment-16695851
]
Weiwei Yang commented on HADOOP-15943:
--
Patch looks good, +1. I will commit this tomorrow if no one objects.
> AliyunOSS: add missing owner & group attributes for oss FileStatus
> --
>
> Key: HADOOP-15943
> URL: https://issues.apache.org/jira/browse/HADOOP-15943
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
>Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3, 3.3.0
>Reporter: wujinhu
>Assignee: wujinhu
>Priority: Major
> Attachments: HADOOP-15943.001.patch, HADOOP-15943.002.patch
>
>
> Owner & group attributes are missing when you list oss objects via hadoop
> command:
> {code:java}
> Found 6 items
> drwxrwxrwx - 0 2018-08-01 21:37 /1024
> drwxrwxrwx - 0 2018-10-30 11:07 /50
> -rw-rw-rw- 1 94070 2018-11-08 21:48 /a
> -rw-rw-rw- 1 2441079322 2018-10-31 10:14 /lineitem.csv
> drwxrwxrwx - 0 1970-01-01 08:00 /tmp
> drwxrwxrwx - 0 1970-01-01 08:00 /user
> {code}
>
> The result should like below(hadoop fs -ls hdfs://master:8020/):
> {code:java}
> Found 5 items
> drwxr-xr-x - hbase hbase 0 2018-11-18 17:31 hdfs://master:8020/hbase
> drwxrwxrwt - hdfs supergroup 0 2018-10-30 11:07 hdfs://master:8020/tmp
> drwxr-xr-x - hdfs supergroup 0 2018-10-30 10:39 hdfs://master:8020/user
> {code}
> However, oss objects do not have owner & group attributes like hadoop files,
> so we assume both owner & group are the current user at the time the FS was
> instantiated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15946) the Connection thread should notify all calls in finally clause before quit.
[ https://issues.apache.org/jira/browse/HADOOP-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HADOOP-15946: - Attachment: HADOOP-15946.patch > the Connection thread should notify all calls in finally clause before quit. > > > Key: HADOOP-15946 > URL: https://issues.apache.org/jira/browse/HADOOP-15946 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jinglun >Priority: Major > Attachments: HADOOP-15946.patch, issue-replay.patch > > > Threads that call Client.call() would wait forever unless the connection > thread notifies them, so the connection thread should try it's best to notify > when it's going to quit. > In Connection.close(), if any Throwable occurs before cleanupCalls(), the > connection thread will quit directly and leave all the waiting threads > waiting forever. So i think doing cleanupCalls() in finally clause might be a > good idea. > I met this problem when i started a hadoop2.6 DataNode with 8 block pools. > The DN successfully reported to 7 Namespaces and failed at the last Namespace > because the connection thread of the heartbeat rpc got a "OOME:Direct buffer > memory" and quit without calling cleanupCalls(). > I think we can move cleanupCalls() to finally clause as a protection, because > i notice in HADOOP-10940 the close of stream is changed to > IOUtils.closeStream(ipcStreams) which catches all Throwable, so the problem i > met was fixed. > issue-replay.patch simulates the case i described above. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15946) the Connection thread should notify all calls in finally clause before quit.
Jinglun created HADOOP-15946: Summary: the Connection thread should notify all calls in finally clause before quit. Key: HADOOP-15946 URL: https://issues.apache.org/jira/browse/HADOOP-15946 Project: Hadoop Common Issue Type: Improvement Reporter: Jinglun Attachments: issue-replay.patch Threads that call Client.call() would wait forever unless the connection thread notifies them, so the connection thread should try it's best to notify when it's going to quit. In Connection.close(), if any Throwable occurs before cleanupCalls(), the connection thread will quit directly and leave all the waiting threads waiting forever. So i think doing cleanupCalls() in finally clause might be a good idea. I met this problem when i started a hadoop2.6 DataNode with 8 block pools. The DN successfully reported to 7 Namespaces and failed at the last Namespace because the connection thread of the heartbeat rpc got a "OOME:Direct buffer memory" and quit without calling cleanupCalls(). I think we can move cleanupCalls() to finally clause as a protection, because i notice in HADOOP-10940 the close of stream is changed to IOUtils.closeStream(ipcStreams) which catches all Throwable, so the problem i met was fixed. issue-replay.patch simulates the case i described above. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
