[jira] [Work logged] (HDDS-1911) Support Prefix ACL operations for OM HA.
[ https://issues.apache.org/jira/browse/HDDS-1911?focusedWorklogId=292609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292609 ] ASF GitHub Bot logged work on HDDS-1911: Author: ASF GitHub Bot Created on: 11/Aug/19 05:35 Start Date: 11/Aug/19 05:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1275: HDDS-1911. Support Prefix ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1275#issuecomment-520201758 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 75 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 36 | Maven dependency ordering for branch | | +1 | mvninstall | 609 | trunk passed | | +1 | compile | 368 | trunk passed | | +1 | checkstyle | 71 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 934 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 433 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 630 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 555 | the patch passed | | +1 | compile | 373 | the patch passed | | +1 | javac | 373 | the patch passed | | +1 | checkstyle | 76 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 746 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 164 | the patch passed | | +1 | findbugs | 649 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 341 | hadoop-hdds in the patch passed. | | -1 | unit | 2238 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 8266 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1275 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 89b1c4b02a75 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d29007f | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/testReport/ | | Max. process+thread count | 4725 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292609) Time Spent: 20m (was: 10m) > Support Prefix ACL operations for OM HA. > > > Key: HDDS-1911 > URL: https://issues.apache.org/jira/browse/HDDS-1911 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee:
[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.
[ https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904561#comment-16904561 ] Wei-Chiu Chuang commented on HDFS-13505: Yes we should update the HDFS Permission Guide. Thanks for catching that [~ayushtkn]! > Turn on HDFS ACLs by default. > - > > Key: HDFS-13505 > URL: https://issues.apache.org/jira/browse/HDFS-13505 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch > > > Turn on HDFS ACLs by default. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1911) Support Prefix ACL operations for OM HA.
[ https://issues.apache.org/jira/browse/HDDS-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1911: - Status: Patch Available (was: Open) > Support Prefix ACL operations for OM HA. > > > Key: HDDS-1911 > URL: https://issues.apache.org/jira/browse/HDDS-1911 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > +-HDDS-1608-+ adds 4 new api for Ozone rpc client. OM HA implementation needs > to handle them. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1911) Support Prefix ACL operations for OM HA.
[ https://issues.apache.org/jira/browse/HDDS-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1911: - Labels: pull-request-available (was: ) > Support Prefix ACL operations for OM HA. > > > Key: HDDS-1911 > URL: https://issues.apache.org/jira/browse/HDDS-1911 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > +-HDDS-1608-+ adds 4 new api for Ozone rpc client. OM HA implementation needs > to handle them. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1911) Support Prefix ACL operations for OM HA.
[ https://issues.apache.org/jira/browse/HDDS-1911?focusedWorklogId=292604=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292604 ] ASF GitHub Bot logged work on HDDS-1911: Author: ASF GitHub Bot Created on: 11/Aug/19 03:16 Start Date: 11/Aug/19 03:16 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1275: HDDS-1911. Support Prefix ACL operations for OM HA. URL: https://github.com/apache/hadoop/pull/1275 (cherry picked from commit 3906b7d233cd276901fc86b24d7d26d676ff985b) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292604) Time Spent: 10m Remaining Estimate: 0h > Support Prefix ACL operations for OM HA. > > > Key: HDDS-1911 > URL: https://issues.apache.org/jira/browse/HDDS-1911 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > +-HDDS-1608-+ adds 4 new api for Ozone rpc client. OM HA implementation needs > to handle them. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904531#comment-16904531
]
Masatake Iwasaki commented on HDFS-14423:
-
Thanks [~jojochuang]. While the patch looks good, the issued reported as "plus
(+ ) characters get turned into spaces" still seems to be exists. I'm looking
into the cause.
{noformat}
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz
webhdfs://localhost/%
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz
webhdfs://localhost/a+b
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -mkdir
webhdfs://localhost/c+d
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls webhdfs://localhost/
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hadoop fs -put 'e+f.txt'
'webhdfs://localhost/'
put: rename `webhdfs://localhost/e+f.txt._COPYING_' to
`webhdfs://localhost/e+f.txt': Input/output error
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls /
Found 6 items
-rw-r--r-- 1 iwasakims supergroup 0 2019-08-09 23:35 /%
-rw-r--r-- 1 iwasakims supergroup 0 2019-08-09 23:50 /a b
drwxr-xr-x - iwasakims supergroup 0 2019-08-09 23:37 /c+d
-rw-r--r-- 1 iwasakims supergroup 5 2019-08-09 23:52 /e
f.txt._COPYING_
drwxrwxrwx - iwasakims supergroup 0 2019-06-14 13:09 /tmp
drwxr-xr-x - iwasakims supergroup 0 2019-06-14 13:10 /user
{noformat}
> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r-- 1 jing supergroup 0 2019-04-12 11:20 /a b
> drwxr-xr-x - jing supergroup 0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the
> right fix is. Note that Hive uses % to escape special characters in partition
> values, so banning % might not be a good option. For example, Hive will
> create a paths like {{table_name/partition_key=%2F}} when
> {{partition_key='/'}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904511#comment-16904511 ] Wei-Chiu Chuang commented on HDFS-14718: There are two things that should get considered: (1) does it break existing applications? TreeMap and HashMap are different in that the output may be in different order. I don't we ever specified the order so this isn't an incompatible change. (2) performance. which one is more performant if you list a million files? > HttpFS: Sort response by key names as WebHDFS does > -- > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13505) Turn on HDFS ACLs by default.
[
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904510#comment-16904510
]
Ayush Saxena edited comment on HDFS-13505 at 8/10/19 8:31 PM:
--
Thanx [~smeng] for the patch. Guess you missed updating in
{{HDFSPermissionsGuide}} as I mentioned above.
[~dineshchitlangia] can you give a check too on the release note, If you have
any objections, or want something to get added.
was (Author: ayushtkn):
Thanx [~smeng] for the patch. Guess you missed updating in
{{HDFSPermissionsGuide}} as I mentioned above.
> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.
[
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904510#comment-16904510
]
Ayush Saxena commented on HDFS-13505:
-
Thanx [~smeng] for the patch. Guess you missed updating in
{{HDFSPermissionsGuide}} as I mentioned above.
> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14595) HDFS-11848 breaks API compatibility
[
https://issues.apache.org/jira/browse/HDFS-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904509#comment-16904509
]
Ayush Saxena commented on HDFS-14595:
-
Thanx [~smeng] Almost LGTM, just like {{TestStripedFileAppend}} seems little
odd place to keep the test. That is for checking append in EC file. May be you
can try {{TestDistributedFileSystem.java}} for the same.
> HDFS-11848 breaks API compatibility
> ---
>
> Key: HDFS-14595
> URL: https://issues.apache.org/jira/browse/HDFS-14595
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Blocker
> Attachments: HDFS-14595.001.patch, HDFS-14595.002.patch,
> HDFS-14595.003.patch, HDFS-14595.004.patch, HDFS-14595.005.patch, hadoop_
> 36e1870eab904d5a6f12ecfb1fdb52ca08d95ac5 to
> b241194d56f97ee372cbec7062bcf155bc3df662 compatibility report.htm
>
>
> Our internal tool caught an API compatibility issue with HDFS-11848.
> HDFS-11848 adds an additional parameter to
> DistributedFileSystem.listOpenFiles(), but it doesn't keep the existing API.
> This can cause issue when upgrading from Hadoop 2.9.0/2.8.3/3.0.0 to
> 3.0.1/3.1.0 and above.
> Suggest:
> (1) Add back the old API (which was added in HDFS-10480), and mark it
> deprecated.
> (2) Update release doc to enforce running API compatibility check for each
> releases.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13270) RBF: Router audit logger
[
https://issues.apache.org/jira/browse/HDFS-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904498#comment-16904498
]
Hadoop QA commented on HDFS-13270:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 10s{color}
| {color:red} hadoop-hdfs-project generated 1 new + 552 unchanged - 0 fixed =
553 total (was 552) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-hdfs-project: The patch generated 34 new
+ 183 unchanged - 0 fixed = 217 total (was 183) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 52s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}205m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| | Write to static field
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.auditLoggers
from instance method new
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer(Configuration,
Router, ActiveNamenodeResolver, FileSubclusterResolver) At
RouterRpcServer.java:from instance method new
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[ https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292564 ] ASF GitHub Bot logged work on HDDS-1927: Author: ASF GitHub Bot Created on: 10/Aug/19 17:58 Start Date: 10/Aug/19 17:58 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520168308 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 15 | Maven dependency ordering for branch | | +1 | mvninstall | 595 | trunk passed | | +1 | compile | 371 | trunk passed | | +1 | checkstyle | 74 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 864 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 165 | trunk passed | | 0 | spotbugs | 420 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 616 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | +1 | mvninstall | 553 | the patch passed | | +1 | compile | 382 | the patch passed | | +1 | javac | 382 | the patch passed | | -0 | checkstyle | 43 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 681 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 94 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13) | | -1 | findbugs | 433 | hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 | unit | 285 | hadoop-hdds in the patch passed. | | -1 | unit | 54 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 5855 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-ozone | | | Possible null pointer dereference of null in org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo) Dereferenced at OmKeyInfo.java:null in org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo) Dereferenced at OmKeyInfo.java:[line 394] | | Failed junit tests | hadoop.ozone.om.helpers.TestOmKeyInfo | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1263 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fb697dc901b3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / e69db45 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/new-findbugs-hadoop-ozone.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/testReport/ | | Max. process+thread count | 557 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to
[jira] [Commented] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.
[ https://issues.apache.org/jira/browse/HDDS-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904480#comment-16904480 ] Hudson commented on HDDS-1366: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17085 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17085/]) HDDS-1366. Add ability in Recon to track the number of small files in an (arp7: rev d29007fb35d6667f9e8f1d9befafe61b19ca7c18) * (add) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/UtilizationService.java * (add) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FileSizeCountTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerKeyService.java * (edit) hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/UtilizationSchemaDefinition.java * (add) hadoop-ozone/ozone-recon/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/TestUtilizationSchemaDefinition.java * (add) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestUtilizationService.java * (add) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFileSizeCountTask.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java > Add ability in Recon to track the number of small files in an Ozone cluster. > > > Key: HDDS-1366 > URL: https://issues.apache.org/jira/browse/HDDS-1366 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Shweta >Priority: Major > Labels: pull-request-available > Time Spent: 12h > Remaining Estimate: 0h > > Ozone users may want to track the number of small files they have in their > cluster and where they are present. Recon can help them with the information > by iterating the OM Key Table and dividing the keys into different buckets > based on the data size. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.
[ https://issues.apache.org/jira/browse/HDDS-1366?focusedWorklogId=292545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292545 ] ASF GitHub Bot logged work on HDDS-1366: Author: ASF GitHub Bot Created on: 10/Aug/19 17:14 Start Date: 10/Aug/19 17:14 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster URL: https://github.com/apache/hadoop/pull/1146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292545) Time Spent: 12h (was: 11h 50m) > Add ability in Recon to track the number of small files in an Ozone cluster. > > > Key: HDDS-1366 > URL: https://issues.apache.org/jira/browse/HDDS-1366 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Shweta >Priority: Major > Labels: pull-request-available > Time Spent: 12h > Remaining Estimate: 0h > > Ozone users may want to track the number of small files they have in their > cluster and where they are present. Recon can help them with the information > by iterating the OM Key Table and dividing the keys into different buckets > based on the data size. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.
[ https://issues.apache.org/jira/browse/HDDS-1366?focusedWorklogId=292544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292544 ] ASF GitHub Bot logged work on HDDS-1366: Author: ASF GitHub Bot Created on: 10/Aug/19 17:14 Start Date: 10/Aug/19 17:14 Worklog Time Spent: 10m Work Description: arp7 commented on issue #1146: HDDS-1366. Add ability in Recon to track the number of small files in an Ozone Cluster URL: https://github.com/apache/hadoop/pull/1146#issuecomment-520165586 +1 the test failures look unrelated. There are a few checkstyle failures in tests, can you file a followup jira to address them right away? Thanks for the contribution @shwetayakkali and thanks @avijayanhwx and @vivekratnavel for the reviews! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292544) Time Spent: 11h 50m (was: 11h 40m) > Add ability in Recon to track the number of small files in an Ozone cluster. > > > Key: HDDS-1366 > URL: https://issues.apache.org/jira/browse/HDDS-1366 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Shweta >Priority: Major > Labels: pull-request-available > Time Spent: 11h 50m > Remaining Estimate: 0h > > Ozone users may want to track the number of small files they have in their > cluster and where they are present. Recon can help them with the information > by iterating the OM Key Table and dividing the keys into different buckets > based on the data size. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[ https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292542 ] ASF GitHub Bot logged work on HDDS-1927: Author: ASF GitHub Bot Created on: 10/Aug/19 16:45 Start Date: 10/Aug/19 16:45 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520163523 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292542) Time Spent: 1h (was: 50m) > Consolidate add/remove Acl into OzoneAclUtil class > -- > > Key: HDDS-1927 > URL: https://issues.apache.org/jira/browse/HDDS-1927 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > This Jira is created based on @xiaoyu comment on HDDS-1884 > Can we abstract these add/remove logic into common AclUtil class as we can > see similar logic in both bucket manager and key manager? For example, > public static boolean addAcl(List existingAcls, OzoneAcl newAcl) > public static boolean removeAcl(List existingAcls, OzoneAcl newAcl) > > But to do this, we need both OmKeyInfo and OMBucketInfo to use list of > OzoneAcl/OzoneAclInfo. > This Jira is to do that refactor, and also address above comment to move > common logic to AclUtils. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?focusedWorklogId=292532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292532 ] ASF GitHub Bot logged work on HDDS-1951: Author: ASF GitHub Bot Created on: 10/Aug/19 15:27 Start Date: 10/Aug/19 15:27 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1273: HDDS-1951. Wrong symbolic release name on 0.4.1 branch URL: https://github.com/apache/hadoop/pull/1273#issuecomment-520157375 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 50 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ ozone-0.4.1 Compile Tests _ | | +1 | mvninstall | 586 | ozone-0.4.1 passed | | +1 | compile | 368 | ozone-0.4.1 passed | | +1 | mvnsite | 0 | ozone-0.4.1 passed | | +1 | shadedclient | 1736 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | ozone-0.4.1 passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 551 | the patch passed | | +1 | compile | 375 | the patch passed | | +1 | javac | 375 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 679 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 297 | hadoop-hdds in the patch passed. | | -1 | unit | 2084 | hadoop-ozone in the patch failed. | | +1 | asflicense | 54 | The patch does not generate ASF License warnings. | | | | 6351 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1273 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux d11e154c4a50 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | ozone-0.4.1 / 8b80ca0 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/testReport/ | | Max. process+thread count | 5408 (vs. ulimit of 5500) | | modules | C: hadoop-ozone U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292532) Time Spent: 20m (was: 10m) > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA
[jira] [Commented] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox
[ https://issues.apache.org/jira/browse/HDDS-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904451#comment-16904451 ] Anu Engineer commented on HDDS-1923: [~nandakumar131], [~elek] Let us see if we can fix this. I will try this out on Monday and post a patch if needed. Thanks > static/docs/start.html page doesn't render correctly on Firefox > --- > > Key: HDDS-1923 > URL: https://issues.apache.org/jira/browse/HDDS-1923 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: documentation >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Anu Engineer >Priority: Blocker > > static/docs/start.html page doesn't render correctly on Firefox -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox
[ https://issues.apache.org/jira/browse/HDDS-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1923: --- Priority: Blocker (was: Major) > static/docs/start.html page doesn't render correctly on Firefox > --- > > Key: HDDS-1923 > URL: https://issues.apache.org/jira/browse/HDDS-1923 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: documentation >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Anu Engineer >Priority: Blocker > > static/docs/start.html page doesn't render correctly on Firefox -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox
[ https://issues.apache.org/jira/browse/HDDS-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDDS-1923: -- Assignee: Anu Engineer > static/docs/start.html page doesn't render correctly on Firefox > --- > > Key: HDDS-1923 > URL: https://issues.apache.org/jira/browse/HDDS-1923 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: documentation >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Anu Engineer >Priority: Major > > static/docs/start.html page doesn't render correctly on Firefox -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13270) RBF: Router audit logger
[ https://issues.apache.org/jira/browse/HDFS-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-13270: - Attachment: HDFS-13270.002.patch > RBF: Router audit logger > > > Key: HDFS-13270 > URL: https://issues.apache.org/jira/browse/HDFS-13270 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Affects Versions: 3.2.0 >Reporter: maobaolong >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-13270.001.patch, HDFS-13270.002.patch > > > We can use router auditlogger to log the client info and cmd, because the > FSNamesystem#Auditlogger's log think the client are all from router. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14625) Make DefaultAuditLogger class in FSnamesystem to Abstract
[ https://issues.apache.org/jira/browse/HDFS-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-14625: - Attachment: HDFS-14625.004.patch > Make DefaultAuditLogger class in FSnamesystem to Abstract > -- > > Key: HDFS-14625 > URL: https://issues.apache.org/jira/browse/HDFS-14625 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14625 (1).patch, HDFS-14625(2).patch, > HDFS-14625.003.patch, HDFS-14625.004.patch, HDFS-14625.patch > > > As per +HDFS-13270+ Audit logger for Router , we can make DefaultAuditLogger > in FSnamesystem to be Abstract and common -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1942) Support copy during S3 multipart upload part creation
[ https://issues.apache.org/jira/browse/HDDS-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-1942: -- Assignee: Elek, Marton > Support copy during S3 multipart upload part creation > - > > Key: HDDS-1942 > URL: https://issues.apache.org/jira/browse/HDDS-1942 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: S3 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > > Uploads a part by copying data from an existing object as data source > Documented here: > https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.
[ https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904445#comment-16904445 ] Siyao Meng commented on HDFS-13505: --- [~dineshchitlangia] I've added release notes to this jira. Please review. Thanks! > Turn on HDFS ACLs by default. > - > > Key: HDFS-13505 > URL: https://issues.apache.org/jira/browse/HDFS-13505 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch > > > Turn on HDFS ACLs by default. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1952) TestMiniChaosOzoneCluster may run until OOME
Doroszlai, Attila created HDDS-1952:
---
Summary: TestMiniChaosOzoneCluster may run until OOME
Key: HDDS-1952
URL: https://issues.apache.org/jira/browse/HDDS-1952
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: test
Reporter: Doroszlai, Attila
{{TestMiniChaosOzoneCluster}} runs load generator on a cluster for supposedly 1
minute, but it may run indefinitely until JVM crashes due to OutOfMemoryError.
In 0.4.1 nightly build it crashed 29/30 times (and no tests were executed in
the remaining one run due to some other error).
Latest:
https://github.com/elek/ozone-ci/blob/3f553ed6ad358ba61a302967617de737d7fea01a/byscane/byscane-nightly-wggqd/integration/output.log#L5661-L5662
When it crashes, it leaves GBs of data lying around.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904437#comment-16904437
]
Siyao Meng commented on HDFS-14148:
---
PR available: https://github.com/apache/hadoop/pull/1274
> HDFS OIV ReverseXML SnapshotSection parser throws exception when there are
> more than one snapshottable directory
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-14148 started by Siyao Meng.
-
> HDFS OIV ReverseXML SnapshotSection parser throws exception when there are
> more than one snapshottable directory
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Summary: HDFS OIV ReverseXML SnapshotSection parser throws exception when
there are more than one snapshottable directory (was: HDFS OIV ReverseXML
SnapshotSection parser would throw exception when there are >= 2 snapshottable
directories)
> HDFS OIV ReverseXML SnapshotSection parser throws exception when there are
> more than one snapshottable directory
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1950) S3 MPU part-list call fails if there are no parts
[
https://issues.apache.org/jira/browse/HDDS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton reassigned HDDS-1950:
--
Assignee: Elek, Marton
> S3 MPU part-list call fails if there are no parts
> -
>
> Key: HDDS-1950
> URL: https://issues.apache.org/jira/browse/HDDS-1950
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> If an S3 multipart upload is created but no part is upload the part list
> can't be called because it throws HTTP 500:
> Create an MPU:
> {code}
> aws s3api --endpoint http://localhost: create-multipart-upload
> --bucket=docker --key=testkeu
> {
> "Bucket": "docker",
> "Key": "testkeu",
> "UploadId": "85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234"
> }
> {code}
> List the parts:
> {code}
> aws s3api --endpoint http://localhost: list-parts --bucket=docker
> --key=testkeu
> --upload-id=85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234
> {code}
> It throws an exception on the server side, because in the
> KeyManagerImpl.listParts the ReplicationType is retrieved from the first
> part:
> {code}
> HddsProtos.ReplicationType replicationType =
> partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
> {code}
> Which is not yet available in this use case.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser would throw exception when there are >= 2 snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Target Version/s: 3.3.0, 3.2.1, 3.1.3
> HDFS OIV ReverseXML SnapshotSection parser would throw exception when there
> are >= 2 snapshottable directories
> --
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser would throw exception when there are >= 2 snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Issue Type: Bug (was: Improvement)
> HDFS OIV ReverseXML SnapshotSection parser would throw exception when there
> are >= 2 snapshottable directories
> --
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser would throw exception when there are >= 2 snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Summary: HDFS OIV ReverseXML SnapshotSection parser would throw exception
when there are >= 2 snapshottable directories (was: HDFS OIV ReverseXML
snapshot issue: Found unknown XML keys in : dir)
> HDFS OIV ReverseXML SnapshotSection parser would throw exception when there
> are >= 2 snapshottable directories
> --
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Description:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
This problem can be easily reproduced when there are at least TWO
snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run
*TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.
was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
This problem can be easily reproduced when there are at least TWO
snapshot-enabled directories. Apply
> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO
>
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Description:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
This problem can be easily reproduced when there are at least TWO
snapshot-enabled directories. Apply
was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
It seems this problem can be easily reproduced when there are at least TWO
snapshot-enabled directories.
{code:title=trunk}
$ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}
> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
>
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Attachment: HDFS-14148.test.001.patch
> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024,
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories.
> {code:title=trunk}
> $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1:
> ExitException
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Attachment: fsimage_024
> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_024, fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories.
> {code:title=trunk}
> $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1:
> ExitException
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Description:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
It seems this problem can be easily reproduced when there are at least TWO
snapshot-enabled directories.
was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
> at
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO
> snapshot-enabled directories.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
[
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14148:
--
Description:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
It seems this problem can be easily reproduced when there are at least TWO
snapshot-enabled directories.
{code:title=trunk}
$ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}
was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}
This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6
It seems this problem can be easily reproduced when there are at least TWO
snapshot-enabled directories.
> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
>
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
> at
>
[jira] [Work logged] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?focusedWorklogId=292520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292520 ] ASF GitHub Bot logged work on HDDS-1951: Author: ASF GitHub Bot Created on: 10/Aug/19 13:40 Start Date: 10/Aug/19 13:40 Worklog Time Spent: 10m Work Description: elek commented on pull request #1273: HDDS-1951. Wrong symbolic release name on 0.4.1 branch URL: https://github.com/apache/hadoop/pull/1273 See: https://issues.apache.org/jira/browse/HDDS-1951 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292520) Time Spent: 10m Remaining Estimate: 0h > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1951: --- Sprint: HDDS Biscayne > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1951: - Labels: pull-request-available (was: ) > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1951: --- Status: Patch Available (was: Open) > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-1951: -- Assignee: Elek, Marton > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Blocker > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch
[ https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1951: --- Summary: Wrong symbolic release name on 0.4.1 branch (was: Wrong symbolic release name on 0.4.1 branc) > Wrong symbolic release name on 0.4.1 branch > --- > > Key: HDDS-1951 > URL: https://issues.apache.org/jira/browse/HDDS-1951 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Priority: Blocker > > Should be Biscayne instead of Crater lake according to the Roadmap: > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1951) Wrong symbolic release name on 0.4.1 branc
Elek, Marton created HDDS-1951: -- Summary: Wrong symbolic release name on 0.4.1 branc Key: HDDS-1951 URL: https://issues.apache.org/jira/browse/HDDS-1951 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Elek, Marton Should be Biscayne instead of Crater lake according to the Roadmap: https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14615) NPE writing edit logs
[
https://issues.apache.org/jira/browse/HDFS-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904417#comment-16904417
]
hemanthboyina commented on HDFS-14615:
--
i think the group name was null here , creating NPE during MKDIR operation
> NPE writing edit logs
> -
>
> Key: HDFS-14615
> URL: https://issues.apache.org/jira/browse/HDFS-14615
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.1
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
>
> Hit a weird bug where writing mkdir op to edit log throws an NPE and NameNode
> crashed
> {noformat}
> 2019-06-26 10:57:27,398 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: write op failed for
> (journal
> JournalAndStream(mgr=FileJournalManager(root=/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn),
>
> stream=EditLogFileOutputStream(/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn/current/edits_inprogress_0598588)))
> java.lang.NullPointerException
> at org.apache.hadoop.io.Text.encode(Text.java:451)
> at org.apache.hadoop.io.Text.encode(Text.java:431)
> at org.apache.hadoop.io.Text.writeString(Text.java:491)
> at
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:104)
> at
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:84)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$MkdirOp.writeFields(FSEditLogOp.java:1654)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4866)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:60)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.write(EditLogFileOutputStream.java:97)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:444)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:440)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:481)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:288)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:232)
> {noformat}
> The stacktrace is similar to SENTRY-555, which is thought to be a Sentry bug
> (authorization provider), but this cluster doesn't have Sentry and therefore
> could be a genuine HDFS bug.
> File this jira to keep a record.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904414#comment-16904414
]
Siyao Meng edited comment on HDFS-14491 at 8/10/19 12:40 PM:
-
Also as shown in your snippet of *InvalidateBlocks#add*, this operation should
be performed for *each* datanode that has the replica. Hence the
*PendingDeletionBlocks* metric does include all replicas(or EC blocks).
was (Author: smeng):
Also as shown in your snippet of *InvalidateBlocks#add*, this operation should
be performed for *each* datanode that has the replica. Hence the
*PendingDeletionBlocks* metric does include all replicas/EC strips.
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904414#comment-16904414
]
Siyao Meng commented on HDFS-14491:
---
Also as shown in your snippet of *InvalidateBlocks#add*, this operation should
be performed for *each* datanode that has the replica. Hence the
*PendingDeletionBlocks* metric does include all replicas/EC strips.
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas
[
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904413#comment-16904413
]
Siyao Meng commented on HDFS-14491:
---
[~jojochuang]
The metric is retrieved via *FSNameSystem#getPendingDeletionBlocks*
[here|https://github.com/apache/hadoop/blob/b964b81f8509ba6cd938bc36f3acb5e3112b7ca2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L4918]
Then it calls
{code:title=BlockManager#getPendingDeletionBlocksCount}
/** Used by metrics */
public long getPendingDeletionBlocksCount() {
return invalidateBlocks.numBlocks();
}
{code}
If we look into how *invalidateBlocks* is changed, we can see in
*BlockManager#addToInvalidates*
{code:title=BlockManager#addToInvalidates}
/**
* Adds block to list of blocks which will be invalidated on all its
* datanodes.
*/
private void addToInvalidates(BlockInfo storedBlock) {
if (!isPopulatingReplQueues()) {
return;
}
StringBuilder datanodes = blockLog.isDebugEnabled()
? new StringBuilder() : null;
for (DatanodeStorageInfo storage : blocksMap.getStorages(storedBlock)) {
if (storage.getState() != State.NORMAL) {
continue;
}
final DatanodeDescriptor node = storage.getDatanodeDescriptor();
final Block b = getBlockOnStorage(storedBlock, storage);
if (b != null) {
invalidateBlocks.add(b, node, false);
if (datanodes != null) {
datanodes.append(node).append(" ");
}
}
}
if (datanodes != null && datanodes.length() != 0) {
blockLog.debug("BLOCK* addToInvalidates: {} {}", storedBlock, datanodes);
}
}
{code}
So we can see that *invalidateBlocks* is indeed incremented for each block on
each datanode. CMIIW
> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During
> the process of the delete, I noticed that the NameNode UI near the top has a
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the
> cluster. When that number was below the total number of blocks, I briefly
> believed I had deleted the entire cluster. In reality, the second number
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and
> where it doesn't. This may also have an impact on the under-replicated count.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14174) Enhance Audit for chown ( internally setOwner)
[ https://issues.apache.org/jira/browse/HDFS-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904411#comment-16904411 ] hemanthboyina commented on HDFS-14174: -- now we are auditing the new owner information need to capture existing owner and new owner ? [~xkrogen] [~jojochuang] > Enhance Audit for chown ( internally setOwner) > - > > Key: HDFS-14174 > URL: https://issues.apache.org/jira/browse/HDFS-14174 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.6.0 >Reporter: Sailesh Patel >Assignee: hemanthboyina >Priority: Minor > > When a hdfs dfs -chown command is executed, the audit log does not capture > the existing owner and the new owner. > Need to capture the old and new owner to allow auditing to be effective > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1950) S3 MPU part-list call fails if there are no parts
[
https://issues.apache.org/jira/browse/HDDS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-1950:
---
Summary: S3 MPU part-list call fails if there are no parts (was: S3 MPU
part list can't be called if there are no parts)
> S3 MPU part-list call fails if there are no parts
> -
>
> Key: HDDS-1950
> URL: https://issues.apache.org/jira/browse/HDDS-1950
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Priority: Major
>
> If an S3 multipart upload is created but no part is upload the part list
> can't be called because it throws HTTP 500:
> Create an MPU:
> {code}
> aws s3api --endpoint http://localhost: create-multipart-upload
> --bucket=docker --key=testkeu
> {
> "Bucket": "docker",
> "Key": "testkeu",
> "UploadId": "85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234"
> }
> {code}
> List the parts:
> {code}
> aws s3api --endpoint http://localhost: list-parts --bucket=docker
> --key=testkeu
> --upload-id=85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234
> {code}
> It throws an exception on the server side, because in the
> KeyManagerImpl.listParts the ReplicationType is retrieved from the first
> part:
> {code}
> HddsProtos.ReplicationType replicationType =
> partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
> {code}
> Which is not yet available in this use case.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1950) S3 MPU part list can't be called if there are no parts
Elek, Marton created HDDS-1950:
--
Summary: S3 MPU part list can't be called if there are no parts
Key: HDDS-1950
URL: https://issues.apache.org/jira/browse/HDDS-1950
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: S3
Reporter: Elek, Marton
If an S3 multipart upload is created but no part is upload the part list can't
be called because it throws HTTP 500:
Create an MPU:
{code}
aws s3api --endpoint http://localhost: create-multipart-upload
--bucket=docker --key=testkeu
{
"Bucket": "docker",
"Key": "testkeu",
"UploadId": "85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234"
}
{code}
List the parts:
{code}
aws s3api --endpoint http://localhost: list-parts --bucket=docker
--key=testkeu
--upload-id=85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234
{code}
It throws an exception on the server side, because in the
KeyManagerImpl.listParts the ReplicationType is retrieved from the first part:
{code}
HddsProtos.ReplicationType replicationType =
partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
{code}
Which is not yet available in this use case.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904410#comment-16904410 ] hemanthboyina commented on HDFS-13123: -- [~elgoiri] [~jojochuang] attached the intial patch , can you review it > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf, > HDFS-13123.patch > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1946) CertificateClient should not persist keys/certs to ozone.metadata.dir
[ https://issues.apache.org/jira/browse/HDDS-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-1946: -- Target Version/s: 0.4.1 (was: 0.4.0) > CertificateClient should not persist keys/certs to ozone.metadata.dir > - > > Key: HDDS-1946 > URL: https://issues.apache.org/jira/browse/HDDS-1946 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Vivek Ratnavel Subramanian >Priority: Major > > For example, when OM and SCM are deployed on the same host with > ozone.metadata.dir defined. SCM can start successfully but OM can not because > the key/cert from OM will collide with SCM. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1949) Missing or error-prone test cleanup
Doroszlai, Attila created HDDS-1949: --- Summary: Missing or error-prone test cleanup Key: HDDS-1949 URL: https://issues.apache.org/jira/browse/HDDS-1949 Project: Hadoop Distributed Data Store Issue Type: Bug Components: test Reporter: Doroszlai, Attila Assignee: Doroszlai, Attila Some integration tests do not clean up after themselves. Some only clean up if the test is successful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-1949) Missing or error-prone test cleanup
[ https://issues.apache.org/jira/browse/HDDS-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-1949 started by Doroszlai, Attila. --- > Missing or error-prone test cleanup > --- > > Key: HDDS-1949 > URL: https://issues.apache.org/jira/browse/HDDS-1949 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > > Some integration tests do not clean up after themselves. Some only clean up > if the test is successful. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-14665 started by Siyao Meng.
-
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14663) HttpFS: LISTSTATUS_BATCH does not return batches
[ https://issues.apache.org/jira/browse/HDFS-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14663 started by Siyao Meng. - > HttpFS: LISTSTATUS_BATCH does not return batches > > > Key: HDFS-14663 > URL: https://issues.apache.org/jira/browse/HDFS-14663 > Project: Hadoop HDFS > Issue Type: Bug > Components: httpfs >Affects Versions: 3.3.0 >Reporter: Stephen O'Donnell >Assignee: Siyao Meng >Priority: Major > > The webhdfs protocol supports a LISTSTATUS_BATCH operation where it can > retrieve the file listing for a large directory in chunks. > When using the webhdfs service embedded in the namenode, this works as > expected, but when using HTTPFS, any call to LISTSTATUS_BATCH simply returns > the entire listing rather than batches, working effectively like LISTSTATUS > instead. > This seems to be because HTTPFS falls back to using the method > org.apache.hadoop.fs.FileSystem#listStatusBatch, which is intended to be > overridden, but the implementation used in HTTPFS has not done that, leading > to this limitation. > This feature (LISTSTATUS_BATCH) was added to HTTPFS by HDFS-10823, but based > on my testing it does not work as intended. I suspect it is because the > listStatusBatch operation was added to the WebHdfsFileSystem and > HttpFSFileSystem as part of the above Jira, but behind the scenes HTTPFS > seems to use DistributeFileSystem and hence it falls back to the default > implementation "org.apache.hadoop.fs.FileSystem#listStatusBatch" which > returns all entries in a single batch. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-1891) Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDDS-1891 started by Siyao Meng.
> Ozone fs shell command should work with default port when port number is not
> specified
> --
>
> Key: HDDS-1891
> URL: https://issues.apache.org/jira/browse/HDDS-1891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> {code:bash|title=Without port number -> Error}
> $ ozone fs -ls o3fs://bucket.volume.localhost/
> -ls: Ozone file system url should be either one of the two forms:
> o3fs://bucket.volume/key OR o3fs://bucket.volume.om-host.example.com:5678/key
> ...
> {code}
> {code:bash|title=With port number -> Success}
> $ ozone fs -ls o3fs://bucket.volume.localhost:9862/
> Found 1 items
> -rw-rw-rw- 1 hadoop hadoop 1485 2019-08-01 21:14
> o3fs://bucket.volume.localhost:9862/README.txt
> {code}
> We expect the first command to attempt port 9862 by default.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14718 started by Siyao Meng. - > HttpFS: Sort response by key names as WebHDFS does > -- > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1054) List Multipart uploads in a bucket
[ https://issues.apache.org/jira/browse/HDDS-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-1054: -- Assignee: Elek, Marton (was: Bharat Viswanadham) > List Multipart uploads in a bucket > -- > > Key: HDDS-1054 > URL: https://issues.apache.org/jira/browse/HDDS-1054 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Elek, Marton >Priority: Blocker > > This Jira is to implement in ozone to list of in-progress multipart uploads > in a bucket. > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html] > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA
[
https://issues.apache.org/jira/browse/HDDS-1910?focusedWorklogId=292505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292505
]
ASF GitHub Bot logged work on HDDS-1910:
Author: ASF GitHub Bot
Created on: 10/Aug/19 10:06
Start Date: 10/Aug/19 10:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #1268: HDDS-1910.
Cannot build hadoop-hdds-config from scratch in IDEA
URL: https://github.com/apache/hadoop/pull/1268#issuecomment-520136555
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 43 | Docker mode activated. |
||| _ Prechecks _ |
| +1 | dupname | 0 | No case conflicting files found. |
| +1 | @author | 0 | The patch does not contain any @author tags. |
| +1 | test4tests | 0 | The patch appears to include 1 new or modified test
files. |
||| _ trunk Compile Tests _ |
| 0 | mvndep | 25 | Maven dependency ordering for branch |
| +1 | mvninstall | 603 | trunk passed |
| +1 | compile | 376 | trunk passed |
| +1 | mvnsite | 0 | trunk passed |
| +1 | shadedclient | 1789 | branch has no errors when building and testing
our client artifacts. |
| +1 | javadoc | 160 | trunk passed |
||| _ Patch Compile Tests _ |
| 0 | mvndep | 25 | Maven dependency ordering for patch |
| +1 | mvninstall | 553 | the patch passed |
| +1 | compile | 375 | the patch passed |
| +1 | javac | 375 | the patch passed |
| +1 | mvnsite | 0 | the patch passed |
| +1 | whitespace | 0 | The patch has no whitespace issues. |
| +1 | xml | 3 | The patch has no ill-formed XML file. |
| +1 | shadedclient | 675 | patch has no errors when building and testing
our client artifacts. |
| +1 | javadoc | 164 | the patch passed |
||| _ Other Tests _ |
| +1 | unit | 281 | hadoop-hdds in the patch passed. |
| -1 | unit | 2240 | hadoop-ozone in the patch failed. |
| +1 | asflicense | 59 | The patch does not generate ASF License warnings. |
| | | 6581 | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures |
| | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory |
| | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
| | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
| | hadoop.ozone.client.rpc.TestCommitWatcher |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1268 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml |
| uname | Linux 01d2fedea183 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fba222a |
| Default Java | 1.8.0_212 |
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/testReport/ |
| Max. process+thread count | 4867 (vs. ulimit of 5500) |
| modules | C: hadoop-hdds/config hadoop-hdds/common U: hadoop-hdds |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 292505)
Time Spent: 0.5h (was: 20m)
> Cannot build hadoop-hdds-config from scratch in IDEA
>
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or
> after {{mvn clean}}) in IDEA fails with the following error:
>
[jira] [Commented] (HDFS-14595) HDFS-11848 breaks API compatibility
[
https://issues.apache.org/jira/browse/HDFS-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904391#comment-16904391
]
Hadoop QA commented on HDFS-14595:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
3s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14595 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977214/HDFS-14595.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6a6f720f66ea 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Work logged] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
[ https://issues.apache.org/jira/browse/HDDS-1948?focusedWorklogId=292498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292498 ] ASF GitHub Bot logged work on HDDS-1948: Author: ASF GitHub Bot Created on: 10/Aug/19 09:15 Start Date: 10/Aug/19 09:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1266: HDDS-1948. S3 MPU can't be created with octet-stream content-type URL: https://github.com/apache/hadoop/pull/1266#issuecomment-520133277 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 593 | trunk passed | | +1 | compile | 375 | trunk passed | | +1 | checkstyle | 76 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 868 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | trunk passed | | 0 | spotbugs | 423 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 620 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 551 | the patch passed | | +1 | compile | 376 | the patch passed | | +1 | javac | 376 | the patch passed | | +1 | checkstyle | 84 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 674 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 168 | the patch passed | | +1 | findbugs | 632 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 296 | hadoop-hdds in the patch failed. | | -1 | unit | 1766 | hadoop-ozone in the patch failed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 7525 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.safemode.TestSCMSafeModeManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1266 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux be2ffc260e5d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / fba222a | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/testReport/ | | Max. process+thread count | 4900 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 292498) Time Spent: 20m (was: 10m) > S3 MPU can't be created with octet-stream content-type > --- > > Key: HDDS-1948 > URL: https://issues.apache.org/jira/browse/HDDS-1948 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: S3 >Reporter: Elek, Marton >
[jira] [Commented] (HDFS-14595) HDFS-11848 breaks API compatibility
[
https://issues.apache.org/jira/browse/HDFS-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904384#comment-16904384
]
Hadoop QA commented on HDFS-14595:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 56s{color}
| {color:red} hadoop-hdfs-project generated 2 new + 552 unchanged - 0 fixed =
554 total (was 552) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14595 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977213/HDFS-14595.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 4818ee8eb1a4 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fba222a |
| maven |
[jira] [Updated] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA
[
https://issues.apache.org/jira/browse/HDDS-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doroszlai, Attila updated HDDS-1910:
Status: Patch Available (was: In Progress)
> Cannot build hadoop-hdds-config from scratch in IDEA
>
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while
> constructing Processor object: javax.annotation.processing.Processor:
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14717) Junit not found in hadoop-dynamometer-infra
[
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904372#comment-16904372
]
Hadoop QA commented on HDFS-14717:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra: The patch generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-dynamometer-infra in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.dynamometer.TestDynamometerInfra |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14717 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977216/HDFS-14717.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 975bc2ae74a0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fba222a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27467/artifact/out/diff-checkstyle-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
|
| unit |
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Description: *Example* See description of HDFS-14665 for an example of LISTSTATUS. *Analysis* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? - I looked into the Git history. It seems it's simply because WebHDFS uses TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. It is not only limited to LISTSTATUS, but ALL other request's JSON serialization. Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap in HttpFS serialization in FSOperations class? was: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? - I looked into the Git history. It seems it's simply because WebHDFS uses TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap in HttpFS serialization in FSOperations class? > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sorr response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Summary: HttpFS: Sorr response by key names as WebHDFS does (was: HttpFS: Sor response by key names as WebHDFS does) > HttpFS: Sorr response by key names as WebHDFS does > -- > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sor response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Summary: HttpFS: Sor response by key names as WebHDFS does (was: HttpFS: Sort LISTSTATUS response by key names as WebHDFS does) > HttpFS: Sor response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Summary: HttpFS: Sort response by key names as WebHDFS does (was: HttpFS: Sorr response by key names as WebHDFS does) > HttpFS: Sort response by key names as WebHDFS does > -- > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Description: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? - I looked into the Git history. It seems it's simply because WebHDFS uses TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap in HttpFS serialization in FSOperations class? was: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? I looked into the Git history. It seems it's simply because WebHDFS uses TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Example: see description of HDFS-14665. > *Root cause* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Description: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? I looked into the Git history. It seems it's simply because WebHDFS uses TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. was: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? e.g. for some performance concerns on HttpFS? > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Example: see description of HDFS-14665. > *Root cause* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Description: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? e.g. for some performance concerns on HttpFS? was: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? e.g. for some performance concern on HttpFS? > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Example: see description of HDFS-14665. > *Root cause* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? e.g. for some performance concerns > on HttpFS? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Description: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? e.g. for some performance concern on HttpFS? was: Example: see description of HDFS-14665. Root cause: WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. Questions: Why the difference? Is this intentional? e.g. for some performance concern on HttpFS? > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Example: see description of HDFS-14665. > *Root cause* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? e.g. for some performance concern on > HttpFS? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Description: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? e.g. for some performance concern on HttpFS? was: Example: see description of HDFS-14665. *Root cause* WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. *Questions* Why the difference? Is this intentional? e.g. for some performance concern on HttpFS? > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Example: see description of HDFS-14665. > *Root cause* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? e.g. for some performance concern on > HttpFS? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14718: -- Summary: HttpFS: Sort LISTSTATUS response by key names as WebHDFS does (was: HttpFS: Sort LISTSTATUS response by key names) > HttpFS: Sort LISTSTATUS response by key names as WebHDFS does > - > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > Example: see description of HDFS-14665. > Root cause: > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > Questions: > Why the difference? Is this intentional? e.g. for some performance concern on > HttpFS? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names
Siyao Meng created HDFS-14718: - Summary: HttpFS: Sort LISTSTATUS response by key names Key: HDFS-14718 URL: https://issues.apache.org/jira/browse/HDFS-14718 Project: Hadoop HDFS Issue Type: Improvement Components: httpfs Reporter: Siyao Meng Assignee: Siyao Meng Example: see description of HDFS-14665. Root cause: WebHDFS is [using a TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] to serialize HdfsFileStatus, while HttpFS [uses a LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] to serialize FileStatus. Questions: Why the difference? Is this intentional? e.g. for some performance concern on HttpFS? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA
[
https://issues.apache.org/jira/browse/HDDS-1910?focusedWorklogId=292485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292485
]
ASF GitHub Bot logged work on HDDS-1910:
Author: ASF GitHub Bot
Created on: 10/Aug/19 08:16
Start Date: 10/Aug/19 08:16
Worklog Time Spent: 10m
Work Description: adoroszlai commented on pull request #1268: HDDS-1910.
Cannot build hadoop-hdds-config from scratch in IDEA
URL: https://github.com/apache/hadoop/pull/1268
## What changes were proposed in this pull request?
`ConfigFileGenerator` is both defined (`.java` source) and configured to be
used (by reference in `META-INF`) in `hadoop-hdds-config` module's `main`
source. Maven build is configured to work around this by explicit `-proc:none`
compiler arg, applied only to `main` compilation, but not to `test`
compilation. Since the annotation processor is only used in tests in this
module, this works fine.
It seems IDEA does not respect this compiler configuration (and cannot even
be configured differently for `main` and `test` compilation).
This change configures the processor for use in `config/test` and
`common/main`. Thus dependent modules still inherit it for main source
processing, while the `config` module can be properly compiled without the hack.
https://issues.apache.org/jira/browse/HDDS-1910
## How was this patch tested?
1. Verified that `hadoop-hdds/config` can be compiled from scratch in IDEA.
To confirm, try to run eg. `TestConfigFileAppender` unit test.
2. Verified that full Maven build still generates the same Ozone config
based on existing annotations:
```
1762 hadoop-hdds/common/target/test-classes/ozone-default-generated.xml
2030 hadoop-hdds/config/target/test-classes/ozone-default-generated.xml
2294 hadoop-hdds/server-scm/target/classes/ozone-default-generated.xml
918 hadoop-ozone/common/target/classes/ozone-default-generated.xml
2298 hadoop-ozone/csi/target/classes/ozone-default-generated.xml
504
hadoop-ozone/ozonefs-lib-current/target/classes/ozone-default-generated.xml
504
hadoop-ozone/ozonefs-lib-legacy/target/classes/libs/ozone-default-generated.xml
504
hadoop-ozone/ozonefs-lib-legacy/target/classes/ozone-default-generated.xml
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 292485)
Time Spent: 10m
Remaining Estimate: 0h
> Cannot build hadoop-hdds-config from scratch in IDEA
>
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while
> constructing Processor object: javax.annotation.processing.Processor:
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA
[
https://issues.apache.org/jira/browse/HDDS-1910?focusedWorklogId=292486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292486
]
ASF GitHub Bot logged work on HDDS-1910:
Author: ASF GitHub Bot
Created on: 10/Aug/19 08:16
Start Date: 10/Aug/19 08:16
Worklog Time Spent: 10m
Work Description: adoroszlai commented on issue #1268: HDDS-1910. Cannot
build hadoop-hdds-config from scratch in IDEA
URL: https://github.com/apache/hadoop/pull/1268#issuecomment-520129550
/label ozone
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 292486)
Time Spent: 20m (was: 10m)
> Cannot build hadoop-hdds-config from scratch in IDEA
>
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while
> constructing Processor object: javax.annotation.processing.Processor:
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA
[
https://issues.apache.org/jira/browse/HDDS-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1910:
-
Labels: pull-request-available (was: )
> Cannot build hadoop-hdds-config from scratch in IDEA
>
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while
> constructing Processor object: javax.annotation.processing.Processor:
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[ https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292484 ] ASF GitHub Bot logged work on HDDS-1927: Author: ASF GitHub Bot Created on: 10/Aug/19 08:12 Start Date: 10/Aug/19 08:12 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1263: HDDS-1927. Consolidate add/remove Acl into OzoneAclUtil class. Contri… URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520129310 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for branch | | +1 | mvninstall | 577 | trunk passed | | +1 | compile | 365 | trunk passed | | +1 | checkstyle | 76 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 864 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 163 | trunk passed | | 0 | spotbugs | 422 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 611 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | +1 | mvninstall | 591 | the patch passed | | +1 | compile | 391 | the patch passed | | +1 | javac | 391 | the patch passed | | -0 | checkstyle | 34 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 686 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 92 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13) | | -1 | findbugs | 427 | hadoop-ozone generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 | unit | 306 | hadoop-hdds in the patch passed. | | -1 | unit | 52 | hadoop-ozone in the patch failed. | | +1 | asflicense | 36 | The patch does not generate ASF License warnings. | | | | 5870 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-ozone | | | Possible null pointer dereference of null in org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo) Dereferenced at OmKeyInfo.java:null in org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo) Dereferenced at OmKeyInfo.java:[line 394] | | Failed junit tests | hadoop.ozone.om.helpers.TestOmKeyInfo | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1263 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ea8778af37e0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / fba222a | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/new-findbugs-hadoop-ozone.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/testReport/ | | Max. process+thread count | 487 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/client hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Component/s: httpfs
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Target Version/s: 3.3.0
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Description:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (should only be none 0 for directories)
"fileId"
"storagePolicy"
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Root cause:
[toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
didn't serialize the HDFS-specific keys from FileStatus.
Also may file another Jira to align the order of the keys in the responses.
was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Root cause:
[toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
didn't serialize the HDFS-specific keys from FileStatus.
Also may file another Jira to align the order of the keys in the responses.
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Summary: HttpFS: LISTSTATUS response is missing HDFS-specific fields (was:
HttpFS: LISTSTATUS result is missing HDFS-specific fields)
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing HDFS-specific fields
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Summary: HttpFS: LISTSTATUS result is missing HDFS-specific fields (was:
HttpFS: LISTSTATUS result is missing some fields for each entry)
> HttpFS: LISTSTATUS result is missing HDFS-specific fields
> -
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
> didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904364#comment-16904364 ] Adam Antal commented on HDFS-14195: --- Thanks for the patch [~suxingfate], and [~jojochuang] for the commit! > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14717) Junit not found in hadoop-dynamometer-infra
[
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevin su updated HDFS-14717:
Status: Patch Available (was: Open)
> Junit not found in hadoop-dynamometer-infra
> ---
>
> Key: HDFS-14717
> URL: https://issues.apache.org/jira/browse/HDFS-14717
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: kevin su
>Assignee: kevin su
>Priority: Major
> Attachments: HDFS-14717.001.patch
>
>
> {code:java}
> $ hadoop jar hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar
> org.apache.hadoop.tools.dynamometer.Client
> {code}
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at org.apache.hadoop.tools.dynamometer.Client.main(Client.java:256)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 7 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14717) Junit not found in hadoop-dynamometer-infra
[
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904361#comment-16904361
]
kevin su commented on HDFS-14717:
-
[~jojochuang] [~smeng] Thanks for your reply
It's due to _*ClassUtil.findContainingJar(Assert.class)*_ can't find Junit,
because Junit is in other directory.
I also found that _*Junit*_ Already package into
_*hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar*_
So I remove _*ClassUtil.findContainingJar(Assert.class),*_ it works
> Junit not found in hadoop-dynamometer-infra
> ---
>
> Key: HDFS-14717
> URL: https://issues.apache.org/jira/browse/HDFS-14717
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: kevin su
>Assignee: kevin su
>Priority: Major
> Attachments: HDFS-14717.001.patch
>
>
> {code:java}
> $ hadoop jar hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar
> org.apache.hadoop.tools.dynamometer.Client
> {code}
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at org.apache.hadoop.tools.dynamometer.Client.main(Client.java:256)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 7 more{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing some fields for each entry
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Description:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Root cause:
[toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
didn't serialize the HDFS-specific keys from FileStatus.
Also may file another Jira to align the order of the keys in the responses.
was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Also may file another Jira to align the order of the keys in the responses.
> HttpFS: LISTSTATUS result is missing some fields for each entry
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Root cause:
>
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing some fields for each entry
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Description:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
childrenNum (directories only?)
fileId
storagePolicy.
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Also may file another Jira to align the order of the keys in the responses.
was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing childrenNum
(directories only I guess), fileId and storagePolicy fields. The same applies
to LISTSTATUS_BATCH, which might be using the same underlying calls to compose
the response.
Also may file another Jira to align the order of the keys in the responses.
> HttpFS: LISTSTATUS result is missing some fields for each entry
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> childrenNum (directories only?)
> fileId
> storagePolicy.
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing some fields for each entry
[
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDFS-14665:
--
Description:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Also may file another Jira to align the order of the keys in the responses.
was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
}
]
}
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
"FileStatuses": {
"FileStatus": [
...
{
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
}
]
}
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
childrenNum (directories only?)
fileId
storagePolicy.
{code}
The same applies to LISTSTATUS_BATCH, which might be using the same underlying
calls to compose the response.
Also may file another Jira to align the order of the keys in the responses.
> HttpFS: LISTSTATUS result is missing some fields for each entry
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
> }
> ]
> }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
> "FileStatuses": {
> "FileStatus": [
> ...
> {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
> }
> ]
> }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same
> underlying calls to compose the response.
> Also may file another Jira to align the order of the keys in the responses.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
[
https://issues.apache.org/jira/browse/HDDS-1948?focusedWorklogId=292471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292471
]
ASF GitHub Bot logged work on HDDS-1948:
Author: ASF GitHub Bot
Created on: 10/Aug/19 07:09
Start Date: 10/Aug/19 07:09
Worklog Time Spent: 10m
Work Description: elek commented on pull request #1266: HDDS-1948. S3 MPU
can't be created with octet-stream content-type
URL: https://github.com/apache/hadoop/pull/1266
This problem is reported offline by [~shaneku...@gmail.com].
When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
Part Upload initialize message with "application/octet-stream" Content-Type.
This Content-Type is missing from the aws-cli which is used to reimplement
s3 endpoint.
The problem is that we use the same rest endpoint for initialize and
complete Multipart Upload request. For the completion we need the
CompleteMultipartUploadRequest parameter which is parsed from the body.
For initialize we have an empty body which can't be serialized to
CompleteMultipartUploadRequest.
The workaround is to set a specific content type from a filter which help up
to create two different REST method for initialize and completion message.
Here is an example to test (using bogus AWS credentials).
{code}
curl -H 'Host:yourhost' -H 'User-Agent:aws-sdk-go/1.15.11 (go1.11.2; linux;
amd64)' -H 'Content-Length:0' -H 'Authorization:AWS4-HMAC-SHA256
Credential=qwe/20190809/ozone/s3/aws4_request,
SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class,
Signature=7726ed63990ba3f4f1f796d4ab263f5d9c3374528840f5e49d106dbef491f22c' -H
'Content-Type:application/octet-stream' -H 'X-Amz-Acl:private' -H
'X-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
-H 'X-Amz-Date:20190809T070142Z' -H 'X-Amz-Storage-Class:STANDARD' -H
'Accept-Encoding:gzip' -X POST
'http://localhost:/docker/docker/registry/v2/repositories/apache/ozone-runner/_uploads/2173f019-09c3-466b-bb7d-c31ce749d826/data?uploads
{code}
Without the patch it returns with HTTP 405 (Not supported Media Type).
See: https://issues.apache.org/jira/browse/HDDS-1948
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 292471)
Time Spent: 10m
Remaining Estimate: 0h
> S3 MPU can't be created with octet-stream content-type
> ---
>
> Key: HDDS-1948
> URL: https://issues.apache.org/jira/browse/HDDS-1948
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This problem is reported offline by [~shaneku...@gmail.com].
> When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
> Part Upload initialize message with "application/octet-stream" Content-Type.
> This Content-Type is missing from the aws-cli which is used to reimplement s3
> endpoint.
> The problem is that we use the same rest endpoint for initialize and complete
> Multipart Upload request. For the completion we need the
> CompleteMultipartUploadRequest parameter which is parsed from the body.
> For initialize we have an empty body which can't be serialized to
> CompleteMultipartUploadRequest.
> The workaround is to set a specific content type from a filter which help up
> to create two different REST method for initialize and completion message.
> Here is an example to test (using bogus AWS credentials).
> {code}
> curl -H 'Host:yourhost' -H 'User-Agent:aws-sdk-go/1.15.11 (go1.11.2; linux;
> amd64)' -H 'Content-Length:0' -H 'Authorization:AWS4-HMAC-SHA256
> Credential=qwe/20190809/ozone/s3/aws4_request,
> SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class,
> Signature=7726ed63990ba3f4f1f796d4ab263f5d9c3374528840f5e49d106dbef491f22c'
> -H 'Content-Type:application/octet-stream' -H 'X-Amz-Acl:private' -H
> 'X-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
> -H 'X-Amz-Date:20190809T070142Z' -H 'X-Amz-Storage-Class:STANDARD' -H
> 'Accept-Encoding:gzip' -X POST
>
[jira] [Updated] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
[
https://issues.apache.org/jira/browse/HDDS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1948:
-
Labels: pull-request-available (was: )
> S3 MPU can't be created with octet-stream content-type
> ---
>
> Key: HDDS-1948
> URL: https://issues.apache.org/jira/browse/HDDS-1948
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
>
> This problem is reported offline by [~shaneku...@gmail.com].
> When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
> Part Upload initialize message with "application/octet-stream" Content-Type.
> This Content-Type is missing from the aws-cli which is used to reimplement s3
> endpoint.
> The problem is that we use the same rest endpoint for initialize and complete
> Multipart Upload request. For the completion we need the
> CompleteMultipartUploadRequest parameter which is parsed from the body.
> For initialize we have an empty body which can't be serialized to
> CompleteMultipartUploadRequest.
> The workaround is to set a specific content type from a filter which help up
> to create two different REST method for initialize and completion message.
> Here is an example to test (using bogus AWS credentials).
> {code}
> curl -H 'Host:yourhost' -H 'User-Agent:aws-sdk-go/1.15.11 (go1.11.2; linux;
> amd64)' -H 'Content-Length:0' -H 'Authorization:AWS4-HMAC-SHA256
> Credential=qwe/20190809/ozone/s3/aws4_request,
> SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class,
> Signature=7726ed63990ba3f4f1f796d4ab263f5d9c3374528840f5e49d106dbef491f22c'
> -H 'Content-Type:application/octet-stream' -H 'X-Amz-Acl:private' -H
> 'X-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
> -H 'X-Amz-Date:20190809T070142Z' -H 'X-Amz-Storage-Class:STANDARD' -H
> 'Accept-Encoding:gzip' -X POST
> 'http://localhost:/docker/docker/registry/v2/repositories/apache/ozone-runner/_uploads/2173f019-09c3-466b-bb7d-c31ce749d826/data?uploads
> {code}
> Without the patch it returns with HTTP 405 (Not supported Media Type).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
[
https://issues.apache.org/jira/browse/HDDS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-1948:
---
Status: Patch Available (was: Open)
> S3 MPU can't be created with octet-stream content-type
> ---
>
> Key: HDDS-1948
> URL: https://issues.apache.org/jira/browse/HDDS-1948
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> This problem is reported offline by [~shaneku...@gmail.com].
> When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
> Part Upload initialize message with "application/octet-stream" Content-Type.
> This Content-Type is missing from the aws-cli which is used to reimplement s3
> endpoint.
> The problem is that we use the same rest endpoint for initialize and complete
> Multipart Upload request. For the completion we need the
> CompleteMultipartUploadRequest parameter which is parsed from the body.
> For initialize we have an empty body which can't be serialized to
> CompleteMultipartUploadRequest.
> The workaround is to set a specific content type from a filter which help up
> to create two different REST method for initialize and completion message.
> Here is an example to test (using bogus AWS credentials).
> {code}
> curl -H 'Host:yourhost' -H 'User-Agent:aws-sdk-go/1.15.11 (go1.11.2; linux;
> amd64)' -H 'Content-Length:0' -H 'Authorization:AWS4-HMAC-SHA256
> Credential=qwe/20190809/ozone/s3/aws4_request,
> SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class,
> Signature=7726ed63990ba3f4f1f796d4ab263f5d9c3374528840f5e49d106dbef491f22c'
> -H 'Content-Type:application/octet-stream' -H 'X-Amz-Acl:private' -H
> 'X-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
> -H 'X-Amz-Date:20190809T070142Z' -H 'X-Amz-Storage-Class:STANDARD' -H
> 'Accept-Encoding:gzip' -X POST
> 'http://localhost:/docker/docker/registry/v2/repositories/apache/ozone-runner/_uploads/2173f019-09c3-466b-bb7d-c31ce749d826/data?uploads
> {code}
> Without the patch it returns with HTTP 405 (Not supported Media Type).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
[
https://issues.apache.org/jira/browse/HDDS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-1948:
---
Description:
This problem is reported offline by [~shaneku...@gmail.com].
When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
Part Upload initialize message with "application/octet-stream" Content-Type.
This Content-Type is missing from the aws-cli which is used to reimplement s3
endpoint.
The problem is that we use the same rest endpoint for initialize and complete
Multipart Upload request. For the completion we need the
CompleteMultipartUploadRequest parameter which is parsed from the body.
For initialize we have an empty body which can't be serialized to
CompleteMultipartUploadRequest.
The workaround is to set a specific content type from a filter which help up to
create two different REST method for initialize and completion message.
Here is an example to test (using bogus AWS credentials).
{code}
curl -H 'Host:yourhost' -H 'User-Agent:aws-sdk-go/1.15.11 (go1.11.2; linux;
amd64)' -H 'Content-Length:0' -H 'Authorization:AWS4-HMAC-SHA256
Credential=qwe/20190809/ozone/s3/aws4_request,
SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class,
Signature=7726ed63990ba3f4f1f796d4ab263f5d9c3374528840f5e49d106dbef491f22c' -H
'Content-Type:application/octet-stream' -H 'X-Amz-Acl:private' -H
'X-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
-H 'X-Amz-Date:20190809T070142Z' -H 'X-Amz-Storage-Class:STANDARD' -H
'Accept-Encoding:gzip' -X POST
'http://localhost:/docker/docker/registry/v2/repositories/apache/ozone-runner/_uploads/2173f019-09c3-466b-bb7d-c31ce749d826/data?uploads
{code}
Without the patch it returns with HTTP 405 (Not supported Media Type).
was:
This problem is reported offline by [~shaneku...@gmail.com].
When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
Part Upload initialize message with "application/octet-stream" Content-Type.
This Content-Type is missing from the aws-cli which is used to reimplement s3
endpoint.
The problem is that we use the same rest endpoint for initialize and complete
Multipart Upload request. For the completion we need the
CompleteMultipartUploadRequest parameter which is parsed from the body.
For initialize we have an empty body which can't be serialized to
CompleteMultipartUploadRequest.
The workaround is to set a specific content type from a filter which help up to
create two different REST method for initialize and completion message.
> S3 MPU can't be created with octet-stream content-type
> ---
>
> Key: HDDS-1948
> URL: https://issues.apache.org/jira/browse/HDDS-1948
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> This problem is reported offline by [~shaneku...@gmail.com].
> When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi
> Part Upload initialize message with "application/octet-stream" Content-Type.
> This Content-Type is missing from the aws-cli which is used to reimplement s3
> endpoint.
> The problem is that we use the same rest endpoint for initialize and complete
> Multipart Upload request. For the completion we need the
> CompleteMultipartUploadRequest parameter which is parsed from the body.
> For initialize we have an empty body which can't be serialized to
> CompleteMultipartUploadRequest.
> The workaround is to set a specific content type from a filter which help up
> to create two different REST method for initialize and completion message.
> Here is an example to test (using bogus AWS credentials).
> {code}
> curl -H 'Host:yourhost' -H 'User-Agent:aws-sdk-go/1.15.11 (go1.11.2; linux;
> amd64)' -H 'Content-Length:0' -H 'Authorization:AWS4-HMAC-SHA256
> Credential=qwe/20190809/ozone/s3/aws4_request,
> SignedHeaders=content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date;x-amz-storage-class,
> Signature=7726ed63990ba3f4f1f796d4ab263f5d9c3374528840f5e49d106dbef491f22c'
> -H 'Content-Type:application/octet-stream' -H 'X-Amz-Acl:private' -H
> 'X-Amz-Content-Sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
> -H 'X-Amz-Date:20190809T070142Z' -H 'X-Amz-Storage-Class:STANDARD' -H
> 'Accept-Encoding:gzip' -X POST
> 'http://localhost:/docker/docker/registry/v2/repositories/apache/ozone-runner/_uploads/2173f019-09c3-466b-bb7d-c31ce749d826/data?uploads
> {code}
> Without the patch it returns with HTTP 405 (Not supported Media Type).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail:
[jira] [Updated] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
[ https://issues.apache.org/jira/browse/HDDS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1948: --- Component/s: S3 > S3 MPU can't be created with octet-stream content-type > --- > > Key: HDDS-1948 > URL: https://issues.apache.org/jira/browse/HDDS-1948 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: S3 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > This problem is reported offline by [~shaneku...@gmail.com]. > When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi > Part Upload initialize message with "application/octet-stream" Content-Type. > This Content-Type is missing from the aws-cli which is used to reimplement s3 > endpoint. > The problem is that we use the same rest endpoint for initialize and complete > Multipart Upload request. For the completion we need the > CompleteMultipartUploadRequest parameter which is parsed from the body. > For initialize we have an empty body which can't be serialized to > CompleteMultipartUploadRequest. > The workaround is to set a specific content type from a filter which help up > to create two different REST method for initialize and completion message. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1948) S3 MPU can't be created with octet-stream content-type
Elek, Marton created HDDS-1948: -- Summary: S3 MPU can't be created with octet-stream content-type Key: HDDS-1948 URL: https://issues.apache.org/jira/browse/HDDS-1948 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Elek, Marton Assignee: Elek, Marton This problem is reported offline by [~shaneku...@gmail.com]. When aws-sdk-go is used to access to s3 gateway of Ozone it sends the Multi Part Upload initialize message with "application/octet-stream" Content-Type. This Content-Type is missing from the aws-cli which is used to reimplement s3 endpoint. The problem is that we use the same rest endpoint for initialize and complete Multipart Upload request. For the completion we need the CompleteMultipartUploadRequest parameter which is parsed from the body. For initialize we have an empty body which can't be serialized to CompleteMultipartUploadRequest. The workaround is to set a specific content type from a filter which help up to create two different REST method for initialize and completion message. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1891) Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-1891:
-
Issue Type: Improvement (was: Bug)
> Ozone fs shell command should work with default port when port number is not
> specified
> --
>
> Key: HDDS-1891
> URL: https://issues.apache.org/jira/browse/HDDS-1891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> {code:bash|title=Without port number -> Error}
> $ ozone fs -ls o3fs://bucket.volume.localhost/
> -ls: Ozone file system url should be either one of the two forms:
> o3fs://bucket.volume/key OR o3fs://bucket.volume.om-host.example.com:5678/key
> ...
> {code}
> {code:bash|title=With port number -> Success}
> $ ozone fs -ls o3fs://bucket.volume.localhost:9862/
> Found 1 items
> -rw-rw-rw- 1 hadoop hadoop 1485 2019-08-01 21:14
> o3fs://bucket.volume.localhost:9862/README.txt
> {code}
> We expect the first command to attempt port 9862 by default.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
