[jira] [Work logged] (HDDS-1911) Support Prefix ACL operations for OM HA.

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1911?focusedWorklogId=292609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292609
 ]

ASF GitHub Bot logged work on HDDS-1911:


Author: ASF GitHub Bot
Created on: 11/Aug/19 05:35
Start Date: 11/Aug/19 05:35
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1275: HDDS-1911. 
Support Prefix ACL operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1275#issuecomment-520201758
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 75 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 0 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 3 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 36 | Maven dependency ordering for branch |
   | +1 | mvninstall | 609 | trunk passed |
   | +1 | compile | 368 | trunk passed |
   | +1 | checkstyle | 71 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 934 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 167 | trunk passed |
   | 0 | spotbugs | 433 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 630 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 21 | Maven dependency ordering for patch |
   | +1 | mvninstall | 555 | the patch passed |
   | +1 | compile | 373 | the patch passed |
   | +1 | javac | 373 | the patch passed |
   | +1 | checkstyle | 76 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 746 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 164 | the patch passed |
   | +1 | findbugs | 649 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 341 | hadoop-hdds in the patch passed. |
   | -1 | unit | 2238 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 45 | The patch does not generate ASF License warnings. |
   | | | 8266 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | 
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures |
   |   | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider |
   |   | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures |
   |   | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
   |   | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.0 Server=19.03.0 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1275 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux 89b1c4b02a75 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / d29007f |
   | Default Java | 1.8.0_212 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/testReport/ |
   | Max. process+thread count | 4725 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: 
hadoop-ozone |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1275/1/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292609)
Time Spent: 20m  (was: 10m)

> Support Prefix ACL operations for OM HA.
> 
>
> Key: HDDS-1911
> URL: https://issues.apache.org/jira/browse/HDDS-1911
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: 

[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.

2019-08-10 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904561#comment-16904561
 ] 

Wei-Chiu Chuang commented on HDFS-13505:


Yes we should update the HDFS Permission Guide. Thanks for catching that 
[~ayushtkn]!

> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14174) Enhance Audit for chown ( internally setOwner)

2019-08-10 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904560#comment-16904560
 ] 

Wei-Chiu Chuang commented on HDFS-14174:


i'd like to know the justification for this change.
Also, if this audit log is changed, it may have to be hidden behind a 
configuration to avoid incompatibility.

> Enhance Audit for chown ( internally setOwner)   
> -
>
> Key: HDFS-14174
> URL: https://issues.apache.org/jira/browse/HDFS-14174
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.6.0
>Reporter: Sailesh Patel
>Assignee: hemanthboyina
>Priority: Minor
>
> When a hdfs dfs -chown  command is executed, the audit log  does not capture 
> the  existing owner and the new owner.    
> Need to capture the old and new owner to allow auditing to be effective
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1911) Support Prefix ACL operations for OM HA.

2019-08-10 Thread Bharat Viswanadham (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDDS-1911:
-
Status: Patch Available  (was: Open)

> Support Prefix ACL operations for OM HA.
> 
>
> Key: HDDS-1911
> URL: https://issues.apache.org/jira/browse/HDDS-1911
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> +-HDDS-1608-+ adds 4 new api for Ozone rpc client. OM HA implementation needs 
> to handle them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1911) Support Prefix ACL operations for OM HA.

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-1911:
-
Labels: pull-request-available  (was: )

> Support Prefix ACL operations for OM HA.
> 
>
> Key: HDDS-1911
> URL: https://issues.apache.org/jira/browse/HDDS-1911
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
>
> +-HDDS-1608-+ adds 4 new api for Ozone rpc client. OM HA implementation needs 
> to handle them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1911) Support Prefix ACL operations for OM HA.

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1911?focusedWorklogId=292604=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292604
 ]

ASF GitHub Bot logged work on HDDS-1911:


Author: ASF GitHub Bot
Created on: 11/Aug/19 03:16
Start Date: 11/Aug/19 03:16
Worklog Time Spent: 10m 
  Work Description: bharatviswa504 commented on pull request #1275: 
HDDS-1911. Support Prefix ACL operations for OM HA.
URL: https://github.com/apache/hadoop/pull/1275
 
 
   (cherry picked from commit 3906b7d233cd276901fc86b24d7d26d676ff985b)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292604)
Time Spent: 10m
Remaining Estimate: 0h

> Support Prefix ACL operations for OM HA.
> 
>
> Key: HDDS-1911
> URL: https://issues.apache.org/jira/browse/HDDS-1911
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> +-HDDS-1608-+ adds 4 new api for Ozone rpc client. OM HA implementation needs 
> to handle them.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.

2019-08-10 Thread Dinesh Chitlangia (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904543#comment-16904543
 ] 

Dinesh Chitlangia commented on HDFS-13505:
--

[~smeng]- Release notes LGTM. Thank you for your kind consideration!


> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14423) Percent (%) and plus (+) characters no longer work in WebHDFS

2019-08-10 Thread Masatake Iwasaki (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904531#comment-16904531
 ] 

Masatake Iwasaki commented on HDFS-14423:
-

Thanks [~jojochuang]. While the patch looks good, the issued reported as "plus 
(+ ) characters get turned into spaces" still seems to be exists. I'm looking 
into the cause.
{noformat}
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz 
webhdfs://localhost/%
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -touchz 
webhdfs://localhost/a+b
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -mkdir 
webhdfs://localhost/c+d
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls webhdfs://localhost/
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hadoop fs -put 'e+f.txt' 
'webhdfs://localhost/'
put: rename `webhdfs://localhost/e+f.txt._COPYING_' to 
`webhdfs://localhost/e+f.txt': Input/output error
[iwasakims@centos7 hadoop-3.3.0-SNAPSHOT]$ bin/hdfs dfs -ls /
Found 6 items
-rw-r--r--   1 iwasakims supergroup  0 2019-08-09 23:35 /%
-rw-r--r--   1 iwasakims supergroup  0 2019-08-09 23:50 /a b
drwxr-xr-x   - iwasakims supergroup  0 2019-08-09 23:37 /c+d
-rw-r--r--   1 iwasakims supergroup  5 2019-08-09 23:52 /e 
f.txt._COPYING_
drwxrwxrwx   - iwasakims supergroup  0 2019-06-14 13:09 /tmp
drwxr-xr-x   - iwasakims supergroup  0 2019-06-14 13:10 /user
{noformat}

> Percent (%) and plus (+) characters no longer work in WebHDFS
> -
>
> Key: HDFS-14423
> URL: https://issues.apache.org/jira/browse/HDFS-14423
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
> Environment: Ubuntu 16.04, but I believe this is irrelevant.
>Reporter: Jing Wang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HDFS-14423.001.patch
>
>
> The following commands with percent (%) no longer work starting with version 
> 3.1:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/%
> $ hadoop/bin/hdfs dfs -cat webhdfs://localhost/%
> cat: URLDecoder: Incomplete trailing escape (%) pattern
> {code}
> Also, plus (+ ) characters get turned into spaces when doing DN operations:
> {code:java}
> $ hadoop/bin/hdfs dfs -touchz webhdfs://localhost/a+b
> $ hadoop/bin/hdfs dfs -mkdir webhdfs://localhost/c+d
> $ hadoop/bin/hdfs dfs -ls /
> Found 4 items
> -rw-r--r--   1 jing supergroup  0 2019-04-12 11:20 /a b
> drwxr-xr-x   - jing supergroup  0 2019-04-12 11:21 /c+d
> {code}
> I can confirm that these commands work correctly on 2.9 and 3.0. Also, the 
> usual hdfs:// client works as expected.
> I suspect a relation with HDFS-13176 or HDFS-13582, but I'm not sure what the 
> right fix is. Note that Hive uses % to escape special characters in partition 
> values, so banning % might not be a good option. For example, Hive will 
> create a paths like {{table_name/partition_key=%2F}} when 
> {{partition_key='/'}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does

2019-08-10 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904511#comment-16904511
 ] 

Wei-Chiu Chuang commented on HDFS-14718:


There are two things that should get considered:
(1) does it break existing applications? TreeMap and HashMap are different in 
that the output may be in different order. I don't we ever specified the order 
so this isn't an incompatible change.
(2) performance.  which one is more performant if you list a million files?

> HttpFS: Sort response by key names as WebHDFS does
> --
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> *Example*
> See description of HDFS-14665 for an example of LISTSTATUS.
> *Analysis*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
> It is not only limited to LISTSTATUS, but ALL other request's JSON 
> serialization.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-13505) Turn on HDFS ACLs by default.

2019-08-10 Thread Ayush Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904510#comment-16904510
 ] 

Ayush Saxena edited comment on HDFS-13505 at 8/10/19 8:31 PM:
--

Thanx [~smeng] for the patch. Guess you missed updating in 
{{HDFSPermissionsGuide}} as I mentioned above.
[~dineshchitlangia] can you give a check too on the release note, If you have 
any objections, or want something to get added.


was (Author: ayushtkn):
Thanx [~smeng] for the patch. Guess you missed updating in 
{{HDFSPermissionsGuide}} as I mentioned above.

> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.

2019-08-10 Thread Ayush Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904510#comment-16904510
 ] 

Ayush Saxena commented on HDFS-13505:
-

Thanx [~smeng] for the patch. Guess you missed updating in 
{{HDFSPermissionsGuide}} as I mentioned above.

> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14595) HDFS-11848 breaks API compatibility

2019-08-10 Thread Ayush Saxena (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904509#comment-16904509
 ] 

Ayush Saxena commented on HDFS-14595:
-

Thanx [~smeng] Almost LGTM, just like {{TestStripedFileAppend}} seems little 
odd place to keep the test. That is for checking append in EC file. May be you 
can try {{TestDistributedFileSystem.java}} for the same.

> HDFS-11848 breaks API compatibility
> ---
>
> Key: HDFS-14595
> URL: https://issues.apache.org/jira/browse/HDFS-14595
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Blocker
> Attachments: HDFS-14595.001.patch, HDFS-14595.002.patch, 
> HDFS-14595.003.patch, HDFS-14595.004.patch, HDFS-14595.005.patch, hadoop_ 
> 36e1870eab904d5a6f12ecfb1fdb52ca08d95ac5 to 
> b241194d56f97ee372cbec7062bcf155bc3df662 compatibility report.htm
>
>
> Our internal tool caught an API compatibility issue with HDFS-11848.
> HDFS-11848 adds an additional parameter to 
> DistributedFileSystem.listOpenFiles(), but it doesn't keep the existing API.
> This can cause issue when upgrading from Hadoop 2.9.0/2.8.3/3.0.0 to 
> 3.0.1/3.1.0 and above.
> Suggest:
> (1) Add back the old API (which was added in HDFS-10480), and mark it 
> deprecated.
> (2) Update release doc to enforce running API compatibility check for each 
> releases.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-13270) RBF: Router audit logger

2019-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904498#comment-16904498
 ] 

Hadoop QA commented on HDFS-13270:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  6s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  3m 10s{color} 
| {color:red} hadoop-hdfs-project generated 1 new + 552 unchanged - 0 fixed = 
553 total (was 552) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 53s{color} | {color:orange} hadoop-hdfs-project: The patch generated 34 new 
+ 183 unchanged - 0 fixed = 217 total (was 183) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 16s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 21s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 52s{color} 
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}205m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
|  |  Write to static field 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.auditLoggers 
from instance method new 
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer(Configuration, 
Router, ActiveNamenodeResolver, FileSubclusterResolver)  At 
RouterRpcServer.java:from instance method new 

[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292567
 ]

ASF GitHub Bot logged work on HDDS-1927:


Author: ASF GitHub Bot
Created on: 10/Aug/19 18:30
Start Date: 10/Aug/19 18:30
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1263: HDDS-1927. 
Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520170426
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 47 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 1 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 5 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 14 | Maven dependency ordering for branch |
   | +1 | mvninstall | 586 | trunk passed |
   | +1 | compile | 371 | trunk passed |
   | +1 | checkstyle | 76 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 854 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 163 | trunk passed |
   | 0 | spotbugs | 420 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 608 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 18 | Maven dependency ordering for patch |
   | +1 | mvninstall | 536 | the patch passed |
   | +1 | compile | 344 | the patch passed |
   | +1 | javac | 344 | the patch passed |
   | +1 | checkstyle | 66 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 663 | patch has no errors when building and testing 
our client artifacts. |
   | -1 | javadoc | 84 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed 
= 15 total (was 13) |
   | +1 | findbugs | 666 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 303 | hadoop-hdds in the patch passed. |
   | -1 | unit | 51 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 34 | The patch does not generate ASF License warnings. |
   | | | 5692 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.ozone.om.helpers.TestOmKeyInfo |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/6/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1263 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux 5e12fa305c5a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / e69db45 |
   | Default Java | 1.8.0_212 |
   | javadoc | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/6/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/6/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/6/testReport/ |
   | Max. process+thread count | 490 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/common hadoop-ozone/client 
hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service 
hadoop-ozone/integration-test U: hadoop-ozone |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/6/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292567)
Time Spent: 1h 20m  (was: 1h 10m)

> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
>  

[jira] [Commented] (HDFS-14625) Make DefaultAuditLogger class in FSnamesystem to Abstract

2019-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904494#comment-16904494
 ] 

Hadoop QA commented on HDFS-14625:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
 6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 36s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 8 new + 177 unchanged - 0 fixed = 185 total (was 177) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 40s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 11s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
46s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 58s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
|   | hadoop.hdfs.server.datanode.TestLargeBlockReport |
|   | hadoop.hdfs.server.balancer.TestBalancer |
|   | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14625 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12977227/HDFS-14625.004.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 27b145f4ce24 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fba222a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/27468/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 

[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292564
 ]

ASF GitHub Bot logged work on HDDS-1927:


Author: ASF GitHub Bot
Created on: 10/Aug/19 17:58
Start Date: 10/Aug/19 17:58
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1263: HDDS-1927. 
Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520168308
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 46 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 1 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 5 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 15 | Maven dependency ordering for branch |
   | +1 | mvninstall | 595 | trunk passed |
   | +1 | compile | 371 | trunk passed |
   | +1 | checkstyle | 74 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 864 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 165 | trunk passed |
   | 0 | spotbugs | 420 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 616 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 25 | Maven dependency ordering for patch |
   | +1 | mvninstall | 553 | the patch passed |
   | +1 | compile | 382 | the patch passed |
   | +1 | javac | 382 | the patch passed |
   | -0 | checkstyle | 43 | hadoop-ozone: The patch generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 681 | patch has no errors when building and testing 
our client artifacts. |
   | -1 | javadoc | 94 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed 
= 15 total (was 13) |
   | -1 | findbugs | 433 | hadoop-ozone generated 1 new + 0 unchanged - 0 fixed 
= 1 total (was 0) |
   ||| _ Other Tests _ |
   | +1 | unit | 285 | hadoop-hdds in the patch passed. |
   | -1 | unit | 54 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 45 | The patch does not generate ASF License warnings. |
   | | | 5855 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | FindBugs | module:hadoop-ozone |
   |  |  Possible null pointer dereference of null in 
org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo)
  Dereferenced at OmKeyInfo.java:null in 
org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo)
  Dereferenced at OmKeyInfo.java:[line 394] |
   | Failed junit tests | hadoop.ozone.om.helpers.TestOmKeyInfo |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1263 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux fb697dc901b3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / e69db45 |
   | Default Java | 1.8.0_212 |
   | checkstyle | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/diff-checkstyle-hadoop-ozone.txt
 |
   | javadoc | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
 |
   | findbugs | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/new-findbugs-hadoop-ozone.html
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/testReport/ |
   | Max. process+thread count | 557 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/common hadoop-ozone/client 
hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service 
hadoop-ozone/integration-test U: hadoop-ozone |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/5/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to 

[jira] [Commented] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.

2019-08-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDDS-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904480#comment-16904480
 ] 

Hudson commented on HDDS-1366:
--

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17085 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/17085/])
HDDS-1366. Add ability in Recon to track the number of small files in an (arp7: 
rev d29007fb35d6667f9e8f1d9befafe61b19ca7c18)
* (add) 
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/UtilizationService.java
* (add) 
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/FileSizeCountTask.java
* (edit) 
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerKeyService.java
* (edit) 
hadoop-ozone/ozone-recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/UtilizationSchemaDefinition.java
* (add) 
hadoop-ozone/ozone-recon/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
* (edit) 
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java
* (edit) 
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/TestUtilizationSchemaDefinition.java
* (add) 
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestUtilizationService.java
* (add) 
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestFileSizeCountTask.java
* (edit) 
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java


> Add ability in Recon to track the number of small files in an Ozone cluster.
> 
>
> Key: HDDS-1366
> URL: https://issues.apache.org/jira/browse/HDDS-1366
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: Ozone Recon
>Reporter: Aravindan Vijayan
>Assignee: Shweta
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> Ozone users may want to track the number of small files they have in their 
> cluster and where they are present. Recon can help them with the information 
> by iterating the OM Key Table and dividing the keys into different buckets 
> based on the data size. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1366?focusedWorklogId=292545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292545
 ]

ASF GitHub Bot logged work on HDDS-1366:


Author: ASF GitHub Bot
Created on: 10/Aug/19 17:14
Start Date: 10/Aug/19 17:14
Worklog Time Spent: 10m 
  Work Description: arp7 commented on pull request #1146: HDDS-1366. Add 
ability in Recon to track the number of small files in an Ozone Cluster
URL: https://github.com/apache/hadoop/pull/1146
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292545)
Time Spent: 12h  (was: 11h 50m)

> Add ability in Recon to track the number of small files in an Ozone cluster.
> 
>
> Key: HDDS-1366
> URL: https://issues.apache.org/jira/browse/HDDS-1366
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: Ozone Recon
>Reporter: Aravindan Vijayan
>Assignee: Shweta
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> Ozone users may want to track the number of small files they have in their 
> cluster and where they are present. Recon can help them with the information 
> by iterating the OM Key Table and dividing the keys into different buckets 
> based on the data size. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1366) Add ability in Recon to track the number of small files in an Ozone cluster.

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1366?focusedWorklogId=292544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292544
 ]

ASF GitHub Bot logged work on HDDS-1366:


Author: ASF GitHub Bot
Created on: 10/Aug/19 17:14
Start Date: 10/Aug/19 17:14
Worklog Time Spent: 10m 
  Work Description: arp7 commented on issue #1146: HDDS-1366. Add ability 
in Recon to track the number of small files in an Ozone Cluster
URL: https://github.com/apache/hadoop/pull/1146#issuecomment-520165586
 
 
   +1 the test failures look unrelated. There are a few checkstyle failures in 
tests, can you file a followup jira to address them right away?
   
   Thanks for the contribution @shwetayakkali and thanks @avijayanhwx and 
@vivekratnavel  for the reviews!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292544)
Time Spent: 11h 50m  (was: 11h 40m)

> Add ability in Recon to track the number of small files in an Ozone cluster.
> 
>
> Key: HDDS-1366
> URL: https://issues.apache.org/jira/browse/HDDS-1366
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: Ozone Recon
>Reporter: Aravindan Vijayan
>Assignee: Shweta
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> Ozone users may want to track the number of small files they have in their 
> cluster and where they are present. Recon can help them with the information 
> by iterating the OM Key Table and dividing the keys into different buckets 
> based on the data size. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292542
 ]

ASF GitHub Bot logged work on HDDS-1927:


Author: ASF GitHub Bot
Created on: 10/Aug/19 16:45
Start Date: 10/Aug/19 16:45
Worklog Time Spent: 10m 
  Work Description: xiaoyuyao commented on issue #1263: HDDS-1927. 
Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520163523
 
 
   /retest
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292542)
Time Spent: 1h  (was: 50m)

> Consolidate add/remove Acl into OzoneAclUtil class
> --
>
> Key: HDDS-1927
> URL: https://issues.apache.org/jira/browse/HDDS-1927
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Xiaoyu Yao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> This Jira is created based on @xiaoyu comment on HDDS-1884
> Can we abstract these add/remove logic into common AclUtil class as we can 
> see similar logic in both bucket manager and key manager? For example,
> public static boolean addAcl(List existingAcls, OzoneAcl newAcl)
> public static boolean removeAcl(List existingAcls, OzoneAcl newAcl)
>  
> But to do this, we need both OmKeyInfo and OMBucketInfo to use list of 
> OzoneAcl/OzoneAclInfo.
> This Jira is to do that refactor, and also address above comment to move 
> common logic to AclUtils.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDDS-1499) OzoneManager Cache

2019-08-10 Thread Anu Engineer (JIRA)


[ 
https://issues.apache.org/jira/browse/HDDS-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904469#comment-16904469
 ] 

Anu Engineer commented on HDDS-1499:


[~nandakumar131] FYI ...

> OzoneManager Cache
> --
>
> Key: HDDS-1499
> URL: https://issues.apache.org/jira/browse/HDDS-1499
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: Ozone Manager
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.4.1, 0.5.0
>
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> In this Jira, we shall implement a cache for Table.
> As with OM HA, we are planning to implement double buffer implementation to 
> flush transaction in a batch, instead of using rocksdb put() for every 
> operation. When this comes in to place we need cache in OzoneManager HA to 
> handle/server the requests for validation/returning responses.
>  
> This Jira will implement Cache as an integral part of the table. In this way 
> users using this table does not need to handle like check cache/db. For this, 
> we can update get API in the table to handle the cache.
>  
> This Jira will implement:
>  # Cache as a part of each Table.
>  # Uses this cache in get().
>  # Exposes api for cleanup, add entries to cache.
> Usage to add the entries in to cache will be done in further jira's.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?focusedWorklogId=292532=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292532
 ]

ASF GitHub Bot logged work on HDDS-1951:


Author: ASF GitHub Bot
Created on: 10/Aug/19 15:27
Start Date: 10/Aug/19 15:27
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1273: HDDS-1951. Wrong 
symbolic release name on 0.4.1 branch
URL: https://github.com/apache/hadoop/pull/1273#issuecomment-520157375
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 50 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 0 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | -1 | test4tests | 0 | The patch doesn't appear to include any new or 
modified tests.  Please justify why no new tests are needed for this patch. 
Also please list what manual steps were performed to verify this patch. |
   ||| _ ozone-0.4.1 Compile Tests _ |
   | +1 | mvninstall | 586 | ozone-0.4.1 passed |
   | +1 | compile | 368 | ozone-0.4.1 passed |
   | +1 | mvnsite | 0 | ozone-0.4.1 passed |
   | +1 | shadedclient | 1736 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 162 | ozone-0.4.1 passed |
   ||| _ Patch Compile Tests _ |
   | +1 | mvninstall | 551 | the patch passed |
   | +1 | compile | 375 | the patch passed |
   | +1 | javac | 375 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | xml | 1 | The patch has no ill-formed XML file. |
   | +1 | shadedclient | 679 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 162 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 297 | hadoop-hdds in the patch passed. |
   | -1 | unit | 2084 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 54 | The patch does not generate ASF License warnings. |
   | | | 6351 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.ozone.TestStorageContainerManager |
   |   | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures |
   |   | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
   |   | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1273 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient xml |
   | uname | Linux d11e154c4a50 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | ozone-0.4.1 / 8b80ca0 |
   | Default Java | 1.8.0_212 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/testReport/ |
   | Max. process+thread count | 5408 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone U: hadoop-ozone |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1273/1/console |
   | versions | git=2.7.4 maven=3.3.9 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292532)
Time Spent: 20m  (was: 10m)

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA

[jira] [Commented] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox

2019-08-10 Thread Anu Engineer (JIRA)


[ 
https://issues.apache.org/jira/browse/HDDS-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904451#comment-16904451
 ] 

Anu Engineer commented on HDDS-1923:


[~nandakumar131], [~elek] Let us see if we can fix this. I will try this out on 
Monday and post a patch if needed. Thanks


> static/docs/start.html page doesn't render correctly on Firefox
> ---
>
> Key: HDDS-1923
> URL: https://issues.apache.org/jira/browse/HDDS-1923
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Anu Engineer
>Priority: Blocker
>
> static/docs/start.html page doesn't render correctly on Firefox



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox

2019-08-10 Thread Anu Engineer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDDS-1923:
---
Priority: Blocker  (was: Major)

> static/docs/start.html page doesn't render correctly on Firefox
> ---
>
> Key: HDDS-1923
> URL: https://issues.apache.org/jira/browse/HDDS-1923
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Anu Engineer
>Priority: Blocker
>
> static/docs/start.html page doesn't render correctly on Firefox



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDDS-1923) static/docs/start.html page doesn't render correctly on Firefox

2019-08-10 Thread Anu Engineer (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer reassigned HDDS-1923:
--

Assignee: Anu Engineer

> static/docs/start.html page doesn't render correctly on Firefox
> ---
>
> Key: HDDS-1923
> URL: https://issues.apache.org/jira/browse/HDDS-1923
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Anu Engineer
>Priority: Major
>
> static/docs/start.html page doesn't render correctly on Firefox



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-13270) RBF: Router audit logger

2019-08-10 Thread hemanthboyina (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hemanthboyina updated HDFS-13270:
-
Attachment: HDFS-13270.002.patch

> RBF: Router audit logger
> 
>
> Key: HDFS-13270
> URL: https://issues.apache.org/jira/browse/HDFS-13270
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-13270.001.patch, HDFS-13270.002.patch
>
>
> We can use router auditlogger to log the client info and cmd, because the 
> FSNamesystem#Auditlogger's log think the client are all from router.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14625) Make DefaultAuditLogger class in FSnamesystem to Abstract

2019-08-10 Thread hemanthboyina (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hemanthboyina updated HDFS-14625:
-
Attachment: HDFS-14625.004.patch

> Make DefaultAuditLogger class in FSnamesystem to Abstract 
> --
>
> Key: HDFS-14625
> URL: https://issues.apache.org/jira/browse/HDFS-14625
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14625 (1).patch, HDFS-14625(2).patch, 
> HDFS-14625.003.patch, HDFS-14625.004.patch, HDFS-14625.patch
>
>
> As per +HDFS-13270+  Audit logger for Router , we can make DefaultAuditLogger 
>  in FSnamesystem to be Abstract and common



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDDS-1942) Support copy during S3 multipart upload part creation

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton reassigned HDDS-1942:
--

Assignee: Elek, Marton

> Support copy during S3 multipart upload part creation
> -
>
> Key: HDDS-1942
> URL: https://issues.apache.org/jira/browse/HDDS-1942
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>  Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>
> Uploads a part by copying data from an existing object as data source
> Documented here:
> https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-13505) Turn on HDFS ACLs by default.

2019-08-10 Thread Siyao Meng (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904445#comment-16904445
 ] 

Siyao Meng commented on HDFS-13505:
---

[~dineshchitlangia] I've added release notes to this jira. Please review. 
Thanks!

> Turn on HDFS ACLs by default.
> -
>
> Key: HDFS-13505
> URL: https://issues.apache.org/jira/browse/HDFS-13505
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13505.00.patch, HDFS-13505.001.patch
>
>
> Turn on HDFS ACLs by default.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDDS-1952) TestMiniChaosOzoneCluster may run until OOME

2019-08-10 Thread Doroszlai, Attila (JIRA)
Doroszlai, Attila created HDDS-1952:
---

 Summary: TestMiniChaosOzoneCluster may run until OOME
 Key: HDDS-1952
 URL: https://issues.apache.org/jira/browse/HDDS-1952
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: test
Reporter: Doroszlai, Attila


{{TestMiniChaosOzoneCluster}} runs load generator on a cluster for supposedly 1 
minute, but it may run indefinitely until JVM crashes due to OutOfMemoryError.

In 0.4.1 nightly build it crashed 29/30 times (and no tests were executed in 
the remaining one run due to some other error).

Latest:
https://github.com/elek/ozone-ci/blob/3f553ed6ad358ba61a302967617de737d7fea01a/byscane/byscane-nightly-wggqd/integration/output.log#L5661-L5662

When it crashes, it leaves GBs of data lying around.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory

2019-08-10 Thread Siyao Meng (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904437#comment-16904437
 ] 

Siyao Meng commented on HDFS-14148:
---

PR available: https://github.com/apache/hadoop/pull/1274

> HDFS OIV ReverseXML SnapshotSection parser throws exception when there are 
> more than one snapshottable directory
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work started] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-14148 started by Siyao Meng.
-
> HDFS OIV ReverseXML SnapshotSection parser throws exception when there are 
> more than one snapshottable directory
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Summary: HDFS OIV ReverseXML SnapshotSection parser throws exception when 
there are more than one snapshottable directory  (was: HDFS OIV ReverseXML 
SnapshotSection parser would throw exception when there are >= 2 snapshottable 
directories)

> HDFS OIV ReverseXML SnapshotSection parser throws exception when there are 
> more than one snapshottable directory
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDDS-1950) S3 MPU part-list call fails if there are no parts

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton reassigned HDDS-1950:
--

Assignee: Elek, Marton

> S3 MPU part-list call fails if there are no parts
> -
>
> Key: HDDS-1950
> URL: https://issues.apache.org/jira/browse/HDDS-1950
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: S3
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> If an S3 multipart upload is created but no part is upload the part list 
> can't be called because it throws HTTP 500:
> Create an MPU:
> {code}
> aws s3api --endpoint http://localhost: create-multipart-upload 
> --bucket=docker --key=testkeu 
> {
> "Bucket": "docker",
> "Key": "testkeu",
> "UploadId": "85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234"
> }
> {code}
> List the parts:
> {code}
> aws s3api --endpoint http://localhost: list-parts  --bucket=docker 
> --key=testkeu 
> --upload-id=85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234
> {code}
> It throws an exception on the server side, because in the 
> KeyManagerImpl.listParts the  ReplicationType is retrieved from the first 
> part:
> {code}
> HddsProtos.ReplicationType replicationType =
> partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
> {code}
> Which is not yet available in this use case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser would throw exception when there are >= 2 snapshottable directories

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Target Version/s: 3.3.0, 3.2.1, 3.1.3

> HDFS OIV ReverseXML SnapshotSection parser would throw exception when there 
> are >= 2 snapshottable directories
> --
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser would throw exception when there are >= 2 snapshottable directories

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Issue Type: Bug  (was: Improvement)

> HDFS OIV ReverseXML SnapshotSection parser would throw exception when there 
> are >= 2 snapshottable directories
> --
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML SnapshotSection parser would throw exception when there are >= 2 snapshottable directories

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Summary: HDFS OIV ReverseXML SnapshotSection parser would throw exception 
when there are >= 2 snapshottable directories  (was: HDFS OIV ReverseXML 
snapshot issue: Found unknown XML keys in : dir)

> HDFS OIV ReverseXML SnapshotSection parser would throw exception when there 
> are >= 2 snapshottable directories
> --
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
> *TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories. Apply HDFS-14148.test.001.patch and run 
*TestOfflineImageViewer#testReverseXmlRoundTrip()* to see my point.

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories. Apply 


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> This problem can be easily reproduced when there are at least TWO 
> 

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

This problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories. Apply 

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.

{code:title=trunk}
$ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> 

[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Attachment: HDFS-14148.test.001.patch

> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14148.test.001.patch, fsimage_024, 
> fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories.
> {code:title=trunk}
> $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: 
> ExitException
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Attachment: fsimage_024

> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_024, fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories.
> {code:title=trunk}
> $ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: 
> ExitException
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
>   at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)
> $ grep -n "" fsimage_0026542.xml
> 228:222049220495
> {code}
> This is also reproduced on latest trunk 
> fba222a85603d6321419aa37bcc48d276dd6c4a6
> It seems this problem can be easily reproduced when there are at least TWO 
> snapshot-enabled directories.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14148) HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14148:
--
Description: 
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.

{code:title=trunk}
$ hdfs oiv -i 24.xml -o 24.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
2019-08-10 06:58:04,444 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}

  was:
The current HDFS OIV tool doesn't seem to support snapshot well when reversing 
XML back to binary.
{code:bash}
$ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
OfflineImageReconstructor failed: Found unknown XML keys in : dir
java.io.IOException: Found unknown XML keys in : dir
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$Node.verifyNoRemainingKeys(OfflineImageReconstructor.java:324)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor$SnapshotSectionProcessor.process(OfflineImageReconstructor.java:1357)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1785)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1840)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:193)
at 
org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:136)

$ grep -n "" fsimage_0026542.xml
228:222049220495
{code}

This is also reproduced on latest trunk fba222a85603d6321419aa37bcc48d276dd6c4a6

It seems this problem can be easily reproduced when there are at least TWO 
snapshot-enabled directories.


> HDFS OIV ReverseXML snapshot issue: Found unknown XML keys in : dir
> 
>
> Key: HDFS-14148
> URL: https://issues.apache.org/jira/browse/HDFS-14148
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: fsimage_441
>
>
> The current HDFS OIV tool doesn't seem to support snapshot well when 
> reversing XML back to binary.
> {code:bash}
> $ hdfs oiv -i fsimage_0026542.xml -o reverse.bin -p ReverseXML
> OfflineImageReconstructor failed: Found unknown XML keys in : dir
> java.io.IOException: Found unknown XML keys in : dir
>   at 
> 

[jira] [Work logged] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?focusedWorklogId=292520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292520
 ]

ASF GitHub Bot logged work on HDDS-1951:


Author: ASF GitHub Bot
Created on: 10/Aug/19 13:40
Start Date: 10/Aug/19 13:40
Worklog Time Spent: 10m 
  Work Description: elek commented on pull request #1273: HDDS-1951. Wrong 
symbolic release name on 0.4.1 branch
URL: https://github.com/apache/hadoop/pull/1273
 
 
   See: https://issues.apache.org/jira/browse/HDDS-1951
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292520)
Time Spent: 10m
Remaining Estimate: 0h

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton updated HDDS-1951:
---
Sprint: HDDS Biscayne

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-1951:
-
Labels: pull-request-available  (was: )

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>  Labels: pull-request-available
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton updated HDDS-1951:
---
Status: Patch Available  (was: Open)

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton reassigned HDDS-1951:
--

Assignee: Elek, Marton

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1951) Wrong symbolic release name on 0.4.1 branch

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton updated HDDS-1951:
---
Summary: Wrong symbolic release name on 0.4.1 branch  (was: Wrong symbolic 
release name on 0.4.1 branc)

> Wrong symbolic release name on 0.4.1 branch
> ---
>
> Key: HDDS-1951
> URL: https://issues.apache.org/jira/browse/HDDS-1951
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>Reporter: Elek, Marton
>Priority: Blocker
>
> Should be Biscayne instead of Crater lake according to the Roadmap:
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDDS-1951) Wrong symbolic release name on 0.4.1 branc

2019-08-10 Thread Elek, Marton (JIRA)
Elek, Marton created HDDS-1951:
--

 Summary: Wrong symbolic release name on 0.4.1 branc
 Key: HDDS-1951
 URL: https://issues.apache.org/jira/browse/HDDS-1951
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
Reporter: Elek, Marton


Should be Biscayne instead of Crater lake according to the Roadmap:

https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Road+Map




--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14615) NPE writing edit logs

2019-08-10 Thread hemanthboyina (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904417#comment-16904417
 ] 

hemanthboyina commented on HDFS-14615:
--

i think the group name was null here , creating NPE during MKDIR operation

> NPE writing edit logs
> -
>
> Key: HDFS-14615
> URL: https://issues.apache.org/jira/browse/HDFS-14615
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 3.1.1
>Reporter: Wei-Chiu Chuang
>Assignee: Lisheng Sun
>Priority: Major
>
> Hit a weird bug where writing mkdir op to edit log throws an NPE and NameNode 
> crashed
> {noformat}
> 2019-06-26 10:57:27,398 ERROR 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: write op failed for 
> (journal 
> JournalAndStream(mgr=FileJournalManager(root=/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn),
>  
> stream=EditLogFileOutputStream(/ssd/work/src/upstream/impala/testdata/cluster/cdh6/node-1/data/dfs/nn/current/edits_inprogress_0598588)))
> java.lang.NullPointerException
> at org.apache.hadoop.io.Text.encode(Text.java:451)
> at org.apache.hadoop.io.Text.encode(Text.java:431)
> at org.apache.hadoop.io.Text.writeString(Text.java:491)
> at 
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:104)
> at 
> org.apache.hadoop.fs.permission.PermissionStatus.write(PermissionStatus.java:84)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$MkdirOp.writeFields(FSEditLogOp.java:1654)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$Writer.writeOp(FSEditLogOp.java:4866)
> at 
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer$TxnBuffer.writeOp(EditsDoubleBuffer.java:157)
> at 
> org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.writeOp(EditsDoubleBuffer.java:60)
> at 
> org.apache.hadoop.hdfs.server.namenode.EditLogFileOutputStream.write(EditLogFileOutputStream.java:97)
> at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$1.apply(JournalSet.java:444)
> at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
> at 
> org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.write(JournalSet.java:440)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.doEditTransaction(FSEditLog.java:481)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync$Edit.logEdit(FSEditLogAsync.java:288)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:232)
> {noformat}
> The stacktrace is similar to SENTRY-555, which is thought to be a Sentry bug 
> (authorization provider), but this cluster doesn't have Sentry and therefore 
> could be a genuine HDFS bug.
> File this jira to keep a record.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas

2019-08-10 Thread Siyao Meng (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904414#comment-16904414
 ] 

Siyao Meng edited comment on HDFS-14491 at 8/10/19 12:40 PM:
-

Also as shown in your snippet of *InvalidateBlocks#add*, this operation should 
be performed for *each* datanode that has the replica. Hence the 
*PendingDeletionBlocks* metric does include all replicas(or EC blocks).


was (Author: smeng):
Also as shown in your snippet of *InvalidateBlocks#add*, this operation should 
be performed for *each* datanode that has the replica. Hence the 
*PendingDeletionBlocks* metric does include all replicas/EC strips.

> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During 
> the process of the delete, I noticed that the NameNode UI near the top has a 
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total 
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the 
> cluster. When that number was below the total number of blocks, I briefly 
> believed I had deleted the entire cluster. In reality, the second number 
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and 
> where it doesn't. This may also have an impact on the under-replicated count.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas

2019-08-10 Thread Siyao Meng (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904414#comment-16904414
 ] 

Siyao Meng commented on HDFS-14491:
---

Also as shown in your snippet of *InvalidateBlocks#add*, this operation should 
be performed for *each* datanode that has the replica. Hence the 
*PendingDeletionBlocks* metric does include all replicas/EC strips.

> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During 
> the process of the delete, I noticed that the NameNode UI near the top has a 
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total 
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the 
> cluster. When that number was below the total number of blocks, I briefly 
> believed I had deleted the entire cluster. In reality, the second number 
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and 
> where it doesn't. This may also have an impact on the under-replicated count.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14491) More Clarity on Namenode UI Around Blocks and Replicas

2019-08-10 Thread Siyao Meng (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904413#comment-16904413
 ] 

Siyao Meng commented on HDFS-14491:
---

[~jojochuang]

The metric is retrieved via *FSNameSystem#getPendingDeletionBlocks* 
[here|https://github.com/apache/hadoop/blob/b964b81f8509ba6cd938bc36f3acb5e3112b7ca2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L4918]

Then it calls
{code:title=BlockManager#getPendingDeletionBlocksCount}
  /** Used by metrics */
  public long getPendingDeletionBlocksCount() {
return invalidateBlocks.numBlocks();
  }
{code}

If we look into how *invalidateBlocks* is changed, we can see in 
*BlockManager#addToInvalidates*
{code:title=BlockManager#addToInvalidates}
  /**
   * Adds block to list of blocks which will be invalidated on all its
   * datanodes.
   */
  private void addToInvalidates(BlockInfo storedBlock) {
if (!isPopulatingReplQueues()) {
  return;
}
StringBuilder datanodes = blockLog.isDebugEnabled()
? new StringBuilder() : null;
for (DatanodeStorageInfo storage : blocksMap.getStorages(storedBlock)) {
  if (storage.getState() != State.NORMAL) {
continue;
  }
  final DatanodeDescriptor node = storage.getDatanodeDescriptor();
  final Block b = getBlockOnStorage(storedBlock, storage);
  if (b != null) {
invalidateBlocks.add(b, node, false);
if (datanodes != null) {
  datanodes.append(node).append(" ");
}
  }
}
if (datanodes != null && datanodes.length() != 0) {
  blockLog.debug("BLOCK* addToInvalidates: {} {}", storedBlock, datanodes);
}
  }
{code}

So we can see that *invalidateBlocks* is indeed incremented for each block on 
each datanode. CMIIW

> More Clarity on Namenode UI Around Blocks and Replicas
> --
>
> Key: HDFS-14491
> URL: https://issues.apache.org/jira/browse/HDFS-14491
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Alan Jackoway
>Assignee: Siyao Meng
>Priority: Minor
> Attachments: HDFS-14491.001.patch
>
>
> I recently deleted more than 1/3 of the files in my HDFS installation. During 
> the process of the delete, I noticed that the NameNode UI near the top has a 
> line like this:
> {quote}44,031,342 files and directories, 38,988,775 blocks = 83,020,117 total 
> filesystem object(s).
> {quote}
> Then lower down had a line like this:
> {quote}Number of Blocks Pending Deletion 4000
> {quote}
> That made it appear that I was deleting more blocks than exist in the 
> cluster. When that number was below the total number of blocks, I briefly 
> believed I had deleted the entire cluster. In reality, the second number 
> includes replicas, while the first does not.
> The UI should be clarified to indicate where "Blocks" includes replicas and 
> where it doesn't. This may also have an impact on the under-replicated count.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14174) Enhance Audit for chown ( internally setOwner)

2019-08-10 Thread hemanthboyina (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904411#comment-16904411
 ] 

hemanthboyina commented on HDFS-14174:
--

now we are auditing the new owner information 
need to capture existing owner and new owner ? [~xkrogen] [~jojochuang] 

> Enhance Audit for chown ( internally setOwner)   
> -
>
> Key: HDFS-14174
> URL: https://issues.apache.org/jira/browse/HDFS-14174
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.6.0
>Reporter: Sailesh Patel
>Assignee: hemanthboyina
>Priority: Minor
>
> When a hdfs dfs -chown  command is executed, the audit log  does not capture 
> the  existing owner and the new owner.    
> Need to capture the old and new owner to allow auditing to be effective
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1950) S3 MPU part-list call fails if there are no parts

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton updated HDDS-1950:
---
Summary: S3 MPU part-list call fails if there are no parts  (was: S3 MPU 
part list can't be called if there are no parts)

> S3 MPU part-list call fails if there are no parts
> -
>
> Key: HDDS-1950
> URL: https://issues.apache.org/jira/browse/HDDS-1950
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: S3
>Reporter: Elek, Marton
>Priority: Major
>
> If an S3 multipart upload is created but no part is upload the part list 
> can't be called because it throws HTTP 500:
> Create an MPU:
> {code}
> aws s3api --endpoint http://localhost: create-multipart-upload 
> --bucket=docker --key=testkeu 
> {
> "Bucket": "docker",
> "Key": "testkeu",
> "UploadId": "85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234"
> }
> {code}
> List the parts:
> {code}
> aws s3api --endpoint http://localhost: list-parts  --bucket=docker 
> --key=testkeu 
> --upload-id=85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234
> {code}
> It throws an exception on the server side, because in the 
> KeyManagerImpl.listParts the  ReplicationType is retrieved from the first 
> part:
> {code}
> HddsProtos.ReplicationType replicationType =
> partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
> {code}
> Which is not yet available in this use case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDDS-1950) S3 MPU part list can't be called if there are no parts

2019-08-10 Thread Elek, Marton (JIRA)
Elek, Marton created HDDS-1950:
--

 Summary: S3 MPU part list can't be called if there are no parts
 Key: HDDS-1950
 URL: https://issues.apache.org/jira/browse/HDDS-1950
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: S3
Reporter: Elek, Marton


If an S3 multipart upload is created but no part is upload the part list can't 
be called because it throws HTTP 500:

Create an MPU:

{code}
aws s3api --endpoint http://localhost: create-multipart-upload 
--bucket=docker --key=testkeu 
{
"Bucket": "docker",
"Key": "testkeu",
"UploadId": "85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234"
}
{code}

List the parts:

{code}
aws s3api --endpoint http://localhost: list-parts  --bucket=docker 
--key=testkeu 
--upload-id=85343e71-4c16-4a75-bb55-01f56a9339b2-102592678478217234
{code}

It throws an exception on the server side, because in the 
KeyManagerImpl.listParts the  ReplicationType is retrieved from the first part:

{code}
HddsProtos.ReplicationType replicationType =
partKeyInfoMap.firstEntry().getValue().getPartKeyInfo().getType();
{code}

Which is not yet available in this use case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster

2019-08-10 Thread hemanthboyina (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904410#comment-16904410
 ] 

hemanthboyina commented on HDFS-13123:
--

[~elgoiri] [~jojochuang] attached the intial patch , can you review it  

> RBF: Add a balancer tool to move data across subcluster 
> 
>
> Key: HDFS-13123
> URL: https://issues.apache.org/jira/browse/HDFS-13123
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Wei Yan
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS Router-Based Federation Rebalancer.pdf, 
> HDFS-13123.patch
>
>
> Follow the discussion in HDFS-12615. This Jira is to track effort for 
> building a rebalancer tool, used by router-based federation to move data 
> among subclusters.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1946) CertificateClient should not persist keys/certs to ozone.metadata.dir

2019-08-10 Thread Nanda kumar (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDDS-1946:
--
Target Version/s: 0.4.1  (was: 0.4.0)

> CertificateClient should not persist keys/certs to ozone.metadata.dir
> -
>
> Key: HDDS-1946
> URL: https://issues.apache.org/jira/browse/HDDS-1946
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Xiaoyu Yao
>Assignee: Vivek Ratnavel Subramanian
>Priority: Major
>
> For example, when OM and SCM are deployed on the same host with 
> ozone.metadata.dir defined. SCM can start successfully but OM can not because 
> the key/cert from OM will collide with SCM.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDDS-1949) Missing or error-prone test cleanup

2019-08-10 Thread Doroszlai, Attila (JIRA)
Doroszlai, Attila created HDDS-1949:
---

 Summary: Missing or error-prone test cleanup
 Key: HDDS-1949
 URL: https://issues.apache.org/jira/browse/HDDS-1949
 Project: Hadoop Distributed Data Store
  Issue Type: Bug
  Components: test
Reporter: Doroszlai, Attila
Assignee: Doroszlai, Attila


Some integration tests do not clean up after themselves.  Some only clean up if 
the test is successful.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work started] (HDDS-1949) Missing or error-prone test cleanup

2019-08-10 Thread Doroszlai, Attila (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDDS-1949 started by Doroszlai, Attila.
---
> Missing or error-prone test cleanup
> ---
>
> Key: HDDS-1949
> URL: https://issues.apache.org/jira/browse/HDDS-1949
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: test
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Major
>
> Some integration tests do not clean up after themselves.  Some only clean up 
> if the test is successful.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work started] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-14665 started by Siyao Meng.
-
> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
>  didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work started] (HDFS-14663) HttpFS: LISTSTATUS_BATCH does not return batches

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-14663 started by Siyao Meng.
-
> HttpFS: LISTSTATUS_BATCH does not return batches
> 
>
> Key: HDFS-14663
> URL: https://issues.apache.org/jira/browse/HDFS-14663
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: httpfs
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Siyao Meng
>Priority: Major
>
> The webhdfs protocol supports a LISTSTATUS_BATCH operation where it can 
> retrieve the file listing for a large directory in chunks.
> When using the webhdfs service embedded in the namenode, this works as 
> expected, but when using HTTPFS, any call to LISTSTATUS_BATCH simply returns 
> the entire listing rather than batches, working effectively like LISTSTATUS 
> instead.
> This seems to be because HTTPFS falls back to using the method 
> org.apache.hadoop.fs.FileSystem#listStatusBatch, which is intended to be 
> overridden, but the implementation used in HTTPFS has not done that, leading 
> to this limitation.
> This feature (LISTSTATUS_BATCH) was added to HTTPFS by HDFS-10823, but based 
> on my testing it does not work as intended. I suspect it is because the 
> listStatusBatch operation was added to the WebHdfsFileSystem and 
> HttpFSFileSystem as part of the above Jira, but behind the scenes HTTPFS 
> seems to use DistributeFileSystem and hence it falls back to the default 
> implementation "org.apache.hadoop.fs.FileSystem#listStatusBatch" which 
> returns all entries in a single batch.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work started] (HDDS-1891) Ozone fs shell command should work with default port when port number is not specified

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDDS-1891 started by Siyao Meng.

> Ozone fs shell command should work with default port when port number is not 
> specified
> --
>
> Key: HDDS-1891
> URL: https://issues.apache.org/jira/browse/HDDS-1891
> Project: Hadoop Distributed Data Store
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> {code:bash|title=Without port number -> Error}
> $ ozone fs -ls o3fs://bucket.volume.localhost/
> -ls: Ozone file system url should be either one of the two forms: 
> o3fs://bucket.volume/key  OR o3fs://bucket.volume.om-host.example.com:5678/key
> ...
> {code}
> {code:bash|title=With port number -> Success}
> $ ozone fs -ls o3fs://bucket.volume.localhost:9862/
> Found 1 items
> -rw-rw-rw-   1 hadoop hadoop   1485 2019-08-01 21:14 
> o3fs://bucket.volume.localhost:9862/README.txt
> {code}
> We expect the first command to attempt port 9862 by default.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work started] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-14718 started by Siyao Meng.
-
> HttpFS: Sort response by key names as WebHDFS does
> --
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> *Example*
> See description of HDFS-14665 for an example of LISTSTATUS.
> *Analysis*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
> It is not only limited to LISTSTATUS, but ALL other request's JSON 
> serialization.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDDS-1054) List Multipart uploads in a bucket

2019-08-10 Thread Elek, Marton (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elek, Marton reassigned HDDS-1054:
--

Assignee: Elek, Marton  (was: Bharat Viswanadham)

> List Multipart uploads in a bucket
> --
>
> Key: HDDS-1054
> URL: https://issues.apache.org/jira/browse/HDDS-1054
> Project: Hadoop Distributed Data Store
>  Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Elek, Marton
>Priority: Blocker
>
> This Jira is to implement in ozone to list of in-progress multipart uploads 
> in a bucket.
> [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadListMPUpload.html]
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1910?focusedWorklogId=292505=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292505
 ]

ASF GitHub Bot logged work on HDDS-1910:


Author: ASF GitHub Bot
Created on: 10/Aug/19 10:06
Start Date: 10/Aug/19 10:06
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1268: HDDS-1910. 
Cannot build hadoop-hdds-config from scratch in IDEA
URL: https://github.com/apache/hadoop/pull/1268#issuecomment-520136555
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 43 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 0 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 1 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 25 | Maven dependency ordering for branch |
   | +1 | mvninstall | 603 | trunk passed |
   | +1 | compile | 376 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 1789 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 160 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 25 | Maven dependency ordering for patch |
   | +1 | mvninstall | 553 | the patch passed |
   | +1 | compile | 375 | the patch passed |
   | +1 | javac | 375 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | xml | 3 | The patch has no ill-formed XML file. |
   | +1 | shadedclient | 675 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 164 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 281 | hadoop-hdds in the patch passed. |
   | -1 | unit | 2240 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 59 | The patch does not generate ASF License warnings. |
   | | | 6581 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures |
   |   | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory |
   |   | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
   |   | hadoop.ozone.client.rpc.TestCommitWatcher |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1268 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient xml |
   | uname | Linux 01d2fedea183 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / fba222a |
   | Default Java | 1.8.0_212 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/testReport/ |
   | Max. process+thread count | 4867 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdds/config hadoop-hdds/common U: hadoop-hdds |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1268/1/console |
   | versions | git=2.7.4 maven=3.3.9 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292505)
Time Spent: 0.5h  (was: 20m)

> Cannot build hadoop-hdds-config from scratch in IDEA
> 
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or 
> after {{mvn clean}}) in IDEA fails with the following error:
> 

[jira] [Commented] (HDFS-14595) HDFS-11848 breaks API compatibility

2019-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904391#comment-16904391
 ] 

Hadoop QA commented on HDFS-14595:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m  
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 50s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  3m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  3m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 12s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m  
3s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 40s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
49s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
|   | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
|   | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
|   | hadoop.hdfs.tools.TestDFSZKFailoverController |
|   | hadoop.hdfs.server.datanode.TestLargeBlockReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14595 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12977214/HDFS-14595.005.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 6a6f720f66ea 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Work logged] (HDDS-1948) S3 MPU can't be created with octet-stream content-type

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1948?focusedWorklogId=292498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292498
 ]

ASF GitHub Bot logged work on HDDS-1948:


Author: ASF GitHub Bot
Created on: 10/Aug/19 09:15
Start Date: 10/Aug/19 09:15
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1266: HDDS-1948. S3 
MPU can't be created with octet-stream content-type 
URL: https://github.com/apache/hadoop/pull/1266#issuecomment-520133277
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 42 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 0 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 5 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | +1 | mvninstall | 593 | trunk passed |
   | +1 | compile | 375 | trunk passed |
   | +1 | checkstyle | 76 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 868 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 170 | trunk passed |
   | 0 | spotbugs | 423 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 620 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | +1 | mvninstall | 551 | the patch passed |
   | +1 | compile | 376 | the patch passed |
   | +1 | javac | 376 | the patch passed |
   | +1 | checkstyle | 84 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 674 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 168 | the patch passed |
   | +1 | findbugs | 632 | the patch passed |
   ||| _ Other Tests _ |
   | -1 | unit | 296 | hadoop-hdds in the patch failed. |
   | -1 | unit | 1766 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 56 | The patch does not generate ASF License warnings. |
   | | | 7525 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdds.scm.safemode.TestSCMSafeModeManager |
   |   | hadoop.ozone.client.rpc.TestOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures |
   |   | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
   |   | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider |
   |   | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1266 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux be2ffc260e5d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / fba222a |
   | Default Java | 1.8.0_212 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/artifact/out/patch-unit-hadoop-hdds.txt
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/testReport/ |
   | Max. process+thread count | 4900 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1266/1/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292498)
Time Spent: 20m  (was: 10m)

> S3 MPU can't be created with octet-stream content-type 
> ---
>
> Key: HDDS-1948
> URL: https://issues.apache.org/jira/browse/HDDS-1948
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: S3
>Reporter: Elek, Marton
>

[jira] [Commented] (HDFS-14595) HDFS-11848 breaks API compatibility

2019-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904384#comment-16904384
 ] 

Hadoop QA commented on HDFS-14595:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 27s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  2m 56s{color} 
| {color:red} hadoop-hdfs-project generated 2 new + 552 unchanged - 0 fixed = 
554 total (was 552) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 47s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 49s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 4s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestBlocksScheduledCounter |
|   | hadoop.hdfs.server.datanode.TestLargeBlockReport |
|   | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14595 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12977213/HDFS-14595.004.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 4818ee8eb1a4 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fba222a |
| maven | 

[jira] [Updated] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA

2019-08-10 Thread Doroszlai, Attila (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doroszlai, Attila updated HDDS-1910:

Status: Patch Available  (was: In Progress)

> Cannot build hadoop-hdds-config from scratch in IDEA
> 
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or 
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while 
> constructing Processor object: javax.annotation.processing.Processor: 
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14717) Junit not found in hadoop-dynamometer-infra

2019-08-10 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904372#comment-16904372
 ] 

Hadoop QA commented on HDFS-14717:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 50s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 14s{color} | {color:orange} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra: The patch generated 1 
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 11s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 25s{color} 
| {color:red} hadoop-dynamometer-infra in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 24s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.dynamometer.TestDynamometerInfra |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14717 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12977216/HDFS-14717.001.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 975bc2ae74a0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fba222a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/27467/artifact/out/diff-checkstyle-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
 |
| unit | 

[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Description: 
*Example*

See description of HDFS-14665 for an example of LISTSTATUS.


*Analysis*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional?

- I looked into the Git history. It seems it's simply because WebHDFS uses 
TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
It is not only limited to LISTSTATUS, but ALL other request's JSON 
serialization.

Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
in HttpFS serialization in FSOperations class?

  was:
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional?

- I looked into the Git history. It seems it's simply because WebHDFS uses 
TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning.

Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
in HttpFS serialization in FSOperations class?


> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> *Example*
> See description of HDFS-14665 for an example of LISTSTATUS.
> *Analysis*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
> It is not only limited to LISTSTATUS, but ALL other request's JSON 
> serialization.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sorr response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Summary: HttpFS: Sorr response by key names as WebHDFS does  (was: HttpFS: 
Sor response by key names as WebHDFS does)

> HttpFS: Sorr response by key names as WebHDFS does
> --
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> *Example*
> See description of HDFS-14665 for an example of LISTSTATUS.
> *Analysis*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
> It is not only limited to LISTSTATUS, but ALL other request's JSON 
> serialization.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sor response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Summary: HttpFS: Sor response by key names as WebHDFS does  (was: HttpFS: 
Sort LISTSTATUS response by key names as WebHDFS does)

> HttpFS: Sor response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> *Example*
> See description of HDFS-14665 for an example of LISTSTATUS.
> *Analysis*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
> It is not only limited to LISTSTATUS, but ALL other request's JSON 
> serialization.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Summary: HttpFS: Sort response by key names as WebHDFS does  (was: HttpFS: 
Sorr response by key names as WebHDFS does)

> HttpFS: Sort response by key names as WebHDFS does
> --
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> *Example*
> See description of HDFS-14665 for an example of LISTSTATUS.
> *Analysis*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. 
> It is not only limited to LISTSTATUS, but ALL other request's JSON 
> serialization.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Description: 
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional?

- I looked into the Git history. It seems it's simply because WebHDFS uses 
TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning.

Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
in HttpFS serialization in FSOperations class?

  was:
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional?

I looked into the Git history. It seems it's simply because WebHDFS uses 
TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning.


> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Example: see description of HDFS-14665.
> *Root cause*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> - I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning.
> Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap 
> in HttpFS serialization in FSOperations class?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Description: 
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional?

I looked into the Git history. It seems it's simply because WebHDFS uses 
TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning.

  was:
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional? e.g. for some performance concerns on 
HttpFS?


> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Example: see description of HDFS-14665.
> *Root cause*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional?
> I looked into the Git history. It seems it's simply because WebHDFS uses 
> TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Description: 
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional? e.g. for some performance concerns on 
HttpFS?

  was:
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional? e.g. for some performance concern on 
HttpFS?


> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Example: see description of HDFS-14665.
> *Root cause*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional? e.g. for some performance concerns 
> on HttpFS?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Description: 
Example: see description of HDFS-14665.

*Root cause*
WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.

*Questions*
Why the difference? Is this intentional? e.g. for some performance concern on 
HttpFS?

  was:
Example: see description of HDFS-14665.

Root cause:
WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.

Questions:
Why the difference? Is this intentional? e.g. for some performance concern on 
HttpFS?


> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Example: see description of HDFS-14665.
> *Root cause*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional? e.g. for some performance concern on 
> HttpFS?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Description: 
Example: see description of HDFS-14665.


*Root cause*

WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.


*Questions*

Why the difference? Is this intentional? e.g. for some performance concern on 
HttpFS?

  was:
Example: see description of HDFS-14665.

*Root cause*
WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.

*Questions*
Why the difference? Is this intentional? e.g. for some performance concern on 
HttpFS?


> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Example: see description of HDFS-14665.
> *Root cause*
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> *Questions*
> Why the difference? Is this intentional? e.g. for some performance concern on 
> HttpFS?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names as WebHDFS does

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14718:
--
Summary: HttpFS: Sort LISTSTATUS response by key names as WebHDFS does  
(was: HttpFS: Sort LISTSTATUS response by key names)

> HttpFS: Sort LISTSTATUS response by key names as WebHDFS does
> -
>
> Key: HDFS-14718
> URL: https://issues.apache.org/jira/browse/HDFS-14718
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> Example: see description of HDFS-14665.
> Root cause:
> WebHDFS is [using a 
> TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
>  to serialize HdfsFileStatus, while HttpFS [uses a 
> LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
>  to serialize FileStatus.
> Questions:
> Why the difference? Is this intentional? e.g. for some performance concern on 
> HttpFS?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Created] (HDFS-14718) HttpFS: Sort LISTSTATUS response by key names

2019-08-10 Thread Siyao Meng (JIRA)
Siyao Meng created HDFS-14718:
-

 Summary: HttpFS: Sort LISTSTATUS response by key names
 Key: HDFS-14718
 URL: https://issues.apache.org/jira/browse/HDFS-14718
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: httpfs
Reporter: Siyao Meng
Assignee: Siyao Meng


Example: see description of HDFS-14665.

Root cause:
WebHDFS is [using a 
TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120]
 to serialize HdfsFileStatus, while HttpFS [uses a 
LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107]
 to serialize FileStatus.

Questions:
Why the difference? Is this intentional? e.g. for some performance concern on 
HttpFS?



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1910?focusedWorklogId=292485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292485
 ]

ASF GitHub Bot logged work on HDDS-1910:


Author: ASF GitHub Bot
Created on: 10/Aug/19 08:16
Start Date: 10/Aug/19 08:16
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on pull request #1268: HDDS-1910. 
Cannot build hadoop-hdds-config from scratch in IDEA
URL: https://github.com/apache/hadoop/pull/1268
 
 
   ## What changes were proposed in this pull request?
   
   `ConfigFileGenerator` is both defined (`.java` source) and configured to be 
used (by reference in `META-INF`) in `hadoop-hdds-config` module's `main` 
source.  Maven build is configured to work around this by explicit `-proc:none` 
compiler arg, applied only to `main` compilation, but not to `test` 
compilation.  Since the annotation processor is only used in tests in this 
module, this works fine.
   
   It seems IDEA does not respect this compiler configuration (and cannot even 
be configured differently for `main` and `test` compilation).
   
   This change configures the processor for use in `config/test` and 
`common/main`.  Thus dependent modules still inherit it for main source 
processing, while the `config` module can be properly compiled without the hack.
   
   https://issues.apache.org/jira/browse/HDDS-1910
   
   ## How was this patch tested?
   
   1. Verified that `hadoop-hdds/config` can be compiled from scratch in IDEA.  
To confirm, try to run eg. `TestConfigFileAppender` unit test.
   2. Verified that full Maven build still generates the same Ozone config 
based on existing annotations:
   ```
1762 hadoop-hdds/common/target/test-classes/ozone-default-generated.xml
2030 hadoop-hdds/config/target/test-classes/ozone-default-generated.xml
2294 hadoop-hdds/server-scm/target/classes/ozone-default-generated.xml
 918 hadoop-ozone/common/target/classes/ozone-default-generated.xml
2298 hadoop-ozone/csi/target/classes/ozone-default-generated.xml
 504 
hadoop-ozone/ozonefs-lib-current/target/classes/ozone-default-generated.xml
 504 
hadoop-ozone/ozonefs-lib-legacy/target/classes/libs/ozone-default-generated.xml
 504 
hadoop-ozone/ozonefs-lib-legacy/target/classes/ozone-default-generated.xml
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292485)
Time Spent: 10m
Remaining Estimate: 0h

> Cannot build hadoop-hdds-config from scratch in IDEA
> 
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or 
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while 
> constructing Processor object: javax.annotation.processing.Processor: 
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1910?focusedWorklogId=292486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292486
 ]

ASF GitHub Bot logged work on HDDS-1910:


Author: ASF GitHub Bot
Created on: 10/Aug/19 08:16
Start Date: 10/Aug/19 08:16
Worklog Time Spent: 10m 
  Work Description: adoroszlai commented on issue #1268: HDDS-1910. Cannot 
build hadoop-hdds-config from scratch in IDEA
URL: https://github.com/apache/hadoop/pull/1268#issuecomment-520129550
 
 
   /label ozone
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 292486)
Time Spent: 20m  (was: 10m)

> Cannot build hadoop-hdds-config from scratch in IDEA
> 
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or 
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while 
> constructing Processor object: javax.annotation.processing.Processor: 
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDDS-1910) Cannot build hadoop-hdds-config from scratch in IDEA

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDDS-1910:
-
Labels: pull-request-available  (was: )

> Cannot build hadoop-hdds-config from scratch in IDEA
> 
>
> Key: HDDS-1910
> URL: https://issues.apache.org/jira/browse/HDDS-1910
> Project: Hadoop Distributed Data Store
>  Issue Type: Bug
>  Components: build
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
>  Labels: pull-request-available
>
> Building {{hadoop-hdds-config}} from scratch (eg. right after checkout or 
> after {{mvn clean}}) in IDEA fails with the following error:
> {code}
> Error:java: Bad service configuration file, or exception thrown while 
> constructing Processor object: javax.annotation.processing.Processor: 
> Provider org.apache.hadoop.hdds.conf.ConfigFileGenerator not found
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class

2019-08-10 Thread ASF GitHub Bot (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDDS-1927?focusedWorklogId=292484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292484
 ]

ASF GitHub Bot logged work on HDDS-1927:


Author: ASF GitHub Bot
Created on: 10/Aug/19 08:12
Start Date: 10/Aug/19 08:12
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on issue #1263: HDDS-1927. 
Consolidate add/remove Acl into OzoneAclUtil class. Contri…
URL: https://github.com/apache/hadoop/pull/1263#issuecomment-520129310
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 46 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 1 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 5 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 29 | Maven dependency ordering for branch |
   | +1 | mvninstall | 577 | trunk passed |
   | +1 | compile | 365 | trunk passed |
   | +1 | checkstyle | 76 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | +1 | shadedclient | 864 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 163 | trunk passed |
   | 0 | spotbugs | 422 | Used deprecated FindBugs config; considering 
switching to SpotBugs. |
   | +1 | findbugs | 611 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 25 | Maven dependency ordering for patch |
   | +1 | mvninstall | 591 | the patch passed |
   | +1 | compile | 391 | the patch passed |
   | +1 | javac | 391 | the patch passed |
   | -0 | checkstyle | 34 | hadoop-ozone: The patch generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) |
   | +1 | mvnsite | 0 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 686 | patch has no errors when building and testing 
our client artifacts. |
   | -1 | javadoc | 92 | hadoop-ozone generated 2 new + 13 unchanged - 0 fixed 
= 15 total (was 13) |
   | -1 | findbugs | 427 | hadoop-ozone generated 1 new + 0 unchanged - 0 fixed 
= 1 total (was 0) |
   ||| _ Other Tests _ |
   | +1 | unit | 306 | hadoop-hdds in the patch passed. |
   | -1 | unit | 52 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 36 | The patch does not generate ASF License warnings. |
   | | | 5870 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | FindBugs | module:hadoop-ozone |
   |  |  Possible null pointer dereference of null in 
org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo)
  Dereferenced at OmKeyInfo.java:null in 
org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getFromProtobuf(OzoneManagerProtocolProtos$KeyInfo)
  Dereferenced at OmKeyInfo.java:[line 394] |
   | Failed junit tests | hadoop.ozone.om.helpers.TestOmKeyInfo |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=19.03.1 Server=19.03.1 base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1263 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle |
   | uname | Linux ea8778af37e0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / fba222a |
   | Default Java | 1.8.0_212 |
   | checkstyle | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/diff-checkstyle-hadoop-ozone.txt
 |
   | javadoc | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
 |
   | findbugs | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/new-findbugs-hadoop-ozone.html
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/testReport/ |
   | Max. process+thread count | 487 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/common hadoop-ozone/client 
hadoop-ozone/ozone-manager hadoop-ozone/objectstore-service 
hadoop-ozone/integration-test U: hadoop-ozone |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1263/4/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   
 

This is an automated message from the Apache Git Service.
To respond to 

[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Component/s: httpfs

> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
>  didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Target Version/s: 3.3.0

> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: httpfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for directories)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
>  didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Description: 
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (should only be none 0 for directories)
"fileId"
"storagePolicy"
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Root cause:

[toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
 didn't serialize the HDFS-specific keys from FileStatus.

Also may file another Jira to align the order of the keys in the responses.

  was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Root cause:

[toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
 didn't serialize the HDFS-specific keys from FileStatus.

Also may file another Jira to align the order of the keys in the responses.


> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (should only be none 0 for 

[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS response is missing HDFS-specific fields

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Summary: HttpFS: LISTSTATUS response is missing HDFS-specific fields  (was: 
HttpFS: LISTSTATUS result is missing HDFS-specific fields)

> HttpFS: LISTSTATUS response is missing HDFS-specific fields
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
>  didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing HDFS-specific fields

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Summary: HttpFS: LISTSTATUS result is missing HDFS-specific fields  (was: 
HttpFS: LISTSTATUS result is missing some fields for each entry)

> HttpFS: LISTSTATUS result is missing HDFS-specific fields
> -
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Root cause:
> [toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
>  didn't serialize the HDFS-specific keys from FileStatus.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output

2019-08-10 Thread Adam Antal (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904364#comment-16904364
 ] 

Adam Antal commented on HDFS-14195:
---

Thanks for the patch [~suxingfate], and [~jojochuang] for the commit!

> OIV: print out storage policy id in oiv Delimited output
> 
>
> Key: HDFS-14195
> URL: https://issues.apache.org/jira/browse/HDFS-14195
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: tools
>Reporter: Wang, Xinglong
>Assignee: Wang, Xinglong
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, 
> HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, 
> HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, 
> HDFS-14195.009.patch, HDFS-14195.010.patch
>
>
> There is lacking of a method to get all folders and files with sort of 
> specified storage policy via command line, like ALL_SSD type.
> By adding storage policy id to oiv output, it will help with oiv 
> post-analysis to have a overview of all folders/files with specified storage 
> policy and to apply internal regulation based on this information.
>  
> Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print 
> out xattr which including storage policy already.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14717) Junit not found in hadoop-dynamometer-infra

2019-08-10 Thread kevin su (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevin su updated HDFS-14717:

Status: Patch Available  (was: Open)

> Junit not found in hadoop-dynamometer-infra
> ---
>
> Key: HDFS-14717
> URL: https://issues.apache.org/jira/browse/HDFS-14717
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: kevin su
>Assignee: kevin su
>Priority: Major
> Attachments: HDFS-14717.001.patch
>
>
> {code:java}
> $ hadoop jar hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar 
> org.apache.hadoop.tools.dynamometer.Client
> {code}
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
>  at org.apache.hadoop.tools.dynamometer.Client.main(Client.java:256)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 7 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-14717) Junit not found in hadoop-dynamometer-infra

2019-08-10 Thread kevin su (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16904361#comment-16904361
 ] 

kevin su commented on HDFS-14717:
-

[~jojochuang] [~smeng] Thanks for your reply 

It's due to  _*ClassUtil.findContainingJar(Assert.class)*_ can't find Junit, 
because Junit is in other directory.

I also found that _*Junit*_ Already package into  
_*hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar*_

So I remove _*ClassUtil.findContainingJar(Assert.class),*_ it works

> Junit not found in hadoop-dynamometer-infra
> ---
>
> Key: HDFS-14717
> URL: https://issues.apache.org/jira/browse/HDFS-14717
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: kevin su
>Assignee: kevin su
>Priority: Major
> Attachments: HDFS-14717.001.patch
>
>
> {code:java}
> $ hadoop jar hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar 
> org.apache.hadoop.tools.dynamometer.Client
> {code}
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
>  at org.apache.hadoop.tools.dynamometer.Client.main(Client.java:256)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 7 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing some fields for each entry

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Description: 
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Root cause:

[toJsonInner|https://github.com/apache/hadoop/blob/17e8cf501b384af93726e4f2e6f5e28c6e3a8f65/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L120]
 didn't serialize the HDFS-specific keys from FileStatus.

Also may file another Jira to align the order of the keys in the responses.

  was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Also may file another Jira to align the order of the keys in the responses.


> HttpFS: LISTSTATUS result is missing some fields for each entry
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Root cause:
> 

[jira] [Updated] (HDFS-14717) Junit not found in hadoop-dynamometer-infra

2019-08-10 Thread kevin su (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevin su updated HDFS-14717:

Attachment: HDFS-14717.001.patch

> Junit not found in hadoop-dynamometer-infra
> ---
>
> Key: HDFS-14717
> URL: https://issues.apache.org/jira/browse/HDFS-14717
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: kevin su
>Assignee: kevin su
>Priority: Major
> Attachments: HDFS-14717.001.patch
>
>
> {code:java}
> $ hadoop jar hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar 
> org.apache.hadoop.tools.dynamometer.Client
> {code}
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
>  at org.apache.hadoop.tools.dynamometer.Client.main(Client.java:256)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 7 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Assigned] (HDFS-14717) Junit not found in hadoop-dynamometer-infra

2019-08-10 Thread kevin su (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevin su reassigned HDFS-14717:
---

Assignee: kevin su

> Junit not found in hadoop-dynamometer-infra
> ---
>
> Key: HDFS-14717
> URL: https://issues.apache.org/jira/browse/HDFS-14717
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: kevin su
>Assignee: kevin su
>Priority: Major
>
> {code:java}
> $ hadoop jar hadoop-dynamometer-infra-3.3.0-SNAPSHOT.jar 
> org.apache.hadoop.tools.dynamometer.Client
> {code}
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
>  at org.apache.hadoop.tools.dynamometer.Client.main(Client.java:256)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
>  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>  ... 7 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing some fields for each entry

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Description: 
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
childrenNum (directories only?)
fileId
storagePolicy.
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Also may file another Jira to align the order of the keys in the responses.

  was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing childrenNum 
(directories only I guess), fileId and storagePolicy fields. The same applies 
to LISTSTATUS_BATCH, which might be using the same underlying calls to compose 
the response.

Also may file another Jira to align the order of the keys in the responses.


> HttpFS: LISTSTATUS result is missing some fields for each entry
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> childrenNum (directories only?)
> fileId
> storagePolicy.
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Updated] (HDFS-14665) HttpFS: LISTSTATUS result is missing some fields for each entry

2019-08-10 Thread Siyao Meng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-14665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyao Meng updated HDFS-14665:
--
Description: 
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
"childrenNum" (directories only?)
"fileId"
"storagePolicy"
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Also may file another Jira to align the order of the keys in the responses.

  was:
WebHDFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"accessTime": 0,
"blockSize": 0,
"childrenNum": 0,
"fileId": 16395,
"group": "hadoop",
"length": 0,
"modificationTime": 1563893395614,
"owner": "mapred",
"pathSuffix": "logs",
"permission": "1777",
"replication": 0,
"storagePolicy": 0,
"type": "DIRECTORY"
  }
]
  }
}
{code}
HttpFS:
{code:java}
GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
{code}
{code}
{
  "FileStatuses": {
"FileStatus": [
...
  {
"pathSuffix": "logs",
"type": "DIRECTORY",
"length": 0,
"owner": "mapred",
"group": "hadoop",
"permission": "1777",
"accessTime": 0,
"modificationTime": 1563893395614,
"blockSize": 0,
"replication": 0
  }
]
  }
}
{code}
You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
{code}
childrenNum (directories only?)
fileId
storagePolicy.
{code}

The same applies to LISTSTATUS_BATCH, which might be using the same underlying 
calls to compose the response.

Also may file another Jira to align the order of the keys in the responses.


> HttpFS: LISTSTATUS result is missing some fields for each entry
> ---
>
> Key: HDFS-14665
> URL: https://issues.apache.org/jira/browse/HDFS-14665
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
>
> WebHDFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "accessTime": 0,
> "blockSize": 0,
> "childrenNum": 0,
> "fileId": 16395,
> "group": "hadoop",
> "length": 0,
> "modificationTime": 1563893395614,
> "owner": "mapred",
> "pathSuffix": "logs",
> "permission": "1777",
> "replication": 0,
> "storagePolicy": 0,
> "type": "DIRECTORY"
>   }
> ]
>   }
> }
> {code}
> HttpFS:
> {code:java}
> GET /webhdfs/v1/tmp/?op=LISTSTATUS=hdfs HTTP/1.1
> {code}
> {code}
> {
>   "FileStatuses": {
> "FileStatus": [
> ...
>   {
> "pathSuffix": "logs",
> "type": "DIRECTORY",
> "length": 0,
> "owner": "mapred",
> "group": "hadoop",
> "permission": "1777",
> "accessTime": 0,
> "modificationTime": 1563893395614,
> "blockSize": 0,
> "replication": 0
>   }
> ]
>   }
> }
> {code}
> You can see the same LISTSTATUS request to HttpFS is missing 3 fields:
> {code}
> "childrenNum" (directories only?)
> "fileId"
> "storagePolicy"
> {code}
> The same applies to LISTSTATUS_BATCH, which might be using the same 
> underlying calls to compose the response.
> Also may file another Jira to align the order of the keys in the responses.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



  1   2   >