[jira] [Updated] (HDFS-11480) Ozone: TestEndpoint task failure

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11480: -- Status: Patch Available (was: Open) > Ozone: TestEndpoint task failure >

[jira] [Updated] (HDFS-11480) Ozone: TestEndpoint task failure

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11480: -- Attachment: HDFS-11480-HDFS-7240.002.patch > Ozone: TestEndpoint task failure >

[jira] [Updated] (HDFS-11480) Ozone: TestEndpoint task failure

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11480: -- Attachment: HDFS-11480-HDFS-7240.001.patch Attach a patch that fixed the TestEndpoint failure. >

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-02-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Attachment: HDFS-11447-HDFS-7240.001.patch Update the name of the patch based [~anu]'s suggestion.

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-02-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Attachment: HDFS-11447.01.patch Attach a initial patch that 1. Define SCM node report protobuf message

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-02-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Attachment: HDFS-11447.01.patch Update patch fixing a new file missing ASF license. > Ozone: SCM:

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-02-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Attachment: (was: HDFS-11447.01.patch) > Ozone: SCM: Send node report along with heartbeat to SCM >

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-02-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Status: Patch Available (was: Open) > Ozone: SCM: Send node report along with heartbeat to SCM >

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-02-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Attachment: (was: HDFS-11447.01.patch) > Ozone: SCM: Send node report along with heartbeat to SCM >

[jira] [Commented] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891512#comment-15891512 ] Xiaoyu Yao commented on HDFS-11484: --- Run the test with the patch 30 times repeatedly using intelliJ and

[jira] [Updated] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11484: -- Attachment: HDFS-11484-HDFS-7240.001.patch > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate >

[jira] [Updated] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11484: -- Status: Open (was: Patch Available) > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate >

[jira] [Updated] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11484: -- Status: Patch Available (was: Open) > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate >

[jira] [Updated] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11484: -- Status: Patch Available (was: Open) > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate >

[jira] [Created] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-11484: - Summary: Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate Key: HDFS-11484 URL: https://issues.apache.org/jira/browse/HDFS-11484 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-11484) Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11484: -- Affects Version/s: HDFS-7240 > Ozone: Fix flaky TestNodeManager#testScmNodeReportUpdate >

[jira] [Updated] (HDFS-11474) Ozone: Tests needs to cleanup levelDB files

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11474: -- Attachment: HDFS-11474-HDFS-7240.001.patch > Ozone: Tests needs to cleanup levelDB files >

[jira] [Updated] (HDFS-11474) Ozone: Tests needs to cleanup levelDB files

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11474: -- Status: Patch Available (was: Open) > Ozone: Tests needs to cleanup levelDB files >

[jira] [Assigned] (HDFS-11473) Ozone : exclude protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-11473: - Assignee: Xiaoyu Yao > Ozone : exclude protobuf files from findbugs check >

[jira] [Updated] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11473: -- Summary: Ozone : exclude cblock protobuf files from findbugs check (was: Ozone : exclude protobuf

[jira] [Updated] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11473: -- Attachment: (was: HDFS-11474-HDFS-7240.001.patch) > Ozone : exclude cblock protobuf files from

[jira] [Updated] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11473: -- Attachment: HDFS-11474-HDFS-7240.001.patch > Ozone : exclude cblock protobuf files from findbugs check

[jira] [Comment Edited] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890618#comment-15890618 ] Xiaoyu Yao edited comment on HDFS-11447 at 3/1/17 5:28 PM: --- I plan to spend some

[jira] [Updated] (HDFS-11474) Ozone: TestContainerMapping needs to cleanup levelDB files

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11474: -- Summary: Ozone: TestContainerMapping needs to cleanup levelDB files (was: Ozone: Tests needs to

[jira] [Commented] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890562#comment-15890562 ] Xiaoyu Yao commented on HDFS-11473: --- {code} CodeWarning UCF Useless control flow in

[jira] [Commented] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890618#comment-15890618 ] Xiaoyu Yao commented on HDFS-11447: --- I plan to spend some time today to fix the ozone test failures.

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report to SCM with heartbeat

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Summary: Ozone: SCM: Send node report to SCM with heartbeat (was: Ozone: SCM: Send node report along

[jira] [Assigned] (HDFS-11474) Ozone: Tests needs to cleanup levelDB files

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-11474: - Assignee: Xiaoyu Yao > Ozone: Tests needs to cleanup levelDB files >

[jira] [Updated] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11473: -- Affects Version/s: HDFS-7240 > Ozone : exclude cblock protobuf files from findbugs check >

[jira] [Updated] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11473: -- Attachment: HDFS-11473.HDFS-7240.001.patch > Ozone : exclude cblock protobuf files from findbugs check

[jira] [Updated] (HDFS-11473) Ozone : exclude cblock protobuf files from findbugs check

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11473: -- Status: Patch Available (was: Open) > Ozone : exclude cblock protobuf files from findbugs check >

[jira] [Updated] (HDFS-11447) Ozone: SCM: Send node report along with heartbeat to SCM

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11447: -- Attachment: HDFS-11447-HDFS-7240.003.patch Thank you [~anu], good catch. Update a new patch with

[jira] [Commented] (HDFS-11474) Ozone: Tests needs to cleanup levelDB files

2017-03-01 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890539#comment-15890539 ] Xiaoyu Yao commented on HDFS-11474: --- This was recently found with

[jira] [Commented] (HDFS-11036) Ozone : reuse Xceiver connection

2017-02-27 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886189#comment-15886189 ] Xiaoyu Yao commented on HDFS-11036: --- Thanks [~vagarychen] for updating the patch. It looks good to me

[jira] [Commented] (HDFS-11463) Add metrics for container operations and export over JMX

2017-02-27 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886149#comment-15886149 ] Xiaoyu Yao commented on HDFS-11463: --- Thanks [~msingh] for working on this. The patch looks good to me,

[jira] [Commented] (HDFS-11036) Ozone : reuse Xceiver connection

2017-02-27 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886509#comment-15886509 ] Xiaoyu Yao commented on HDFS-11036: --- [~vagarychen], we should add exceptions for ozone related

[jira] [Updated] (HDFS-10475) Adding metrics for long FSD lock

2016-08-24 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10475: -- Summary: Adding metrics for long FSD lock (was: Adding metrics and warn/debug logs for long FSD lock)

[jira] [Commented] (HDFS-10475) Adding metrics and warn/debug logs for long FSD lock

2016-08-24 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435977#comment-15435977 ] Xiaoyu Yao commented on HDFS-10475: --- Thanks [~zhz] for the heads up. This log part is covered by

[jira] [Commented] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-06 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468871#comment-15468871 ] Xiaoyu Yao commented on HDFS-10757: --- Thanks [~arun197] and [~jnp] for the discussion. Adding a new

[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-06 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.00.patch Attach a low risk patch to make KMSCP stateless for discussion. >

[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-07 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Status: Patch Available (was: Reopened) > KMSClientProvider combined with KeyProviderCache can result

[jira] [Created] (HDFS-10818) KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request

2016-08-30 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10818: - Summary: KerberosAuthenticationHandler#authenticate should not rebuild SPN based on client request Key: HDFS-10818 URL: https://issues.apache.org/jira/browse/HDFS-10818

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-09-14 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15490754#comment-15490754 ] Xiaoyu Yao commented on HDFS-10690: --- Thanks [~fenghua_hu] for providing the new numbers. Considering the

[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-09 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.01.patch Thanks [~jnp] for the feedback. bq. However, in a client context, I

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523763#comment-15523763 ] Xiaoyu Yao commented on HDFS-10690: --- [~fenghua_hu], thanks for updating the patch. The patch v05 looks

[jira] [Commented] (HDFS-10897) Ozone: SCM: Add NodeManager

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524153#comment-15524153 ] Xiaoyu Yao commented on HDFS-10897: --- Thanks [~anu] for working on this and post the patch. I took a look

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-09-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530221#comment-15530221 ] Xiaoyu Yao commented on HDFS-10690: --- [~fenghua_hu]: the patch v06 looks good. I plan to commit it by EOD

[jira] [Commented] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

2016-09-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530300#comment-15530300 ] Xiaoyu Yao commented on HDFS-10906: --- Agree. Regrading adding Kerberos enabled trash related unit test

[jira] [Commented] (HDFS-10922) Adding additional unit tests for Trash

2016-09-29 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533327#comment-15533327 ] Xiaoyu Yao commented on HDFS-10922: --- [~cheersyang], Sounds good to me. Thank you! > Adding additional

[jira] [Updated] (HDFS-10922) Adding additional unit tests for Trash

2016-09-29 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10922: -- Assignee: Weiwei Yang > Adding additional unit tests for Trash > --

[jira] [Created] (HDFS-10930) Refactor Datanode IO operations into a single class

2016-09-29 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10930: - Summary: Refactor Datanode IO operations into a single class Key: HDFS-10930 URL: https://issues.apache.org/jira/browse/HDFS-10930 Project: Hadoop HDFS Issue

[jira] [Comment Edited] (HDFS-10922) Adding additional unit tests for Trash

2016-10-05 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549350#comment-15549350 ] Xiaoyu Yao edited comment on HDFS-10922 at 10/5/16 5:12 PM: bq. Now working on

[jira] [Commented] (HDFS-10922) Adding additional unit tests for Trash

2016-10-05 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549350#comment-15549350 ] Xiaoyu Yao commented on HDFS-10922: --- bq. Now working on #4 about namenode restart. One question, the

[jira] [Updated] (HDFS-10930) Refactor Datanode IO operations into a single class

2016-10-04 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Attachment: HDFS-10930.01.patch Attach a initial patch that refactors IO related operations into

[jira] [Assigned] (HDFS-4169) Add per-disk latency metrics to DataNode

2016-10-04 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-4169: Assignee: Xiaoyu Yao > Add per-disk latency metrics to DataNode >

[jira] [Updated] (HDFS-4169) Add per-disk latency metrics to DataNode

2016-10-04 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-4169: - Description: Currently, if one of the drives on the DataNode is slow, it's hard to determine what the

[jira] [Created] (HDFS-10958) Add instrumentation hooks around Datanode disk IO

2016-10-04 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10958: - Summary: Add instrumentation hooks around Datanode disk IO Key: HDFS-10958 URL: https://issues.apache.org/jira/browse/HDFS-10958 Project: Hadoop HDFS Issue Type:

[jira] [Created] (HDFS-10959) Adding per disk IO statistics in DataNode.

2016-10-04 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10959: - Summary: Adding per disk IO statistics in DataNode. Key: HDFS-10959 URL: https://issues.apache.org/jira/browse/HDFS-10959 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-10930) Refactor: Wrap Datanode IO related operations

2016-10-04 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Summary: Refactor: Wrap Datanode IO related operations (was: Refactor Datanode IO operations into a

[jira] [Updated] (HDFS-10959) Adding per disk IO statistics in DataNode.

2016-10-04 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10959: -- Description: This ticket is opened to support per disk IO statistics in DataNode based on HDFS-10930.

[jira] [Updated] (HDFS-10930) Refactor: Wrap Datanode IO related operations

2016-10-07 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Status: Patch Available (was: Open) > Refactor: Wrap Datanode IO related operations >

[jira] [Comment Edited] (HDFS-10922) Adding additional unit tests for Trash

2016-10-05 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549350#comment-15549350 ] Xiaoyu Yao edited comment on HDFS-10922 at 10/5/16 5:20 PM: bq. Now working on

[jira] [Updated] (HDFS-10922) Adding additional unit tests for Trash (II)

2016-10-05 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10922: -- Summary: Adding additional unit tests for Trash (II) (was: Adding additional unit tests for Trash) >

[jira] [Commented] (HDFS-10883) `getTrashRoot`'s behavior is not consistent in DFS after enabling EZ.

2016-09-21 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510734#comment-15510734 ] Xiaoyu Yao commented on HDFS-10883: --- [~yuanbo], thanks for reporting the issue and posting the patches.

[jira] [Commented] (HDFS-10883) `getTrashRoot`'s behavior is not consistent in DFS after enabling EZ.

2016-09-22 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514555#comment-15514555 ] Xiaoyu Yao commented on HDFS-10883: --- bq. I think the second behavior is more consistent, because of the

[jira] [Commented] (HDFS-9688) Test the effect of nested encryption zones in HDFS downgrade

2016-09-20 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507691#comment-15507691 ] Xiaoyu Yao commented on HDFS-9688: -- [~ctrezzo], move the the root of encryption zone is allowed from

[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-20 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HDFS-10757.02.patch > KMSClientProvider combined with KeyProviderCache can result in wrong

[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-20 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: (was: HADOOP-10757.02.patch) > KMSClientProvider combined with KeyProviderCache can

[jira] [Updated] (HDFS-10757) KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used

2016-09-20 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10757: -- Attachment: HADOOP-10757.02.patch Thanks [~xiaochen] for the review. Attached a new patch to address

[jira] [Updated] (HDFS-10748) TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout

2016-08-25 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10748: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved

[jira] [Commented] (HDFS-10748) TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout

2016-08-25 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437123#comment-15437123 ] Xiaoyu Yao commented on HDFS-10748: --- Thanks [~linyiqun] for working on this. The patch v02 LGTM, +1. I

[jira] [Commented] (HDFS-10475) Adding metrics for long FSD lock

2016-08-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439661#comment-15439661 ] Xiaoyu Yao commented on HDFS-10475: --- Sure, feel free to reassign this one to your colleagues. Thanks! >

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-08-23 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433269#comment-15433269 ] Xiaoyu Yao edited comment on HDFS-10690 at 8/23/16 5:40 PM: [~fenghua_hu], I

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-08-23 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433269#comment-15433269 ] Xiaoyu Yao commented on HDFS-10690: --- [~fenghua_hu], I understand the tradeoff between memory footprint

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-08-23 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433988#comment-15433988 ] Xiaoyu Yao commented on HDFS-10690: --- bq. I'll write a micro benchmark to compare LinkedMap with TreeMap

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-08-23 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434094#comment-15434094 ] Xiaoyu Yao commented on HDFS-10690: --- I mean the LinkedMap patch should just work without revision.

[jira] [Commented] (HDFS-10922) Adding additional unit tests for Trash

2016-09-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530965#comment-15530965 ] Xiaoyu Yao commented on HDFS-10922: --- Propose to add the following test cases: # test users can delete

[jira] [Created] (HDFS-10922) Adding additional unit tests for Trash

2016-09-28 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10922: - Summary: Adding additional unit tests for Trash Key: HDFS-10922 URL: https://issues.apache.org/jira/browse/HDFS-10922 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Commented] (HDFS-10913) Refactor BlockReceiver by introducing faults injector to enhance testability of detecting slow mirrors

2016-09-28 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530416#comment-15530416 ] Xiaoyu Yao commented on HDFS-10913: --- Thanks [~xiaobingo] for reporting the issue and posting the patch

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-09-30 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536651#comment-15536651 ] Xiaoyu Yao edited comment on HDFS-10690 at 9/30/16 6:34 PM: [~fenghua_hu], we

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-09-30 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536651#comment-15536651 ] Xiaoyu Yao commented on HDFS-10690: --- [~fenghua_hu], we will need to update

[jira] [Updated] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache

2016-10-03 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10690: -- Summary: Optimize insertion/removal of replica in ShortCircuitCache (was: Optimize insertion/removal

[jira] [Updated] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache

2016-10-03 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10690: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved

[jira] [Updated] (HDFS-10930) Refactor Datanode IO operations into a single class

2016-10-03 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Description: Datanode IO (Disk/Network) related operations and instrumentations are currently spilled

[jira] [Commented] (HDFS-10930) Refactor Datanode IO operations into a single class

2016-10-03 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543709#comment-15543709 ] Xiaoyu Yao commented on HDFS-10930: --- No, the goal is not to merge all these classes above into one

[jira] [Commented] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.

2016-09-30 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536813#comment-15536813 ] Xiaoyu Yao commented on HDFS-10912: --- Thanks @Anu for reporting the issue and posting the patch. It looks

[jira] [Created] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

2016-09-26 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10906: - Summary: Add unit tests for Trash with HDFS encryption zones Key: HDFS-10906 URL: https://issues.apache.org/jira/browse/HDFS-10906 Project: Hadoop HDFS Issue

[jira] [Commented] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525143#comment-15525143 ] Xiaoyu Yao commented on HDFS-10906: --- Below is a proposed test plan for list of test cases to add: #

[jira] [Comment Edited] (HDFS-10906) Add unit tests for Trash with HDFS encryption zones

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525143#comment-15525143 ] Xiaoyu Yao edited comment on HDFS-10906 at 9/27/16 5:50 AM: Below is a

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525167#comment-15525167 ] Xiaoyu Yao commented on HDFS-10690: --- Thanks [~fenghua_hu] for updating the patch. +1 for patch v6

[jira] [Commented] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524298#comment-15524298 ] Xiaoyu Yao commented on HDFS-10903: --- Thanks [~liuml07] for reporting this and [~vagarychen] for posting

[jira] [Comment Edited] (HDFS-10903) Replace config key literal strings with config key names II: hadoop hdfs

2016-09-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524298#comment-15524298 ] Xiaoyu Yao edited comment on HDFS-10903 at 9/26/16 9:58 PM: Thanks [~liuml07]

[jira] [Created] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log

2016-09-30 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-10941: - Summary: Improve BlockManager#processMisReplicatesAsync log Key: HDFS-10941 URL: https://issues.apache.org/jira/browse/HDFS-10941 Project: Hadoop HDFS Issue Type:

[jira] [Comment Edited] (HDFS-10984) Expose nntop output as metrics

2016-10-10 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563190#comment-15563190 ] Xiaoyu Yao edited comment on HDFS-10984 at 10/10/16 7:16 PM: - Thanks [~swagle]

[jira] [Commented] (HDFS-10984) Expose nntop output as metrics

2016-10-10 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563190#comment-15563190 ] Xiaoyu Yao commented on HDFS-10984: --- Thanks [~swagle] for reporting the issue and posting the patch. The

[jira] [Commented] (HDFS-10984) Expose nntop output as metrics

2016-10-10 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563376#comment-15563376 ] Xiaoyu Yao commented on HDFS-10984: --- bq. Confused about point 4. I have linked the two tickets, did you

[jira] [Updated] (HDFS-10930) Refactor: Wrap Datanode IO related operations

2016-10-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Attachment: HDFS-10930.03.patch Based on offline discussion with [~arpitagarwal], update the patch to

[jira] [Commented] (HDFS-10930) Refactor: Wrap Datanode IO related operations

2016-10-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608894#comment-15608894 ] Xiaoyu Yao commented on HDFS-10930: --- The instrumentation wrappers are moved into non-static methods of

[jira] [Updated] (HDFS-10930) Refactor: Wrap Datanode IO related operations

2016-10-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-10930: -- Status: Patch Available (was: Open) > Refactor: Wrap Datanode IO related operations >

[jira] [Commented] (HDFS-11001) Ozone:SCM: Add support for registerNode in SCM

2016-10-26 Thread Xiaoyu Yao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609977#comment-15609977 ] Xiaoyu Yao commented on HDFS-11001: --- +1 for the latest patch. > Ozone:SCM: Add support for

<    6   7   8   9   10   11   12   13   14   15   >