[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-11-06 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
   Status: Resolved  (was: Patch Available)

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>  Labels: OzonePostMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch, 
> HDFS-12443-HDFS-7240.003.patch, HDFS-12443-HDFS-7240.004.patch, 
> HDFS-12443-HDFS-7240.005.patch, HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-11-04 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.005.patch

Attach the updated patch for fixing checkstyles warnings and adding some 
helpful logs.

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>Priority: Major
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch, 
> HDFS-12443-HDFS-7240.003.patch, HDFS-12443-HDFS-7240.004.patch, 
> HDFS-12443-HDFS-7240.005.patch, HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-11-04 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.004.patch

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>Priority: Major
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch, 
> HDFS-12443-HDFS-7240.003.patch, HDFS-12443-HDFS-7240.004.patch, 
> HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-11-02 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.003.patch

Attach the new patch.
Please have a look when you get a chanceļ¼Œ [~cheersyang].

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>Priority: Major
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch, 
> HDFS-12443-HDFS-7240.003.patch, HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-10-30 Thread Weiwei Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated HDFS-12443:
---
Attachment: HDFS-12443-SCM-blockdeletion-throttle.pdf

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch, 
> HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-10-29 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.002.patch

Attach the same patch to re-trigger Jenkins.

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-10-26 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.002.patch

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch, 
> HDFS-12443-HDFS-7240.002.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-10-26 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Status: Patch Available  (was: Open)

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-10-26 Thread Yiqun Lin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-12443:
-
Attachment: HDFS-12443-HDFS-7240.001.patch

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>  Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch
>
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-09-26 Thread Anu Engineer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-12443:

Labels: OzonePostMerge  (was: ozoneMerge)

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: OzonePostMerge
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-09-13 Thread Weiwei Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated HDFS-12443:
---
Labels: ozoneMerge  (was: )

> Ozone: Improve SCM block deletion throttling algorithm 
> ---
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: ozoneMerge
>
> Currently SCM scans delLog to send deletion transactions to datanode 
> periodically, the throttling algorithm is simple, it scans at most 
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
> will only get 1 TX to proceed in an interval, this will make the deletion 
> slow. An improvement to this is to make this throttling by datanode, e.g 50 
> TXs per datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org