[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Attachment: HDFS-11888-HDFS-7240.004.patch Patch 003 contains some unexpected YARN changes from IntelliJ refactor/rename of ContainerInfo class. I'm uploading patch 004 that fixed it. > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch, > HDFS-11888-HDFS-7240.002.patch, HDFS-11888-HDFS-7240.003.patch, > HDFS-11888-HDFS-7240.004.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Attachment: HDFS-11888-HDFS-7240.003.patch > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch, > HDFS-11888-HDFS-7240.002.patch, HDFS-11888-HDFS-7240.003.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Attachment: HDFS-11888-HDFS-7240.002.patch > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch, > HDFS-11888-HDFS-7240.002.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Attachment: (was: HDFS-12181-HDFS-7240.002.patch) > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch, > HDFS-11888-HDFS-7240.002.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Attachment: HDFS-12181-HDFS-7240.002.patch > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch, > HDFS-12181-HDFS-7240.002.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Status: Patch Available (was: Open) > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Attachment: HDFS-11888-HDFS-7240.001.patch > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11888-HDFS-7240.001.patch > > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11888) Ozone: SCM: use container state machine for open containers allocated for key/blcoks
[ https://issues.apache.org/jira/browse/HDFS-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-11888: -- Summary: Ozone: SCM: use container state machine for open containers allocated for key/blcoks (was: Ozone: SCM: allocateBlcok does not set the creationFlag correctly when the open container pool size is greater than 1) > Ozone: SCM: use container state machine for open containers allocated for > key/blcoks > - > > Key: HDFS-11888 > URL: https://issues.apache.org/jira/browse/HDFS-11888 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > SCM BlockManager provision a pool of containers upon block creation request > that can't be satisfied with current open containers in the pool. However, > only one container is returned with the creationFlag to the client. The other > container provisioned in the same batch will not have this flag. Client can't > assume to use these containers that has not been created on SCM datanodes, > This ticket is opened to fix the issue by persist the createContainerNeeded > flag for the provisioned containers. The flag will be eventually cleared by > processing container report from datanode when the container report handler > is fully implemented on SCM. > For now, we will use a default batch size of 1 for > ozone.scm.container.provision_batch_size so that the container will be > created on demand upon the first block allocation into the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
