[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11154: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~vagarychen] Thanks for contributing the patch. I have committed this to the ozone branch. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: HDFS-7240 > > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch, > HDFS-11154-HDFS-7240.007.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.007.patch Fix the checkstyle warning > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch, > HDFS-11154-HDFS-7240.007.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.006.patch The cblock test persistence test fail is related. Fix in v006 patch > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.005.patch The v005 patch was somehow checked by Jenkins twice. The first run was fine, all the warnings were unrelated. The second run complained about a number of things but the report links are all dead. So I suspect it hit some Jenkins internal issues in the second run. Upload v005 patch a second time to trigger one more Jenkins check. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.005.patch The compilation error was caused by missing recent updates of the branch. Fixed in v005 patch. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.004.patch Thanks [~anu] for the follow up comments. Uploaded v004 patch to fix all three issues. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.003.patch fix the findbug warnings on encoding in v003 patches. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11154: Status: Patch Available (was: Open) > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.002.patch > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.001.patch > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
