[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812682#comment-15812682 ] Yishan Yang commented on HBASE-17437: - Post patch to review board, https://reviews.apache.org/r/55352/ > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812588#comment-15812588 ] Yishan Yang commented on HBASE-17437: - About hbase.regionserver.hlog.dir.perms, since we differentiate logDir out of rootDir, in hbase, master used to set hbase.rootdir.perms by master, here, we just add one more perm parameter for logDir. > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812496#comment-15812496 ] Yishan Yang commented on HBASE-17437: - Will address comments and put new patch into review board, thanks! > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812424#comment-15812424 ] Yishan Yang commented on HBASE-17437: - Yeah, we used to use WAL, but from doc, default.xml, etc. On code side, it always use WAL, but from user side, hlog has much more occurrence. (Like 100 to 16). We already deployed it into our production and a lot of customer are using it right now. We passed integration tests , performance tests, unit tests. Up till now, there is no complain about this deployment yet. Thanks for your feed back! > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812416#comment-15812416 ] Yishan Yang commented on HBASE-17437: - We already put this patch in production, integration test, unit test, performance tests are all passed. For replication, it is working. Master branch has multiple commits related to "HBASE-17132 Cleanup deprecated code for WAL", so that part of the classes and unit tests need not be changed. That's the reason branch-1.2 is larger than master branch.. Will put it into review board, thanks! > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration, wal >Affects Versions: 1.2.4 >Reporter: Yishan Yang >Assignee: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying a WAL directory outside of the root directory
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishan Yang updated HBASE-17437: Summary: Support specifying a WAL directory outside of the root directory (was: Support specifying WAL directory to different directory with different file system (For example: S3)) > Support specifying a WAL directory outside of the root directory > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.2.5 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying WAL directory to different directory with different file system (For example: S3)
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishan Yang updated HBASE-17437: Description: Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems (such as Amazon S3) don’t support append or consistent writes. These two properties are imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily need the same consistency guarantees (since writes are cached locally and if writes fail, they can always be replayed from the WAL). This JIRA aims to allow users to configure a log directory (for WALs) that is outside of the root directory or even in a different FileSystem. The default value will still put the log directory under the root directory. was:Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems (such as Amazon S3) don’t support append or consistent writes. These two properties are imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily need the same consistency guarantees (since writes are cached locally and if writes fail, they can always be replayed from the WAL). > Support specifying WAL directory to different directory with different file > system (For example: S3) > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.2.5 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). > > This JIRA aims to allow users to configure a log directory (for WALs) that is > outside of the root directory or even in a different FileSystem. The default > value will still put the log directory under the root directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying WAL directory to different directory with different file system (For example: S3)
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishan Yang updated HBASE-17437: Attachment: hbase-17437-branch-1.2.patch hbase-17437-master.patch > Support specifying WAL directory to different directory with different file > system (For example: S3) > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.2.5 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17437) Support specifying WAL directory to different directory with different file system (For example: S3)
[ https://issues.apache.org/jira/browse/HBASE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yishan Yang updated HBASE-17437: Status: Patch Available (was: Open) > Support specifying WAL directory to different directory with different file > system (For example: S3) > > > Key: HBASE-17437 > URL: https://issues.apache.org/jira/browse/HBASE-17437 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Yishan Yang > Labels: patch > Fix For: 2.0.0, 1.2.5 > > Attachments: hbase-17437-branch-1.2.patch, hbase-17437-master.patch > > > Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some > FileSystems (such as Amazon S3) don’t support append or consistent writes. > These two properties are imperative for the WAL in order to avoid loss of > writes. However, StoreFiles don’t necessarily need the same consistency > guarantees (since writes are cached locally and if writes fail, they can > always be replayed from the WAL). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17437) Support specifying WAL directory to different directory with different file system (For example: S3)
Yishan Yang created HBASE-17437: --- Summary: Support specifying WAL directory to different directory with different file system (For example: S3) Key: HBASE-17437 URL: https://issues.apache.org/jira/browse/HBASE-17437 Project: HBase Issue Type: Bug Affects Versions: 1.2.4 Reporter: Yishan Yang Fix For: 2.0.0, 1.2.5 Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems (such as Amazon S3) don’t support append or consistent writes. These two properties are imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily need the same consistency guarantees (since writes are cached locally and if writes fail, they can always be replayed from the WAL). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
