[jira] [Created] (HBASE-27547) Close store file readers after region warmup
Eungsop Yoo created HBASE-27547: --- Summary: Close store file readers after region warmup Key: HBASE-27547 URL: https://issues.apache.org/jira/browse/HBASE-27547 Project: HBase Issue Type: Bug Reporter: Eungsop Yoo We recently upgraded one of our clusters from version 1.4.13 to 2.4.15. After that we noticed that the number of open FD was increased 10 times more than before, from 1.3k to 15k. Each RS has about 1000 regions. I thouht that it is OK if the number of open FD keeps steady. But the number was increased when regions were balanced. I found that there is a bug related to the region warmup process, which leaks opened FDs, and I fixed it. Please review the PR. But I still have a question about why HBase 2.4 opens much more than 1.4. I am trying to find the answer, but not yet. Is there any person who can answer the question? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27123) Splitting region may become RIT for 20 minutes when meta region is moved during split
[ https://issues.apache.org/jira/browse/HBASE-27123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-27123: Description: When meta region is moved during region split, the spitting region may become RIT(Region in Transition) for 20 minutes. And then the RegionServer has the RIT will fail. We are facing the issue during RegionServer rolling restart on HBase 1.4.13. I found that HBase 1.7.1 has the same issue, too. It is OK on HBase 2.4.12. This is the fix. was: When meta region is moved during region split, the spitting region may become RIT(Region in Transition) for 20 minutes. And then the RegionServer has the RIT will fail. We are facing the issue during RegionServer rolling restart in HBase 1.4.13. I found that HBase 1.7.1 has the same issue, too. This is the fix. > Splitting region may become RIT for 20 minutes when meta region is moved > during split > - > > Key: HBASE-27123 > URL: https://issues.apache.org/jira/browse/HBASE-27123 > Project: HBase > Issue Type: Bug > Components: Admin, Coprocessors >Affects Versions: 1.4.13, 1.7.1 >Reporter: Eungsop Yoo >Priority: Minor > > When meta region is moved during region split, the spitting region may become > RIT(Region in Transition) for 20 minutes. And then the RegionServer has the > RIT will fail. We are facing the issue during RegionServer rolling restart on > HBase 1.4.13. I found that HBase 1.7.1 has the same issue, too. It is OK on > HBase 2.4.12. > This is the fix. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27123) Splitting region may become RIT for 20 minutes when meta region is moved during split
Eungsop Yoo created HBASE-27123: --- Summary: Splitting region may become RIT for 20 minutes when meta region is moved during split Key: HBASE-27123 URL: https://issues.apache.org/jira/browse/HBASE-27123 Project: HBase Issue Type: Bug Components: Admin, Coprocessors Affects Versions: 1.7.1, 1.4.13 Reporter: Eungsop Yoo When meta region is moved during region split, the spitting region may become RIT(Region in Transition) for 20 minutes. And then the RegionServer has the RIT will fail. We are facing the issue during RegionServer rolling restart in HBase 1.4.13. I found that HBase 1.7.1 has the same issue, too. This is the fix. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-22530: Status: Patch Available (was: Open) fix checkstyle error > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.4 >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.1.patch, HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-22530: Status: Open (was: Patch Available) > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.4 >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.1.patch, HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-22530: Attachment: HBASE-22530.1.patch > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.4 >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.1.patch, HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-22530: Affects Version/s: 2.1.4 Status: Patch Available (was: Open) trivial patch > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.4 >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-22530: Comment: was deleted (was: trivial patch) > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854316#comment-16854316 ] Eungsop Yoo commented on HBASE-22530: - trivial patch > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
[ https://issues.apache.org/jira/browse/HBASE-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-22530: Attachment: HBASE-22530.patch > The metrics of store files count of region are returned to clients incorrectly > -- > > Key: HBASE-22530 > URL: https://issues.apache.org/jira/browse/HBASE-22530 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-22530.patch > > > There is a mistake on mapping metrics; the count of stores is mapped to the > count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22530) The metrics of store files count of region are returned to clients incorrectly
Eungsop Yoo created HBASE-22530: --- Summary: The metrics of store files count of region are returned to clients incorrectly Key: HBASE-22530 URL: https://issues.apache.org/jira/browse/HBASE-22530 Project: HBase Issue Type: Bug Reporter: Eungsop Yoo There is a mistake on mapping metrics; the count of stores is mapped to the count of store files. So clients cannot retreive the metrics correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148108#comment-15148108 ] Eungsop Yoo commented on HBASE-15267: - Yes. The read requests count for Get and Scan are inconsistent. > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147955#comment-15147955 ] Eungsop Yoo commented on HBASE-15267: - Suppose that there are three rows, a, b and c(deleted but not major compacted yet). And d is not existing row. The below table is the test result of the read requests metric for some Scan and Get operations. || Operation || As is(read requests/filtered read requests) || This patch(read requests/filtered read requests) || | Scan() | 2/1 | 2/1 | | Scan(c, d) | 0/1 | 0/1 | | Scan(d) | 0/0 | 0/0 | | Get(c) | 1/1 | 0/1 | | Get(d) | 1/0 | 0/0 | I thought that read requests metric for Scan is correct, so I suggested this patch for Get to be fixed. > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Attachment: HBASE-15267-v1.patch Test for scan not existing row is added. > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Status: Patch Available (was: Open) > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267-v1.patch, HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Status: Open (was: Patch Available) > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146884#comment-15146884 ] Eungsop Yoo commented on HBASE-15267: - Currently (without this patch) Scan for not existing row does not increase the read requests count, but Get for not existing row increase the read requests count. It seems to be inconsistent. How do you think about this? > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146880#comment-15146880 ] Eungsop Yoo commented on HBASE-15267: - OK. My explanation was not enough. My intention was that how many gets were issued that returning no record is countered as filtered read requests metric, which was add in [HBASE-15197|https://issues.apache.org/jira/browse/HBASE-15197]. But in current patch, the requests for not existing rows is not countered at all. I'll try to count it as filtered read requests metric. How about this? > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Status: Patch Available (was: Open) > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
[ https://issues.apache.org/jira/browse/HBASE-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15267: Attachment: HBASE-15267.patch > Read requests count metric is increased when Get operation does not return > any record > - > > Key: HBASE-15267 > URL: https://issues.apache.org/jira/browse/HBASE-15267 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15267.patch > > > Read requests count is increased when Get operation returns no record by > filtering out. In such cases, Get for deleted record, Get for TTL expired > record and Get for filtering out. > So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15267) Read requests count metric is increased when Get operation does not return any record
Eungsop Yoo created HBASE-15267: --- Summary: Read requests count metric is increased when Get operation does not return any record Key: HBASE-15267 URL: https://issues.apache.org/jira/browse/HBASE-15267 Project: HBase Issue Type: Bug Reporter: Eungsop Yoo Priority: Minor Read requests count is increased when Get operation returns no record by filtering out. In such cases, Get for deleted record, Get for TTL expired record and Get for filtering out. So I fixed the bug and added some more test cases for read metrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Status: Open (was: Patch Available) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197.patch, HBASE-15197.patch, > WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Status: Open (was: Patch Available) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197.patch, > HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Attachment: HBASE-15197-v1.patch The comment is modified. > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197.patch, > HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Status: Patch Available (was: Open) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197.patch, > HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Attachment: HBASE-15197-v2.patch > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Status: Patch Available (was: Open) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15197-v1.patch, HBASE-15197-v2.patch, > HBASE-15197.patch, HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[
https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126917#comment-15126917
]
Eungsop Yoo commented on HBASE-15197:
-
After running below command at the root directory of HBase project, the
compilation errors are removed.
{code}
mvn clean install -DskipTests
{code}
I think Test environment should do this before running test.
> Expose filtered read requests metric to metrics framework and Web UI
>
>
> Key: HBASE-15197
> URL: https://issues.apache.org/jira/browse/HBASE-15197
> Project: HBase
> Issue Type: Improvement
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15197.patch, WebUI-screenshot.png
>
>
> Filtered read requests metric for scan is added in
> [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be
> retrieved from Scan object only. I think that it would be more informative
> when it is exposed to metrics framework. So I suggest to add filtered read
> requests metric and to expose it metrics framework and Web UI.
> ps. I think I found a bug; read requests count increased when Get operation
> returns no record by filtering out. Should it be fixed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125962#comment-15125962 ] Eungsop Yoo commented on HBASE-15197: - I have no problem in my PC for running tests. What should I do in this case? > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127624#comment-15127624 ] Eungsop Yoo commented on HBASE-15197: - I don't have a login account for builds.apache.org. Someone please help me. > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
Eungsop Yoo created HBASE-15197: --- Summary: Expose filtered read requests metric to metrics framework and Web UI Key: HBASE-15197 URL: https://issues.apache.org/jira/browse/HBASE-15197 Project: HBase Issue Type: Improvement Reporter: Eungsop Yoo Priority: Minor Filtered read requests metric for scan is added in [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be retrieved from Scan object only. I think that it would be more informative when it is exposed to metrics framework. So I suggest to add filtered read requests metric and to expose it metrics framework and Web UI. ps. I think I found a bug; read requests count increased when Get operation returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Attachment: HBASE-15197.patch > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Status: Patch Available (was: Open) > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15197) Expose filtered read requests metric to metrics framework and Web UI
[ https://issues.apache.org/jira/browse/HBASE-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15197: Attachment: WebUI-screenshot.png > Expose filtered read requests metric to metrics framework and Web UI > > > Key: HBASE-15197 > URL: https://issues.apache.org/jira/browse/HBASE-15197 > Project: HBase > Issue Type: Improvement >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15197.patch, WebUI-screenshot.png > > > Filtered read requests metric for scan is added in > [HBASE-5980|https://issues.apache.org/jira/browse/HBASE-5980]. But it can be > retrieved from Scan object only. I think that it would be more informative > when it is exposed to metrics framework. So I suggest to add filtered read > requests metric and to expose it metrics framework and Web UI. > ps. I think I found a bug; read requests count increased when Get operation > returns no record by filtering out. Should it be fixed? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15115232#comment-15115232
]
Eungsop Yoo commented on HBASE-15055:
-
Without that change, the running for a single test case "testCompactionRatio"
will fail.
I set the value to minsize because it is equal to
HConstants.HREGION_MEMSTORE_FLUSH_SIZE in that test class.
The default value of hbase.hstore.compaction.min.size is set by
HConstants.HREGION_MEMSTORE_FLUSH_SIZE at CompactionConfiguration.
{code}
...
package org.apache.hadoop.hbase.regionserver.compactions;
...
public class CompactionConfiguration {
...
CompactionConfiguration(Configuration conf, StoreConfigInformation
storeConfigInfo) {
...
minCompactSize = conf.getLong(HBASE_HSTORE_COMPACTION_MIN_SIZE_KEY,
storeConfigInfo.getMemstoreFlushSize());
...
{code}
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Assignee: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch,
> HBASE-15055-v11.patch, HBASE-15055-v2.patch, HBASE-15055-v3.patch,
> HBASE-15055-v4.patch, HBASE-15055-v5.patch, HBASE-15055-v6.patch,
> HBASE-15055-v7.patch, HBASE-15055-v8.patch, HBASE-15055-v9.patch,
> HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v11.patch To address comments of Anoop and Ted, the patch (including some test cases) is updated. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v11.patch, HBASE-15055-v2.patch, HBASE-15055-v3.patch, > HBASE-15055-v4.patch, HBASE-15055-v5.patch, HBASE-15055-v6.patch, > HBASE-15055-v7.patch, HBASE-15055-v8.patch, HBASE-15055-v9.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v11.patch, HBASE-15055-v2.patch, HBASE-15055-v3.patch, > HBASE-15055-v4.patch, HBASE-15055-v5.patch, HBASE-15055-v6.patch, > HBASE-15055-v7.patch, HBASE-15055-v8.patch, HBASE-15055-v9.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Release Note: If the major compaction period has elapsed, we should run a major compaction irrespective of hbase.hstore.compaction.max.size. (was: Do not skip large files when the sum of the size of TTL expired store files is greater than threshold. Major compaction should be triggered to delete TTL expired records. One parameter is added, "hbase.hstore.expired.size.ratio". When 0.0 is set to this parameter, major compaction is triggered if there is any TTL expired records. When 1.0 is set, all of large files are skipped regardless of the existence of TTL expired records. The default is 0.5, major compaction is triggered if above 50% of the store files in size have TTL expired records. ) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v11.patch, HBASE-15055-v2.patch, HBASE-15055-v3.patch, > HBASE-15055-v4.patch, HBASE-15055-v5.patch, HBASE-15055-v6.patch, > HBASE-15055-v7.patch, HBASE-15055-v8.patch, HBASE-15055-v9.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095330#comment-15095330 ] Eungsop Yoo commented on HBASE-15055: - Right. "isMajorCompaction" should not be skipped. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095331#comment-15095331 ] Eungsop Yoo commented on HBASE-15055: - OK > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v10.patch To address Anoop's comment, the new patch is submitted. In this patch, RatioBasedCompactionPolicy keeps all candidates and runs a major compaction irrespective of hbase.hstore.compaction.max.size if the major compaction period has elapsed. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, > HBASE-15055-v2.patch, HBASE-15055-v3.patch, HBASE-15055-v4.patch, > HBASE-15055-v5.patch, HBASE-15055-v6.patch, HBASE-15055-v7.patch, > HBASE-15055-v8.patch, HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092601#comment-15092601 ] Eungsop Yoo commented on HBASE-15055: - I see. I will update the patch to trigger MC even if max size is specified. It is expecred that will resolve the problem related to TTL expirarion too. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v9.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090249#comment-15090249 ] Eungsop Yoo commented on HBASE-15055: - testMajorCompactionWithTtlAndCompactionMaxSize2 is removed. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055-v9.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v8.patch Test cases are modified to verify "expiredRatio" variable has a proper value. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055-v8.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v7.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055-v7.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088271#comment-15088271 ] Eungsop Yoo commented on HBASE-15055: - OMG. I will fix it. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088280#comment-15088280 ] Eungsop Yoo commented on HBASE-15055: - Yes. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v4.patch One parameter is added, "hbase.hstore.expired.size.ratio". When 0.0 is set to this parameter, major compaction is triggered if there is any TTL expired records. When 1.0 is set, all of large files are skipped regardless of the existence of TTL expired records. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085326#comment-15085326 ] Eungsop Yoo commented on HBASE-15055: - The default is 0.5, major compaction is triggered if above 50% of the store files in size have TTL expired records. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085378#comment-15085378 ] Eungsop Yoo commented on HBASE-15055: - Is this what I should be doing? > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v5.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Release Note: Do not skip large files when the sum of the size of TTL expired store files is greater than threshold. Major compaction should be triggered to delete TTL expired records. One parameter is added, "hbase.hstore.expired.size.ratio". When 0.0 is set to this parameter, major compaction is triggered if there is any TTL expired records. When 1.0 is set, all of large files are skipped regardless of the existence of TTL expired records. The default is 0.5, major compaction is triggered if above 50% of the store files in size have TTL expired records. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086733#comment-15086733 ] Eungsop Yoo commented on HBASE-15055: - I will check it out. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086757#comment-15086757 ] Eungsop Yoo commented on HBASE-15055: - The new logic does not know whether this compaction is major or not. It just tries to promote minor compaction to major compaction to remove TTL expired records from disks. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v6.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Assignee: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055-v4.patch, HBASE-15055-v5.patch, > HBASE-15055-v6.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Patch Available (was: Open) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727-v4.patch, HBASE-14727.patch, > rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Attachment: HBASE-14727-v4.patch I removed unnecessary codes more. > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727-v4.patch, HBASE-14727.patch, > rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14727) Add block cache stats for regions
[
https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085040#comment-15085040
]
Eungsop Yoo commented on HBASE-14727:
-
{quote}
149 /** the size of cache for the region */
150 optional uint64 cache_size = 22;
What is a region cache size? We usually do the cache by regionserver, not by
region. Is this the amount of the cache occupied by the region summing the size
of all the cached blocks for a region? Is so, tighten the comment/description
I'd say.
{quote}
Yes, I'll update it.
{quote}
Do we need this:
143 /** the number of cache miss count for the region */
144 optional uint64 cache_miss_count = 20;
We have the cache hit number and elsewhere we have overall hits so miss would
be the difference? Do we need to send this one over?
{quote}
Instead of adding request(hit + miss) count, I added this to PB. It is used to
calculate cache hit ratio.(It is mentioned later some more.)
The new fields are added to PB to expose new metrics to clients out of region
servers. So now clients can query the metrics related to block cache for
regions by calling Admin.getClusterStatus().getLoad() like other metrics such
as read request count, write request count, etc.
{quote}
I think we do this elsewhere but that doesn't mean it a good idea (smile):
155 /** the current cache hit ratio for the region */
156 optional float cache_hit_ratio = 24;
The above can be calculated from cache hit rate or overall hit rate
... but this latter might be ok.
{quote}
This can be calculated at client side with hit and miss count for a region. So
this can be removed from PB.
{quote}
When you add region to BlockCacheKey, is it the name or the id that you use? It
looks like the name given you are parsing the file path. The file path will not
always have the region id in it (it won't in hbase 2.0.0). Do you have to get
it from the filepath?
{quote}
I used the the encoded region name parsed form the file path. Do you mean that
parsing the encoded region name from the file path is impossible in HBase 2.0.0?
If it is impossible, I need to find another way.
{quote}
What does the CacheStatsChore do? Why we need it now?
{quote}
Block cache metrics for regions are implemented in CacheStats class. The
instance of CacheStats belongs to BlockCache and the instance of BlockCache is
a singleton. I need to update the metrics when regions are moved or split. But
I could not update the block cache metrics for regions as the region metrics
belong to HRegion. So I made the chore to clean up the stat entries related to
regions that are not online. Is there any suggestion?
> Add block cache stats for regions
> -
>
> Key: HBASE-14727
> URL: https://issues.apache.org/jira/browse/HBASE-14727
> Project: HBase
> Issue Type: New Feature
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch,
> HBASE-14727-v3.patch, HBASE-14727-v4.patch, HBASE-14727.patch,
> rs-web-ui-v2.png
>
>
> The stats of block cache are calculated at region server level only. So at
> region level, we could not know about the numbers of blocks cached, the sizes
> of cached blocks, the numbers of cache hit, etc. I suggest to add the stats
> of block cache at region level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Attachment: rs-web-ui-v2.png > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727.patch, rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Attachment: HBASE-14727-v3.patch > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727.patch, rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Attachment: (was: rs-web-ui.png) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727.patch, rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Patch Available (was: Open) I removed unnecessary codes. > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727.patch, rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Open (was: Patch Available) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727.patch, rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-14727: Status: Open (was: Patch Available) > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727-v3.patch, HBASE-14727.patch, rs-web-ui-v2.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080694#comment-15080694 ] Eungsop Yoo commented on HBASE-14727: - I'm sorry. I missed your comment. I'll do it. > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14727) Add block cache stats for regions
[ https://issues.apache.org/jira/browse/HBASE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080695#comment-15080695 ] Eungsop Yoo commented on HBASE-14727: - I'm sorry. I missed your comment. I'll do it. > Add block cache stats for regions > - > > Key: HBASE-14727 > URL: https://issues.apache.org/jira/browse/HBASE-14727 > Project: HBase > Issue Type: New Feature >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-14727-v1.patch, HBASE-14727-v2.patch, > HBASE-14727.patch, rs-web-ui.png > > > The stats of block cache are calculated at region server level only. So at > region level, we could not know about the numbers of blocks cached, the sizes > of cached blocks, the numbers of cache hit, etc. I suggest to add the stats > of block cache at region level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v3.patch Test cases are updated to verify that major compactions should be triggered only if some TTL expirations. > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055-v3.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v2.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075731#comment-15075731
]
Eungsop Yoo commented on HBASE-15055:
-
I see. It may cause more major compactions.
But the increased number of times of major compactions would be limited at most
once (per each region) per HConstants.MAJOR_COMPACTION_PERIOD(7 days) by the
below code.
{code}
public class RatioBasedCompactionPolicy extends CompactionPolicy {
...
public boolean isMajorCompaction(final Collection filesToCompact)
throws IOException {
...
long lowTimestamp = StoreUtils.getLowestTimestamp(filesToCompact);
long now = System.currentTimeMillis();
if (lowTimestamp > 0l && lowTimestamp < (now - mcTime)) {
...
{code}
I think it isn't too much. How about you?
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch,
> HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Attachment: HBASE-15055-v1.patch > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075621#comment-15075621
]
Eungsop Yoo commented on HBASE-15055:
-
{quote}
We already have TestMajorCompaction
Can the new test be folded into TestMajorCompaction ?
{code}
136 long oldest = getOldest(candidates.get(pos),
System.currentTimeMillis());
{code}
Use EnvironmentEdgeManager
{quote}
OK. I did them.
{quote}
138 if (cfTtl != HConstants.FOREVER && oldest > cfTtl)
139 return candidates;
We should check the remaining StoreFile instead of returning immediately, right
?
{quote}
No. We have to return immediately if there is any TTL expiration because of the
below code.
All candidates should be kept.
{code}
101 isAllFiles = candidateFiles.size() == candidateSelection.size();
{code}
> Major compaction is not triggered when both of TTL and
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in
> candidate selection. It causes skipping of major compaction. So the TTL
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no
> TTL expired record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Patch Available (was: Open) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
[ https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eungsop Yoo updated HBASE-15055: Status: Open (was: Patch Available) > Major compaction is not triggered when both of TTL and > hbase.hstore.compaction.max.size are set > --- > > Key: HBASE-15055 > URL: https://issues.apache.org/jira/browse/HBASE-15055 > Project: HBase > Issue Type: Bug >Reporter: Eungsop Yoo >Priority: Minor > Attachments: HBASE-15055-v1.patch, HBASE-15055-v2.patch, > HBASE-15055.patch > > > Some large files may be skipped by hbase.hstore.compaction.max.size in > candidate selection. It causes skipping of major compaction. So the TTL > expired records are still remained in the disks and keep consuming disks. > To resolve this issue, I suggest that to skip large files only if there is no > TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set
Eungsop Yoo created HBASE-15055: --- Summary: Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set Key: HBASE-15055 URL: https://issues.apache.org/jira/browse/HBASE-15055 Project: HBase Issue Type: Bug Reporter: Eungsop Yoo Priority: Minor Some large files may be skipped by hbase.hstore.compaction.max.size in candidate selection. It causes skipping of major compaction. So the TTL expired records are still remained in the disks and keep consuming disks. To resolve this issue, I suggest that to skip large files only if there is no TTL expired record. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names
[
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012172#comment-15012172
]
Eungsop Yoo commented on HBASE-14824:
-
I'm done.
> HBaseAdmin.mergeRegions should use full region names instead of encoded
> region names
>
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824-v1.patch, HBASE-14824-v2.patch,
> HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally.
> HBaseAdmin.getRegion() requires the full region name. So
> MetaTableAccessor.getRegion always returns null and this causes one more meta
> table scan.
> {code}
> Pair getRegion(final byte[] regionName) throws
> IOException {
> if (regionName == null) {
> throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
> MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in
> HBaseAdmin.mergeRegions().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names
[
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008584#comment-15008584
]
Eungsop Yoo commented on HBASE-14824:
-
Yes, it is.
> HBaseAdmin.mergeRegions should use full region names instead of encoded
> region names
>
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally.
> HBaseAdmin.getRegion() requires the full region name. So
> MetaTableAccessor.getRegion always returns null and this causes one more meta
> table scan.
> {code}
> Pair getRegion(final byte[] regionName) throws
> IOException {
> if (regionName == null) {
> throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
> MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in
> HBaseAdmin.mergeRegions().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names
[
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eungsop Yoo updated HBASE-14824:
Status: Open (was: Patch Available)
> HBaseAdmin.mergeRegions should use full region names instead of encoded
> region names
>
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally.
> HBaseAdmin.getRegion() requires the full region name. So
> MetaTableAccessor.getRegion always returns null and this causes one more meta
> table scan.
> {code}
> Pair getRegion(final byte[] regionName) throws
> IOException {
> if (regionName == null) {
> throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
> MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in
> HBaseAdmin.mergeRegions().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names
[
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eungsop Yoo updated HBASE-14824:
Attachment: HBASE-14824-v1.patch
> HBaseAdmin.mergeRegions should use full region names instead of encoded
> region names
>
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
> Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824-v1.patch, HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally.
> HBaseAdmin.getRegion() requires the full region name. So
> MetaTableAccessor.getRegion always returns null and this causes one more meta
> table scan.
> {code}
> Pair getRegion(final byte[] regionName) throws
> IOException {
> if (regionName == null) {
> throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
> MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in
> HBaseAdmin.mergeRegions().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
