[jira] [Assigned] (HBASE-28463) Time Based Priority for BucketCache
[ https://issues.apache.org/jira/browse/HBASE-28463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar reassigned HBASE-28463: - Assignee: Rahul Agarkar > Time Based Priority for BucketCache > --- > > Key: HBASE-28463 > URL: https://issues.apache.org/jira/browse/HBASE-28463 > Project: HBase > Issue Type: New Feature > Components: BucketCache >Reporter: Janardhan Hungund >Assignee: Rahul Agarkar >Priority: Major > > This Jira introduces the feature of time-based data tiering in HBase to > optimize storage efficiency and access performance by segregating data based > on its recency. By keeping recent data in the bucket cache (backed by faster > storage types like SSDs) and evicting older data, the system aims to provide > a more flexible control over the cache allocation and eviction logic via > configuration, allowing for defining time priorities for cached data. > The need for a more extensive cache allocation mechanism becomes even more > critical on HBase deployments where cache access reflects on significant > performance gains, such as when using cloud storage as the underlying file > system. > The data is segregated into hot or cold categories based on its age. The > recent data within a specific time range (configured as hot-data-age) is > treated as hot and is stored in the ephemeral cache, while the older data is > stored and accessed from the cloud storage. > This feature intends to provide the TCO gains by optimizing the utilization > of high cost bucket cache. Perfect fit for the use cases that have the > date-based data writes while the scans focus on the recently written data. > Please find the detailed design document of the feature attached with the > Jira. > Thanks, > Janardhan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28186) Rebase CacheAwareBalance related commits into master branch
[ https://issues.apache.org/jira/browse/HBASE-28186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790199#comment-17790199 ] Rahul Agarkar commented on HBASE-28186: --- Hi [~wchevreuil] , I have raised the PR (#5541) for merging these changes into branch-2. > Rebase CacheAwareBalance related commits into master branch > --- > > Key: HBASE-28186 > URL: https://issues.apache.org/jira/browse/HBASE-28186 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28097) Add documentation section for the Cache Aware balancer function
[ https://issues.apache.org/jira/browse/HBASE-28097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-28097: -- Status: Patch Available (was: In Progress) > Add documentation section for the Cache Aware balancer function > --- > > Key: HBASE-28097 > URL: https://issues.apache.org/jira/browse/HBASE-28097 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Rahul Agarkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-28097) Add documentation section for the Cache Aware balancer function
[ https://issues.apache.org/jira/browse/HBASE-28097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-28097 started by Rahul Agarkar. - > Add documentation section for the Cache Aware balancer function > --- > > Key: HBASE-28097 > URL: https://issues.apache.org/jira/browse/HBASE-28097 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Rahul Agarkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27999: -- Attachment: (was: HBASE-27999 Cache Aware Load Balancer.docx) > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase uses ephemeral cache to cache the blocks by reading them from the slow > storages and storing them to the bucket cache. This cache is warmed up > everytime a region server is started. Depending on the data size and the > configured cache size, the cache warm up can take anywhere between a few > minutes to few hours. Doing this everytime the region server starts can be a > very expensive process. To eliminate this, HBASE-27313 implemented the cache > persistence feature where the region servers periodically persist the blocks > cached in the bucket cache. This persisted information is then used to > resurrect the cache in the event of a region server restart because of normal > restart or crash. > This feature aims at enhancing this capability of HBase to enable the > balancer implementation considers the cache allocation of each region on > region servers when calculating a new assignment plan and uses the > region/region server cache allocation info reported by region servers which > takes into account to calculate the percentage of HFiles cached for each > region on the hosting server, and then use that as another factor when > deciding on an optimal, new assignment plan. > > A design document describing the balancer can be found at > https://docs.google.com/document/d/1A8-eVeRhZjwL0hzFw9wmXl8cGP4BFomSlohX2QcaFg4/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27999: -- Description: HBase uses ephemeral cache to cache the blocks by reading them from the slow storages and storing them to the bucket cache. This cache is warmed up everytime a region server is started. Depending on the data size and the configured cache size, the cache warm up can take anywhere between a few minutes to few hours. Doing this everytime the region server starts can be a very expensive process. To eliminate this, HBASE-27313 implemented the cache persistence feature where the region servers periodically persist the blocks cached in the bucket cache. This persisted information is then used to resurrect the cache in the event of a region server restart because of normal restart or crash. This feature aims at enhancing this capability of HBase to enable the balancer implementation considers the cache allocation of each region on region servers when calculating a new assignment plan and uses the region/region server cache allocation info reported by region servers which takes into account to calculate the percentage of HFiles cached for each region on the hosting server, and then use that as another factor when deciding on an optimal, new assignment plan. A design document describing the balancer can be found at https://docs.google.com/document/d/1A8-eVeRhZjwL0hzFw9wmXl8cGP4BFomSlohX2QcaFg4/edit?usp=sharing was: HBase uses ephemeral cache to cache the blocks by reading them from the slow storages and storing them to the bucket cache. This cache is warmed up everytime a region server is started. Depending on the data size and the configured cache size, the cache warm up can take anywhere between a few minutes to few hours. Doing this everytime the region server starts can be a very expensive process. To eliminate this, HBASE-27313 implemented the cache persistence feature where the region servers periodically persist the blocks cached in the bucket cache. This persisted information is then used to resurrect the cache in the event of a region server restart because of normal restart or crash. This feature aims at enhancing this capability of HBase to enable the balancer implementation considers the cache allocation of each region on region servers when calculating a new assignment plan and uses the region/region server cache allocation info reported by region servers which takes into account to calculate the percentage of HFiles cached for each region on the hosting server, and then use that as another factor when deciding on an optimal, new assignment plan.. > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase uses ephemeral cache to cache the blocks by reading them from the slow > storages and storing them to the bucket cache. This cache is warmed up > everytime a region server is started. Depending on the data size and the > configured cache size, the cache warm up can take anywhere between a few > minutes to few hours. Doing this everytime the region server starts can be a > very expensive process. To eliminate this, HBASE-27313 implemented the cache > persistence feature where the region servers periodically persist the blocks > cached in the bucket cache. This persisted information is then used to > resurrect the cache in the event of a region server restart because of normal > restart or crash. > This feature aims at enhancing this capability of HBase to enable the > balancer implementation considers the cache allocation of each region on > region servers when calculating a new assignment plan and uses the > region/region server cache allocation info reported by region servers which > takes into account to calculate the percentage of HFiles cached for each > region on the hosting server, and then use that as another factor when > deciding on an optimal, new assignment plan. > > A design document describing the balancer can be found at > https://docs.google.com/document/d/1A8-eVeRhZjwL0hzFw9wmXl8cGP4BFomSlohX2QcaFg4/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999 ] Rahul Agarkar deleted comment on HBASE-27999: --- was (Author: ragarkar): A design document for this balancer can be found at https://docs.google.com/document/d/1A8-eVeRhZjwL0hzFw9wmXl8cGP4BFomSlohX2QcaFg4/edit?usp=sharing > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase uses ephemeral cache to cache the blocks by reading them from the slow > storages and storing them to the bucket cache. This cache is warmed up > everytime a region server is started. Depending on the data size and the > configured cache size, the cache warm up can take anywhere between a few > minutes to few hours. Doing this everytime the region server starts can be a > very expensive process. To eliminate this, HBASE-27313 implemented the cache > persistence feature where the region servers periodically persist the blocks > cached in the bucket cache. This persisted information is then used to > resurrect the cache in the event of a region server restart because of normal > restart or crash. > This feature aims at enhancing this capability of HBase to enable the > balancer implementation considers the cache allocation of each region on > region servers when calculating a new assignment plan and uses the > region/region server cache allocation info reported by region servers which > takes into account to calculate the percentage of HFiles cached for each > region on the hosting server, and then use that as another factor when > deciding on an optimal, new assignment plan.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758364#comment-17758364 ] Rahul Agarkar commented on HBASE-27999: --- A design document for this balancer can be found at https://docs.google.com/document/d/1A8-eVeRhZjwL0hzFw9wmXl8cGP4BFomSlohX2QcaFg4/edit?usp=sharing > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > Attachments: HBASE-27999 Cache Aware Load Balancer.docx > > > HBase uses ephemeral cache to cache the blocks by reading them from the slow > storages and storing them to the bucket cache. This cache is warmed up > everytime a region server is started. Depending on the data size and the > configured cache size, the cache warm up can take anywhere between a few > minutes to few hours. Doing this everytime the region server starts can be a > very expensive process. To eliminate this, HBASE-27313 implemented the cache > persistence feature where the region servers periodically persist the blocks > cached in the bucket cache. This persisted information is then used to > resurrect the cache in the event of a region server restart because of normal > restart or crash. > This feature aims at enhancing this capability of HBase to enable the > balancer implementation considers the cache allocation of each region on > region servers when calculating a new assignment plan and uses the > region/region server cache allocation info reported by region servers which > takes into account to calculate the percentage of HFiles cached for each > region on the hosting server, and then use that as another factor when > deciding on an optimal, new assignment plan.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27389) Add cost function in balancer to consider the cost of building bucket cache before moving regions
[ https://issues.apache.org/jira/browse/HBASE-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27389: -- Description: HBase currently uses StochasticLoadBalancer to determine the cost of moving the regions from one RS to another. Each cost functions give a result between 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer iterates through each cost function and comes up with the total cost. Now, the balancer will create multiple balancing plans on random actions and try to compute the cost of each plan as if they are executed, if the cost of the plan is less than the initial cost, the plan is executed. Implement a new "CacheAwareCostFunction" which takes into account if the region is fully cached and return the highest cost if the plan suggests moving this region. was: HBase currently uses StochasticLoadBalancer to determine the cost of moving the regions from one RS to another. Each cost functions give a result between 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer iterates through each cost function and comes up with the total cost. Now, the balancer will create multiple balancing plans on random actions and try to compute the cost of each plan as if they are executed, if the cost of the plan is less than the initial cost, the plan is executed. Implement a new "PrefetchCacheCostFunction" which takes into account if the region is fully cached and return the highest cost if the plan suggests moving this region. > Add cost function in balancer to consider the cost of building bucket cache > before moving regions > - > > Key: HBASE-27389 > URL: https://issues.apache.org/jira/browse/HBASE-27389 > Project: HBase > Issue Type: Task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase currently uses StochasticLoadBalancer to determine the cost of moving > the regions from one RS to another. Each cost functions give a result between > 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer > iterates through each cost function and comes up with the total cost. Now, > the balancer will create multiple balancing plans on random actions and try > to compute the cost of each plan as if they are executed, if the cost of the > plan is less than the initial cost, the plan is executed. > Implement a new "CacheAwareCostFunction" which takes into account if the > region is fully cached and return the highest cost if the plan suggests > moving this region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27999: -- Attachment: HBASE-27999 Cache Aware Load Balancer.docx > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > Attachments: HBASE-27999 Cache Aware Load Balancer.docx > > > HBase uses ephemeral cache to cache the blocks by reading them from the slow > storages and storing them to the bucket cache. This cache is warmed up > everytime a region server is started. Depending on the data size and the > configured cache size, the cache warm up can take anywhere between a few > minutes to few hours. Doing this everytime the region server starts can be a > very expensive process. To eliminate this, HBASE-27313 implemented the cache > persistence feature where the region servers periodically persist the blocks > cached in the bucket cache. This persisted information is then used to > resurrect the cache in the event of a region server restart because of normal > restart or crash. > This feature aims at enhancing this capability of HBase to enable the > balancer implementation considers the cache allocation of each region on > region servers when calculating a new assignment plan and uses the > region/region server cache allocation info reported by region servers which > takes into account to calculate the percentage of HFiles cached for each > region on the hosting server, and then use that as another factor when > deciding on an optimal, new assignment plan.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27999: -- Description: HBase uses ephemeral cache to cache the blocks by reading them from the slow storages and storing them to the bucket cache. This cache is warmed up everytime a region server is started. Depending on the data size and the configured cache size, the cache warm up can take anywhere between a few minutes to few hours. Doing this everytime the region server starts can be a very expensive process. To eliminate this, HBASE-27313 implemented the cache persistence feature where the region servers periodically persist the blocks cached in the bucket cache. This persisted information is then used to resurrect the cache in the event of a region server restart because of normal restart or crash. This feature aims at enhancing this capability of HBase to enable the balancer implementation considers the cache allocation of each region on region servers when calculating a new assignment plan and uses the region/region server cache allocation info reported by region servers which takes into account to calculate the percentage of HFiles cached for each region on the hosting server, and then use that as another factor when deciding on an optimal, new assignment plan.. > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase uses ephemeral cache to cache the blocks by reading them from the slow > storages and storing them to the bucket cache. This cache is warmed up > everytime a region server is started. Depending on the data size and the > configured cache size, the cache warm up can take anywhere between a few > minutes to few hours. Doing this everytime the region server starts can be a > very expensive process. To eliminate this, HBASE-27313 implemented the cache > persistence feature where the region servers periodically persist the blocks > cached in the bucket cache. This persisted information is then used to > resurrect the cache in the event of a region server restart because of normal > restart or crash. > This feature aims at enhancing this capability of HBase to enable the > balancer implementation considers the cache allocation of each region on > region servers when calculating a new assignment plan and uses the > region/region server cache allocation info reported by region servers which > takes into account to calculate the percentage of HFiles cached for each > region on the hosting server, and then use that as another factor when > deciding on an optimal, new assignment plan.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27999) Implement cache aware load balancer
[ https://issues.apache.org/jira/browse/HBASE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27999: -- Summary: Implement cache aware load balancer (was: Implement cache prefetch aware load balancer) > Implement cache aware load balancer > --- > > Key: HBASE-27999 > URL: https://issues.apache.org/jira/browse/HBASE-27999 > Project: HBase > Issue Type: Sub-task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27997) Enhance prefetch executor to record region prefetch information along with the list of hfiles prefetched
[ https://issues.apache.org/jira/browse/HBASE-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27997: -- Description: HBASE-27313 implemented the prefetch persistence feature where it persists the list of hFiles prefetched in the bucket cache. This information is used to reconstruct the cache in the event of a server restart/crash. Currently, only the list of hFiles is persisted. However, for the new PrefetchAwareLoadBalancer (work in progress) to work, we need the information about how much a region is prefetched on a region server. This Jira introduces an additional map in the prefetch executor to maintain the information about how much a region has been prefetched on that region server. The size of region server prefetched is calculated as the total size of all hFiles prefetched for that region. > Enhance prefetch executor to record region prefetch information along with > the list of hfiles prefetched > > > Key: HBASE-27997 > URL: https://issues.apache.org/jira/browse/HBASE-27997 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBASE-27313 implemented the prefetch persistence feature where it persists > the list of hFiles prefetched in the bucket cache. This information is used > to reconstruct the cache in the event of a server restart/crash. > Currently, only the list of hFiles is persisted. > However, for the new PrefetchAwareLoadBalancer (work in progress) to work, we > need the information about how much a region is prefetched on a region server. > This Jira introduces an additional map in the prefetch executor to maintain > the information about how much a region has been prefetched on that region > server. The size of region server prefetched is calculated as the total size > of all hFiles prefetched for that region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27997) Enhance prefetch executor to record region prefetch information along with the list of hfiles prefetched
[ https://issues.apache.org/jira/browse/HBASE-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27997 started by Rahul Agarkar. - > Enhance prefetch executor to record region prefetch information along with > the list of hfiles prefetched > > > Key: HBASE-27997 > URL: https://issues.apache.org/jira/browse/HBASE-27997 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27998) Enhance region metrics to include prefetch ratio for each region
[ https://issues.apache.org/jira/browse/HBASE-27998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27998 started by Rahul Agarkar. - > Enhance region metrics to include prefetch ratio for each region > > > Key: HBASE-27998 > URL: https://issues.apache.org/jira/browse/HBASE-27998 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27999) Implement cache prefetch aware load balancer
Rahul Agarkar created HBASE-27999: - Summary: Implement cache prefetch aware load balancer Key: HBASE-27999 URL: https://issues.apache.org/jira/browse/HBASE-27999 Project: HBase Issue Type: Sub-task Components: Balancer Reporter: Rahul Agarkar Assignee: Rahul Agarkar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27998) Enhance region metrics to include prefetch ratio for each region
Rahul Agarkar created HBASE-27998: - Summary: Enhance region metrics to include prefetch ratio for each region Key: HBASE-27998 URL: https://issues.apache.org/jira/browse/HBASE-27998 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Rahul Agarkar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HBASE-27998) Enhance region metrics to include prefetch ratio for each region
[ https://issues.apache.org/jira/browse/HBASE-27998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar reassigned HBASE-27998: - Assignee: Rahul Agarkar > Enhance region metrics to include prefetch ratio for each region > > > Key: HBASE-27998 > URL: https://issues.apache.org/jira/browse/HBASE-27998 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27997) Enhance prefetch executor to record region prefetch information along with the list of hfiles prefetched
Rahul Agarkar created HBASE-27997: - Summary: Enhance prefetch executor to record region prefetch information along with the list of hfiles prefetched Key: HBASE-27997 URL: https://issues.apache.org/jira/browse/HBASE-27997 Project: HBase Issue Type: Sub-task Components: BucketCache Reporter: Rahul Agarkar Assignee: Rahul Agarkar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27820) HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar
[ https://issues.apache.org/jira/browse/HBASE-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27820: -- Description: With some recent Atlas changes for supporting HTTP based hook support, HBase is not starting because of conflicts between jersey jars and rs-api jar. This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. HBase uses shaded jersey jars and hence does not need to use this jar directly. However, it still adds this jar to the CLASSPATH while starting the server. Atlas on the other hand is using a non-shaded version of javax.ws.rs-api jar which causes this conflict and causes the hbase server fail while initializing atlas co-processor. Since hbase is using shaded jersey jar and not using this jar directly, it should be removed from the bundle as it may cause similar conflicts with other client applications potentially using it. was: With some recent Atlas changes for supporting HTTP based hook support, HBase is not starting because of conflicts between jersey jars and rs-api jar. This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. HBase uses shaded jersey jars and hence does not need to use this jar directly. However, it still adds this jar to the CLASSPATH while starting the server. Atlas on the other hand is using a non-shaded version of javax.ws.rs-api jar which causes this conflict and causes the hbase server fail while initializing atlas co-processor. This problem is not seen if only hbase server is started. > HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar > -- > > Key: HBASE-27820 > URL: https://issues.apache.org/jira/browse/HBASE-27820 > Project: HBase > Issue Type: Task > Components: scripts >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > With some recent Atlas changes for supporting HTTP based hook support, HBase > is not starting because of conflicts between jersey jars and rs-api jar. > This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. > HBase uses shaded jersey jars and hence does not need to use this jar > directly. However, it still adds this jar to the CLASSPATH while starting the > server. Atlas on the other hand is using a non-shaded version of > javax.ws.rs-api jar which causes this conflict and causes the hbase server > fail while initializing atlas co-processor. > Since hbase is using shaded jersey jar and not using this jar directly, it > should be removed from the bundle as it may cause similar conflicts with > other client applications potentially using it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27820) HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar
[ https://issues.apache.org/jira/browse/HBASE-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27820: -- Description: With some recent Atlas changes for supporting HTTP based hook support, HBase is not starting because of conflicts between jersey jars and rs-api jar. This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. HBase uses shaded jersey jars and hence does not need to use this jar directly. However, it still adds this jar to the CLASSPATH while starting the server. Atlas on the other hand is using a non-shaded version of javax.ws.rs-api jar which causes this conflict and causes the hbase server fail while initializing atlas co-processor. This problem is not seen if only hbase server is started. was: With some recent Atlas changes, HBase is not starting because of conflicts between jersey jars and rs-api jar. This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. > HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar > -- > > Key: HBASE-27820 > URL: https://issues.apache.org/jira/browse/HBASE-27820 > Project: HBase > Issue Type: Task > Components: scripts >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > With some recent Atlas changes for supporting HTTP based hook support, HBase > is not starting because of conflicts between jersey jars and rs-api jar. > This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. > HBase uses shaded jersey jars and hence does not need to use this jar > directly. However, it still adds this jar to the CLASSPATH while starting the > server. Atlas on the other hand is using a non-shaded version of > javax.ws.rs-api jar which causes this conflict and causes the hbase server > fail while initializing atlas co-processor. > This problem is not seen if only hbase server is started. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27820) HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar
[ https://issues.apache.org/jira/browse/HBASE-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27820 started by Rahul Agarkar. - > HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar > -- > > Key: HBASE-27820 > URL: https://issues.apache.org/jira/browse/HBASE-27820 > Project: HBase > Issue Type: Task > Components: scripts >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > With some recent Atlas changes, HBase is not starting because of conflicts > between jersey jars and rs-api jar. > This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27820) HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar
Rahul Agarkar created HBASE-27820: - Summary: HBase is not starting due to Jersey library conflicts with javax.ws.rs.api jar Key: HBASE-27820 URL: https://issues.apache.org/jira/browse/HBASE-27820 Project: HBase Issue Type: Task Components: scripts Reporter: Rahul Agarkar Assignee: Rahul Agarkar With some recent Atlas changes, HBase is not starting because of conflicts between jersey jars and rs-api jar. This Jira is to exclude the javax.ws.rs-api.jar from the HBase classpath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26813) Remove javax.ws.rs-api dependency
[
https://issues.apache.org/jira/browse/HBASE-26813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713131#comment-17713131
]
Rahul Agarkar commented on HBASE-26813:
---
Hello [~ndimiduk] , do we have any plans to fix this issue? I was looking at a
similar issue and tried a simple fix where I modified the bin/hbase script to
filter out the javax.ws.rs-api* jars while adding the jar files to the
CLASSPATH. This seems to be working for me, although, not sure if this is the
right approach.
Thanks.
> Remove javax.ws.rs-api dependency
> -
>
> Key: HBASE-26813
> URL: https://issues.apache.org/jira/browse/HBASE-26813
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> I see that we still have {{javax.ws.rs-api}} as a dependency in our
> hbase-http pom. If, for example, the {{ClientBuilder}} from this jar is used,
> it'll instantiate whatever non-shaded jersey client is on the class path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27389) Add cost function in balancer to consider the cost of building bucket cache before moving regions
[ https://issues.apache.org/jira/browse/HBASE-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707765#comment-17707765 ] Rahul Agarkar commented on HBASE-27389: --- The pull request has been updated with the following: # Added system tests # Fixed issues seen while testing the changes > Add cost function in balancer to consider the cost of building bucket cache > before moving regions > - > > Key: HBASE-27389 > URL: https://issues.apache.org/jira/browse/HBASE-27389 > Project: HBase > Issue Type: Task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase currently uses StochasticLoadBalancer to determine the cost of moving > the regions from one RS to another. Each cost functions give a result between > 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer > iterates through each cost function and comes up with the total cost. Now, > the balancer will create multiple balancing plans on random actions and try > to compute the cost of each plan as if they are executed, if the cost of the > plan is less than the initial cost, the plan is executed. > Implement a new "PrefetchCacheCostFunction" which takes into account if the > region is fully cached and return the highest cost if the plan suggests > moving this region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HBASE-27389) Add cost function in balancer to consider the cost of building bucket cache before moving regions
[ https://issues.apache.org/jira/browse/HBASE-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-27389 started by Rahul Agarkar. - > Add cost function in balancer to consider the cost of building bucket cache > before moving regions > - > > Key: HBASE-27389 > URL: https://issues.apache.org/jira/browse/HBASE-27389 > Project: HBase > Issue Type: Task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase currently uses StochasticLoadBalancer to determine the cost of moving > the regions from one RS to another. Each cost functions give a result between > 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer > iterates through each cost function and comes up with the total cost. Now, > the balancer will create multiple balancing plans on random actions and try > to compute the cost of each plan as if they are executed, if the cost of the > plan is less than the initial cost, the plan is executed. > Implement a new "PrefetchCacheCostFunction" which takes into account if the > region is fully cached and return the highest cost if the plan suggests > moving this region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27389) Add cost function in balancer to consider the cost of building bucket cache before moving regions
[ https://issues.apache.org/jira/browse/HBASE-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27389: -- Status: Patch Available (was: In Progress) > Add cost function in balancer to consider the cost of building bucket cache > before moving regions > - > > Key: HBASE-27389 > URL: https://issues.apache.org/jira/browse/HBASE-27389 > Project: HBase > Issue Type: Task > Components: Balancer >Reporter: Rahul Agarkar >Assignee: Rahul Agarkar >Priority: Major > > HBase currently uses StochasticLoadBalancer to determine the cost of moving > the regions from one RS to another. Each cost functions give a result between > 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer > iterates through each cost function and comes up with the total cost. Now, > the balancer will create multiple balancing plans on random actions and try > to compute the cost of each plan as if they are executed, if the cost of the > plan is less than the initial cost, the plan is executed. > Implement a new "PrefetchCacheCostFunction" which takes into account if the > region is fully cached and return the highest cost if the plan suggests > moving this region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27389) Add cost function in balancer to consider the cost of building bucket cache before moving regions
[ https://issues.apache.org/jira/browse/HBASE-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Agarkar updated HBASE-27389: -- I am working on this and will raise a PR soon. > Add cost function in balancer to consider the cost of building bucket cache > before moving regions > - > > Key: HBASE-27389 > URL: https://issues.apache.org/jira/browse/HBASE-27389 > Project: HBase > Issue Type: Task > Components: Balancer >Reporter: Rahul Agarkar >Priority: Major > > HBase currently uses StochasticLoadBalancer to determine the cost of moving > the regions from one RS to another. Each cost functions give a result between > 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer > iterates through each cost function and comes up with the total cost. Now, > the balancer will create multiple balancing plans on random actions and try > to compute the cost of each plan as if they are executed, if the cost of the > plan is less than the initial cost, the plan is executed. > Implement a new "PrefetchCacheCostFunction" which takes into account if the > region is fully cached and return the highest cost if the plan suggests > moving this region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27389) Add cost function in balancer to consider the cost of building bucket cache before moving regions
Rahul Agarkar created HBASE-27389: - Summary: Add cost function in balancer to consider the cost of building bucket cache before moving regions Key: HBASE-27389 URL: https://issues.apache.org/jira/browse/HBASE-27389 Project: HBase Issue Type: Task Components: Balancer Reporter: Rahul Agarkar HBase currently uses StochasticLoadBalancer to determine the cost of moving the regions from one RS to another. Each cost functions give a result between 0 and 1, with 0 being the lowest cost and 1 being the cost. The balancer iterates through each cost function and comes up with the total cost. Now, the balancer will create multiple balancing plans on random actions and try to compute the cost of each plan as if they are executed, if the cost of the plan is less than the initial cost, the plan is executed. Implement a new "PrefetchCacheCostFunction" which takes into account if the region is fully cached and return the highest cost if the plan suggests moving this region. -- This message was sent by Atlassian Jira (v8.20.10#820010)
