[jira] [Updated] (HBASE-28447) New configuration to override the hfile specific blocksize
[
https://issues.apache.org/jira/browse/HBASE-28447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gourab Taparia updated HBASE-28447:
---
Priority: Minor (was: Critical)
> New configuration to override the hfile specific blocksize
> --
>
> Key: HBASE-28447
> URL: https://issues.apache.org/jira/browse/HBASE-28447
> Project: HBase
> Issue Type: Improvement
>Reporter: Gourab Taparia
>Assignee: Gourab Taparia
>Priority: Minor
>
> Right now there is no config attached to the HFile block size by which we can
> override the default. The default is set to 64 KB in
> HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this
> value.
> Since the BLOCKSIZE is tracked at the column family level - we will need to
> respect the CFD value first. Also, configuration settings are also something
> that can be set in schema, at the column or table level, and will override
> the relevant values from the site file. Below is the precedence order we can
> use to get the final blocksize value :
> {code:java}
> ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides
> > site configuration > HConstants.DEFAULT_BLOCKSIZE{code}
> PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize”
> however that is specific to map-reduce jobs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28447) New configuration to override the hfile specific blocksize
[
https://issues.apache.org/jira/browse/HBASE-28447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gourab Taparia updated HBASE-28447:
---
Summary: New configuration to override the hfile specific blocksize (was:
New configuration to configure with hfile specific blocksize)
> New configuration to override the hfile specific blocksize
> --
>
> Key: HBASE-28447
> URL: https://issues.apache.org/jira/browse/HBASE-28447
> Project: HBase
> Issue Type: Improvement
>Reporter: Gourab Taparia
>Assignee: Gourab Taparia
>Priority: Critical
>
> Right now there is no config attached to the HFile block size by which we can
> override the default. The default is set to 64 KB in
> HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this
> value.
> Since the BLOCKSIZE is tracked at the column family level - we will need to
> respect the CFD value first. Also, configuration settings are also something
> that can be set in schema, at the column or table level, and will override
> the relevant values from the site file. Below is the precedence order we can
> use to get the final blocksize value :
> {code:java}
> ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides
> > site configuration > HConstants.DEFAULT_BLOCKSIZE{code}
> PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize”
> however that is specific to map-reduce jobs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HBASE-28447) New configuration to configure with hfile specific blocksize
[
https://issues.apache.org/jira/browse/HBASE-28447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gourab Taparia reassigned HBASE-28447:
--
Assignee: Gourab Taparia
> New configuration to configure with hfile specific blocksize
>
>
> Key: HBASE-28447
> URL: https://issues.apache.org/jira/browse/HBASE-28447
> Project: HBase
> Issue Type: Improvement
>Reporter: Gourab Taparia
>Assignee: Gourab Taparia
>Priority: Critical
>
> Right now there is no config attached to the HFile block size by which we can
> override the default. The default is set to 64 KB in
> HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this
> value.
> Since the BLOCKSIZE is tracked at the column family level - we will need to
> respect the CFD value first. Also, configuration settings are also something
> that can be set in schema, at the column or table level, and will override
> the relevant values from the site file. Below is the precedence order we can
> use to get the final blocksize value :
> {code:java}
> ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides
> > site configuration > HConstants.DEFAULT_BLOCKSIZE{code}
> PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize”
> however that is specific to map-reduce jobs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-28447) New configuration to configure with hfile specific blocksize
[
https://issues.apache.org/jira/browse/HBASE-28447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gourab Taparia updated HBASE-28447:
---
Description:
Right now there is no config attached to the HFile block size by which we can
override the default. The default is set to 64 KB in
HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this
value.
Since the BLOCKSIZE is tracked at the column family level - we will need to
respect the CFD value first. Also, configuration settings are also something
that can be set in schema, at the column or table level, and will override the
relevant values from the site file. Below is the precedence order we can use to
get the final blocksize value :
{code:java}
ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides >
site configuration > HConstants.DEFAULT_BLOCKSIZE{code}
PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize”
however that is specific to map-reduce jobs.
was:
Right now there is no config attached to the HFile block size by which we can
override the default. The default is set to 64 KB in
HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this
value.
Since the BLOCKSIZE is tracked at the column family level - we will need to
respect the CFD value first. Also, configuration settings are also something
that can be set in schema, at the column or table level, and will override the
relevant values from the site file. Below is the precedence order we can use to
get the final blocksize value :
ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides >
site configuration > HConstants.DEFAULT_BLOCKSIZE
PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize”
however that is specific to map-reduce jobs.
> New configuration to configure with hfile specific blocksize
>
>
> Key: HBASE-28447
> URL: https://issues.apache.org/jira/browse/HBASE-28447
> Project: HBase
> Issue Type: Improvement
>Reporter: Gourab Taparia
>Priority: Critical
>
> Right now there is no config attached to the HFile block size by which we can
> override the default. The default is set to 64 KB in
> HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this
> value.
> Since the BLOCKSIZE is tracked at the column family level - we will need to
> respect the CFD value first. Also, configuration settings are also something
> that can be set in schema, at the column or table level, and will override
> the relevant values from the site file. Below is the precedence order we can
> use to get the final blocksize value :
> {code:java}
> ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides
> > site configuration > HConstants.DEFAULT_BLOCKSIZE{code}
> PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize”
> however that is specific to map-reduce jobs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-28447) New configuration to configure with hfile specific blocksize
Gourab Taparia created HBASE-28447: -- Summary: New configuration to configure with hfile specific blocksize Key: HBASE-28447 URL: https://issues.apache.org/jira/browse/HBASE-28447 Project: HBase Issue Type: Improvement Reporter: Gourab Taparia Right now there is no config attached to the HFile block size by which we can override the default. The default is set to 64 KB in HConstants.DEFAULT_BLOCKSIZE . We need a new config which can control this value. Since the BLOCKSIZE is tracked at the column family level - we will need to respect the CFD value first. Also, configuration settings are also something that can be set in schema, at the column or table level, and will override the relevant values from the site file. Below is the precedence order we can use to get the final blocksize value : ColumnFamilyDescriptor.BLOCKSIZE > schema level site configuration overrides > site configuration > HConstants.DEFAULT_BLOCKSIZE PS: There is one related config “hbase.mapreduce.hfileoutputformat.blocksize” however that is specific to map-reduce jobs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28431) Cleaning up binary assemblies and diagnostic tools
[ https://issues.apache.org/jira/browse/HBASE-28431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828606#comment-17828606 ] Cheng Pan commented on HBASE-28431: --- > Do those work with the cloud connectors ? [~stoty] Specific to AWS S3, yes, see HADOOP-16080, Azure and GCP should work too. The pre-built Spark dist shippes Hadoop shaded client since 3.2.0, at least I haven't seen cloud storage incompatible reporting yet. For demonstration, there is a docker compose based playground maintained by the Apache Kyuubi project, which has out-out-box configurations of Spark and S3 (MinIO), you can check the classpath and run some examples of Spark jobs for verification. https://github.com/apache/kyuubi/tree/master/docker/playground BTW, the Jira notification is a little bit noisy, I blocked all messages except to AT ME. > Cleaning up binary assemblies and diagnostic tools > -- > > Key: HBASE-28431 > URL: https://issues.apache.org/jira/browse/HBASE-28431 > Project: HBase > Issue Type: Umbrella >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Priority: Major > > As discussed on the mailing list, the current binary assembly has several > problems. > The discussed improvements: > * Provide assembly versions without transitive Hadoop dependencies > * Remove test JARs and their dependencies from the assemblies > * Move useful diagnostic tools into the runtime jars > See [https://lists.apache.org/thread/s784s8778cys2vx0scbd28cl2lm1g6vt] for > details. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28124) Missing fields in Scan.toJSON
[
https://issues.apache.org/jira/browse/HBASE-28124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828604#comment-17828604
]
Hudson commented on HBASE-28124:
Results for branch branch-2.4
[build #709 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/709/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/709/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/709/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/709/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.4/709/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Missing fields in Scan.toJSON
> -
>
> Key: HBASE-28124
> URL: https://issues.apache.org/jira/browse/HBASE-28124
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Chandra Sekhar K
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> Let's do an audit and add whatever's missing. I noticed getLimit is missing
> at least.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28124) Missing fields in Scan.toJSON
[
https://issues.apache.org/jira/browse/HBASE-28124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828598#comment-17828598
]
Hudson commented on HBASE-28124:
Results for branch branch-2.6
[build #78 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Missing fields in Scan.toJSON
> -
>
> Key: HBASE-28124
> URL: https://issues.apache.org/jira/browse/HBASE-28124
> Project: HBase
> Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Chandra Sekhar K
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> Let's do an audit and add whatever's missing. I noticed getLimit is missing
> at least.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[
https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828599#comment-17828599
]
Hudson commented on HBASE-28415:
Results for branch branch-2.6
[build #78 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/78/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove Curator dependency from hbase-endpoint
> -
>
> Key: HBASE-28415
> URL: https://issues.apache.org/jira/browse/HBASE-28415
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-beta-1
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.6.0, 3.0.0-beta-2
>
>
> TestRpcControllerFactory used to erronously import some classes from
> Curator's relocated Guava.
> This has been fixed now, but the (compile scope) dependency for curator has
> not been removed.
> I propose removing the dependency in master, branch-3 and branch-2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28446) Remove the reflection for ByteBufferPositionedReadable
[ https://issues.apache.org/jira/browse/HBASE-28446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828530#comment-17828530 ] Wei-Chiu Chuang commented on HBASE-28446: - That's a really good point. Thanks [~stoty] I was just thinking that Hadoop 3.1/3.2 was EOL. > Remove the reflection for ByteBufferPositionedReadable > -- > > Key: HBASE-28446 > URL: https://issues.apache.org/jira/browse/HBASE-28446 > Project: HBase > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Minor > > HBASE-21946 used reflection to access the ByteBufferPositionedReadable API > that's only available in Hadoop 3.3. > Now that HBase branch-2.6 and above updated hadoop three dependency to 3.3, > we can get rid of the reflection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28427) FNFE related to 'master:store' when moving archived hfiles to global archived dir
[
https://issues.apache.org/jira/browse/HBASE-28427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828508#comment-17828508
]
Hudson commented on HBASE-28427:
Results for branch master
[build #1034 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1034/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1034/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1034/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/1034/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> FNFE related to 'master:store' when moving archived hfiles to global archived
> dir
> -
>
> Key: HBASE-28427
> URL: https://issues.apache.org/jira/browse/HBASE-28427
> Project: HBase
> Issue Type: Improvement
> Components: master
> Environment: Centos7
> hbase-4.0.0-alpha-1-SNAPSHOT
>Reporter: guluo
>Assignee: guluo
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9
>
>
> Sometimes we encounter the following log.
> {code:java}
> 2024-02-26T18:11:34,934 WARN [master:store-Store-Compactor]
> region.MasterRegionFlusherAndCompactor: Failed to move archived hfiles from
> file:/root/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/MasterData/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/state
> to global dir
> file:/root/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/statejava.io.FileNotFoundException:
> File
> file:/root/hbase-4.0.0-alpha-1-SNAPSHOT/tmp/hbase/MasterData/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/state
> does not exist at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:733)
> ~[hadoop-common-3.3.5.jar:?] at
> org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2045)
> ~[hadoop-common-3.3.5.jar:?] at
> org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2089)
> ~[hadoop-common-3.3.5.jar:?] at
> org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:970)
> ~[hadoop-common-3.3.5.jar:?] at
> org.apache.hadoop.hbase.master.region.MasterRegionUtils.moveFilesUnderDir(MasterRegionUtils.java:42)
> ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT] at
> org.apache.hadoop.hbase.master.region.MasterRegionFlusherAndCompactor.moveHFileToGlobalArchiveDir(MasterRegionFlusherAndCompactor.java:145)
> ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT] at
> org.apache.hadoop.hbase.master.region.MasterRegionFlusherAndCompactor.compact(MasterRegionFlusherAndCompactor.java:157)
> ~[hbase-server-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_231] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_231] at java.lang.Thread.run(Thread.java:748)
> ~[?:1.8.0_231] {code}
> Analysis:
> Against 'master:store', Archived hfiles would first be placed in
> \{hbase.rootdir}/MasterData/archive/ after compacting, and then moved to
> global archived dir.
> 'master:store' has 3 column families:proc,rs,state, if there is empty in one
> of column families,such as state, no files in this column families will be
> archived, and
> \{hbase.rootdir}/MasterData/archive/data/master/store/1595e783b53d99cd5eef43b6debb2682/state
> would do not exist.
> Finally, we will get FileNotFoundException when moving these archived hfiles
> to global archived dir.
>
> Although，Current code can avoid this effecting by catching Exception, However
> it is maybe better to check the dir exists before moving.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28446) Remove the reflection for ByteBufferPositionedReadable
[ https://issues.apache.org/jira/browse/HBASE-28446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828502#comment-17828502 ] Istvan Toth commented on HBASE-28446: - Does building with Hadoop 3.3 by default mean that we drop support for earlier Hadoop versions ? Previous HBase versions supported multiple Hadoop versions. https://hbase.apache.org/book.html#hadoop > Remove the reflection for ByteBufferPositionedReadable > -- > > Key: HBASE-28446 > URL: https://issues.apache.org/jira/browse/HBASE-28446 > Project: HBase > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Minor > > HBASE-21946 used reflection to access the ByteBufferPositionedReadable API > that's only available in Hadoop 3.3. > Now that HBase branch-2.6 and above updated hadoop three dependency to 3.3, > we can get rid of the reflection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5764: URL: https://github.com/apache/hbase/pull/5764#issuecomment-2007918628 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 1s | master passed | | +1 :green_heart: | compile | 1m 16s | master passed | | +1 :green_heart: | shadedjars | 5m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 1m 10s | the patch passed | | +1 :green_heart: | javac | 1m 10s | the patch passed | | +1 :green_heart: | shadedjars | 5m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 31s | hbase-client in the patch passed. | | -1 :x: | unit | 236m 3s | hbase-server in the patch failed. | | | | 264m 11s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5764 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cd73ba12977a 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0763a74096 | | Default Java | Eclipse Adoptium-11.0.17+8 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/testReport/ | | Max. process+thread count | 4831 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5764: URL: https://github.com/apache/hbase/pull/5764#issuecomment-2007916489 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 13s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 32s | master passed | | +1 :green_heart: | compile | 1m 1s | master passed | | +1 :green_heart: | shadedjars | 5m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 30s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | shadedjars | 5m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 20s | hbase-client in the patch passed. | | -1 :x: | unit | 236m 57s | hbase-server in the patch failed. | | | | 262m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5764 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c721f15886be 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0763a74096 | | Default Java | Temurin-1.8.0_352-b08 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/testReport/ | | Max. process+thread count | 5218 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[ https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828484#comment-17828484 ] Andrew Kyle Purtell commented on HBASE-28415: - I have no concerns about it being removed. Sure, I may cherry pick it. We can file a new issue for that work. > Remove Curator dependency from hbase-endpoint > - > > Key: HBASE-28415 > URL: https://issues.apache.org/jira/browse/HBASE-28415 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > TestRpcControllerFactory used to erronously import some classes from > Curator's relocated Guava. > This has been fixed now, but the (compile scope) dependency for curator has > not been removed. > I propose removing the dependency in master, branch-3 and branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5764: URL: https://github.com/apache/hbase/pull/5764#issuecomment-2007869879 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 56s | master passed | | +1 :green_heart: | compile | 1m 9s | master passed | | +1 :green_heart: | shadedjars | 5m 26s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 50s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 5m 23s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 27s | hbase-client in the patch passed. | | -1 :x: | unit | 209m 24s | hbase-server in the patch failed. | | | | 236m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5764 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux ac7902d6f083 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0763a74096 | | Default Java | Eclipse Adoptium-17.0.10+7 | | unit | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/testReport/ | | Max. process+thread count | 4899 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/console | | versions | git=2.34.1 maven=3.8.6 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-28446) Remove the reflection for ByteBufferPositionedReadable
Wei-Chiu Chuang created HBASE-28446: --- Summary: Remove the reflection for ByteBufferPositionedReadable Key: HBASE-28446 URL: https://issues.apache.org/jira/browse/HBASE-28446 Project: HBase Issue Type: Improvement Reporter: Wei-Chiu Chuang HBASE-21946 used reflection to access the ByteBufferPositionedReadable API that's only available in Hadoop 3.3. Now that HBase branch-2.6 and above updated hadoop three dependency to 3.3, we can get rid of the reflection. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[ https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828439#comment-17828439 ] Istvan Toth commented on HBASE-28415: - Thank you, [~bbeaudreault], that was my original proposal as well. > Remove Curator dependency from hbase-endpoint > - > > Key: HBASE-28415 > URL: https://issues.apache.org/jira/browse/HBASE-28415 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > TestRpcControllerFactory used to erronously import some classes from > Curator's relocated Guava. > This has been fixed now, but the (compile scope) dependency for curator has > not been removed. > I propose removing the dependency in master, branch-3 and branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26611) Changing SFT implementation on disabled table is dangerous
[ https://issues.apache.org/jira/browse/HBASE-26611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26611: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Changing SFT implementation on disabled table is dangerous > -- > > Key: HBASE-26611 > URL: https://issues.apache.org/jira/browse/HBASE-26611 > Project: HBase > Issue Type: Sub-task > Components: HFile, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As our current design is to have a MIGRATION SFT so when region opens, we > will mirror the store files to the dst SFT. > But for a disabled table, the region will not be reopened so we have no > chance to mirror the store files list, so after user enable the table, it may > find out that all the store files are gone... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26611) Changing SFT implementation on disabled table is dangerous
[ https://issues.apache.org/jira/browse/HBASE-26611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828415#comment-17828415 ] Bryan Beaudreault commented on HBASE-26611: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Changing SFT implementation on disabled table is dangerous > -- > > Key: HBASE-26611 > URL: https://issues.apache.org/jira/browse/HBASE-26611 > Project: HBase > Issue Type: Sub-task > Components: HFile, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As our current design is to have a MIGRATION SFT so when region opens, we > will mirror the store files to the dst SFT. > But for a disabled table, the region will not be reopened so we have no > chance to mirror the store files list, so after user enable the table, it may > find out that all the store files are gone... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-25988) Store the store file list by a file
[ https://issues.apache.org/jira/browse/HBASE-25988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828406#comment-17828406 ] Bryan Beaudreault commented on HBASE-25988: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Store the store file list by a file > --- > > Key: HBASE-25988 > URL: https://issues.apache.org/jira/browse/HBASE-25988 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26079) Use StoreFileTracker when splitting and merging
[ https://issues.apache.org/jira/browse/HBASE-26079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26079: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Use StoreFileTracker when splitting and merging > --- > > Key: HBASE-26079 > URL: https://issues.apache.org/jira/browse/HBASE-26079 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26064) Introduce a StoreFileTracker to abstract the store file tracking logic
[ https://issues.apache.org/jira/browse/HBASE-26064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828405#comment-17828405 ] Bryan Beaudreault commented on HBASE-26064: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Introduce a StoreFileTracker to abstract the store file tracking logic > -- > > Key: HBASE-26064 > URL: https://issues.apache.org/jira/browse/HBASE-26064 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As mentioned in the parent issue, we could abstract a storefile metadata > system, then we could use different backend to store the storefile list, and > also it is possible to generalize the migration between different tracking > system. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26064) Introduce a StoreFileTracker to abstract the store file tracking logic
[ https://issues.apache.org/jira/browse/HBASE-26064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26064: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Introduce a StoreFileTracker to abstract the store file tracking logic > -- > > Key: HBASE-26064 > URL: https://issues.apache.org/jira/browse/HBASE-26064 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As mentioned in the parent issue, we could abstract a storefile metadata > system, then we could use different backend to store the storefile list, and > also it is possible to generalize the migration between different tracking > system. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-25988) Store the store file list by a file
[ https://issues.apache.org/jira/browse/HBASE-25988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-25988: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Store the store file list by a file > --- > > Key: HBASE-25988 > URL: https://issues.apache.org/jira/browse/HBASE-25988 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26079) Use StoreFileTracker when splitting and merging
[ https://issues.apache.org/jira/browse/HBASE-26079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828404#comment-17828404 ] Bryan Beaudreault commented on HBASE-26079: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Use StoreFileTracker when splitting and merging > --- > > Key: HBASE-26079 > URL: https://issues.apache.org/jira/browse/HBASE-26079 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26246) Persist the StoreFileTracker configurations to TableDescriptor when creating table
[ https://issues.apache.org/jira/browse/HBASE-26246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828402#comment-17828402 ] Bryan Beaudreault commented on HBASE-26246: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Persist the StoreFileTracker configurations to TableDescriptor when creating > table > -- > > Key: HBASE-26246 > URL: https://issues.apache.org/jira/browse/HBASE-26246 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As discussed in this section in the design doc: > https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.78r2mdeyquug > If we use different SFT implementation at master side and region server side, > it is likely to cause data loss, which is a very serious misconfiguration > problem. > A possible solution is to make sure that master and region server always load > the configurations about StoreEngine from the same place. To archive this, a > possible way is to always set the StoreEngine configurations to the > TableDescriptor, even if user does not explicitly set it when creating a > table. > And also, when upgrading, we should check whether the existing tables have > StoreEngine configurations, if not, we need to set them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26224) Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations
[ https://issues.apache.org/jira/browse/HBASE-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828403#comment-17828403 ] Bryan Beaudreault commented on HBASE-26224: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Introduce a MigrationStoreFileTracker to support migrating from different > store file tracker implementations > > > Key: HBASE-26224 > URL: https://issues.apache.org/jira/browse/HBASE-26224 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As mentioned in this section in the design doc: > https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.vv41957yf4i -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26248) Should find a suitable way to let users specify the store file tracker implementation
[ https://issues.apache.org/jira/browse/HBASE-26248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828401#comment-17828401 ] Bryan Beaudreault commented on HBASE-26248: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Should find a suitable way to let users specify the store file tracker > implementation > - > > Key: HBASE-26248 > URL: https://issues.apache.org/jira/browse/HBASE-26248 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Now all the implementations are marked as IA.Private, so there is no safe way > for users to specify them. > But for simplify, maybe we should not expose the full class name of the > implementation classes to users. Just follow the way in WALFactory, to > introduce an alias name for the implemention, for example > DEFAULT, FILE, REGION, etc. > Thoughts? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26263) [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables
[ https://issues.apache.org/jira/browse/HBASE-26263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828400#comment-17828400 ] Bryan Beaudreault commented on HBASE-26263: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > [Rolling Upgrading] Persist the StoreFileTracker configurations to > TableDescriptor for existing tables > -- > > Key: HBASE-26263 > URL: https://issues.apache.org/jira/browse/HBASE-26263 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Reporter: Duo Zhang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26246) Persist the StoreFileTracker configurations to TableDescriptor when creating table
[ https://issues.apache.org/jira/browse/HBASE-26246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26246: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Persist the StoreFileTracker configurations to TableDescriptor when creating > table > -- > > Key: HBASE-26246 > URL: https://issues.apache.org/jira/browse/HBASE-26246 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Reporter: Duo Zhang >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As discussed in this section in the design doc: > https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.78r2mdeyquug > If we use different SFT implementation at master side and region server side, > it is likely to cause data loss, which is a very serious misconfiguration > problem. > A possible solution is to make sure that master and region server always load > the configurations about StoreEngine from the same place. To archive this, a > possible way is to always set the StoreEngine configurations to the > TableDescriptor, even if user does not explicitly set it when creating a > table. > And also, when upgrading, we should check whether the existing tables have > StoreEngine configurations, if not, we need to set them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26224) Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations
[ https://issues.apache.org/jira/browse/HBASE-26224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26224: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Introduce a MigrationStoreFileTracker to support migrating from different > store file tracker implementations > > > Key: HBASE-26224 > URL: https://issues.apache.org/jira/browse/HBASE-26224 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As mentioned in this section in the design doc: > https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.vv41957yf4i -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26248) Should find a suitable way to let users specify the store file tracker implementation
[ https://issues.apache.org/jira/browse/HBASE-26248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26248: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Should find a suitable way to let users specify the store file tracker > implementation > - > > Key: HBASE-26248 > URL: https://issues.apache.org/jira/browse/HBASE-26248 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Now all the implementations are marked as IA.Private, so there is no safe way > for users to specify them. > But for simplify, maybe we should not expose the full class name of the > implementation classes to users. Just follow the way in WALFactory, to > introduce an alias name for the implemention, for example > DEFAULT, FILE, REGION, etc. > Thoughts? -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
Apache-HBase commented on PR #5764: URL: https://github.com/apache/hbase/pull/5764#issuecomment-2007454716 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 2m 58s | master passed | | +1 :green_heart: | compile | 3m 10s | master passed | | +1 :green_heart: | checkstyle | 0m 50s | master passed | | +1 :green_heart: | spotless | 0m 43s | branch has no errors when running spotless:check. | | +1 :green_heart: | spotbugs | 2m 9s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 49s | the patch passed | | +1 :green_heart: | compile | 3m 3s | the patch passed | | -0 :warning: | javac | 2m 27s | hbase-server generated 2 new + 193 unchanged - 2 fixed = 195 total (was 195) | | -0 :warning: | checkstyle | 0m 35s | hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 5m 9s | Patch does not cause any errors with Hadoop 3.3.6. | | -1 :x: | spotless | 0m 37s | patch has 40 errors when running spotless:check, run spotless:apply to fix. | | +1 :green_heart: | spotbugs | 3m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 19s | The patch does not generate ASF License warnings. | | | | 35m 44s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/5764 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile | | uname | Linux e41e43784673 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0763a74096 | | Default Java | Eclipse Adoptium-11.0.17+8 | | javac | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt | | checkstyle | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotless | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/artifact/yetus-general-check/output/patch-spotless.txt | | Max. process+thread count | 79 (vs. ulimit of 3) | | modules | C: hbase-client hbase-server U: . | | Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5764/2/console | | versions | git=2.34.1 maven=3.8.6 spotbugs=4.7.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26264) Add more checks to prevent misconfiguration on store file tracker
[ https://issues.apache.org/jira/browse/HBASE-26264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828399#comment-17828399 ] Bryan Beaudreault commented on HBASE-26264: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Add more checks to prevent misconfiguration on store file tracker > - > > Key: HBASE-26264 > URL: https://issues.apache.org/jira/browse/HBASE-26264 > Project: HBase > Issue Type: Sub-task > Components: conf, HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > For example, when creating new tables, if the store file tracker is > configured to MigrationStoreFileTracker, we should fail the creation and tell > users you should not use this store file tracker for a new table. > And when switching the implementation for store file tracker, we should let > end users first switch to MigrationStoreFileTracker, and then to the final > target store file tracker. If end users configure wrong store file tracker, > we should fail the alter table operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26263) [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables
[ https://issues.apache.org/jira/browse/HBASE-26263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26263: -- Fix Version/s: 2.5.0 (was: 2.6.0) > [Rolling Upgrading] Persist the StoreFileTracker configurations to > TableDescriptor for existing tables > -- > > Key: HBASE-26263 > URL: https://issues.apache.org/jira/browse/HBASE-26263 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Reporter: Duo Zhang >Assignee: Zhuoyue Huang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26271) Cleanup the broken store files under data directory
[ https://issues.apache.org/jira/browse/HBASE-26271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828398#comment-17828398 ] Bryan Beaudreault commented on HBASE-26271: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Cleanup the broken store files under data directory > --- > > Key: HBASE-26271 > URL: https://issues.apache.org/jira/browse/HBASE-26271 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As for some new store file tracker implementation, we allow flush/compaction > to write directly to data directory, so if we crash in the middle, there will > be broken store files left in the data directory. > We should find a proper way to delete these broken files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26264) Add more checks to prevent misconfiguration on store file tracker
[ https://issues.apache.org/jira/browse/HBASE-26264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26264: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Add more checks to prevent misconfiguration on store file tracker > - > > Key: HBASE-26264 > URL: https://issues.apache.org/jira/browse/HBASE-26264 > Project: HBase > Issue Type: Sub-task > Components: conf, HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > For example, when creating new tables, if the store file tracker is > configured to MigrationStoreFileTracker, we should fail the creation and tell > users you should not use this store file tracker for a new table. > And when switching the implementation for store file tracker, we should let > end users first switch to MigrationStoreFileTracker, and then to the final > target store file tracker. If end users configure wrong store file tracker, > we should fail the alter table operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26280) Use store file tracker when snapshoting
[ https://issues.apache.org/jira/browse/HBASE-26280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828397#comment-17828397 ] Bryan Beaudreault commented on HBASE-26280: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Use store file tracker when snapshoting > --- > > Key: HBASE-26280 > URL: https://issues.apache.org/jira/browse/HBASE-26280 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26271) Cleanup the broken store files under data directory
[ https://issues.apache.org/jira/browse/HBASE-26271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26271: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Cleanup the broken store files under data directory > --- > > Key: HBASE-26271 > URL: https://issues.apache.org/jira/browse/HBASE-26271 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As for some new store file tracker implementation, we allow flush/compaction > to write directly to data directory, so if we crash in the middle, there will > be broken store files left in the data directory. > We should find a proper way to delete these broken files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26280) Use store file tracker when snapshoting
[ https://issues.apache.org/jira/browse/HBASE-26280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26280: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Use store file tracker when snapshoting > --- > > Key: HBASE-26280 > URL: https://issues.apache.org/jira/browse/HBASE-26280 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26326) CreateTableProcedure fails when FileBasedStoreFileTracker is set in global config
[
https://issues.apache.org/jira/browse/HBASE-26326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828395#comment-17828395
]
Bryan Beaudreault commented on HBASE-26326:
---
Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there.
> CreateTableProcedure fails when FileBasedStoreFileTracker is set in global
> config
> -
>
> Key: HBASE-26326
> URL: https://issues.apache.org/jira/browse/HBASE-26326
> Project: HBase
> Issue Type: Sub-task
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> CreateTableProcedure tries to instantiate SFT impl in its preCreate method so
> that the given SFT impl can update the tbl descriptor with all its required
> configs. When global config is set to FILE, FileBasedStoreFileTracker
> currently needs store dir path at its construction time (via the passed
> StoreContext param), but we don't have this info at this stage, so
> FileBasedStoreFileTracker get's an NPE:
> {noformat}
> 2021-10-01 14:28:21,310 ERROR
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
> runtime exception: pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION,
> locked=true; CreateTableProcedure table=test-3
> java.lang.UnsupportedOperationException: Constructor threw an exception for
> org.apache.hadoop.hbase.regionserver.storefiletracker.MigrationStoreFileTracker
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:66)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:72)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory.create(StoreFileTrackerFactory.java:107)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory.persistTrackerConfig(StoreFileTrackerFactory.java:164)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.preCreate(CreateTableProcedure.java:274)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:102)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1662)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1409)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1979)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:58)
> ... 12 more
> Caused by: java.lang.UnsupportedOperationException: Constructor threw an
> exception for
> org.apache.hadoop.hbase.regionserver.storefiletracker.FileBasedStoreFileTracker
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:66)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:72)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory.createForMigration(StoreFileTrackerFactory.java:157)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.MigrationStoreFileTracker.(MigrationStoreFileTracker.java:50)
> ... 17 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:58)
> ... 20 more
> Caused by:
[jira] [Commented] (HBASE-26286) Add support for specifying store file tracker when restoring or cloning snapshot
[ https://issues.apache.org/jira/browse/HBASE-26286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828396#comment-17828396 ] Bryan Beaudreault commented on HBASE-26286: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Add support for specifying store file tracker when restoring or cloning > snapshot > > > Key: HBASE-26286 > URL: https://issues.apache.org/jira/browse/HBASE-26286 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Reporter: Duo Zhang >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As discussed in HBASE-26280. > https://issues.apache.org/jira/browse/HBASE-26280?focusedCommentId=17414894=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17414894 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26286) Add support for specifying store file tracker when restoring or cloning snapshot
[ https://issues.apache.org/jira/browse/HBASE-26286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26286: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Add support for specifying store file tracker when restoring or cloning > snapshot > > > Key: HBASE-26286 > URL: https://issues.apache.org/jira/browse/HBASE-26286 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Reporter: Duo Zhang >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As discussed in HBASE-26280. > https://issues.apache.org/jira/browse/HBASE-26280?focusedCommentId=17414894=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17414894 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26326) CreateTableProcedure fails when FileBasedStoreFileTracker is set in global config
[
https://issues.apache.org/jira/browse/HBASE-26326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-26326:
--
Fix Version/s: 2.5.0
(was: 2.6.0)
> CreateTableProcedure fails when FileBasedStoreFileTracker is set in global
> config
> -
>
> Key: HBASE-26326
> URL: https://issues.apache.org/jira/browse/HBASE-26326
> Project: HBase
> Issue Type: Sub-task
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> CreateTableProcedure tries to instantiate SFT impl in its preCreate method so
> that the given SFT impl can update the tbl descriptor with all its required
> configs. When global config is set to FILE, FileBasedStoreFileTracker
> currently needs store dir path at its construction time (via the passed
> StoreContext param), but we don't have this info at this stage, so
> FileBasedStoreFileTracker get's an NPE:
> {noformat}
> 2021-10-01 14:28:21,310 ERROR
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught
> runtime exception: pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION,
> locked=true; CreateTableProcedure table=test-3
> java.lang.UnsupportedOperationException: Constructor threw an exception for
> org.apache.hadoop.hbase.regionserver.storefiletracker.MigrationStoreFileTracker
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:66)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:72)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory.create(StoreFileTrackerFactory.java:107)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory.persistTrackerConfig(StoreFileTrackerFactory.java:164)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.preCreate(CreateTableProcedure.java:274)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:102)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:58)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1662)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1409)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1979)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:58)
> ... 12 more
> Caused by: java.lang.UnsupportedOperationException: Constructor threw an
> exception for
> org.apache.hadoop.hbase.regionserver.storefiletracker.FileBasedStoreFileTracker
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:66)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:72)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.StoreFileTrackerFactory.createForMigration(StoreFileTrackerFactory.java:157)
> at
> org.apache.hadoop.hbase.regionserver.storefiletracker.MigrationStoreFileTracker.(MigrationStoreFileTracker.java:50)
> ... 17 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:58)
> ... 20 more
> Caused by: java.lang.NullPointerException
> at
[jira] [Commented] (HBASE-26328) Clone snapshot doesn't load reference files into FILE SFT impl
[ https://issues.apache.org/jira/browse/HBASE-26328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828394#comment-17828394 ] Bryan Beaudreault commented on HBASE-26328: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Clone snapshot doesn't load reference files into FILE SFT impl > -- > > Key: HBASE-26328 > URL: https://issues.apache.org/jira/browse/HBASE-26328 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > After cloning a snapshot from a FILE SFT enabled table, noticed that none of > the cloned table files were added into the FILE SFT meta files, in fact, FILE > SFT meta dir didn't even got created. Scanning this cloned table gives no > results, as none of the files are tracked. I believe we need to call > StoreFileTracker.add during the snapshot cloning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26454) CreateTableProcedure still relies on temp dir and renames when creating table FS layout
[ https://issues.apache.org/jira/browse/HBASE-26454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26454: -- Fix Version/s: 2.5.0 (was: 2.6.0) > CreateTableProcedure still relies on temp dir and renames when creating table > FS layout > --- > > Key: HBASE-26454 > URL: https://issues.apache.org/jira/browse/HBASE-26454 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As pointed by [~elserj] when reviewing some other issue, internally, > CreateTableProcedure still creates the FS layout on temp folder first, then > rename to actual dir. I think this is unnecessary, we could create it > straight in the final dir, just like we did in [CloneSnapshotProcedure > here|https://github.com/apache/hbase/pull/3749/files#r728424375]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26477) Fix MiniMapReduceCluster failure under Java17
[
https://issues.apache.org/jira/browse/HBASE-26477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828391#comment-17828391
]
Bryan Beaudreault commented on HBASE-26477:
---
Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there.
> Fix MiniMapReduceCluster failure under Java17
> -
>
> Key: HBASE-26477
> URL: https://issues.apache.org/jira/browse/HBASE-26477
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> MiniMapReduceCluster in HBaseTestingUtil.java will not work. Find the error
> message in the container log:
> {code:java}
> Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make
> protected final java.lang.Class
> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
> throws java.lang.ClassFormatError accessible: module java.base does not
> "opens java.lang" to unnamed module @755c9148
> at
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
> ~[?:?]
> at
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
> ~[?:?]
> at java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
> ~[?:?]
> at java.lang.reflect.Method.setAccessible(Method.java:193) ~[?:?]
> at
> com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56)
> ~[guice-4.0.jar:?]
> at
> java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
> at
> com.google.inject.internal.cglib.core.$ReflectUtils.(ReflectUtils.java:46)
> ~[guice-4.0.jar:?]
> {code}
> The container of MiniMapReduceCluster is running in an independent JVM. So
> that we need to pass the JVM opts to the container JVM.
> Impacted UTs:
> TestSecureExportSnapshot
> TestMobSecureExportSnapshot
> TestVerifyReplicationCrossDiffHdfs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-26328) Clone snapshot doesn't load reference files into FILE SFT impl
[ https://issues.apache.org/jira/browse/HBASE-26328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26328: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Clone snapshot doesn't load reference files into FILE SFT impl > -- > > Key: HBASE-26328 > URL: https://issues.apache.org/jira/browse/HBASE-26328 > Project: HBase > Issue Type: Sub-task > Components: HFile, snapshots >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > After cloning a snapshot from a FILE SFT enabled table, noticed that none of > the cloned table files were added into the FILE SFT meta files, in fact, FILE > SFT meta dir didn't even got created. Scanning this cloned table gives no > results, as none of the files are tracked. I believe we need to call > StoreFileTracker.add during the snapshot cloning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26386) Refactor StoreFileTracker implementations to expose the set method
[ https://issues.apache.org/jira/browse/HBASE-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26386: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Refactor StoreFileTracker implementations to expose the set method > -- > > Key: HBASE-26386 > URL: https://issues.apache.org/jira/browse/HBASE-26386 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As when clone/restore a snapshot, we need to set the store file list directly > to StoreFileTracker, which means we need this method in upper layer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26386) Refactor StoreFileTracker implementations to expose the set method
[ https://issues.apache.org/jira/browse/HBASE-26386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828393#comment-17828393 ] Bryan Beaudreault commented on HBASE-26386: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Refactor StoreFileTracker implementations to expose the set method > -- > > Key: HBASE-26386 > URL: https://issues.apache.org/jira/browse/HBASE-26386 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As when clone/restore a snapshot, we need to set the store file list directly > to StoreFileTracker, which means we need this method in upper layer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26454) CreateTableProcedure still relies on temp dir and renames when creating table FS layout
[ https://issues.apache.org/jira/browse/HBASE-26454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828392#comment-17828392 ] Bryan Beaudreault commented on HBASE-26454: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > CreateTableProcedure still relies on temp dir and renames when creating table > FS layout > --- > > Key: HBASE-26454 > URL: https://issues.apache.org/jira/browse/HBASE-26454 > Project: HBase > Issue Type: Sub-task >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As pointed by [~elserj] when reviewing some other issue, internally, > CreateTableProcedure still creates the FS layout on temp folder first, then > rename to actual dir. I think this is unnecessary, we could create it > straight in the final dir, just like we did in [CloneSnapshotProcedure > here|https://github.com/apache/hbase/pull/3749/files#r728424375]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26477) Fix MiniMapReduceCluster failure under Java17
[
https://issues.apache.org/jira/browse/HBASE-26477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-26477:
--
Fix Version/s: 2.5.0
(was: 2.6.0)
> Fix MiniMapReduceCluster failure under Java17
> -
>
> Key: HBASE-26477
> URL: https://issues.apache.org/jira/browse/HBASE-26477
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> MiniMapReduceCluster in HBaseTestingUtil.java will not work. Find the error
> message in the container log:
> {code:java}
> Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make
> protected final java.lang.Class
> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
> throws java.lang.ClassFormatError accessible: module java.base does not
> "opens java.lang" to unnamed module @755c9148
> at
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
> ~[?:?]
> at
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
> ~[?:?]
> at java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
> ~[?:?]
> at java.lang.reflect.Method.setAccessible(Method.java:193) ~[?:?]
> at
> com.google.inject.internal.cglib.core.$ReflectUtils$2.run(ReflectUtils.java:56)
> ~[guice-4.0.jar:?]
> at
> java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
> at
> com.google.inject.internal.cglib.core.$ReflectUtils.(ReflectUtils.java:46)
> ~[guice-4.0.jar:?]
> {code}
> The container of MiniMapReduceCluster is running in an independent JVM. So
> that we need to pass the JVM opts to the container JVM.
> Impacted UTs:
> TestSecureExportSnapshot
> TestMobSecureExportSnapshot
> TestVerifyReplicationCrossDiffHdfs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-26530) Backport HBASE-26524 Support remove coprocessor by class name via alter table command
[ https://issues.apache.org/jira/browse/HBASE-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26530: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Backport HBASE-26524 Support remove coprocessor by class name via alter table > command > -- > > Key: HBASE-26530 > URL: https://issues.apache.org/jira/browse/HBASE-26530 > Project: HBase > Issue Type: Task > Components: Coprocessors, shell >Affects Versions: 2.5.0, 2.6.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > porting HBASE-26524 to branch-2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26530) Backport HBASE-26524 Support remove coprocessor by class name via alter table command
[ https://issues.apache.org/jira/browse/HBASE-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828390#comment-17828390 ] Bryan Beaudreault commented on HBASE-26530: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Backport HBASE-26524 Support remove coprocessor by class name via alter table > command > -- > > Key: HBASE-26530 > URL: https://issues.apache.org/jira/browse/HBASE-26530 > Project: HBase > Issue Type: Task > Components: Coprocessors, shell >Affects Versions: 2.5.0, 2.6.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > porting HBASE-26524 to branch-2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26578) CompactedHFilesDischarger should be shutdown when aborting regionserver
[ https://issues.apache.org/jira/browse/HBASE-26578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828389#comment-17828389 ] Bryan Beaudreault commented on HBASE-26578: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > CompactedHFilesDischarger should be shutdown when aborting regionserver > --- > > Key: HBASE-26578 > URL: https://issues.apache.org/jira/browse/HBASE-26578 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 3.0.0-alpha-1, 2.0.0 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > The CompactedHFilesDischarger chore service should be shutdown when > regionserver aborting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26578) CompactedHFilesDischarger should be shutdown when aborting regionserver
[ https://issues.apache.org/jira/browse/HBASE-26578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26578: -- Fix Version/s: 2.5.0 (was: 2.6.0) > CompactedHFilesDischarger should be shutdown when aborting regionserver > --- > > Key: HBASE-26578 > URL: https://issues.apache.org/jira/browse/HBASE-26578 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 3.0.0-alpha-1, 2.0.0 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > The CompactedHFilesDischarger chore service should be shutdown when > regionserver aborting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26587) Introduce a new Admin API to change SFT implementation
[ https://issues.apache.org/jira/browse/HBASE-26587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26587: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Introduce a new Admin API to change SFT implementation > -- > > Key: HBASE-26587 > URL: https://issues.apache.org/jira/browse/HBASE-26587 > Project: HBase > Issue Type: Sub-task > Components: Admin, API >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Now changing SFT implementation requires two MTP, and there are also lots of > rules on how to correctly set the MIGRATION SFT, I think we'd better > introduce a special method to implement the complicated logic by code,to make > our users life easier. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26585) Add SFT configuration to META table descriptor when creating META
[ https://issues.apache.org/jira/browse/HBASE-26585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26585: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Add SFT configuration to META table descriptor when creating META > - > > Key: HBASE-26585 > URL: https://issues.apache.org/jira/browse/HBASE-26585 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-2, 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > META doesn't get created via CreateTableProcedure, but rather during master > initialisation on cluster's first run, so changes implemented on HBASE-26246 > don't take effect for meta table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26587) Introduce a new Admin API to change SFT implementation
[ https://issues.apache.org/jira/browse/HBASE-26587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828388#comment-17828388 ] Bryan Beaudreault commented on HBASE-26587: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Introduce a new Admin API to change SFT implementation > -- > > Key: HBASE-26587 > URL: https://issues.apache.org/jira/browse/HBASE-26587 > Project: HBase > Issue Type: Sub-task > Components: Admin, API >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Now changing SFT implementation requires two MTP, and there are also lots of > rules on how to correctly set the MIGRATION SFT, I think we'd better > introduce a special method to implement the complicated logic by code,to make > our users life easier. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26585) Add SFT configuration to META table descriptor when creating META
[ https://issues.apache.org/jira/browse/HBASE-26585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828386#comment-17828386 ] Bryan Beaudreault commented on HBASE-26585: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Add SFT configuration to META table descriptor when creating META > - > > Key: HBASE-26585 > URL: https://issues.apache.org/jira/browse/HBASE-26585 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-2, 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > META doesn't get created via CreateTableProcedure, but rather during master > initialisation on cluster's first run, so changes implemented on HBASE-26246 > don't take effect for meta table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26639) The implementation of TestMergesSplitsAddToTracker is problematic
[ https://issues.apache.org/jira/browse/HBASE-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828385#comment-17828385 ] Bryan Beaudreault commented on HBASE-26639: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > The implementation of TestMergesSplitsAddToTracker is problematic > - > > Key: HBASE-26639 > URL: https://issues.apache.org/jira/browse/HBASE-26639 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As it will clear the trackedFiles map in TestStoreFileTracker for every test > but we will also store the files for system tables, so it will cause NPE. Not > sure why on master the it will not fail but on branch-2 at least > testSplitLoadsFromTracker will fail... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26586) Should not rely on the global config when setting SFT implementation for a table while upgrading
[ https://issues.apache.org/jira/browse/HBASE-26586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26586: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Should not rely on the global config when setting SFT implementation for a > table while upgrading > > > Key: HBASE-26586 > URL: https://issues.apache.org/jira/browse/HBASE-26586 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3 > > > If a table does not have a SFT implementation in its table descriptor, it > means we are upgrading from an older version where SFT is not implemented, so > the SFT implementation for the table should always be DEFAULT. > A misconfigure here could cause data loss. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26586) Should not rely on the global config when setting SFT implementation for a table while upgrading
[ https://issues.apache.org/jira/browse/HBASE-26586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828384#comment-17828384 ] Bryan Beaudreault commented on HBASE-26586: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Should not rely on the global config when setting SFT implementation for a > table while upgrading > > > Key: HBASE-26586 > URL: https://issues.apache.org/jira/browse/HBASE-26586 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3 > > > If a table does not have a SFT implementation in its table descriptor, it > means we are upgrading from an older version where SFT is not implemented, so > the SFT implementation for the table should always be DEFAULT. > A misconfigure here could cause data loss. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26639) The implementation of TestMergesSplitsAddToTracker is problematic
[ https://issues.apache.org/jira/browse/HBASE-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26639: -- Fix Version/s: 2.5.0 (was: 2.6.0) > The implementation of TestMergesSplitsAddToTracker is problematic > - > > Key: HBASE-26639 > URL: https://issues.apache.org/jira/browse/HBASE-26639 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > As it will clear the trackedFiles map in TestStoreFileTracker for every test > but we will also store the files for system tables, so it will cause NPE. Not > sure why on master the it will not fail but on branch-2 at least > testSplitLoadsFromTracker will fail... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26654) ModifyTableDescriptorProcedure shoud load TableDescriptor while executing
[ https://issues.apache.org/jira/browse/HBASE-26654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828383#comment-17828383 ] Bryan Beaudreault commented on HBASE-26654: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > ModifyTableDescriptorProcedure shoud load TableDescriptor while executing > - > > Key: HBASE-26654 > URL: https://issues.apache.org/jira/browse/HBASE-26654 > Project: HBase > Issue Type: Bug > Components: master, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Found this problem when implementing HBASE-26587. We should not pass the old > table descriptor when constructing the procedure, as we may have some other > MTPs before us so when executing, the table descriptor may have already been > changed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26654) ModifyTableDescriptorProcedure shoud load TableDescriptor while executing
[ https://issues.apache.org/jira/browse/HBASE-26654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26654: -- Fix Version/s: 2.5.0 (was: 2.6.0) > ModifyTableDescriptorProcedure shoud load TableDescriptor while executing > - > > Key: HBASE-26654 > URL: https://issues.apache.org/jira/browse/HBASE-26654 > Project: HBase > Issue Type: Bug > Components: master, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Found this problem when implementing HBASE-26587. We should not pass the old > table descriptor when constructing the procedure, as we may have some other > MTPs before us so when executing, the table descriptor may have already been > changed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26674) Should modify filesCompacting under storeWriteLock
[
https://issues.apache.org/jira/browse/HBASE-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-26674:
--
Fix Version/s: 2.5.0
(was: 2.6.0)
> Should modify filesCompacting under storeWriteLock
> --
>
> Key: HBASE-26674
> URL: https://issues.apache.org/jira/browse/HBASE-26674
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> The stacktrace
> {noformat}
> java.lang.IllegalArgumentException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.getCurrentEligibleFiles(SortedCompactionPolicy.java:173)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.preSelectCompactionForCoprocessor(SortedCompactionPolicy.java:44)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.preSelect(DefaultStoreEngine.java:130)
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1438)
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1419)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2238)
> at
> org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver.test(TestWriteHeavyIncrementObserver.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:38)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Haven't seen it flaky before. And this is a runtime exception in our non test
> code base, which seems critical.
> Not sure if it has the same root cause with HBASE-26644.
> Need to dig more.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26675) Data race on Compactor.writer
[ https://issues.apache.org/jira/browse/HBASE-26675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828381#comment-17828381 ] Bryan Beaudreault commented on HBASE-26675: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Data race on Compactor.writer > - > > Key: HBASE-26675 > URL: https://issues.apache.org/jira/browse/HBASE-26675 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > In Compactor.getCompactionTargets, we use synchronized on the writer instance > but it does not help resolve the data race problem, because in other places > we do not use synchronized... > And since it is not a big deal to the upper layer an old snapshot, we could > just make the writer instance volatile and use it directly in the method, > just make sure we always use the same instance in the method, i.e, assign it > to a local var first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26675) Data race on Compactor.writer
[ https://issues.apache.org/jira/browse/HBASE-26675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26675: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Data race on Compactor.writer > - > > Key: HBASE-26675 > URL: https://issues.apache.org/jira/browse/HBASE-26675 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > In Compactor.getCompactionTargets, we use synchronized on the writer instance > but it does not help resolve the data race problem, because in other places > we do not use synchronized... > And since it is not a big deal to the upper layer an old snapshot, we could > just make the writer instance volatile and use it directly in the method, > just make sure we always use the same instance in the method, i.e, assign it > to a local var first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26674) Should modify filesCompacting under storeWriteLock
[
https://issues.apache.org/jira/browse/HBASE-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828382#comment-17828382
]
Bryan Beaudreault commented on HBASE-26674:
---
Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there.
> Should modify filesCompacting under storeWriteLock
> --
>
> Key: HBASE-26674
> URL: https://issues.apache.org/jira/browse/HBASE-26674
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3
>
>
> The stacktrace
> {noformat}
> java.lang.IllegalArgumentException
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:131)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.getCurrentEligibleFiles(SortedCompactionPolicy.java:173)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.preSelectCompactionForCoprocessor(SortedCompactionPolicy.java:44)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.preSelect(DefaultStoreEngine.java:130)
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1438)
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1419)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2238)
> at
> org.apache.hadoop.hbase.coprocessor.example.TestWriteHeavyIncrementObserver.test(TestWriteHeavyIncrementObserver.java:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:38)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Haven't seen it flaky before. And this is a runtime exception in our non test
> code base, which seems critical.
> Not sure if it has the same root cause with HBASE-26644.
> Need to dig more.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-26690) Modify FSTableDescriptors to not rely on renaming when writing TableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-26690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26690: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Modify FSTableDescriptors to not rely on renaming when writing TableDescriptor > -- > > Key: HBASE-26690 > URL: https://issues.apache.org/jira/browse/HBASE-26690 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26700) The way we bypass broken track file is not enough in StoreFileListFile
[ https://issues.apache.org/jira/browse/HBASE-26700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828378#comment-17828378 ] Bryan Beaudreault commented on HBASE-26700: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > The way we bypass broken track file is not enough in StoreFileListFile > -- > > Key: HBASE-26700 > URL: https://issues.apache.org/jira/browse/HBASE-26700 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3 > > > When implementing HBASE-26690, I found that a pb message can be parsed with > partial bytes, if we stop at some special point. > So we can not rely on the pb parser to always throw > InvalidProtocolBufferException for us, we need to detect the incomplete file > by ourselves. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26690) Modify FSTableDescriptors to not rely on renaming when writing TableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-26690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828379#comment-17828379 ] Bryan Beaudreault commented on HBASE-26690: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Modify FSTableDescriptors to not rely on renaming when writing TableDescriptor > -- > > Key: HBASE-26690 > URL: https://issues.apache.org/jira/browse/HBASE-26690 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26700) The way we bypass broken track file is not enough in StoreFileListFile
[ https://issues.apache.org/jira/browse/HBASE-26700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26700: -- Fix Version/s: 2.5.0 (was: 2.6.0) > The way we bypass broken track file is not enough in StoreFileListFile > -- > > Key: HBASE-26700 > URL: https://issues.apache.org/jira/browse/HBASE-26700 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.5.0, 3.0.0-alpha-3 > > > When implementing HBASE-26690, I found that a pb message can be parsed with > partial bytes, if we stop at some special point. > So we can not rely on the pb parser to always throw > InvalidProtocolBufferException for us, we need to detect the incomplete file > by ourselves. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26707) Reduce number of renames during bulkload
[ https://issues.apache.org/jira/browse/HBASE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26707: -- Fix Version/s: 2.5.0 > Reduce number of renames during bulkload > > > Key: HBASE-26707 > URL: https://issues.apache.org/jira/browse/HBASE-26707 > Project: HBase > Issue Type: Sub-task >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Make sure we only do a single rename operation during bulkload when > StoreEngine does not require the the use of tmp directories. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26837) Set SFT config when creating TableDescriptor in TestCloneSnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828375#comment-17828375 ] Bryan Beaudreault commented on HBASE-26837: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Set SFT config when creating TableDescriptor in TestCloneSnapshotProcedure > -- > > Key: HBASE-26837 > URL: https://issues.apache.org/jira/browse/HBASE-26837 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > See posts in HBASE-26836. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26707) Reduce number of renames during bulkload
[ https://issues.apache.org/jira/browse/HBASE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828377#comment-17828377 ] Bryan Beaudreault commented on HBASE-26707: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Reduce number of renames during bulkload > > > Key: HBASE-26707 > URL: https://issues.apache.org/jira/browse/HBASE-26707 > Project: HBase > Issue Type: Sub-task >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > Make sure we only do a single rename operation during bulkload when > StoreEngine does not require the the use of tmp directories. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26707) Reduce number of renames during bulkload
[ https://issues.apache.org/jira/browse/HBASE-26707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26707: -- Fix Version/s: (was: 2.6.0) > Reduce number of renames during bulkload > > > Key: HBASE-26707 > URL: https://issues.apache.org/jira/browse/HBASE-26707 > Project: HBase > Issue Type: Sub-task >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Make sure we only do a single rename operation during bulkload when > StoreEngine does not require the the use of tmp directories. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26881) Backport HBASE-25368 to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828374#comment-17828374 ] Bryan Beaudreault commented on HBASE-26881: --- Replaced 2.6.0 fixVersion with 2.5.0, since it first landed there. > Backport HBASE-25368 to branch-2 > - > > Key: HBASE-26881 > URL: https://issues.apache.org/jira/browse/HBASE-26881 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.5.0, 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0 > > > HBASE-26640 introduced two extra paths under master:store table: > ".initializing" and ".initialized", in order to control when such store has > been completed started for SFT. > Problem is that TestHFileProcedurePrettyPrinter uses > RegionInfo.isEncodedRegionName to determine if a given child path in the > table dir is a valid region dir. Current implementation for > RegionInfo.isEncodedRegionName considers ".initializing" and ".initialized" > as valid region encoded names, thus the test ends up picking one of the flag > dirs to list hfiles that should had been modified when validating the test > outcome. > Further improvements have been made to RegionInfo.isEncodedRegionName in > HBASE-25368 to proper validate region names, but those weren't backported to > branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26881) Backport HBASE-25368 to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26881: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Backport HBASE-25368 to branch-2 > - > > Key: HBASE-26881 > URL: https://issues.apache.org/jira/browse/HBASE-26881 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.5.0, 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 2.5.0 > > > HBASE-26640 introduced two extra paths under master:store table: > ".initializing" and ".initialized", in order to control when such store has > been completed started for SFT. > Problem is that TestHFileProcedurePrettyPrinter uses > RegionInfo.isEncodedRegionName to determine if a given child path in the > table dir is a valid region dir. Current implementation for > RegionInfo.isEncodedRegionName considers ".initializing" and ".initialized" > as valid region encoded names, thus the test ends up picking one of the flag > dirs to list hfiles that should had been modified when validating the test > outcome. > Further improvements have been made to RegionInfo.isEncodedRegionName in > HBASE-25368 to proper validate region names, but those weren't backported to > branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-26837) Set SFT config when creating TableDescriptor in TestCloneSnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-26837: -- Fix Version/s: 2.5.0 (was: 2.6.0) > Set SFT config when creating TableDescriptor in TestCloneSnapshotProcedure > -- > > Key: HBASE-26837 > URL: https://issues.apache.org/jira/browse/HBASE-26837 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-3 > > > See posts in HBASE-26836. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[ https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28415: -- Resolution: Fixed Status: Resolved (was: Patch Available) In fact, I resolve it now without the 2.5.9 fixVersion. But Andrew feel free to reopen/cherry-pick if you think it makes sense. > Remove Curator dependency from hbase-endpoint > - > > Key: HBASE-28415 > URL: https://issues.apache.org/jira/browse/HBASE-28415 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > TestRpcControllerFactory used to erronously import some classes from > Curator's relocated Guava. > This has been fixed now, but the (compile scope) dependency for curator has > not been removed. > I propose removing the dependency in master, branch-3 and branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[ https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28415: -- Fix Version/s: (was: 2.5.9) > Remove Curator dependency from hbase-endpoint > - > > Key: HBASE-28415 > URL: https://issues.apache.org/jira/browse/HBASE-28415 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > TestRpcControllerFactory used to erronously import some classes from > Curator's relocated Guava. > This has been fixed now, but the (compile scope) dependency for curator has > not been removed. > I propose removing the dependency in master, branch-3 and branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[ https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28415: -- Fix Version/s: (was: 2.4.18) > Remove Curator dependency from hbase-endpoint > - > > Key: HBASE-28415 > URL: https://issues.apache.org/jira/browse/HBASE-28415 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2, 2.5.9 > > > TestRpcControllerFactory used to erronously import some classes from > Curator's relocated Guava. > This has been fixed now, but the (compile scope) dependency for curator has > not been removed. > I propose removing the dependency in master, branch-3 and branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-28415) Remove Curator dependency from hbase-endpoint
[ https://issues.apache.org/jira/browse/HBASE-28415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828366#comment-17828366 ] Bryan Beaudreault commented on HBASE-28415: --- I'm not sure if this should go back to patch releases. I've merged the branch-2 and branch-3 backports, and have cherry-picked it to branch-2.6. I'm trying to audit our fixVersions to ensure that anything with 2.6.0 fixVersion exists in branch-2.6. I don't think we're doing 2.4.x releases anymore, so I removed that fixVersion. [~apurtell] do you want this in 2.5.9? If so, we can cherry-pick and resolve as-is. Or we can remove that fixVersion and resolve. > Remove Curator dependency from hbase-endpoint > - > > Key: HBASE-28415 > URL: https://issues.apache.org/jira/browse/HBASE-28415 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-beta-1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2, 2.5.9 > > > TestRpcControllerFactory used to erronously import some classes from > Curator's relocated Guava. > This has been fixed now, but the (compile scope) dependency for curator has > not been removed. > I propose removing the dependency in master, branch-3 and branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28415 Remove Curator dependency from hbase-endpoint (#5738) [hbase]
bbeaudreault commented on PR #5749: URL: https://github.com/apache/hbase/pull/5749#issuecomment-2007365062 I've cherry-picked this to branch-2.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28415 Remove Curator dependency from hbase-endpoint (#5738) [hbase]
bbeaudreault merged PR #5749: URL: https://github.com/apache/hbase/pull/5749 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28415 Remove Curator dependency from hbase-endpoint (#5738) [hbase]
bbeaudreault merged PR #5750: URL: https://github.com/apache/hbase/pull/5750 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-28186) Rebase CacheAwareBalance related commits into master branch
[ https://issues.apache.org/jira/browse/HBASE-28186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828363#comment-17828363 ] Bryan Beaudreault commented on HBASE-28186: --- [~wchevreuil] I've removed the fixVersions from this JIRA. Let me know if that's an issue for you. I don't think it's useful to include this Jira in the changes.md because it's just marking the rebasing of the real feature in HBASE-27999. > Rebase CacheAwareBalance related commits into master branch > --- > > Key: HBASE-28186 > URL: https://issues.apache.org/jira/browse/HBASE-28186 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6, 4.0.0-alpha-1 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28186) Rebase CacheAwareBalance related commits into master branch
[ https://issues.apache.org/jira/browse/HBASE-28186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28186: -- Fix Version/s: (was: 2.6.0) (was: 3.0.0-beta-1) > Rebase CacheAwareBalance related commits into master branch > --- > > Key: HBASE-28186 > URL: https://issues.apache.org/jira/browse/HBASE-28186 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.6, 4.0.0-alpha-1 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28124) Missing fields in Scan.toJSON
[ https://issues.apache.org/jira/browse/HBASE-28124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28124. --- Fix Version/s: 2.6.0 2.4.18 3.0.0-beta-2 2.5.9 Hadoop Flags: Reviewed Resolution: Fixed Pushed to all active branches. Thanks [~chandrasekhar.k] for contributing! > Missing fields in Scan.toJSON > - > > Key: HBASE-28124 > URL: https://issues.apache.org/jira/browse/HBASE-28124 > Project: HBase > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Chandra Sekhar K >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 2.4.18, 3.0.0-beta-2, 2.5.9 > > > Let's do an audit and add whatever's missing. I noticed getLimit is missing > at least. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27110) Move replication peer storage from zookeeper to other storage systems
[ https://issues.apache.org/jira/browse/HBASE-27110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828360#comment-17828360 ] Bryan Beaudreault commented on HBASE-27110: --- Nevermind. This Jira is not an umbrella, but it was treated as such. I see the code was committed for different jiras. All good here > Move replication peer storage from zookeeper to other storage systems > - > > Key: HBASE-27110 > URL: https://issues.apache.org/jira/browse/HBASE-27110 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > This is a more specific issue based on the works which are already done in > HBASE-15867. > There are several candidates for storing replication peer. > 1. A new family of master local region > 2. On FileSystem > 3. A new family of hbase:meta > Either choice has pros and cons. We need to decide which way to go first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27110) Move replication peer storage from zookeeper to other storage systems
[ https://issues.apache.org/jira/browse/HBASE-27110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828359#comment-17828359 ] Bryan Beaudreault commented on HBASE-27110: --- [~zhangduo] I see fixVersion has 2.6.0, but I don't see this code in branch-2 or branch-2.6. Was that an error? > Move replication peer storage from zookeeper to other storage systems > - > > Key: HBASE-27110 > URL: https://issues.apache.org/jira/browse/HBASE-27110 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > This is a more specific issue based on the works which are already done in > HBASE-15867. > There are several candidates for storing replication peer. > 1. A new family of master local region > 2. On FileSystem > 3. A new family of hbase:meta > Either choice has pros and cons. We need to decide which way to go first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
guluo2016 commented on PR #5764: URL: https://github.com/apache/hbase/pull/5764#issuecomment-2007340839 Updating modifiers from public to private, and triggering test case running again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] HBASE-28443 Return too slow when scanning a table with non-existing REGION_REPLICA_ID [hbase]
guluo2016 commented on PR #5764: URL: https://github.com/apache/hbase/pull/5764#issuecomment-2007331599 Strange，the expired test case was been executed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-28442) Assertion failure in NewVersionBehaviorTracker during major compaction
[
https://issues.apache.org/jira/browse/HBASE-28442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Beaudreault updated HBASE-28442:
--
Fix Version/s: (was: 2.6.0)
> Assertion failure in NewVersionBehaviorTracker during major compaction
> --
>
> Key: HBASE-28442
> URL: https://issues.apache.org/jira/browse/HBASE-28442
> Project: HBase
> Issue Type: Bug
>Reporter: Kadir Ozdemir
>Priority: Major
> Attachments:
> Test-for-the-assert-failure-in-NewVersionBehaviorTra.patch
>
>
> The attached test inserts put, and delete mutations that include all
> types of delete markers. These mutations are written to multiple files using
> multiple memstore flushes. Then the test runs minor compaction, and finally
> major compaction. When KeepDeleteCells is set to true and new version
> behavior is enabled, the test fails with the following assertion failure:
> {code:java}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.regionserver.querymatcher.NewVersionBehaviorTracker.checkVersions(NewVersionBehaviorTracker.java:310)
> at
> org.apache.hadoop.hbase.regionserver.querymatcher.MajorCompactionScanQueryMatcher.match(MajorCompactionScanQueryMatcher.java:78)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:640)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:446)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:367)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:62)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1177)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2407)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2262)
> at
> org.apache.hadoop.hbase.regionserver.TestStoreFileWriter.testCompactedFiles(TestStoreFileWriter.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748) {code}
> The attached
> [^Test-for-the-assert-failure-in-NewVersionBehaviorTra.patch]test is a
> simplified version of the test in the
[jira] [Commented] (HBASE-28442) Assertion failure in NewVersionBehaviorTracker during major compaction
[
https://issues.apache.org/jira/browse/HBASE-28442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828343#comment-17828343
]
Bryan Beaudreault commented on HBASE-28442:
---
Removed 2.6.0 fixVersion
> Assertion failure in NewVersionBehaviorTracker during major compaction
> --
>
> Key: HBASE-28442
> URL: https://issues.apache.org/jira/browse/HBASE-28442
> Project: HBase
> Issue Type: Bug
>Reporter: Kadir Ozdemir
>Priority: Major
> Attachments:
> Test-for-the-assert-failure-in-NewVersionBehaviorTra.patch
>
>
> The attached test inserts put, and delete mutations that include all
> types of delete markers. These mutations are written to multiple files using
> multiple memstore flushes. Then the test runs minor compaction, and finally
> major compaction. When KeepDeleteCells is set to true and new version
> behavior is enabled, the test fails with the following assertion failure:
> {code:java}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.regionserver.querymatcher.NewVersionBehaviorTracker.checkVersions(NewVersionBehaviorTracker.java:310)
> at
> org.apache.hadoop.hbase.regionserver.querymatcher.MajorCompactionScanQueryMatcher.match(MajorCompactionScanQueryMatcher.java:78)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:640)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:446)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:367)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:62)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:122)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1177)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2407)
> at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2262)
> at
> org.apache.hadoop.hbase.regionserver.TestStoreFileWriter.testCompactedFiles(TestStoreFileWriter.java:187)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748) {code}
> The attached
> [^Test-for-the-assert-failure-in-NewVersionBehaviorTra.patch]test is a
>
[jira] [Resolved] (HBASE-28338) Bounded leak of FSDataInputStream buffers from checksum switching
[ https://issues.apache.org/jira/browse/HBASE-28338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault resolved HBASE-28338. --- Fix Version/s: 2.6.0 3.0.0-beta-2 Resolution: Fixed > Bounded leak of FSDataInputStream buffers from checksum switching > - > > Key: HBASE-28338 > URL: https://issues.apache.org/jira/browse/HBASE-28338 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.6.0, 3.0.0-beta-2 > > > In FSDataInputStreamWrapper, the unbuffer() method caches an unbuffer > instance the first time it is called. When an FSDataInputStreamWrapper is > initialized, it has hbase checksum disabled. > In HFileInfo.initTrailerAndContext we get the stream, read the trailer, then > call unbuffer. At this point, checksums have not been enabled yet via > prepareForBlockReader. So the call to unbuffer() caches the current > non-checksum stream as the unbuffer instance. > Later, in initMetaAndIndex we do a similar thing. This time, > prepareForBlockReader has been called, so we are now using hbase checksums. > When initMetaAndIndex calls unbuffer(), it uses the old unbuffer instance > which actually has been closed when we switched to hbase checksums. So that > call does nothing, and the new no-checksum input stream is never unbuffered. > I haven't seen this cause an issue with normal hdfs replication (though > haven't gone looking). It's very problematic for Erasure Coding because > DFSStripedInputStream holds a large buffer (numDataBlocks * cellSize, so 6mb > for RS-6-3-1024k) that is only used for stream reads NOT pread. The > FSDataInputStreamWrapper we are talking about here is only used for pread in > hbase, so those 6mb buffers just hang around totally unused but > unreclaimable. Since there is an input stream per StoreFile, this can add up > very quickly on big servers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] HBASE-28124 Missing fields in Scan.toJSON [hbase]
Apache9 merged PR #5678: URL: https://github.com/apache/hbase/pull/5678 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
