[jira] [Created] (HBASE-28386) Potential Data loss issue for the HFLog
Dong0829 created HBASE-28386: Summary: Potential Data loss issue for the HFLog Key: HBASE-28386 URL: https://issues.apache.org/jira/browse/HBASE-28386 Project: HBase Issue Type: Bug Components: wal Reporter: Dong0829 Attachments: Screenshot 2024-02-19 at 10.53.20.png I am going through the WAL implementation HFLog, and based on the code reading, as my understanding there maybe potential data loss for example in below scenarios: # Append record A3 already into the stream, and but the syncFuture S3 does not reach to the SyncRunner yet # The SyncRunner run the sync and call sync on writer, but failed, all the data in stream lost including A3 # Because the syncFuture S3 does not reach to the syncRunner, it can not release it and mark it failed # In next run() of the SyncRunner, the new sync() succeeded, it will release and mark S3 as succeeded, but actually it should failed which cause data loss. Can anyone help to check above above is true? Am I missing anything regarding this process? !Screenshot 2024-02-19 at 10.53.20.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HBASE-27614) Region Reopen failure when the openNum has issue
[
https://issues.apache.org/jira/browse/HBASE-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684396#comment-17684396
]
Dong0829 edited comment on HBASE-27614 at 2/6/23 3:32 AM:
--
Thanks [~zhangduo] for looking into this
Maybe I did not explain the issue clearly, changing TTL does NOT cause the
seqNum go back. The context is:
# For some reason(we suspect its the WAL data loss or HFile loss during HBase
migrate), the seqNum during open region was much smaller than
{{seqnumDuringOpen in the meta}}
# Above mismatch cause the reopen keep re-opening the same region even its
already opened successfully
If the "The seqNum should not go backward", then as I said, for the open
procedure([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]),
should we consider the open succeeded if the openSeqNum is smaller than the
one in region state?
I am using hbase-2.4.13, you mean instead of the set the seqNum based on the
wal and hfile, now there is a separate file to tracking the max sequence id? If
yes, may I know from which version?
was (Author: li0829):
Thanks [~zhangduo] for looking into this
Maybe I did not explain the issue clearly, changing TTL does NOT cause the
seqNum go back. The context is:
# For some reason(we suspect its the WAL data loss or HFile log during HBase
migrate), the seqNum during open region was much smaller than
{{seqnumDuringOpen in the meta}}
# Above mismatch cause the reopen keep re-opening the same region even its
already opened successfully
If the "The seqNum should not go backward", then as I said, for the open
procedure([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]),
should we consider the open succeeded if the openSeqNum is smaller than the
one in region state?
I am using hbase-2.4.13, you mean instead of the set the seqNum based on the
wal and hfile, now there is a separate file to tracking the max sequence id? If
yes, may I know from which version?
> Region Reopen failure when the openNum has issue
>
>
> Key: HBASE-27614
> URL: https://issues.apache.org/jira/browse/HBASE-27614
> Project: HBase
> Issue Type: Bug
>Reporter: Dong0829
>Assignee: Dong0829
>Priority: Major
>
> We faced the issue when change the TTL for the hbase table and a lot of
> regions keep reopen and tons of TRSP created, after troubleshooting, we found
> some logic issue for the region reopen procedure logic.
> In the reopen process, it will check the seqNum to confirm if the region
> reopened successfully or not. If the seqNum accident become bigger than the
> current HFile and WAL (because of the data loss), there will be issue and
> unnecessary loop for the region close/open
>
> We should be able to optimize the logic, more details
> For this regionOpenedWithoutPersistingToMeta, should we just update the
> OpenSeqNum when the new one is bigger than the old one?
> As the region already opened, we should update the OpenSeqNum no matter its
> bigger or smaller, otherwise, we should not just return WARN but failed the
> open, right?
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
>
> Above does matter because for the
> checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
> if the seq is smaller, the region will be returned and keep reopening. So
> we should either update the logic in regionOpenedWithoutPersistingToMeta or
> checkReopened to make sure the region reopen works properly if the seqNum has
> issue
>
>
> Reproduce steps:
>
> 1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
> {{create 'test', 'info'}}
> {{put 'test', 'fool', 'info:cat', 'test'}}
> {{2. Manually update one region row for this test table in hbase:meta on the
> column, for example:}}
> {{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
> 'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
> 3. Modify the table TTL :
> alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
>
> You will see the region keep reopening
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-27614) Region Reopen failure when the openNum has issue
[
https://issues.apache.org/jira/browse/HBASE-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684396#comment-17684396
]
Dong0829 commented on HBASE-27614:
--
Thanks [~zhangduo] for looking into this
Maybe I did not explain the issue clearly, changing TTL does NOT cause the
seqNum go back. The context is:
# For some reason(we suspect its the WAL data loss or HFile log during HBase
migrate), the seqNum during open region was much smaller than
{{seqnumDuringOpen in the meta}}
# Above mismatch cause the reopen keep re-opening the same region even its
already opened successfully
If the "The seqNum should not go backward", then as I said, for the open
procedure([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]),
should we consider the open succeeded if the openSeqNum is smaller than the
one in region state?
I am using hbase-2.4.13, you mean instead of the set the seqNum based on the
wal and hfile, now there is a separate file to tracking the max sequence id? If
yes, may I know from which version?
> Region Reopen failure when the openNum has issue
>
>
> Key: HBASE-27614
> URL: https://issues.apache.org/jira/browse/HBASE-27614
> Project: HBase
> Issue Type: Bug
>Reporter: Dong0829
>Assignee: Dong0829
>Priority: Major
>
> We faced the issue when change the TTL for the hbase table and a lot of
> regions keep reopen and tons of TRSP created, after troubleshooting, we found
> some logic issue for the region reopen procedure logic.
> In the reopen process, it will check the seqNum to confirm if the region
> reopened successfully or not. If the seqNum accident become bigger than the
> current HFile and WAL (because of the data loss), there will be issue and
> unnecessary loop for the region close/open
>
> We should be able to optimize the logic, more details
> For this regionOpenedWithoutPersistingToMeta, should we just update the
> OpenSeqNum when the new one is bigger than the old one?
> As the region already opened, we should update the OpenSeqNum no matter its
> bigger or smaller, otherwise, we should not just return WARN but failed the
> open, right?
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
>
> Above does matter because for the
> checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
> if the seq is smaller, the region will be returned and keep reopening. So
> we should either update the logic in regionOpenedWithoutPersistingToMeta or
> checkReopened to make sure the region reopen works properly if the seqNum has
> issue
>
>
> Reproduce steps:
>
> 1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
> {{create 'test', 'info'}}
> {{put 'test', 'fool', 'info:cat', 'test'}}
> {{2. Manually update one region row for this test table in hbase:meta on the
> column, for example:}}
> {{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
> 'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
> 3. Modify the table TTL :
> alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
>
> You will see the region keep reopening
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-27614) Region Reopen failure when the openNum has issue
[
https://issues.apache.org/jira/browse/HBASE-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong0829 updated HBASE-27614:
-
Description:
We faced the issue when change the TTL for the hbase table and a lot of regions
keep reopen and tons of TRSP created, after troubleshooting, we found some
logic issue for the region reopen procedure logic.
In the reopen process, it will check the seqNum to confirm if the region
reopened successfully or not. If the seqNum accident become bigger than the
current HFile and WAL (because of the data loss), there will be issue and
unnecessary loop for the region close/open
We should be able to optimize the logic, more details
For this regionOpenedWithoutPersistingToMeta, should we just update the
OpenSeqNum when the new one is bigger than the old one?
As the region already opened, we should update the OpenSeqNum no matter its
bigger or smaller, otherwise, we should not just return WARN but failed the
open, right?
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
Above does matter because for the
checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
if the seq is smaller, the region will be returned and keep reopening. So we
should either update the logic in regionOpenedWithoutPersistingToMeta or
checkReopened to make sure the region reopen works properly if the seqNum has
issue
Reproduce steps:
1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
{{create 'test', 'info'}}
{{put 'test', 'fool', 'info:cat', 'test'}}
{{2. Manually update one region row for this test table in hbase:meta on the
column, for example:}}
{{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
3. Modify the table TTL :
alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
You will see the region keep reopening
was:
We faced the issue when change the TTL for the hbase table and a lot of regions
keep reopen and tons of TRSP created, after troubleshooting, we found some
issue for the region reopen procedure logic.
In the reopen process, it will check the seqNum to confirm if the region
reopened successfully or not. If the seqNum accident become bigger than the
current HFile and WAL (because of the data loss), there will be issue and
unnecessary loop for the region close/open
We should be able to optimize the logic, more details
For this regionOpenedWithoutPersistingToMeta, should we just update the
OpenSeqNum when the new one is bigger than the old one?
As the region already opened, we should update the OpenSeqNum no matter its
bigger or smaller, otherwise, we should not just return WARN but failed the
open, right?
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
Above does matter because for the
checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
if the seq is smaller, the region will be returned and keep reopening. So we
should either update the logic in regionOpenedWithoutPersistingToMeta or
checkReopened to make sure the region reopen works properly if the seqNum has
issue
Reproduce steps:
1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
{{create 'test', 'info'}}
{{put 'test', 'fool', 'info:cat', 'test'}}
{{2. Manually update one region row for this test table in hbase:meta on the
column, for example:}}
{{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
3. Modify the table TTL :
alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
You will see the region keep reopening
> Region Reopen failure when the openNum has issue
>
>
> Key: HBASE-27614
> URL: https://issues.apache.org/jira/browse/HBASE-27614
> Project: HBase
> Issue Type: Bug
>Reporter: Dong0829
>Assignee: Dong0829
>Priority: Major
>
> We faced the issue when change the TTL for the hbase table and a lot of
> regions keep reopen and tons of TRSP created, after troubleshooting, we found
> some logic issue for the region reopen procedure logic.
> In the reopen process, it will check the seqNum to confirm if the region
> reopened successfully or not. If the seqNum accident become bigger than the
> current HFile and WAL (because of the data loss), there will be issue and
> unnecessary loop for the region close/open
>
> We should be able to
[jira] [Updated] (HBASE-27614) Region Reopen failure when the openNum has issue
[
https://issues.apache.org/jira/browse/HBASE-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong0829 updated HBASE-27614:
-
Description:
We faced the issue when change the TTL for the hbase table and a lot of regions
keep reopen and tons of TRSP created, after troubleshooting, we found some
issue for the region reopen procedure logic.
In the reopen process, it will check the seqNum to confirm if the region
reopened successfully or not. If the seqNum accident become bigger than the
current HFile and WAL (because of the data loss), there will be issue and
unnecessary loop for the region close/open
We should be able to optimize the logic, more details
For this regionOpenedWithoutPersistingToMeta, should we just update the
OpenSeqNum when the new one is bigger than the old one?
As the region already opened, we should update the OpenSeqNum no matter its
bigger or smaller, otherwise, we should not just return WARN but failed the
open, right?
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
Above does matter because for the
checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
if the seq is smaller, the region will be returned and keep reopening. So we
should either update the logic in regionOpenedWithoutPersistingToMeta or
checkReopened to make sure the region reopen works properly if the seqNum has
issue
Reproduce steps:
1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
{{create 'test', 'info'}}
{{put 'test', 'fool', 'info:cat', 'test'}}
{{2. Manually update one region row for this test table in hbase:meta on the
column, for example:}}
{{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
3. Modify the table TTL :
alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
You will see the region keep reopening
was:
We faced the issue when change the TTL for the hbase table and a lot of regions
keep reopen and tons of TRSP created, after troubleshooting, we found some
issue for the region reopen procedure logic.
In the reopen process, it will check the seqNum to confirm if the region
reopened successfully or not. If the seqNum accident become bigger than the
current HFile and WAL (because of the data loss), there will be issue and
unnecessary loop for the region close/open
We should be able to optimize the logic, more details
For this regionOpenedWithoutPersistingToMeta, should we just update the
OpenSeqNum when the new one is bigger than the old one?
As the region already opened, we should update the OpenSeqNum no matter its
bigger or smaller, otherwise, we should not just return WARN but failed the
open, right?
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
Above does matter because for the
checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
if the seq is smaller, the region will be returned and keep reopening. So we
should either update the logic in regionOpenedWithoutPersistingToMeta or
checkReopened to make sure the region reopen works properly if the seqNum has
issue
Reproduce steps:
1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
{{create 'test', 'info'}}
{{put 'test', 'fool', 'info:cat', 'test'}}
{{2. Manually update one region row for this test table in hbase:meta on the
column, for example:}}
{{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
{{3. Modify the table TTL :
alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
{{}}
You will see the region keep reopening {{}}
{{}}
> Region Reopen failure when the openNum has issue
>
>
> Key: HBASE-27614
> URL: https://issues.apache.org/jira/browse/HBASE-27614
> Project: HBase
> Issue Type: Bug
>Reporter: Dong0829
>Assignee: Dong0829
>Priority: Major
>
> We faced the issue when change the TTL for the hbase table and a lot of
> regions keep reopen and tons of TRSP created, after troubleshooting, we found
> some issue for the region reopen procedure logic.
> In the reopen process, it will check the seqNum to confirm if the region
> reopened successfully or not. If the seqNum accident become bigger than the
> current HFile and WAL (because of the data loss), there will be issue and
> unnecessary loop for the region close/open
>
> We should be able to
[jira] [Created] (HBASE-27614) Region Reopen failure when the openNum has issue
Dong0829 created HBASE-27614:
Summary: Region Reopen failure when the openNum has issue
Key: HBASE-27614
URL: https://issues.apache.org/jira/browse/HBASE-27614
Project: HBase
Issue Type: Bug
Reporter: Dong0829
Assignee: Dong0829
We faced the issue when change the TTL for the hbase table and a lot of regions
keep reopen and tons of TRSP created, after troubleshooting, we found some
issue for the region reopen procedure logic.
In the reopen process, it will check the seqNum to confirm if the region
reopened successfully or not. If the seqNum accident become bigger than the
current HFile and WAL (because of the data loss), there will be issue and
unnecessary loop for the region close/open
We should be able to optimize the logic, more details
For this regionOpenedWithoutPersistingToMeta, should we just update the
OpenSeqNum when the new one is bigger than the old one?
As the region already opened, we should update the OpenSeqNum no matter its
bigger or smaller, otherwise, we should not just return WARN but failed the
open, right?
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/OpenRegionProcedure.java#L81]
Above does matter because for the
checkReopened([https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java#L312]),
if the seq is smaller, the region will be returned and keep reopening. So we
should either update the logic in regionOpenedWithoutPersistingToMeta or
checkReopened to make sure the region reopen works properly if the seqNum has
issue
Reproduce steps:
1. {{{}Create a test table and put some data, for example:{}}}{{{}test{}}}
{{create 'test', 'info'}}
{{put 'test', 'fool', 'info:cat', 'test'}}
{{2. Manually update one region row for this test table in hbase:meta on the
column, for example:}}
{{put 'hbase:meta', 'test,,1673406566311.3eb4d3e0258bd06f4639a595920c7673.',
'info:seqnumDuringOpen', "\x00\x00\x00\x00\x00\x10\x00\x05"}}
{{3. Modify the table TTL :
alter 'test', \{NAME=>'info' , TTL => '63244800'}}}
{{}}
You will see the region keep reopening {{}}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HBASE-27245) Expose the archive option to end user
[ https://issues.apache.org/jira/browse/HBASE-27245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong0829 updated HBASE-27245: - Affects Version/s: 3.0.0-alpha-3 > Expose the archive option to end user > - > > Key: HBASE-27245 > URL: https://issues.apache.org/jira/browse/HBASE-27245 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 2.5.0, 3.0.0-alpha-3 >Reporter: Dong0829 >Priority: Minor > > For the delete table procedure, we already have archive option, but the > archive now will already be true and not provided to end user. > Actually its valuable to provide this option to the end user because some > time, customer does not want this table archived when drop the table, it will > waste the storage and time if hbase keep archive the table. Especially for > the users who are using AWS s3 or Azure as storage which will take lots time > for the rename and delete operation for big tables -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-27245) Expose the archive option to end user
[ https://issues.apache.org/jira/browse/HBASE-27245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong0829 updated HBASE-27245: - Description: For the delete table procedure, we already have archive option, but the archive now will already be true and not provided to end user. Actually its valuable to provide this option to the end user because some time, customer does not want this table archived when drop the table, it will waste the storage and time if hbase keep archive the table. Especially for the users who are using AWS s3 or Azure as storage which will take lots time for the rename and delete operation for big tables was: For the delete table procedure, we already have archive option, but the archive now will already be true and not provided to end user. Actually its valuable to provide this option to the end user because some time, customer does not want this table archived when drop the table, it will waste the storage and time if hbase keep archive the table. Especially for the users who are using AWS s3 or Azure as storage which will take more time for the rename and delete operation > Expose the archive option to end user > - > > Key: HBASE-27245 > URL: https://issues.apache.org/jira/browse/HBASE-27245 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 2.5.0 >Reporter: Dong0829 >Priority: Minor > > For the delete table procedure, we already have archive option, but the > archive now will already be true and not provided to end user. > Actually its valuable to provide this option to the end user because some > time, customer does not want this table archived when drop the table, it will > waste the storage and time if hbase keep archive the table. Especially for > the users who are using AWS s3 or Azure as storage which will take lots time > for the rename and delete operation for big tables -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27245) Expose the archive option to end user
Dong0829 created HBASE-27245: Summary: Expose the archive option to end user Key: HBASE-27245 URL: https://issues.apache.org/jira/browse/HBASE-27245 Project: HBase Issue Type: New Feature Components: master Affects Versions: 2.5.0 Reporter: Dong0829 For the delete table procedure, we already have archive option, but the archive now will already be true and not provided to end user. Actually its valuable to provide this option to the end user because some time, customer does not want this table archived when drop the table, it will waste the storage and time if hbase keep archive the table. Especially for the users who are using AWS s3 or Azure as storage which will take more time for the rename and delete operation -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27244) bin/hbase still use slf4j-log4j while reload4j in place
[ https://issues.apache.org/jira/browse/HBASE-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571683#comment-17571683 ] Dong0829 commented on HBASE-27244: -- Yes, I am preparing the PR > bin/hbase still use slf4j-log4j while reload4j in place > --- > > Key: HBASE-27244 > URL: https://issues.apache.org/jira/browse/HBASE-27244 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.4.13 >Reporter: Dong0829 >Priority: Major > > For the bin/hbase file, there are some class path set which will handle the > slf4j-log4j before, but now in hbase branch 2, we upgrade to reload4j, but > the hbase file not upgrade, for example, if we run mapredcp: > hbase mapredcp: > /usr/lib/hbase/lib/shaded-clients/hbase-shaded-mapreduce-2.4.12.jar:/usr/lib/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/usr/lib/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/usr/lib/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/usr/lib/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/reload4j-1.2.19.jar:/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar > Among above, > /usr/lib/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar should > be excluded but as slf4j-log4j before > This will impact the downstream users -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27244) bin/hbase still use slf4j-log4j while reload4j in place
Dong0829 created HBASE-27244: Summary: bin/hbase still use slf4j-log4j while reload4j in place Key: HBASE-27244 URL: https://issues.apache.org/jira/browse/HBASE-27244 Project: HBase Issue Type: Bug Components: shell Affects Versions: 2.4.13 Reporter: Dong0829 For the bin/hbase file, there are some class path set which will handle the slf4j-log4j before, but now in hbase branch 2, we upgrade to reload4j, but the hbase file not upgrade, for example, if we run mapredcp: hbase mapredcp: /usr/lib/hbase/lib/shaded-clients/hbase-shaded-mapreduce-2.4.12.jar:/usr/lib/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/usr/lib/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/usr/lib/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/usr/lib/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/reload4j-1.2.19.jar:/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.33.jar:/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar Among above, /usr/lib/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar should be excluded but as slf4j-log4j before This will impact the downstream users -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HBASE-27116) Non-interactive missing for the hbase-shell
[ https://issues.apache.org/jira/browse/HBASE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553851#comment-17553851 ] Dong0829 edited comment on HBASE-27116 at 6/14/22 1:45 AM: --- After a add back the eval_io, its working well for the non-interactive https://github.com/apache/hbase/pull/4531 was (Author: li0829): https://github.com/apache/hbase/pull/4531 > Non-interactive missing for the hbase-shell > --- > > Key: HBASE-27116 > URL: https://issues.apache.org/jira/browse/HBASE-27116 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.4.12 >Reporter: Dong0829 >Priority: Major > > During test 2.4.12, we found a lot of our integration testing failure for the > non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the > "shell.eval_io" for the non-interactive shell, this caused the input/output > handle behavior for shell is the same for interactive and non-interactive > which caused unexpected error, for example: > > ``` > echo -e "disable_all '.*'\ny" | hbase shell -n > =Old version 2.4.4= > t1 > Disable the above 1 tables (y/n)? > 1 tables successfully disabled > Took 0.7811 seconds > =New version 2.4.12= > hbase:001:0> disable_all '.*'t1 > test_tableDisable the above 2 tables (y/n)? > Took 0.5270 seconds > NoMethodError: undefined method `chomp' for nil:NilClass > command at > uri:classloader:/shell/commands/disable_all.rb:41 > command_safe at uri:classloader:/shell/commands.rb:49 > translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 > command_safe at uri:classloader:/shell/commands.rb:49 > > run at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 >at classpath:/jar-bootstrap.rb:223}} > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-27116) Non-interactive missing for the hbase-shell
[ https://issues.apache.org/jira/browse/HBASE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong0829 updated HBASE-27116: - Status: Patch Available (was: Open) https://github.com/apache/hbase/pull/4531 > Non-interactive missing for the hbase-shell > --- > > Key: HBASE-27116 > URL: https://issues.apache.org/jira/browse/HBASE-27116 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.4.12 >Reporter: Dong0829 >Priority: Major > > During test 2.4.12, we found a lot of our integration testing failure for the > non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the > "shell.eval_io" for the non-interactive shell, this caused the input/output > handle behavior for shell is the same for interactive and non-interactive > which caused unexpected error, for example: > > ``` > echo -e "disable_all '.*'\ny" | hbase shell -n > =Old version 2.4.4= > t1 > Disable the above 1 tables (y/n)? > 1 tables successfully disabled > Took 0.7811 seconds > =New version 2.4.12= > hbase:001:0> disable_all '.*'t1 > test_tableDisable the above 2 tables (y/n)? > Took 0.5270 seconds > NoMethodError: undefined method `chomp' for nil:NilClass > command at > uri:classloader:/shell/commands/disable_all.rb:41 > command_safe at uri:classloader:/shell/commands.rb:49 > translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 > command_safe at uri:classloader:/shell/commands.rb:49 > > run at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 >at classpath:/jar-bootstrap.rb:223}} > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-27116) Non-interactive missing for the hbase-shell
[ https://issues.apache.org/jira/browse/HBASE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong0829 updated HBASE-27116: - Description: During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same for interactive and non-interactive which caused unexpected error, for example: ``` echo -e "disable_all '.*'\ny" | hbase shell -n =Old version 2.4.4= t1 Disable the above 1 tables (y/n)? 1 tables successfully disabled Took 0.7811 seconds =New version 2.4.12= hbase:001:0> disable_all '.*'t1 test_tableDisable the above 2 tables (y/n)? Took 0.5270 seconds NoMethodError: undefined method `chomp' for nil:NilClass command at uri:classloader:/shell/commands/disable_all.rb:41 command_safe at uri:classloader:/shell/commands.rb:49 translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 command_safe at uri:classloader:/shell/commands.rb:49 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:223}} ``` was: During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same which caused unexpected error, for example: ``` echo -e "disable_all '.*'\ny" | hbase shell -n =Old version 2.4.4= t1 Disable the above 1 tables (y/n)? 1 tables successfully disabled Took 0.7811 seconds =New version 2.4.12= hbase:001:0> disable_all '.*'t1 test_tableDisable the above 2 tables (y/n)? Took 0.5270 seconds NoMethodError: undefined method `chomp' for nil:NilClass command at uri:classloader:/shell/commands/disable_all.rb:41 command_safe at uri:classloader:/shell/commands.rb:49 translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 command_safe at uri:classloader:/shell/commands.rb:49 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:223}} ``` > Non-interactive missing for the hbase-shell > --- > > Key: HBASE-27116 > URL: https://issues.apache.org/jira/browse/HBASE-27116 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.4.12 >Reporter: Dong0829 >Priority: Major > > During test 2.4.12, we found a lot of our integration testing failure for the > non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the > "shell.eval_io" for the non-interactive shell, this caused the input/output > handle behavior for shell is the same for interactive and non-interactive > which caused unexpected error, for example: > > ``` > echo -e "disable_all '.*'\ny" | hbase shell -n > =Old version 2.4.4= > t1 > Disable the above 1 tables (y/n)? > 1 tables successfully disabled > Took 0.7811 seconds > =New version 2.4.12= > hbase:001:0> disable_all '.*'t1 > test_tableDisable the above 2 tables (y/n)? > Took 0.5270 seconds > NoMethodError: undefined method `chomp' for nil:NilClass > command at > uri:classloader:/shell/commands/disable_all.rb:41 > command_safe at uri:classloader:/shell/commands.rb:49 > translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 > command_safe at uri:classloader:/shell/commands.rb:49 > > run at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 >at classpath:/jar-bootstrap.rb:223}} > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (HBASE-27116) Non-interactive missing for the hbase-shell
[ https://issues.apache.org/jira/browse/HBASE-27116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong0829 updated HBASE-27116: - Description: During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same which caused unexpected error, for example: ``` echo -e "disable_all '.*'\ny" | hbase shell -n =Old version 2.4.4= t1 Disable the above 1 tables (y/n)? 1 tables successfully disabled Took 0.7811 seconds =New version 2.4.12= hbase:001:0> disable_all '.*'t1 test_tableDisable the above 2 tables (y/n)? Took 0.5270 seconds NoMethodError: undefined method `chomp' for nil:NilClass command at uri:classloader:/shell/commands/disable_all.rb:41 command_safe at uri:classloader:/shell/commands.rb:49 translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 command_safe at uri:classloader:/shell/commands.rb:49 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:223}} ``` was: During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same which caused unexpected error, for example: ``` echo -e "disable_all '.*'\ny" | hbase shell -n =Old version 2.4.4= t1 Disable the above 1 tables (y/n)?1 tables successfully disabled Took 0.7811 seconds =New version 2.4.12= hbase:001:0> disable_all '.*'t1 test_tableDisable the above 2 tables (y/n)? Took 0.5270 seconds NoMethodError: undefined method `chomp' for nil:NilClass command at uri:classloader:/shell/commands/disable_all.rb:41 command_safe at uri:classloader:/shell/commands.rb:49 translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 command_safe at uri:classloader:/shell/commands.rb:49 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:223}} ``` > Non-interactive missing for the hbase-shell > --- > > Key: HBASE-27116 > URL: https://issues.apache.org/jira/browse/HBASE-27116 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.4.12 >Reporter: Dong0829 >Priority: Major > > During test 2.4.12, we found a lot of our integration testing failure for the > non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the > "shell.eval_io" for the non-interactive shell, this caused the input/output > handle behavior for shell is the same which caused unexpected error, for > example: > > ``` > echo -e "disable_all '.*'\ny" | hbase shell -n > =Old version 2.4.4= > t1 > Disable the above 1 tables (y/n)? > 1 tables successfully disabled > Took 0.7811 seconds > =New version 2.4.12= > hbase:001:0> disable_all '.*'t1 > test_tableDisable the above 2 tables (y/n)? > Took 0.5270 seconds > NoMethodError: undefined method `chomp' for nil:NilClass > command at > uri:classloader:/shell/commands/disable_all.rb:41 > command_safe at uri:classloader:/shell/commands.rb:49 > translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 > command_safe at uri:classloader:/shell/commands.rb:49 > > run at > uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 >at classpath:/jar-bootstrap.rb:223}} > ``` -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27116) Non-interactive missing for the hbase-shell
Dong0829 created HBASE-27116: Summary: Non-interactive missing for the hbase-shell Key: HBASE-27116 URL: https://issues.apache.org/jira/browse/HBASE-27116 Project: HBase Issue Type: Bug Components: shell Affects Versions: 2.4.12 Reporter: Dong0829 During test 2.4.12, we found a lot of our integration testing failure for the non-interactive mode in hbase-shell, looks like in 2.4.12, we removed the "shell.eval_io" for the non-interactive shell, this caused the input/output handle behavior for shell is the same which caused unexpected error, for example: ``` echo -e "disable_all '.*'\ny" | hbase shell -n =Old version 2.4.4= t1 Disable the above 1 tables (y/n)?1 tables successfully disabled Took 0.7811 seconds =New version 2.4.12= hbase:001:0> disable_all '.*'t1 test_tableDisable the above 2 tables (y/n)? Took 0.5270 seconds NoMethodError: undefined method `chomp' for nil:NilClass command at uri:classloader:/shell/commands/disable_all.rb:41 command_safe at uri:classloader:/shell/commands.rb:49 translate_hbase_exceptions at uri:classloader:/shell/commands.rb:122 command_safe at uri:classloader:/shell/commands.rb:49 run at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/irb.rb:427 at classpath:/jar-bootstrap.rb:223}} ``` -- This message was sent by Atlassian Jira (v8.20.7#820007)
