[jira] [Commented] (HBASE-20006) TestRestoreSnapshotFromClientWithRegionReplicas is flakey
[
https://issues.apache.org/jira/browse/HBASE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447588#comment-16447588
]
Toshihiro Suzuki commented on HBASE-20006:
--
Thank you for reviewing the patch [~busbey] [~stack].
> TestRestoreSnapshotFromClientWithRegionReplicas is flakey
> -
>
> Key: HBASE-20006
> URL: https://issues.apache.org/jira/browse/HBASE-20006
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Reporter: stack
>Assignee: Toshihiro Suzuki
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 1.5.0
>
> Attachments: HBASE-20006.branch-2.001.patch,
> HBASE-20006.master.001.patch, HBASE-20006.master.002.patch,
> HBASE-20006.master.003.patch, HBASE-20006.master.003.patch
>
>
> Failing 10% of the time. Interestingly, it is below that causes fail. We go
> to split but it is already split. We will then fail the split with an
> internal assert which messes up procedures; at a minimum we should just not
> split (this is in the prepare stage).
> {code}
> 2018-02-15 23:21:42,162 INFO [PEWorker-12]
> procedure.MasterProcedureScheduler(571): pid=105,
> state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
> table=testOnlineSnapshotAfterSplittingRegions-1518736887838,
> parent=3f850cea7d71a7ebd019f2f009efca4d,
> daughterA=06b5e6366efbef155d70e56cfdf58dc9,
> daughterB=8c175de1b33765a5683ac1e502edb0bd,
> table=testOnlineSnapshotAfterSplittingRegions-1518736887838,
> testOnlineSnapshotAfterSplittingRegions-1518736887838,,1518736887882.3f850cea7d71a7ebd019f2f009efca4d.
> 2018-02-15 23:21:42,162 INFO [PEWorker-12]
> assignment.SplitTableRegionProcedure(440): Split of {ENCODED =>
> 3f850cea7d71a7ebd019f2f009efca4d, NAME =>
> 'testOnlineSnapshotAfterSplittingRegions-1518736887838,,1518736887882.3f850cea7d71a7ebd019f2f009efca4d.',
> STARTKEY => '', ENDKEY => '1'} skipped; state is already SPLIT
> 2018-02-15 23:21:42,163 ERROR [PEWorker-12]
> procedure2.ProcedureExecutor(1480): CODE-BUG: Uncaught runtime exception:
> pid=105, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure
> table=testOnlineSnapshotAfterSplittingRegions-1518736887838,
> parent=3f850cea7d71a7ebd019f2f009efca4d,
> daughterA=06b5e6366efbef155d70e56cfdf58dc9,
> daughterB=8c175de1b33765a5683ac1e502edb0bd
> java.lang.AssertionError: split region should have an exception here
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:228)
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:89)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1455)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1224)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1734)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20414) TestLockProcedure#testMultipleLocks may fail on slow machine
[
https://issues.apache.org/jira/browse/HBASE-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447580#comment-16447580
]
ramkrishna.s.vasudevan commented on HBASE-20414:
I haven't checked the test case in detail. But i can see the
{code:java}
Thread.sleep(LOCAL_LOCKS_TIMEOUT / 2);{code}
in multiple places. The place that you fix now is enough to make this test
generally reliable?
> TestLockProcedure#testMultipleLocks may fail on slow machine
>
>
> Key: HBASE-20414
> URL: https://issues.apache.org/jira/browse/HBASE-20414
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20414.v1.txt
>
>
> Here was recent failure :
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/172/testReport/junit/org.apache.hadoop.hbase.master.locking/TestLockProcedure/health_checks___yetus_jdk8_hadoop2_checks___testMultipleLocks/
> {code}
> java.lang.AssertionError: expected: but was:
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.sendHeartbeatAndCheckLocked(TestLockProcedure.java:221)
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.testMultipleLocks(TestLockProcedure.java:311)
> {code}
> In the test output, we can see this:
> {code}
> 2018-04-13 20:19:54,230 DEBUG [Time-limited test]
> locking.TestLockProcedure(225): Proc id 22 : LOCKED.
> ...
> 2018-04-13 20:19:55,529 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(865): Stored pid=26, state=RUNNABLE;
> org.apache.hadoop.hbase.master.locking.LockProcedure
> regions=a7f9adfd047350eabb199a39564ba4db,c1e297609590b707233a2f9c8bb51fa1,
> type=EXCLUSIVE
> 2018-04-13 20:19:56,230 DEBUG [ProcExecTimeout] locking.LockProcedure(207):
> Timeout failure ProcedureEvent for pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE, ready=false, [pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE]
> {code}
> After the pid=26 log, the code does this (1 second wait):
> {code}
> // Assert tables & region locks are waiting because of namespace lock.
> Thread.sleep(HEARTBEAT_TIMEOUT / 2);
> {code}
> On a slow machine (in the case above), there was only 730 msec between the
> queueing of regionsLock2 and the WAITING_TIMEOUT state of the nsLock. The 1
> second wait was too long, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20472) InfoServer doesnot honour any acl set by the admin
[
https://issues.apache.org/jira/browse/HBASE-20472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20472:
---
Fix Version/s: 3.0.0
Status: Patch Available (was: Open)
> InfoServer doesnot honour any acl set by the admin
> --
>
> Key: HBASE-20472
> URL: https://issues.apache.org/jira/browse/HBASE-20472
> Project: HBase
> Issue Type: Bug
> Components: security, UI
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-20472.master.001.patch
>
>
> The adminsAcl property can be used to restrict access to certain sections of
> the web UI only to a particular set of users/groups. But in hbase, adminAcl
> variable for InfoServer is always null, rendering it to not honour any acl
> set by the admin. In fact I could not find any property in hbase to specify
> acl list for web server.
> *Analysis*:
> * *InfoSever* object forgets(?) to set any *adminAcl* in the builder object
> for http server.
> {code:java}
> public InfoServer(String name, String bindAddress, int port, boolean findPort,
> final Configuration c) {
> .
> .
>
> HttpServer.Builder builder =
> new org.apache.hadoop.hbase.http.HttpServer.Builder();
> .
> .
> this.httpServer = builder.build();
> }{code}
> [See InfoServer
> constructor|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/InfoServer.java#L55]
> * http server retreives a null value and sets it as adminsAcl, which is
> passed to *createWebAppContext*() method
> {code:java}
> private HttpServer(final Builder b) throws IOException {
> .
> .
> .
> this.adminsAcl = b.adminsAcl;
> this.webAppContext = createWebAppContext(b.name, b.conf, adminsAcl,
> appDir);
>
> .
> .
> }{code}
> [See L527
> HttpServer.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L527]
> * This method next sets *ADMIN_ACL* attribute for the servlet context to
> *null*
> {code:java}
> private static WebAppContext createWebAppContext(String name,
> Configuration conf, AccessControlList adminsAcl, final String appDir) {
> WebAppContext ctx = new WebAppContext();
> .
> .
> ctx.getServletContext().setAttribute(ADMINS_ACL, adminsAcl);
> .
> .
> }
> {code}
> * Now any page having *HttpServer.hasAdministratorAccess*() will allow
> access to everyone, making this check useless.
> {code:java}
> @Override
> public void doGet(HttpServletRequest request, HttpServletResponse response
> ) throws ServletException, IOException {
> // Do the authorization
> if (!HttpServer.hasAdministratorAccess(getServletContext(), request,
> response)) {
> return;
> }
> .
> .
> }{code}
> [For example See L104
> LogLevel.java|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/log/LogLevel.java#L104]
> * *hasAdministratorAccess()* checks for the following and returns true, in
> any case as *ADMIN_ACL* is always *null*
> {code:java}
> public static boolean hasAdministratorAccess(
> ServletContext servletContext, HttpServletRequest request,
> HttpServletResponse response) throws IOException {
> .
> .
> if (servletContext.getAttribute(ADMINS_ACL) != null &&
> !userHasAdministratorAccess(servletContext, remoteUser)) {
> response.sendError(HttpServletResponse.SC_UNAUTHORIZED, "User "
> + remoteUser + " is unauthorized to access this page.");
>return false;
> }
> return true;
> }{code}
> [See line 1196 in
> HttpServer|https://github.com/apache/hbase/blob/46cb5dfa226892fd2580f26ce9ce77225bd7e67c/hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java#L1196]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[
https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447564#comment-16447564
]
Hudson commented on HBASE-20470:
Results for branch branch-2.0
[build #211 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [2.0.0RC1] has broken unit tests...
>
>
> Key: HBASE-20470
> URL: https://issues.apache.org/jira/browse/HBASE-20470
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-20470.branch-2.0.001.patch,
> HBASE-20470.branch-2.0.002.patch, HBASE-20470.branch-2.0.003.patch,
> HBASE-20470.branch-2.0.003.patch
>
>
> Found by [~uagashe] I think its because some depend on IMC and it was
> disabled just before I made RC1. Let me try a nothing change and see.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20462) Put up 2.0.0RC1
[
https://issues.apache.org/jira/browse/HBASE-20462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447566#comment-16447566
]
Hudson commented on HBASE-20462:
Results for branch branch-2.0
[build #211 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Put up 2.0.0RC1
> ---
>
> Key: HBASE-20462
> URL: https://issues.apache.org/jira/browse/HBASE-20462
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20465) Fix TestEnableRSGroup flaky
[
https://issues.apache.org/jira/browse/HBASE-20465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447565#comment-16447565
]
Hudson commented on HBASE-20465:
Results for branch branch-2.0
[build #211 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/211//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix TestEnableRSGroup flaky
> ---
>
> Key: HBASE-20465
> URL: https://issues.apache.org/jira/browse/HBASE-20465
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20465.master.001.patch
>
>
> Most recent {{TestEnableRSGroup}} tests failed on branch-2.0 according to our
> flaky dashboard.
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.rsgroup.TestEnableRSGroup.testEnableRSGroup(TestEnableRSGroup.java:80)
> {code}
> TestEnableRSGroup:80:
> {code:java}
> // check if master started successfully
> assertTrue(TEST_UTIL.getMiniHBaseCluster().getMaster() != null);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19024) Configurable default durability for synchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447560#comment-16447560 ] ramkrishna.s.vasudevan commented on HBASE-19024: So one question regarding the performance numbers here wrt hsync, Since the sync calls could have been executed with hflush for some of the batch mutations the degradation that was specified here was lesser than when all the batches are going through hsync? > Configurable default durability for synchronous WAL > --- > > Key: HBASE-19024 > URL: https://issues.apache.org/jira/browse/HBASE-19024 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Vikas Vishwakarma >Assignee: Harshal Jain >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-19024-master.v10.patch, > HBASE-19024.branch-1.2.001.patch, HBASE-19024.branch-1.2.002.patch, > HBASE-19024.branch-1.2.003.patch, HBASE-19024.branch-1.2.004.patch, > HBASE-19024.branch-1.2.005.patch, branch-1.branch-1.patch, > branch-1.v1.branch-1.patch, master.patch, master.v2.patch, master.v3.patch, > master.v5.patch, master.v5.patch, master.v6.patch, master.v9.patch > > > At present we do not have an option to hsync WAL edits to the disk for better > durability. In our local tests we see 10-15% latency impact of using hsync > instead of hflush which is not very high. > We should have a configurable option to hysnc WAL edits instead of just > sync/hflush which will call the corresponding API on the hadoop side. > Currently HBase handles both SYNC_WAL and FSYNC_WAL as the same calling > FSDataOutputStream sync/hflush on the hadoop side. This can be modified to > let FSYNC_WAL call hsync on the hadoop side instead of sync/hflush. We can > keep the default value to sync as the current behavior and hsync can be > enabled based on explicit configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447554#comment-16447554 ] Chance Li commented on HBASE-20312: --- [~mdrob], sir, I will separate this patch to two parts as [~carp84] suggested after completing the document. I have to pay attention to the precommit and review first, and now failed UT cases seem to irrelevant, I try to analyze it. separating patch is more easy, as these two parts are very logically clear in this patch, and I've always been focusing on readability. Thanks for great suggestion and review. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default
[ https://issues.apache.org/jira/browse/HBASE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447539#comment-16447539 ] Guangxu Cheng commented on HBASE-20474: --- a simple patch.Thanks > Show non-RPC tasks on master/regionserver Web UI by default > > > Key: HBASE-20474 > URL: https://issues.apache.org/jira/browse/HBASE-20474 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20474.master.001.patch > > > Now, when opening master or regionserver pages, all tasks will be displayed > on the page, however, but in most cases we will pay more attention to non-RPC > tasks. > In addition, if all tasks are displayed by default, a large number of pages > will be occupied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default
[ https://issues.apache.org/jira/browse/HBASE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20474: -- Fix Version/s: 3.0.0 Status: Patch Available (was: Open) > Show non-RPC tasks on master/regionserver Web UI by default > > > Key: HBASE-20474 > URL: https://issues.apache.org/jira/browse/HBASE-20474 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20474.master.001.patch > > > Now, when opening master or regionserver pages, all tasks will be displayed > on the page, however, but in most cases we will pay more attention to non-RPC > tasks. > In addition, if all tasks are displayed by default, a large number of pages > will be occupied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default
[ https://issues.apache.org/jira/browse/HBASE-20474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated HBASE-20474: -- Attachment: HBASE-20474.master.001.patch > Show non-RPC tasks on master/regionserver Web UI by default > > > Key: HBASE-20474 > URL: https://issues.apache.org/jira/browse/HBASE-20474 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: HBASE-20474.master.001.patch > > > Now, when opening master or regionserver pages, all tasks will be displayed > on the page, however, but in most cases we will pay more attention to non-RPC > tasks. > In addition, if all tasks are displayed by default, a large number of pages > will be occupied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20474) Show non-RPC tasks on master/regionserver Web UI by default
Guangxu Cheng created HBASE-20474: - Summary: Show non-RPC tasks on master/regionserver Web UI by default Key: HBASE-20474 URL: https://issues.apache.org/jira/browse/HBASE-20474 Project: HBase Issue Type: Improvement Components: UI Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now, when opening master or regionserver pages, all tasks will be displayed on the page, however, but in most cases we will pay more attention to non-RPC tasks. In addition, if all tasks are displayed by default, a large number of pages will be occupied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447538#comment-16447538
]
Vikas Vishwakarma commented on HBASE-17958:
---
Actually [~apurtell] [~lhofhansl] [~Apache9] I think the issue is this one
HBASE-10993 already reported and fixed by [~stack] in HBASE-15971. Profiling
and comparing 0.98 vs 1.3 is also showing this as the major diff in hot
threads. Now in the request hbase.ipc.server.callqueue.type default has been
set to fifo so it should be fixed. I have verified we are not setting it to
deadline in our hbase-site.xml. But still we have very similar characteristics.
Let me look into this further.
I will log a separate ticket to update all the profiling snapshots and result
details, since it is not related to this request and this is turning into a
long discussion.
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 1.4.0, 2.0.0
>
> Attachments: 0001-add-one-ut-testWithColumnCountGetFilter.patch,
> 17958-add.txt, HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch,
> HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch, HBASE-17958-v1.patch,
> HBASE-17958-v2.patch, HBASE-17958-v3.patch, HBASE-17958-v4.patch,
> HBASE-17958-v5.patch, HBASE-17958-v6.patch, HBASE-17958-v7.patch,
> HBASE-17958-v7.patch
>
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[
https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447536#comment-16447536
]
Hudson commented on HBASE-20470:
Results for branch branch-2
[build #648 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [2.0.0RC1] has broken unit tests...
>
>
> Key: HBASE-20470
> URL: https://issues.apache.org/jira/browse/HBASE-20470
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-20470.branch-2.0.001.patch,
> HBASE-20470.branch-2.0.002.patch, HBASE-20470.branch-2.0.003.patch,
> HBASE-20470.branch-2.0.003.patch
>
>
> Found by [~uagashe] I think its because some depend on IMC and it was
> disabled just before I made RC1. Let me try a nothing change and see.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20465) Fix TestEnableRSGroup flaky
[
https://issues.apache.org/jira/browse/HBASE-20465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447537#comment-16447537
]
Hudson commented on HBASE-20465:
Results for branch branch-2
[build #648 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/648//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix TestEnableRSGroup flaky
> ---
>
> Key: HBASE-20465
> URL: https://issues.apache.org/jira/browse/HBASE-20465
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20465.master.001.patch
>
>
> Most recent {{TestEnableRSGroup}} tests failed on branch-2.0 according to our
> flaky dashboard.
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.rsgroup.TestEnableRSGroup.testEnableRSGroup(TestEnableRSGroup.java:80)
> {code}
> TestEnableRSGroup:80:
> {code:java}
> // check if master started successfully
> assertTrue(TEST_UTIL.getMiniHBaseCluster().getMaster() != null);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20471) Recheck the design and implementation of FSYNC_WAL durability for WAL
[
https://issues.apache.org/jira/browse/HBASE-20471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446692#comment-16446692
]
Yu Li edited comment on HBASE-20471 at 4/23/18 3:32 AM:
Some of my thoughts:
1. Only allow setting {{SKIP_WAL/ASYNC_WAL/SYNC_WAL}} through
{{Mutation#setDurability}} API
* We can separate FSYNC_WAL from SYNC_WAL writes only if we flush the
mutations one by one, but actually we're grouping (and we should, for the sake
of performance) the writes through disruptor mechanism, which makes it possible
that a mutation with SYNC_WAL durability is actually fsync'ed, vice versa. So
for sync write I suggest we only allow setting SYNC_WAL per mutation, and
decide the detailed sync mode through cluster-level setting (for now).
* To keep backward compatibility, if user set FSYNC_WAL, we should change
it to SYNC_WAL in {{Mutation#setDurability}}
* We should add some document about this change in our ref-guide.
2. Instead, we allow user to set the cluster-level way of sync through
hbase-site.xml, hflush or hsync.
3. In the future we may allow user to use a dedicated WAL per table/CF and set
its sync mode through table/CF descriptor.
was (Author: carp84):
Some of my thoughts:
1. Only allow setting {{SYNC_WAL}} through {{Mutation#setDurability}} API
* We can separate FSYNC_WAL from SYNC_WAL writes only if we flush the
mutations one by one, but actually we're grouping (and we should, for the sake
of performance) the writes through disruptor mechanism, which makes it possible
that a mutation with SYNC_WAL durability is actually fsync'ed, vice versa.
* To keep backward compatibility, if user set FSYNC_WAL, we should change
it to SYNC_WAL in {{Mutation#setDurability}}
* We should add some document about this change in our ref-guide.
2. Instead, we allow user to set the cluster-level way of sync through
hbase-site.xml, hflush or hsync.
3. In the future we may allow user to use a dedicated WAL per table/CF and set
its sync mode through table/CF descriptor.
> Recheck the design and implementation of FSYNC_WAL durability for WAL
> -
>
> Key: HBASE-20471
> URL: https://issues.apache.org/jira/browse/HBASE-20471
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Priority: Major
>
> This is something derived from discussion in HBASE-19024 around [this
> comment|https://issues.apache.org/jira/browse/HBASE-19024?focusedCommentId=16445592=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16445592]
> We have been supplying user the API to set durability per mutation for a long
> time, by design the SYNC_WAL durability to call {{FSDataOutputStream#hflush}}
> and FSYNC_WAL {{FSDataOutputStream#hsync}}, while in implementation we have
> been calling hflush for FSYNC_WAL also until HBASE-19024. Although
> HBASE-19024 tried to fix the syntax with good willing, the implementation
> there cannot assure the FSYNC_WAL edits are truly hsync'ed due to the
> disruptor mechanism used in WAL implementation. Here in this JIRA we aim to
> have more discussion and give it a complete solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20471) Recheck the design and implementation of FSYNC_WAL durability for WAL
[
https://issues.apache.org/jira/browse/HBASE-20471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447525#comment-16447525
]
Yu Li commented on HBASE-20471:
---
bq. We do expect SKIP_WAL to take effect on a per RPC basis
Yes I agree, by "Only allow setting SYNC_WAL..." I actually meant for sync mode
we only allow SYNC_WAL and no FSYNC_WAL per RPC. I think 1) don't write WAL; 2)
write WAL asynchronously and 3) write WAL synchronously could over all needs
from user perspective, and the detailed sync mode should be a
per-cluster/per-table/per-CF setting rather than per RPC. Let me edit my old
comment to make it clear (smile)
> Recheck the design and implementation of FSYNC_WAL durability for WAL
> -
>
> Key: HBASE-20471
> URL: https://issues.apache.org/jira/browse/HBASE-20471
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Priority: Major
>
> This is something derived from discussion in HBASE-19024 around [this
> comment|https://issues.apache.org/jira/browse/HBASE-19024?focusedCommentId=16445592=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16445592]
> We have been supplying user the API to set durability per mutation for a long
> time, by design the SYNC_WAL durability to call {{FSDataOutputStream#hflush}}
> and FSYNC_WAL {{FSDataOutputStream#hsync}}, while in implementation we have
> been calling hflush for FSYNC_WAL also until HBASE-19024. Although
> HBASE-19024 tried to fix the syntax with good willing, the implementation
> there cannot assure the FSYNC_WAL edits are truly hsync'ed due to the
> disruptor mechanism used in WAL implementation. Here in this JIRA we aim to
> have more discussion and give it a complete solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12943: --- Fix Version/s: (was: 1.4.4) 1.4.5 > Set sun.net.inetaddr.ttl in HBase > - > > Key: HBASE-12943 > URL: https://issues.apache.org/jira/browse/HBASE-12943 > Project: HBase > Issue Type: Bug >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.5 > > Attachments: 12943-1-master.txt > > > The default value of config: sun.net.inetaddr.ttl is -1 and the java > processes will cache the mapping of hostname to ip address forever, See: > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > But things go wrong when a regionserver with same hostname and different ip > address rejoins the hbase cluster. The HMaster will get wrong ip address of > the regionserver from this cache and every region assignment to this > regionserver will be blocked for a time because the HMaster can't communicate > with the regionserver. > A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong > cache expired. > Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14223:
---
Fix Version/s: (was: 1.4.4)
1.4.5
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.2.8, 2.0.0, 1.4.5
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:50,497
>
[jira] [Updated] (HBASE-17890) FuzzyRowFilter fail if unaligned support is false
[
https://issues.apache.org/jira/browse/HBASE-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-17890:
---
Fix Version/s: (was: 1.4.4)
1.4.5
> FuzzyRowFilter fail if unaligned support is false
> -
>
> Key: HBASE-17890
> URL: https://issues.apache.org/jira/browse/HBASE-17890
> Project: HBase
> Issue Type: Sub-task
> Components: util
>Affects Versions: 1.2.5, 2.0.0
>Reporter: Jerry He
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.3.3, 1.2.8, 1.4.5
>
> Attachments: HBASE-17890.v0.branch-1.patch, HBASE-17890.v0.patch,
> HBASE-17890.v1.branch-1.patch, HBASE-17890.v1.patch, HBASE-17890.v2.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch, HBASE-17890.v3.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch
>
>
> When unaligned support is false, FuzzyRow tests fail:
> {noformat}
> Failed tests:
> TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186
> expected:<10> but was:<0>
> TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
> TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but
> was:
> TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343
> expected:<6250> but was:<0>
> TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445
> expected:<5> but was:<0>
> TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
> {noformat}
> This can be reproduced in the case described in HBASE-17869. Or on a platform
> really without unaligned support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17885: --- Fix Version/s: (was: 1.4.4) 1.4.5 > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.5 > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16071:
---
Fix Version/s: (was: 1.4.4)
1.4.5
> The VisibilityLabelFilter and AccessControlFilter should not count the
> "delete cell"
>
>
> Key: HBASE-16071
> URL: https://issues.apache.org/jira/browse/HBASE-16071
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 1.5.0, 1.3.3, 2.0.0, 1.4.5
>
> Attachments: HBASE-16071-v1.patch, HBASE-16071-v2.patch,
> HBASE-16071-v3.patch
>
>
> The VisibilityLabelFilter will see and count the "delete cell" if the
> scan.isRaw() returns true, so the (put) cell will be skipped if it has lower
> version than "delete cell"
> The critical code is shown below:
> {code:title=VisibilityLabelFilter.java|borderStyle=solid}
> public ReturnCode filterKeyValue(Cell cell) throws IOException {
> if (curFamily.getBytes() == null
> || !(CellUtil.matchingFamily(cell, curFamily.getBytes(),
> curFamily.getOffset(),
> curFamily.getLength( {
> curFamily.set(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength());
> // For this family, all the columns can have max of
> curFamilyMaxVersions versions. No need to
> // consider the older versions for visibility label check.
> // Ideally this should have been done at a lower layer by HBase (?)
> curFamilyMaxVersions = cfVsMaxVersions.get(curFamily);
> // Family is changed. Just unset curQualifier.
> curQualifier.unset();
> }
> if (curQualifier.getBytes() == null
> || !(CellUtil.matchingQualifier(cell, curQualifier.getBytes(),
> curQualifier.getOffset(),
> curQualifier.getLength( {
> curQualifier.set(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength());
> curQualMetVersions = 0;
> }
> curQualMetVersions++;
> if (curQualMetVersions > curFamilyMaxVersions) {
> return ReturnCode.SKIP;
> }
> return this.expEvaluator.evaluate(cell) ? ReturnCode.INCLUDE :
> ReturnCode.SKIP;
> }
> {code}
> [VisibilityLabelFilter.java|https://github.com/apache/hbase/blob/d7a4499dfc8b3936a0eca867589fc2b23b597866/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelFilter.java]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-13160) SplitLogWorker does not pick up the task immediately
[
https://issues.apache.org/jira/browse/HBASE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-13160:
---
Fix Version/s: (was: 1.4.4)
1.4.5
> SplitLogWorker does not pick up the task immediately
>
>
> Key: HBASE-13160
> URL: https://issues.apache.org/jira/browse/HBASE-13160
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.5
>
> Attachments: hbase-13160_v1.patch
>
>
> We were reading some code with Jeffrey, and we realized that the
> SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
> sleeps, but have another mechanism to learn about new tasks, but does not
> make affective use of the zk notification.
> I have a simple patch which might improve this area.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18228) HBCK improvements
[ https://issues.apache.org/jira/browse/HBASE-18228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18228: --- Fix Version/s: (was: 1.4.4) 1.4.5 2.1.0 3.0.0 > HBCK improvements > - > > Key: HBASE-18228 > URL: https://issues.apache.org/jira/browse/HBASE-18228 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Lars Hofhansl >Assignee: Karan Mehta >Priority: Critical > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5 > > Attachments: HBASE-18228.branch-1.3.patch > > > We just had a prod issue and running HBCK the way we did actually causes more > problems. > In part HBCK did stuff we did not expect, in part we had little visibility > into what HBCK was doing, and in part the logging was confusing. > I'm proposing 2 improvements: > 1. A dry-run mode. Run, and just list what would have been done. > 2. An interactive mode. Run, and for each action request Y/N user input. So > that a user can opt-out of stuff. > [~jmhsieh], FYI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20414) TestLockProcedure#testMultipleLocks may fail on slow machine
[
https://issues.apache.org/jira/browse/HBASE-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447506#comment-16447506
]
Ted Yu commented on HBASE-20414:
bq. does it have chance the value of (HEARTBEAT_TIMEOUT-(now-start)-10) is
negative
I don't think so. That means 1990 < now - start. From the flaky test scenario
given above, there was enough buffer for the above expression to be positive.
> TestLockProcedure#testMultipleLocks may fail on slow machine
>
>
> Key: HBASE-20414
> URL: https://issues.apache.org/jira/browse/HBASE-20414
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20414.v1.txt
>
>
> Here was recent failure :
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/172/testReport/junit/org.apache.hadoop.hbase.master.locking/TestLockProcedure/health_checks___yetus_jdk8_hadoop2_checks___testMultipleLocks/
> {code}
> java.lang.AssertionError: expected: but was:
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.sendHeartbeatAndCheckLocked(TestLockProcedure.java:221)
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.testMultipleLocks(TestLockProcedure.java:311)
> {code}
> In the test output, we can see this:
> {code}
> 2018-04-13 20:19:54,230 DEBUG [Time-limited test]
> locking.TestLockProcedure(225): Proc id 22 : LOCKED.
> ...
> 2018-04-13 20:19:55,529 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(865): Stored pid=26, state=RUNNABLE;
> org.apache.hadoop.hbase.master.locking.LockProcedure
> regions=a7f9adfd047350eabb199a39564ba4db,c1e297609590b707233a2f9c8bb51fa1,
> type=EXCLUSIVE
> 2018-04-13 20:19:56,230 DEBUG [ProcExecTimeout] locking.LockProcedure(207):
> Timeout failure ProcedureEvent for pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE, ready=false, [pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE]
> {code}
> After the pid=26 log, the code does this (1 second wait):
> {code}
> // Assert tables & region locks are waiting because of namespace lock.
> Thread.sleep(HEARTBEAT_TIMEOUT / 2);
> {code}
> On a slow machine (in the case above), there was only 730 msec between the
> queueing of regionsLock2 and the WAITING_TIMEOUT state of the nsLock. The 1
> second wait was too long, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18415) The local timeout may cause Admin to submit duplicate request
[ https://issues.apache.org/jira/browse/HBASE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18415: --- Fix Version/s: (was: 1.4.4) 1.4.5 > The local timeout may cause Admin to submit duplicate request > - > > Key: HBASE-18415 > URL: https://issues.apache.org/jira/browse/HBASE-18415 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.5 > > Attachments: HBASE-18415.branch-1.ut.patch, > HBASE-18415.branch-1.v0.patch, HBASE-18415.branch-1.v1.patch, > HBASE-18415.branch-1.v2.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v3.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v4.patch, HBASE-18415.branch-1.v4.patch, > HBASE-18415.branch-1.v4.patch > > > After a timeout occurs on first request, client will retry the request with > distinct group/nonce. The second request may bring the TableXXXException back > if the first request have changed the table state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18549) Unclaimed replication queues can go undetected
[ https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18549: --- Fix Version/s: (was: 1.4.4) 1.4.5 > Unclaimed replication queues can go undetected > -- > > Key: HBASE-18549 > URL: https://issues.apache.org/jira/browse/HBASE-18549 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Ashu Pachauri >Priority: Critical > Fix For: 1.5.0, 1.3.3, 1.4.5 > > > We have come across this situation multiple times where a zookeeper issues > can cause NodeFailoverWorker to fail picking up replication queue for a dead > region server silently. One example is when the znode size for a particular > queue exceed jute.maxBuffer value. > There can be other situations that may lead to this and just go undetected. > We need to have a metric for number of unclaimed replication queues. This > will help in mitigating the problem through alerting on the metric and > identifying underlying issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18639) nightly jobs that need to do jdk7 + jdk8 need more than 6 hours
[ https://issues.apache.org/jira/browse/HBASE-18639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18639: --- Fix Version/s: (was: 1.4.4) 1.4.5 > nightly jobs that need to do jdk7 + jdk8 need more than 6 hours > --- > > Key: HBASE-18639 > URL: https://issues.apache.org/jira/browse/HBASE-18639 > Project: HBase > Issue Type: Task > Components: test >Reporter: Sean Busbey >Priority: Major > Labels: beginner > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.5 > > > looks like the branches that need to do 2 jdks will need more than 6 hours, > at least until we can parallelize things. > e.g. > * > [branch-1.2|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.2/buildTimeTrend] > * > [branch-1.3|https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1.3/buildTimeTrend] > branch-1.4 and branch-1 are failing in jdk7 checks, so they don't go that > long yet, but once we get those tests fixed presumably they'll need the time > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20447) Only fail cacheBlock if block collisions aren't related to next block metadata
[ https://issues.apache.org/jira/browse/HBASE-20447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20447: --- Fix Version/s: (was: 1.4.4) 1.4.5 > Only fail cacheBlock if block collisions aren't related to next block metadata > -- > > Key: HBASE-20447 > URL: https://issues.apache.org/jira/browse/HBASE-20447 > Project: HBase > Issue Type: Bug > Components: BlockCache, BucketCache >Affects Versions: 1.4.3, 2.0.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20447.branch-1.001.patch, > HBASE-20447.master.001.patch > > > This is the issue I was originally having here: > [http://mail-archives.apache.org/mod_mbox/hbase-dev/201802.mbox/%3CCAN+qs_Pav=md_aoj4xji+kcnetubg2xou2ntxv1g6m8-5vn...@mail.gmail.com%3E] > > When we pread, we don't force the read to read all of the next block header. > However, when we get into a race condition where two opener threads try to > cache the same block and one thread read all of the next block header and the > other one didn't, it will fail the open process. This is especially important > in a splitting case where it will potentially fail the split process. > Instead, in the caches, we should only fail if the required blocks are > different. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20026) Add 1.4 release line to the JDK and Hadoop expectation tables
[ https://issues.apache.org/jira/browse/HBASE-20026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20026: --- Fix Version/s: (was: 1.4.4) 1.4.5 > Add 1.4 release line to the JDK and Hadoop expectation tables > - > > Key: HBASE-20026 > URL: https://issues.apache.org/jira/browse/HBASE-20026 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 1.4.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 1.4.5 > > > the ref guide currently doesn't have any expectations listed for branch-1.4 > releases around JDK and Hadoop versions. > either add it, or maybe update the existing entries so we have "1.2, 1.3, > 1.4" in a single entry. unless we're ready to include something different > among them. (Maybe note the default Hadoop we ship with? Or Hadoop 2.8.2+ > moving to S maybe? if we've actually done any of the legwork.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19444) RSGroups test units cannot be concurrently executed
[ https://issues.apache.org/jira/browse/HBASE-19444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19444: --- Fix Version/s: (was: 1.4.4) 1.4.5 > RSGroups test units cannot be concurrently executed > --- > > Key: HBASE-19444 > URL: https://issues.apache.org/jira/browse/HBASE-19444 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Priority: Minor > Fix For: 1.5.0, 2.0.0, 1.4.5 > > > TestRSGroups and friends cannot be concurrently executed or they are very > likely to flake, failing with constraint exceptions. If executed serially all > units pass. Fix for concurrent execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20468) RPC quota requests ineffective due to not counting multi-actions
[ https://issues.apache.org/jira/browse/HBASE-20468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20468: --- Fix Version/s: (was: 1.4.4) 1.4.5 > RPC quota requests ineffective due to not counting multi-actions > > > Key: HBASE-20468 > URL: https://issues.apache.org/jira/browse/HBASE-20468 > Project: HBase > Issue Type: Bug > Components: rpc >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.1.0, 1.5.0, 1.3.3, 1.2.8, 2.0.1, 1.4.5 > > > Was digging into a problem with [~ankit.singhal] where setting RPC quotas on > number of requests wasn't having any effect on a multi-Get > Ankit did enough digging to find that this was because each RPC was being > treated as one request instead of the number of requests contained within the > RPC itself. > Thinking as an operator, this is a pretty ineffective control because a user > could just craft their API usage easily to work around any kind of limits I > want to set to control their impact on the system. TimeBasedLimiter is > assuming that one call to the quota code can only count as one request which > I think is just wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20446) Allow building HBase 1.x against Hadoop 3.1.0
[ https://issues.apache.org/jira/browse/HBASE-20446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447501#comment-16447501 ] Andrew Purtell commented on HBASE-20446: [~mdrob] What is the current "for production" release(s) of Hadoop >= 3 ? > Allow building HBase 1.x against Hadoop 3.1.0 > - > > Key: HBASE-20446 > URL: https://issues.apache.org/jira/browse/HBASE-20446 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 1.5.0 > > Attachments: 20446.txt > > > Simple change, just leaving it here in case somebody needs this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20471) Recheck the design and implementation of FSYNC_WAL durability for WAL
[
https://issues.apache.org/jira/browse/HBASE-20471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447499#comment-16447499
]
Andrew Purtell commented on HBASE-20471:
After HBASE-19024 the default durability setting becomes FSYNC_WAL if
_hbase.wal.hsync_ is true, so all edits in the batch will have the same
setting, and will be fsynced (unless the user changes the Durability setting,
and then it's unclear, but we don't expect this for the normal case). The way
the new setting _hbase.wal.hsync_ works was good enough for HBASE-19024 but
agreed it is hardly ideal. If the user assumes SYNC_WAL or FSYNC_WAL is applied
to every mutation, this is not correct and should be documented. We do expect
SKIP_WAL to take effect on a per RPC basis, that is a Put should skip the WAL
if SKIP_WAL, or every mutation in a batch or RowMutations should skip the WAL
if the RPC carries a Durability attribute with SKIP_WAL selected. If that
assumption is now somehow not correct, we need to fix that.
> Recheck the design and implementation of FSYNC_WAL durability for WAL
> -
>
> Key: HBASE-20471
> URL: https://issues.apache.org/jira/browse/HBASE-20471
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Priority: Major
>
> This is something derived from discussion in HBASE-19024 around [this
> comment|https://issues.apache.org/jira/browse/HBASE-19024?focusedCommentId=16445592=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16445592]
> We have been supplying user the API to set durability per mutation for a long
> time, by design the SYNC_WAL durability to call {{FSDataOutputStream#hflush}}
> and FSYNC_WAL {{FSDataOutputStream#hsync}}, while in implementation we have
> been calling hflush for FSYNC_WAL also until HBASE-19024. Although
> HBASE-19024 tried to fix the syntax with good willing, the implementation
> there cannot assure the FSYNC_WAL edits are truly hsync'ed due to the
> disruptor mechanism used in WAL implementation. Here in this JIRA we aim to
> have more discussion and give it a complete solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20471) Recheck the design and implementation of FSYNC_WAL durability for WAL
[
https://issues.apache.org/jira/browse/HBASE-20471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447499#comment-16447499
]
Andrew Purtell edited comment on HBASE-20471 at 4/23/18 2:59 AM:
-
After HBASE-19024 the default durability setting becomes FSYNC_WAL if
_hbase.wal.hsync_ is true, so all edits in the batch will have the same
setting, and will be fsynced (unless the user changes the Durability setting,
and then it's unclear, but we don't expect this for the normal case). The way
the new setting _hbase.wal.hsync_ works was good enough for HBASE-19024 but
agreed it is hardly ideal. If the user assumes SYNC_WAL or FSYNC_WAL is applied
to every mutation, this is not correct and should be documented.
We do expect SKIP_WAL to take effect on a per RPC basis, that is a Put should
skip the WAL if SKIP_WAL, or every mutation in a batch or RowMutations should
skip the WAL if the RPC carries a Durability attribute with SKIP_WAL selected.
If that assumption is now somehow not correct, we need to fix that.
was (Author: apurtell):
After HBASE-19024 the default durability setting becomes FSYNC_WAL if
_hbase.wal.hsync_ is true, so all edits in the batch will have the same
setting, and will be fsynced (unless the user changes the Durability setting,
and then it's unclear, but we don't expect this for the normal case). The way
the new setting _hbase.wal.hsync_ works was good enough for HBASE-19024 but
agreed it is hardly ideal. If the user assumes SYNC_WAL or FSYNC_WAL is applied
to every mutation, this is not correct and should be documented. We do expect
SKIP_WAL to take effect on a per RPC basis, that is a Put should skip the WAL
if SKIP_WAL, or every mutation in a batch or RowMutations should skip the WAL
if the RPC carries a Durability attribute with SKIP_WAL selected. If that
assumption is now somehow not correct, we need to fix that.
> Recheck the design and implementation of FSYNC_WAL durability for WAL
> -
>
> Key: HBASE-20471
> URL: https://issues.apache.org/jira/browse/HBASE-20471
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Priority: Major
>
> This is something derived from discussion in HBASE-19024 around [this
> comment|https://issues.apache.org/jira/browse/HBASE-19024?focusedCommentId=16445592=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16445592]
> We have been supplying user the API to set durability per mutation for a long
> time, by design the SYNC_WAL durability to call {{FSDataOutputStream#hflush}}
> and FSYNC_WAL {{FSDataOutputStream#hsync}}, while in implementation we have
> been calling hflush for FSYNC_WAL also until HBASE-19024. Although
> HBASE-19024 tried to fix the syntax with good willing, the implementation
> there cannot assure the FSYNC_WAL edits are truly hsync'ed due to the
> disruptor mechanism used in WAL implementation. Here in this JIRA we aim to
> have more discussion and give it a complete solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20414) TestLockProcedure#testMultipleLocks may fail on slow machine
[
https://issues.apache.org/jira/browse/HBASE-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447497#comment-16447497
]
Chia-Ping Tsai commented on HBASE-20414:
My concern is the test is still based on timer...Let me try the test with your
patch locally.
BTW, does it have chance the value of (HEARTBEAT_TIMEOUT-(now-start)-10) is
negative? If so, that will cause IllegalArgumentException.
> TestLockProcedure#testMultipleLocks may fail on slow machine
>
>
> Key: HBASE-20414
> URL: https://issues.apache.org/jira/browse/HBASE-20414
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20414.v1.txt
>
>
> Here was recent failure :
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/172/testReport/junit/org.apache.hadoop.hbase.master.locking/TestLockProcedure/health_checks___yetus_jdk8_hadoop2_checks___testMultipleLocks/
> {code}
> java.lang.AssertionError: expected: but was:
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.sendHeartbeatAndCheckLocked(TestLockProcedure.java:221)
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.testMultipleLocks(TestLockProcedure.java:311)
> {code}
> In the test output, we can see this:
> {code}
> 2018-04-13 20:19:54,230 DEBUG [Time-limited test]
> locking.TestLockProcedure(225): Proc id 22 : LOCKED.
> ...
> 2018-04-13 20:19:55,529 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(865): Stored pid=26, state=RUNNABLE;
> org.apache.hadoop.hbase.master.locking.LockProcedure
> regions=a7f9adfd047350eabb199a39564ba4db,c1e297609590b707233a2f9c8bb51fa1,
> type=EXCLUSIVE
> 2018-04-13 20:19:56,230 DEBUG [ProcExecTimeout] locking.LockProcedure(207):
> Timeout failure ProcedureEvent for pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE, ready=false, [pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE]
> {code}
> After the pid=26 log, the code does this (1 second wait):
> {code}
> // Assert tables & region locks are waiting because of namespace lock.
> Thread.sleep(HEARTBEAT_TIMEOUT / 2);
> {code}
> On a slow machine (in the case above), there was only 730 msec between the
> queueing of regionsLock2 and the WAITING_TIMEOUT state of the nsLock. The 1
> second wait was too long, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20414) TestLockProcedure#testMultipleLocks may fail on slow machine
[
https://issues.apache.org/jira/browse/HBASE-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447496#comment-16447496
]
Ted Yu commented on HBASE-20414:
See the comment in code above the {{Thread.sleep}} call.
I think the sleep is necessary.
> TestLockProcedure#testMultipleLocks may fail on slow machine
>
>
> Key: HBASE-20414
> URL: https://issues.apache.org/jira/browse/HBASE-20414
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20414.v1.txt
>
>
> Here was recent failure :
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/172/testReport/junit/org.apache.hadoop.hbase.master.locking/TestLockProcedure/health_checks___yetus_jdk8_hadoop2_checks___testMultipleLocks/
> {code}
> java.lang.AssertionError: expected: but was:
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.sendHeartbeatAndCheckLocked(TestLockProcedure.java:221)
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.testMultipleLocks(TestLockProcedure.java:311)
> {code}
> In the test output, we can see this:
> {code}
> 2018-04-13 20:19:54,230 DEBUG [Time-limited test]
> locking.TestLockProcedure(225): Proc id 22 : LOCKED.
> ...
> 2018-04-13 20:19:55,529 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(865): Stored pid=26, state=RUNNABLE;
> org.apache.hadoop.hbase.master.locking.LockProcedure
> regions=a7f9adfd047350eabb199a39564ba4db,c1e297609590b707233a2f9c8bb51fa1,
> type=EXCLUSIVE
> 2018-04-13 20:19:56,230 DEBUG [ProcExecTimeout] locking.LockProcedure(207):
> Timeout failure ProcedureEvent for pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE, ready=false, [pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE]
> {code}
> After the pid=26 log, the code does this (1 second wait):
> {code}
> // Assert tables & region locks are waiting because of namespace lock.
> Thread.sleep(HEARTBEAT_TIMEOUT / 2);
> {code}
> On a slow machine (in the case above), there was only 730 msec between the
> queueing of regionsLock2 and the WAITING_TIMEOUT state of the nsLock. The 1
> second wait was too long, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20468) RPC quota requests ineffective due to not counting multi-actions
[ https://issues.apache.org/jira/browse/HBASE-20468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447491#comment-16447491 ] Andrew Purtell commented on HBASE-20468: +1 Yep we've had this exact same thought over where I work. [~vincentpoon] did some investigation last year. > RPC quota requests ineffective due to not counting multi-actions > > > Key: HBASE-20468 > URL: https://issues.apache.org/jira/browse/HBASE-20468 > Project: HBase > Issue Type: Bug > Components: rpc >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.1.0, 1.5.0, 1.3.3, 1.2.8, 1.4.4, 2.0.1 > > > Was digging into a problem with [~ankit.singhal] where setting RPC quotas on > number of requests wasn't having any effect on a multi-Get > Ankit did enough digging to find that this was because each RPC was being > treated as one request instead of the number of requests contained within the > RPC itself. > Thinking as an operator, this is a pretty ineffective control because a user > could just craft their API usage easily to work around any kind of limits I > want to set to control their impact on the system. TimeBasedLimiter is > assuming that one call to the quota code can only count as one request which > I think is just wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20414) TestLockProcedure#testMultipleLocks may fail on slow machine
[
https://issues.apache.org/jira/browse/HBASE-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447488#comment-16447488
]
Chia-Ping Tsai commented on HBASE-20414:
Is the sleep necessary? If not, Could we remove the sleeper in order to avoid
the timeout?
> TestLockProcedure#testMultipleLocks may fail on slow machine
>
>
> Key: HBASE-20414
> URL: https://issues.apache.org/jira/browse/HBASE-20414
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20414.v1.txt
>
>
> Here was recent failure :
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/172/testReport/junit/org.apache.hadoop.hbase.master.locking/TestLockProcedure/health_checks___yetus_jdk8_hadoop2_checks___testMultipleLocks/
> {code}
> java.lang.AssertionError: expected: but was:
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.sendHeartbeatAndCheckLocked(TestLockProcedure.java:221)
> at
> org.apache.hadoop.hbase.master.locking.TestLockProcedure.testMultipleLocks(TestLockProcedure.java:311)
> {code}
> In the test output, we can see this:
> {code}
> 2018-04-13 20:19:54,230 DEBUG [Time-limited test]
> locking.TestLockProcedure(225): Proc id 22 : LOCKED.
> ...
> 2018-04-13 20:19:55,529 DEBUG [Time-limited test]
> procedure2.ProcedureExecutor(865): Stored pid=26, state=RUNNABLE;
> org.apache.hadoop.hbase.master.locking.LockProcedure
> regions=a7f9adfd047350eabb199a39564ba4db,c1e297609590b707233a2f9c8bb51fa1,
> type=EXCLUSIVE
> 2018-04-13 20:19:56,230 DEBUG [ProcExecTimeout] locking.LockProcedure(207):
> Timeout failure ProcedureEvent for pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE, ready=false, [pid=22, state=WAITING_TIMEOUT;
> org.apache.hadoop.hbase.master.locking.LockProcedure, namespace=namespace,
> type=EXCLUSIVE]
> {code}
> After the pid=26 log, the code does this (1 second wait):
> {code}
> // Assert tables & region locks are waiting because of namespace lock.
> Thread.sleep(HEARTBEAT_TIMEOUT / 2);
> {code}
> On a slow machine (in the case above), there was only 730 msec between the
> queueing of regionsLock2 and the WAITING_TIMEOUT state of the nsLock. The 1
> second wait was too long, leading to assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447481#comment-16447481
]
Andrew Purtell commented on HBASE-17958:
Ok just to confirm - 1.4 and up, not 1.3. So the perf issue we are seeing is in
1.3. You'll probably want to check out branch-1.3 instead of branch-1 Lars.
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 1.4.0, 2.0.0
>
> Attachments: 0001-add-one-ut-testWithColumnCountGetFilter.patch,
> 17958-add.txt, HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch,
> HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch, HBASE-17958-v1.patch,
> HBASE-17958-v2.patch, HBASE-17958-v3.patch, HBASE-17958-v4.patch,
> HBASE-17958-v5.patch, HBASE-17958-v6.patch, HBASE-17958-v7.patch,
> HBASE-17958-v7.patch
>
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-20463) Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL change and document"
[
https://issues.apache.org/jira/browse/HBASE-20463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-20463:
--
Comment: was deleted
(was: Looks like the shell is still broken:
{code:java}
ArgumentError: wrong number of arguments (0 for 1)
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /home/lars/dev/test/hbase/bin/../bin/hirb.rb:38{code}
Although that looks all ruby internal... Might be environmental, but pretty
sure that this has worked a while ago.
NM was me... I'll delete this comment. (had to do with Java 9, apologies))
> Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL
> change and document"
> --
>
> Key: HBASE-20463
> URL: https://issues.apache.org/jira/browse/HBASE-20463
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 1.5.0, 1.4.4
>
> Attachments: HBASE-20463-branch-1.v0.patch, HBASE-20463.0.patch
>
>
> Hope you don't mind my making an issue for fixing branch-1 breakage [~busbey]
> (and [~apurtell]).
> See parent for discussion on breakage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20463) Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL change and document"
[
https://issues.apache.org/jira/browse/HBASE-20463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447477#comment-16447477
]
Lars Hofhansl edited comment on HBASE-20463 at 4/23/18 2:34 AM:
Looks like the shell is still broken:
{code:java}
ArgumentError: wrong number of arguments (0 for 1)
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /home/lars/dev/test/hbase/bin/../bin/hirb.rb:38{code}
Although that looks all ruby internal... Might be environmental, but pretty
sure that this has worked a while ago.
NM was me... I'll delete this comment. (had to do with Java 9, apologies)
was (Author: lhofhansl):
Looks like the shell is still broken:
{code:java}
ArgumentError: wrong number of arguments (0 for 1)
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /home/lars/dev/test/hbase/bin/../bin/hirb.rb:38{code}
Although that looks all ruby internal... Might be environmental, but pretty
sure that this has worked a while ago.
NM was me... I'll delete this comment.
> Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL
> change and document"
> --
>
> Key: HBASE-20463
> URL: https://issues.apache.org/jira/browse/HBASE-20463
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 1.5.0, 1.4.4
>
> Attachments: HBASE-20463-branch-1.v0.patch, HBASE-20463.0.patch
>
>
> Hope you don't mind my making an issue for fixing branch-1 breakage [~busbey]
> (and [~apurtell]).
> See parent for discussion on breakage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20463) Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL change and document"
[
https://issues.apache.org/jira/browse/HBASE-20463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447477#comment-16447477
]
Lars Hofhansl edited comment on HBASE-20463 at 4/23/18 2:33 AM:
Looks like the shell is still broken:
{code:java}
ArgumentError: wrong number of arguments (0 for 1)
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /home/lars/dev/test/hbase/bin/../bin/hirb.rb:38{code}
Although that looks all ruby internal... Might be environmental, but pretty
sure that this has worked a while ago.
NM was me... I'll delete this comment.
was (Author: lhofhansl):
Looks like the shell is still broken:
{code:java}
ArgumentError: wrong number of arguments (0 for 1)
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /home/lars/dev/test/hbase/bin/../bin/hirb.rb:38{code}
Although that looks all ruby internal... Might be environmental, but pretty
sure that this has worked a while ago.
> Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL
> change and document"
> --
>
> Key: HBASE-20463
> URL: https://issues.apache.org/jira/browse/HBASE-20463
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 1.5.0, 1.4.4
>
> Attachments: HBASE-20463-branch-1.v0.patch, HBASE-20463.0.patch
>
>
> Hope you don't mind my making an issue for fixing branch-1 breakage [~busbey]
> (and [~apurtell]).
> See parent for discussion on breakage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20463) Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL change and document"
[
https://issues.apache.org/jira/browse/HBASE-20463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447477#comment-16447477
]
Lars Hofhansl commented on HBASE-20463:
---
Looks like the shell is still broken:
{code:java}
ArgumentError: wrong number of arguments (0 for 1)
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:10
method_added at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/core_ext/object.rb:129
Pattern at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:2
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:1
require at org/jruby/RubyKernel.java:1062
(root) at
file:/home/lars/dev/test/hbase-1.5.0-SNAPSHOT/lib/jruby-complete-1.6.8.jar!/builtin/java/java.util.regex.rb:42
(root) at /home/lars/dev/test/hbase/bin/../bin/hirb.rb:38{code}
Although that looks all ruby internal... Might be environmental, but pretty
sure that this has worked a while ago.
> Fix breakage introduced on branch-1 by HBASE-20276 "[shell] Revert shell REPL
> change and document"
> --
>
> Key: HBASE-20463
> URL: https://issues.apache.org/jira/browse/HBASE-20463
> Project: HBase
> Issue Type: Bug
> Components: shell
>Reporter: stack
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 1.5.0, 1.4.4
>
> Attachments: HBASE-20463-branch-1.v0.patch, HBASE-20463.0.patch
>
>
> Hope you don't mind my making an issue for fixing branch-1 breakage [~busbey]
> (and [~apurtell]).
> See parent for discussion on breakage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447467#comment-16447467
]
Reid Chan commented on HBASE-20151:
---
ping [~chia7712], would you mind taking a look at this patch, thanks in advance.
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447456#comment-16447456
]
Vikas Vishwakarma commented on HBASE-17958:
---
[~lhofhansl] [~Apache9] we are ok on this request for now. We have observed the
issue against 1.3 build. This patch seems to be present only in 1.4 . I
verified that this is not cherry picked in our internal 1.3 light fork also.
Once we are done with debugging the 1.3 issue, we anyways have a plan to do a
0.98 vs 1.3 vs 1.4 benchmark. Will get back on this request when we do the 1.4
benchmark. [~apurtell]
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 1.4.0, 2.0.0
>
> Attachments: 0001-add-one-ut-testWithColumnCountGetFilter.patch,
> 17958-add.txt, HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch,
> HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch, HBASE-17958-v1.patch,
> HBASE-17958-v2.patch, HBASE-17958-v3.patch, HBASE-17958-v4.patch,
> HBASE-17958-v5.patch, HBASE-17958-v6.patch, HBASE-17958-v7.patch,
> HBASE-17958-v7.patch
>
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447453#comment-16447453
]
Chia-Ping Tsai commented on HBASE-19746:
Assume we remove the default impl of getType() in 2.0. User who implemented
their own version of Cell need to revise their code (extend the getType()) in
order to succeed to update hbase from 1.x to 2.0. (and I assume this change
break the API compatibility)
If we keep default impl of getType() in 2.0, user will be doable to compile
their code without adding impl to getType() after updating hbase from 1 to 2.
And then they must impl the getType() in 3.0 because getType() will be abstract
in 3.0. I think this scenario is reasonable because updating from 1 to 3 have
no any compatibility guarantee. I mean we can't expect the code based on hbase
1.x is compatible with hbase 3.0 API. Of course, we need to add more clear
comment to getTypeByte to suggest user to implement getType() and remove the
"override" annotation from getTypeByte if they want to migrate code from 2 to 3
smoothly.
[~lars_francke] Please correct me if i misunderstand the API compatibility of
hbase. If there are no other users who have their variant of Cell, reverting
this patch won't hurt hbase I'd say.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20473) Ineffective INFO logging adjustment in HFilePerformanceEvaluation
Ted Yu created HBASE-20473:
--
Summary: Ineffective INFO logging adjustment in
HFilePerformanceEvaluation
Key: HBASE-20473
URL: https://issues.apache.org/jira/browse/HBASE-20473
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
{code}
// Disable verbose INFO logging from org.apache.hadoop.io.compress.CodecPool
static {
System.setProperty("org.apache.commons.logging.Log",
"org.apache.commons.logging.impl.SimpleLog");
{code}
The above code has no effect since we're migrating away from commons-logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20438) Add an HBase antipattern check for reintroducing commons-logging
[
https://issues.apache.org/jira/browse/HBASE-20438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20438:
--
Fix Version/s: (was: 2.0.1)
2.0.0
> Add an HBase antipattern check for reintroducing commons-logging
>
>
> Key: HBASE-20438
> URL: https://issues.apache.org/jira/browse/HBASE-20438
> Project: HBase
> Issue Type: Improvement
> Components: dependencies, test
>Affects Versions: 3.0.0, 2.1.0, 2.0.0
>Reporter: Sean Busbey
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-20438.master.001.patch
>
>
> We moved to slf4j in HBASE-10092, but looking at our source tree we've had
> some regression back to commons-logging:
> {code}
> $ git grep -E "org.apache.commons.logging.Log(Factory|;)"
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/zksyncer/ClientZKSyncer.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/zksyncer/ClientZKSyncer.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileArchiverNotifierImpl.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileArchiverNotifierImpl.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionSizeReportingChore.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionSizeReportingChore.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionSizeStoreImpl.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionSizeStoreImpl.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/StoreHotnessProtector.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterPortAssignment.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterPortAssignment.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFlushFromClient.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFlushFromClient.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSeparateClientZKCluster.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSeparateClientZKCluster.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestFailedProcCleanup.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestFailedProcCleanup.java:import
> org.apache.commons.logging.LogFactory;
> hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDisabledWAL.java:import
> org.apache.commons.logging.Log;
> hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDisabledWAL.java:import
> org.apache.commons.logging.LogFactory;
> {code}
> We should do the same kind of check that we do to avoid e.g. the Hadoop
> annotations
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20443) Use checkstyle to ban imports from commons-collections 3
[
https://issues.apache.org/jira/browse/HBASE-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20443:
--
Fix Version/s: (was: 2.0.1)
2.0.0
> Use checkstyle to ban imports from commons-collections 3
>
>
> Key: HBASE-20443
> URL: https://issues.apache.org/jira/browse/HBASE-20443
> Project: HBase
> Issue Type: Task
> Components: dependencies, test
>Reporter: Sean Busbey
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-20443.master.001.patch,
> HBASE-20443.master.002.patch
>
>
> we upgraded to commons-collections 4 in HBASE-18704 and then to an
> internal-only hbase-thirdparty version in HBASE-20223, but we've regressed:
> {code}
> $ git grep "import org.apache.commons.collections"
> hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/master/BackupLogCleaner.java:import
> org.apache.commons.collections.MapUtils;
> hbase-client/src/main/java/org/apache/hadoop/hbase/client/RowMutations.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ZKReplicationQueueStorage.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java:import
> org.apache.commons.collections.CollectionUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java:import
> org.apache.commons.collections.MapUtils;
> {code}
> we should add the same kind of check as we do for e.g. hadoop annotations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20441) Use checkstyle to ban imports from commons-lang 2
[
https://issues.apache.org/jira/browse/HBASE-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20441:
--
Fix Version/s: (was: 2.0.1)
2.0.0
> Use checkstyle to ban imports from commons-lang 2
> -
>
> Key: HBASE-20441
> URL: https://issues.apache.org/jira/browse/HBASE-20441
> Project: HBase
> Issue Type: Task
> Components: dependencies, test
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-20441.master.001.patch
>
>
> We updated to commons-lang 3 in HBASE-18674 but we've regressed:
> {code}
> $ git grep org.apache.commons.lang\\.
> hbase-common/src/main/java/org/apache/hadoop/hbase/net/Address.java:import
> org.apache.commons.lang.StringUtils;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileArchiverNotifierFactoryImpl.java:import
> org.apache.commons.lang.builder.HashCodeBuilder;
> hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileArchiverNotifierImpl.java:import
> org.apache.commons.lang.builder.HashCodeBuilder;
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHdfsSnapshotHRegion.java:import
> org.apache.commons.lang.StringUtils;
> hbase-server/src/test/java/org/apache/hadoop/hbase/util/compaction/TestMajorCompactionRequest.java:import
> org.apache.commons.lang.RandomStringUtils;
> {code}
> we should add the same kind of check as we do for e.g. hadoop annotations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19547) HBase fails building on AArch64 due to asciidoctor-maven-plugin.
[
https://issues.apache.org/jira/browse/HBASE-19547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19547:
--
Fix Version/s: (was: 2.0.1)
2.0.0
> HBase fails building on AArch64 due to asciidoctor-maven-plugin.
>
>
> Key: HBASE-19547
> URL: https://issues.apache.org/jira/browse/HBASE-19547
> Project: HBase
> Issue Type: Bug
> Components: build, documentation, website
>Affects Versions: 3.0.0
>Reporter: Yuqi Gu
>Assignee: Yuqi Gu
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-19547.patch
>
>
> HBase fails building on AArch64 due to asciidoctor-maven-plugin.
> {code:java}
> mvn clean site package install -DskipTests
> [ERROR] Failed to execute
> goal org.asciidoctor:asciidoctor-maven-plugin:1.5.5:
> process-asciidoc (output-pdf) on project hbase:
> Execution output-pdf of goal
> org.asciidoctor:asciidoctor-maven-plugin:1.5.5:
> process-asciidoc failed: (NotImplementedError)
> fstat unimplemented unsupported or native support failed to load ->
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20459) Majority of scan CPU time in HBase-1 spent in size estimation
[ https://issues.apache.org/jira/browse/HBASE-20459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20459: -- Fix Version/s: (was: 2.0.1) 2.0.0 > Majority of scan CPU time in HBase-1 spent in size estimation > - > > Key: HBASE-20459 > URL: https://issues.apache.org/jira/browse/HBASE-20459 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 1.4.3 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 1.5.0, 1.4.4, 2.0.0 > > Attachments: 20459-v2.txt, 20459.2.0.txt, 20459.txt, > HBASE-20459.branch-2.0.001.patch, Screenshot_20180419_162559.png > > > See attached screenshot. Will look into a fix later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20465) Fix TestEnableRSGroup flaky
[
https://issues.apache.org/jira/browse/HBASE-20465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447400#comment-16447400
]
stack edited comment on HBASE-20465 at 4/22/18 10:40 PM:
-
Pushed to branch-2.0+. Lets try it. Thanks [~balazs.meszaros] for digging in on
this flakie that never seems to leave the flakies' charts.
was (Author: stack):
Pushed to branch-2.0+. Lets try it. Thanks for digging in on this flakie that
never seems to leave the flakies' charts.
> Fix TestEnableRSGroup flaky
> ---
>
> Key: HBASE-20465
> URL: https://issues.apache.org/jira/browse/HBASE-20465
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20465.master.001.patch
>
>
> Most recent {{TestEnableRSGroup}} tests failed on branch-2.0 according to our
> flaky dashboard.
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.rsgroup.TestEnableRSGroup.testEnableRSGroup(TestEnableRSGroup.java:80)
> {code}
> TestEnableRSGroup:80:
> {code:java}
> // check if master started successfully
> assertTrue(TEST_UTIL.getMiniHBaseCluster().getMaster() != null);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20465) Fix TestEnableRSGroup flaky
[
https://issues.apache.org/jira/browse/HBASE-20465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20465:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2.0+. Lets try it. Thanks for digging in on this flakie that
never seems to leave the flakies' charts.
> Fix TestEnableRSGroup flaky
> ---
>
> Key: HBASE-20465
> URL: https://issues.apache.org/jira/browse/HBASE-20465
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20465.master.001.patch
>
>
> Most recent {{TestEnableRSGroup}} tests failed on branch-2.0 according to our
> flaky dashboard.
> {code}
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.rsgroup.TestEnableRSGroup.testEnableRSGroup(TestEnableRSGroup.java:80)
> {code}
> TestEnableRSGroup:80:
> {code:java}
> // check if master started successfully
> assertTrue(TEST_UTIL.getMiniHBaseCluster().getMaster() != null);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[ https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20470: -- Resolution: Fixed Assignee: stack Status: Resolved (was: Patch Available) Pushed to branch-2.0+ after fixing checkstyle. > [2.0.0RC1] has broken unit tests... > > > Key: HBASE-20470 > URL: https://issues.apache.org/jira/browse/HBASE-20470 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-20470.branch-2.0.001.patch, > HBASE-20470.branch-2.0.002.patch, HBASE-20470.branch-2.0.003.patch, > HBASE-20470.branch-2.0.003.patch > > > Found by [~uagashe] I think its because some depend on IMC and it was > disabled just before I made RC1. Let me try a nothing change and see. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447285#comment-16447285
]
Chia-Ping Tsai commented on HBASE-19746:
{quote}I do not understand your comments about API compatibility. What exactly
do you mean here?
{quote}
You can compile your code without any changing after updating the hbase version
from 1.x to 2.0. It means you can change the version of hbase dependencies and
then still succeed to build the project.
{quote}It's a major version so we are allowed to break API. We do this either
in 2.0 by _not_ adding a default or in 3.0 by adding the _default_ method and
removing it later. So for me that's not an argument.
{quote}
Pardon me. I just catch your point. :( So you prefer to remove the default
impl (The API is break here) to enforce user to implement the #getType in 2.0
rather than delaying the break to 3.0? If so, I'm fine to revert this patch.
{quote}It's getting late here so I'll wait until the US wakes up on Monday and
will see if anyone else has an opinion. If no one jumps in I'll ping the dev
list.
{quote}
:)
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447278#comment-16447278
]
Lars Francke commented on HBASE-19746:
--
Ha, I thought we had it but apparently not :)
I do not understand your comments about API compatibility. What exactly do you
mean here? It's a major version so we are allowed to break API. We do this
either in 2.0 by _not_ adding a default or in 3.0 by adding the _default_
method and removing it later. So for me that's not an argument.
I think we just disagree on how to proceed?
It's getting late here so I'll wait until the US wakes up on Monday and will
see if anyone else has an opinion. If no one jumps in I'll ping the dev list.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447275#comment-16447275
]
Chia-Ping Tsai commented on HBASE-19746:
{quote}I'm advocating removing the default implementation of getType for 2.0.
Basically reverting this patch and instead implementing getType in all the Cell
implementations.
{quote}
As you mentioned, it's probably already too late for that. Also, I'm not sure
whether breaking the API compatibility for inheritance is right.
{quote}That said: If I understand this correctly then all we're disagreeing on
is whether we want to break BC from 1 -> 2 or from 2 ->3. In my opinion it
should be sooner rather than later and if I understand you correctly you'd
rather do it later and leave the default implementation in there.
{quote}
I prefer to remove the default impl in 3.0 because removing it in 2.0 breaks
not only BC but also API compatibility. Breaking the BC in major version is
acceptable. Breaking API compatibility make user fail to compile their code
without any changes after updating hbase from 1.x to 2.0.
{quote}My reasoning for doing it sooner rather than later is that we have no
way to add a compile-time warning that warns of the impeding removal of the
default implementation. To be honest I have no idea how to proceed.
What we do agree on is that the Javadoc for getType needs to be clarified,
right?
{quote}
You're absolutely right.
{quote}Actually I thought of one thing: We could add @deprecated to getType()
with an explanation. It's a bit ambiguous but it would warn users at least. And
then in 3.0 remove the default implementation along with the @deprecated tag.
{quote}
This workaround is ok to me if we can add clear comment to getType()
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447250#comment-16447250
]
Lars Francke edited comment on HBASE-19746 at 4/22/18 2:27 PM:
---
Actually I thought of one thing: We could add @deprecated to getType() with an
explanation. It's a bit ambiguous but it would warn users at least. And then in
3.0 remove the default implementation along with the @deprecated tag.
was (Author: lars_francke):
Actually I thought of one thing: We could add @deprecated to getType() with an
explanation. It's a bit ambiguous but it would warn users at least.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447250#comment-16447250
]
Lars Francke commented on HBASE-19746:
--
Actually I thought of one thing: We could add @deprecated to getType() with an
explanation. It's a bit ambiguous but it would warn users at least.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447248#comment-16447248
]
Lars Francke commented on HBASE-19746:
--
Sorry, I didn't make myself clear.
I'm not advocating removing getTypeByte for 2.0, this one can go in 3.0. We're
both agreeing on that one :)
I'm advocating removing the default implementation of getType for 2.0.
Basically reverting this patch and instead implementing getType in all the Cell
implementations.
I know this is anecdotal (just like yours I assume?) but I haven't seen a
single customer who's implemented their own version of Cell but I agree with
your reasoning.
That said: If I understand this correctly then all we're disagreeing on is
whether we want to break BC from 1 -> 2 or from 2 ->3. In my opinion it should
be sooner rather than later and if I understand you correctly you'd rather do
it later and leave the default implementation in there.
My reasoning for doing it sooner rather than later is that we have no way to
add a compile-time warning that warns of the impeding removal of the default
implementation. To be honest I have no idea how to proceed.
What we do agree on is that the Javadoc for getType needs to be clarified,
right?
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447227#comment-16447227
]
Chia-Ping Tsai commented on HBASE-19746:
{quote}I'd even be happy to try and provide a patch to remove it for 2.0 but
it's probably already too late for that?
{quote}
This is what I tried to do...Unfortunately, we encouraged user to write their
Cell impl for a long time since all of Put, Delete, Increment and Append have
the method - add(Cell). Hence, many users who had built custom Cell will be
hurt if we remove the #getTypeByte from Cell in 2.0. For example, our
application have 1x Cell impl so the compile error burst my build when I just
updated the hbase version from 1.x to 2.0-snapshot. By contrast, we don't
encourage user to build their custom Table so adding a new stuff to Table is
ok. The inheritance to Cell is a ambiguous stuff against our compatibility
rule. That is why I raised HBASE-18519 to add the cell builder to prevent user
from writing custom Cell impl, and the purpose of HBASE-19535 is to write the
new compatibility rule...
The deprecated cycle applied to Cell is weird I think. However, it approve us
to remove the #getTypeByte in 3.0 and this behavior doesn't break our rule
totally...
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[
https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447226#comment-16447226
]
Hadoop QA commented on HBASE-20470:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} hbase-server: The patch generated 1 new + 26 unchanged
- 0 fixed = 27 total (was 26) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 58s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}150m
6s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:369877d |
| JIRA Issue | HBASE-20470 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920196/HBASE-20470.branch-2.0.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7acc6b1c547e 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.0 / 9d0a2e850e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/12592/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12592/testReport/ |
| Max. process+thread count | 4093 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
|
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447217#comment-16447217
]
Lars Francke commented on HBASE-19746:
--
Thanks for helping clarify. In that case my -0 stands. In Table for example we
also added stuff (checkAndMutate) without adding a default.
I'd even be happy to try and provide a patch to remove it for 2.0 but it's
probably already too late for that?
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[ https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447179#comment-16447179 ] stack commented on HBASE-20470: --- Retry. TestReplicationKillSlaveRS.xml.[failed-to-read] flakey. > [2.0.0RC1] has broken unit tests... > > > Key: HBASE-20470 > URL: https://issues.apache.org/jira/browse/HBASE-20470 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: HBASE-20470.branch-2.0.001.patch, > HBASE-20470.branch-2.0.002.patch, HBASE-20470.branch-2.0.003.patch, > HBASE-20470.branch-2.0.003.patch > > > Found by [~uagashe] I think its because some depend on IMC and it was > disabled just before I made RC1. Let me try a nothing change and see. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[ https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20470: -- Attachment: HBASE-20470.branch-2.0.003.patch > [2.0.0RC1] has broken unit tests... > > > Key: HBASE-20470 > URL: https://issues.apache.org/jira/browse/HBASE-20470 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: HBASE-20470.branch-2.0.001.patch, > HBASE-20470.branch-2.0.002.patch, HBASE-20470.branch-2.0.003.patch, > HBASE-20470.branch-2.0.003.patch > > > Found by [~uagashe] I think its because some depend on IMC and it was > disabled just before I made RC1. Let me try a nothing change and see. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20459) Majority of scan CPU time in HBase-1 spent in size estimation
[
https://issues.apache.org/jira/browse/HBASE-20459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447177#comment-16447177
]
Hudson commented on HBASE-20459:
Results for branch branch-2.0
[build #208 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/208/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/208//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/208//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/208//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Majority of scan CPU time in HBase-1 spent in size estimation
> -
>
> Key: HBASE-20459
> URL: https://issues.apache.org/jira/browse/HBASE-20459
> Project: HBase
> Issue Type: Improvement
> Components: Performance, scan
>Affects Versions: 1.4.3
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 1.5.0, 1.4.4, 2.0.1
>
> Attachments: 20459-v2.txt, 20459.2.0.txt, 20459.txt,
> HBASE-20459.branch-2.0.001.patch, Screenshot_20180419_162559.png
>
>
> See attached screenshot. Will look into a fix later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20459) Majority of scan CPU time in HBase-1 spent in size estimation
[
https://issues.apache.org/jira/browse/HBASE-20459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447175#comment-16447175
]
Hudson commented on HBASE-20459:
Results for branch branch-2
[build #645 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/645/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/645//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/645//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/645//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Majority of scan CPU time in HBase-1 spent in size estimation
> -
>
> Key: HBASE-20459
> URL: https://issues.apache.org/jira/browse/HBASE-20459
> Project: HBase
> Issue Type: Improvement
> Components: Performance, scan
>Affects Versions: 1.4.3
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 1.5.0, 1.4.4, 2.0.1
>
> Attachments: 20459-v2.txt, 20459.2.0.txt, 20459.txt,
> HBASE-20459.branch-2.0.001.patch, Screenshot_20180419_162559.png
>
>
> See attached screenshot. Will look into a fix later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447172#comment-16447172
]
Chia-Ping Tsai commented on HBASE-19746:
{quote}In 3.0 we need to remove the default implementation, right? Because we
can't get to getTypeByte anymore which we need for the implementation.
{quote}
Yep.
{quote}So we'll break BC in 3.0 for two different reasons.
{quote}
You are right.
{quote}I still don't like it but that is also not documented anywhere. Either
we undeprecate getTypeByte or we need to mark getType as deprecated somehow
because it will effectively change in 3.0.
{quote}
Agreed. Perhaps we should file a Jira to doc how to update from 1.x to 2.0?
IIRC, there are some issues but I'm not sure whether the Cell changes are
included.
{quote}You're saying that you suggested the user to implement getType but
that's not the case as far as I can tell. There's no such suggestion. And
unlike a "@deprecation" tag we cannot really enforce such a thing at compile
time.
{quote}
You are right again. Are there any better way to reduce the pain of removing
methods from Public class?
{quote}What are we gaining by including the default implementation?
{quote}
API compatibility? IIRC, recompiling should work when updating to next major
release.
{quote}The alternative is to implement it in all our implementing classes in
2.0 (which we'll otherwise have to do in 3.0), right?
{quote}
Yep. But it will introduce many duplicate code.
{quote}The BC issue doesn't really matter as we'll have to do it anyway,
whether we do it in 2.0 or 3.0 doesn't really make a difference, no?
{quote}
Ya... the cell impl user must update their code for either 2.0 or 3.0 since we
will do remove a method from Cell in 3.0. The API compatibility will break in
3.0 but it is valid since we have deprecated the getTypeByte...ya, it is a
unfriendly way...
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447159#comment-16447159
]
Lars Francke commented on HBASE-19746:
--
Thanks. Understood.
That brings me back to the original question though.
In 3.0 we need to remove the default implementation, right? Because we can't
get to getTypeByte anymore which we need for the implementation.
So we'll break BC in 3.0 for two different reasons. I still don't like it but
that is also not documented anywhere. Either we undeprecate getTypeByte or we
need to mark getType as deprecated somehow because it will effectively change
in 3.0.
You're saying that you suggested the user to implement getType but that's not
the case as far as I can tell. There's no such suggestion. And unlike a
"@deprecation" tag we cannot really enforce such a thing at compile time.
What are we gaining by including the default implementation? The alternative is
to implement it in all our implementing classes in 2.0 (which we'll otherwise
have to do in 3.0), right? The BC issue doesn't really matter as we'll have to
do it anyway, whether we do it in 2.0 or 3.0 doesn't really make a difference,
no?
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447142#comment-16447142
]
Chia-Ping Tsai edited comment on HBASE-19746 at 4/22/18 8:25 AM:
-
{quote}In that case we'd need to deprecate `Cell#getType()` in 2.0 but that has
not happened. It's not marked as deprecated. If that's the intention I can
provide a patch to do so hoping [~stack] can still include it in 2.0.
{quote}
My bad. It is typo. The method should be removed in 3.0 is Cell#getTypeByte.
{quote}So we remove both #getTypeByte and #getType in 3.0?
{quote}
Only Cell#getTypeByte will be removed in 3.0
was (Author: chia7712):
{quote}In that case we'd need to deprecate `Cell#getType()` in 2.0 but that has
not happened. It's not marked as deprecated. If that's the intention I can
provide a patch to do so hoping [~stack] can still include it in 2.0.
{quote}
My bad. It is typo. The method should be removed in 3.0 is Cell#getTypeByte.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447122#comment-16447122
]
Chia-Ping Tsai edited comment on HBASE-19746 at 4/22/18 8:25 AM:
-
hi [~lars_francke] Thanks for the nice question.
{quote}So for 3.0 we remove the default implementation again?
{quote}
Not really. The API which will be removed in 3.0 is -Cell#getType- (typo...It
should be Cell#getTypeByte). We have deprecated the Cell#getTypeByte and
suggested user to implement the Cell#getType also in 2.0. Hence, we can remove
Cell#getTypeByte in 3.0.
{quote}And you do this to not break API compatibility between 1.x and 2.x?
{quote}
Yep. this issue is to keep the bc of Public class (Cell).
{quote}As far as I understand the compatibility rules in the book (which I
still have on my todo list to clarify) we don't guarantee any compatibility
between Major releases.
{quote}
IIRC, our rules to Public class is we must deprecate the API in a whole major
release before we really remove them in next major release. Also, we should
introduce the replacement for deprecated APIs.
{quote}Otherwise how would we ever add new methods to interfaces for which we
cannot provide a default implementation?
{quote}
Ya, we have discussed this issue before - how we keep the bc for the
inheritance. see HBASE-19535. If we can introduce an annotation to say the bc
of inheritance to specified Public class isn't guaranteed, our life will be
more easier. BTW, the BC is fine if the impl is from hbase.
{quote}If this is the case I'm -0 to include this. Doesn't this just shift the
problem to the next version?
{quote}
Not really. We create a room to remove the Cell#getType in 3.0.
{quote} There's nothing notifying implementors of the Cell interface of this
new default method (unlike with @deprecated things)
{quote}
I expect that hbase user should check the doc of deprecated API to find out the
replacement. The comment is shown below.
{code:java}
/**
* @return The byte representation of the KeyValue.TYPE of this cell: one of
Put, Delete, etc
* @deprecated As of HBase-2.0. Will be removed in HBase-3.0. Use {@link
#getType()}.
*/{code}
Of course, it would be better to have a more readable way to reminder user to
use new APIs.
{quote}There's also nothing in the release notes and the Javadoc for getType
also doesn't notify users about what's going to happen.
{quote}
You are right. We make a considerable change to Cell APIs (RawCell,
ExtendedCell, CelBuilder, and so on) but no fat docs are added. I can file a
Jira to write the use case to hbase doc for the new usage of Cell.
was (Author: chia7712):
hi [~lars_francke] Thanks for the nice question.
{quote}So for 3.0 we remove the default implementation again?
{quote}
Not really. The API which will be removed in 3.0 is Cell#getType. We have
deprecated the Cell#getTypeByte and suggested user to implement the
Cell#getType also in 2.0. Hence, we can remove Cell#getTypeByte in 3.0.
{quote}And you do this to not break API compatibility between 1.x and 2.x?
{quote}
Yep. this issue is to keep the bc of Public class (Cell).
{quote}As far as I understand the compatibility rules in the book (which I
still have on my todo list to clarify) we don't guarantee any compatibility
between Major releases.
{quote}
IIRC, our rules to Public class is we must deprecate the API in a whole major
release before we really remove them in next major release. Also, we should
introduce the replacement for deprecated APIs.
{quote}Otherwise how would we ever add new methods to interfaces for which we
cannot provide a default implementation?
{quote}
Ya, we have discussed this issue before - how we keep the bc for the
inheritance. see HBASE-19535. If we can introduce an annotation to say the bc
of inheritance to specified Public class isn't guaranteed, our life will be
more easier. BTW, the BC is fine if the impl is from hbase.
{quote}If this is the case I'm -0 to include this. Doesn't this just shift the
problem to the next version?
{quote}
Not really. We create a room to remove the Cell#getType in 3.0.
{quote} There's nothing notifying implementors of the Cell interface of this
new default method (unlike with @deprecated things)
{quote}
I expect that hbase user should check the doc of deprecated API to find out the
replacement. The comment is shown below.
{code:java}
/**
* @return The byte representation of the KeyValue.TYPE of this cell: one of
Put, Delete, etc
* @deprecated As of HBase-2.0. Will be removed in HBase-3.0. Use {@link
#getType()}.
*/{code}
Of course, it would be better to have a more readable way to reminder user to
use new APIs.
{quote}There's also nothing in the release notes and the Javadoc for getType
also doesn't notify users about what's going to happen.
{quote}
You are right. We make a considerable change to Cell APIs (RawCell,
ExtendedCell,
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447142#comment-16447142
]
Chia-Ping Tsai commented on HBASE-19746:
{quote}In that case we'd need to deprecate `Cell#getType()` in 2.0 but that has
not happened. It's not marked as deprecated. If that's the intention I can
provide a patch to do so hoping [~stack] can still include it in 2.0.
{quote}
My bad. It is typo. The method should be removed in 3.0 is Cell#getTypeByte.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447136#comment-16447136
]
Lars Francke commented on HBASE-19746:
--
Thanks for the quick reply. Unfortunately I still don't get it.
{quote}Not really. The API which will be removed in 3.0 is Cell#getType. We
have deprecated the Cell#getTypeByte and suggested user to implement the
Cell#getType also in 2.0. Hence, we can remove Cell#getTypeByte in 3.0.{quote}
A few things:
In that case we'd need to deprecate `Cell#getType()` in 2.0 but that has not
happened. It's not marked as deprecated. If that's the intention I can provide
a patch to do so hoping [~stack] can still include it in 2.0.
So we remove both #getTypeByte and #getType in 3.0?
And I don't see a suggestion to implement getType anywhere tbh.
{quote}Yep. this issue is to keep the bc of Public class (Cell).{quote}
But we _won't_ keep it, no? With your plan we'll break BC with 3.0 anyway?
{quote}IIRC, our rules to Public class is we must deprecate the API in a whole
major release before we really remove them in next major release. Also, we
should introduce the replacement for deprecated APIs.{quote}
We discussed this at length last time I did a round of deprecation removals
(see HBASE-13462 and ) and we found out that the rules are ambiguous and that
everyone reads them differently. So we have both options.
{quote}I expect that hbase user should check the doc of deprecated API to find
out the replacement. The comment is shown below.{quote}
True! For getCellByte there's a deprecation tag but there's nothing in there
noting that the default implementation - or indeed the whole method - of
getType will go away.
Reading all of this I'm still at -0 on this. If I read you correctly the
_least_ we need to do is add a @deprecated tag to getType and an explanatory
note. But to be honest...why remove one method in favor of another just to
immediately deprecate that other method. I don't see how breaking BC in 3.0 for
a single method (removal of getTypeByte) is better than for two (getTypeByte +
getType), granted it's also not much worth but if we're going to take it away
anyway I'd rather not expose a new method that people then might rely on.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[
https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447129#comment-16447129
]
Hadoop QA commented on HBASE-20312:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 21 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} The patch hbase-common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} hbase-server: The patch generated 0 new + 470
unchanged - 2 fixed = 470 total (was 472) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 2s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
26s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}162m 30s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestFromClientSide3 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20312 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920186/HBASE-20312-master.v9.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 07f81b3d617b 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (HBASE-19746) Add default impl to Cell#getType
[
https://issues.apache.org/jira/browse/HBASE-19746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447122#comment-16447122
]
Chia-Ping Tsai commented on HBASE-19746:
hi [~lars_francke] Thanks for the nice question.
{quote}So for 3.0 we remove the default implementation again?
{quote}
Not really. The API which will be removed in 3.0 is Cell#getType. We have
deprecated the Cell#getTypeByte and suggested user to implement the
Cell#getType also in 2.0. Hence, we can remove Cell#getTypeByte in 3.0.
{quote}And you do this to not break API compatibility between 1.x and 2.x?
{quote}
Yep. this issue is to keep the bc of Public class (Cell).
{quote}As far as I understand the compatibility rules in the book (which I
still have on my todo list to clarify) we don't guarantee any compatibility
between Major releases.
{quote}
IIRC, our rules to Public class is we must deprecate the API in a whole major
release before we really remove them in next major release. Also, we should
introduce the replacement for deprecated APIs.
{quote}Otherwise how would we ever add new methods to interfaces for which we
cannot provide a default implementation?
{quote}
Ya, we have discussed this issue before - how we keep the bc for the
inheritance. see HBASE-19535. If we can introduce an annotation to say the bc
of inheritance to specified Public class isn't guaranteed, our life will be
more easier. BTW, the BC is fine if the impl is from hbase.
{quote}If this is the case I'm -0 to include this. Doesn't this just shift the
problem to the next version?
{quote}
Not really. We create a room to remove the Cell#getType in 3.0.
{quote} There's nothing notifying implementors of the Cell interface of this
new default method (unlike with @deprecated things)
{quote}
I expect that hbase user should check the doc of deprecated API to find out the
replacement. The comment is shown below.
{code:java}
/**
* @return The byte representation of the KeyValue.TYPE of this cell: one of
Put, Delete, etc
* @deprecated As of HBase-2.0. Will be removed in HBase-3.0. Use {@link
#getType()}.
*/{code}
Of course, it would be better to have a more readable way to reminder user to
use new APIs.
{quote}There's also nothing in the release notes and the Javadoc for getType
also doesn't notify users about what's going to happen.
{quote}
You are right. We make a considerable change to Cell APIs (RawCell,
ExtendedCell, CelBuilder, and so on) but no fat docs are added. I can file a
Jira to write the use case to hbase doc for the new usage of Cell.
> Add default impl to Cell#getType
>
>
> Key: HBASE-19746
> URL: https://issues.apache.org/jira/browse/HBASE-19746
> Project: HBase
> Issue Type: Sub-task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-19746.v0.patch, HBASE-19746.v1.patch,
> HBASE-19746.v1.qa.patch
>
>
> Noticed this issue when migrating the app to branch-2.
> {{Cell}} is IA.Public so it should obey our compatibility rules. Not sure
> whether any related discussion had be in HBASE-19112. It worthwhile, however,
> to raise this issue again.
> FYI [~anoopsamjohn] [~ram_krish] [~stack]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20335) nightly jobs no longer contain machine information
[
https://issues.apache.org/jira/browse/HBASE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447115#comment-16447115
]
Hudson commented on HBASE-20335:
Results for branch HBASE-20335
[build #12 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20335/12/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20335/12//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20335/12//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20335/12//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> nightly jobs no longer contain machine information
> --
>
> Key: HBASE-20335
> URL: https://issues.apache.org/jira/browse/HBASE-20335
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-20335.0.patch, HBASE-20335.1.patch
>
>
> something is up with nightly jobs. they no longer have the machine
> information from HBASE-19228.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20470) [2.0.0RC1] has broken unit tests...
[
https://issues.apache.org/jira/browse/HBASE-20470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447114#comment-16447114
]
Hadoop QA commented on HBASE-20470:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
25s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
13s{color} | {color:red} hbase-server: The patch generated 1 new + 26 unchanged
- 0 fixed = 27 total (was 26) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 35s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:369877d |
| JIRA Issue | HBASE-20470 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920185/HBASE-20470.branch-2.0.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b1a5cf64beee 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.0 / 9d0a2e850e |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/12590/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12590/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-17958) Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
[
https://issues.apache.org/jira/browse/HBASE-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447109#comment-16447109
]
Vikas Vishwakarma commented on HBASE-17958:
---
sure [~lhofhansl] [~Apache9] I will look into it. let's see if this helps
narrow down the issue, then we can try the suggested solutions.
> Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
>
>
> Key: HBASE-17958
> URL: https://issues.apache.org/jira/browse/HBASE-17958
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 1.4.0, 2.0.0
>
> Attachments: 0001-add-one-ut-testWithColumnCountGetFilter.patch,
> 17958-add.txt, HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch,
> HBASE-17958-branch-1.patch, HBASE-17958-branch-1.patch, HBASE-17958-v1.patch,
> HBASE-17958-v2.patch, HBASE-17958-v3.patch, HBASE-17958-v4.patch,
> HBASE-17958-v5.patch, HBASE-17958-v6.patch, HBASE-17958-v7.patch,
> HBASE-17958-v7.patch
>
>
> {code}
> ScanQueryMatcher.MatchCode qcode = matcher.match(cell);
> qcode = optimize(qcode, cell);
> {code}
> The optimize method may change the MatchCode from SEEK_NEXT_COL/SEEK_NEXT_ROW
> to SKIP. But it still pass the next cell to ScanQueryMatcher. It will get
> wrong result when use some filter, etc. ColumnCountGetFilter. It just count
> the columns's number. If pass a same column to this filter, the count result
> will be wrong. So we should avoid passing cell to ScanQueryMatcher when
> optimize SEEK to SKIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
