[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299663#comment-16299663 ] stack commented on HBASE-19148: --- .005 enables normalizer by default. Removes balance by table. Lets see if the latter is responsible for the timeouts we're seeing in the retries here. > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch, HBASE-19148.master.003.patch, > HBASE-19148.master.004 (1).patch, HBASE-19148.master.004.patch, > HBASE-19148.master.005.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19148: -- Attachment: HBASE-19148.master.005.patch > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch, HBASE-19148.master.003.patch, > HBASE-19148.master.004 (1).patch, HBASE-19148.master.004.patch, > HBASE-19148.master.005.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[ https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19559: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0-beta-1 Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks [~zghaobac] for reviewing. > Fix TestLogRolling.testLogRollOnDatanodeDeath > - > > Key: HBASE-19559 > URL: https://issues.apache.org/jira/browse/HBASE-19559 > Project: HBase > Issue Type: Bug > Components: test, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19559.patch, HBASE-19559.patch > > > It appears several times on the failing list of pre commit result. > The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-10092) Move to slf4j
[
https://issues.apache.org/jira/browse/HBASE-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-10092:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Release Note:
We now have slf4j as our front-end. Be careful adding logging from here on out;
make sure it slf4j.
>From here on out, as us devs go, we need to convert log messages from being
>'guarded' -- i.e. surrounded by if (LOG.isDebugEnabled...) -- to instead being
>parameterized log messages. e.g. the latter rather than the former in the
>below:
logger.debug("The new entry is "+entry+".");
logger.debug("The new entry is {}.", entry);
See [1] for background on perf benefits.
Note, FATAL log level is not present in slf4j. It is noted as a Marker but
won't show in logs as a LEVEL.
1. https://www.slf4j.org/faq.html#logging_performance
Status: Resolved (was: Patch Available)
Pushed to branch-2 and master.
Filed HBASE-19577 for changing backend off log4j1, an EOL'd tech.
> Move to slf4j
> -
>
> Key: HBASE-10092
> URL: https://issues.apache.org/jira/browse/HBASE-10092
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Balazs Meszaros
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 10092.txt, 10092v2.txt, HBASE-10092-preview-v0.patch,
> HBASE-10092.master.001.patch, HBASE-10092.master.002.patch,
> HBASE-10092.master.003.patch, HBASE-10092.master.004.patch, HBASE-10092.patch
>
>
> Allows logging with less friction. See http://logging.apache.org/log4j/2.x/
> This rather radical transition can be done w/ minor change given they have an
> adapter for apache's logging, the one we use. They also have and adapter for
> slf4j so we likely can remove at least some of the 4 versions of this module
> our dependencies make use of.
> I made a start in attached patch but am currently stuck in maven dependency
> resolve hell courtesy of our slf4j. Fixing will take some concentration and
> a good net connection, an item I currently lack. Other TODOs are that will
> need to fix our little log level setting jsp page -- will likely have to undo
> our use of hadoop's tool here -- and the config system changes a little.
> I will return to this project soon. Will bring numbers.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19577) Move off log4j1 as our logging backend.
stack created HBASE-19577: - Summary: Move off log4j1 as our logging backend. Key: HBASE-19577 URL: https://issues.apache.org/jira/browse/HBASE-19577 Project: HBase Issue Type: Sub-task Components: logging Reporter: stack See HBASE-10092 for discussion. We have inserted slf4j as our frontend. Need to swap out the 5-year-old log4j1. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15992) Preserve original KeeperException when converted to external exceptions
[
https://issues.apache.org/jira/browse/HBASE-15992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299657#comment-16299657
]
Austin Yan commented on HBASE-15992:
Hi Mr.Andrew,
I met a very similar issue when I run my program for a while(around 20 days),I
restarted my program,it gets back to normal,but days later,it got same issue.My
HBase version is 1.0.2.
When in bad condition,below log is printed serveral time per seconds.
Could you please advise is it same issue to HBase-15992 ?Do we have any patch
so far?
Thanks.
Below is error log:
[ERROR 2017-12-20 10:07:00]
{org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:725} -
hconnection-0x18093947-0x940897f49f808d29,
quorum=sparkstn03ts:24002,sparkstn02ts:24002,sparkstn01ts:24002,
baseZNode=/hbase Received unexpected KeeperException, re-throwing
exceptionorg.apache.zookeeper.KeeperException$AuthFailedException:
KeeperErrorCode = AuthFailed for /hbase/meta-region-server at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1611) at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:360)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:746) at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:482)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:600)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1188)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:304)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:211)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:185)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1252)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1158)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:304)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:323)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:298)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:807) at
com.cmb.zz1.api.zz1hbs.ZZ1HBS_Table.scan(ZZ1HBS_Table.java:448) at
com.cmb.zz1.api.zz1hbs.ZZ1HBS_Table.scan(ZZ1HBS_Table.java:439) at
com.cmb.zz1.api.zz1hbs.ZZ1HBS_Table.scan(ZZ1HBS_Table.java:410) at
com.cmb.adr.wke.adrqimls.ADRQIMLS.rtvImgRcd(ADRQIMLS.java:228) at
com.cmb.adr.wke.adrqimls.ADRQIMLS.qryImgLst(ADRQIMLS.java:151) at
com.cmb.adr.wke.adrqimls.ADRQIMLS.execute(ADRQIMLS.java:44) at
com.cmb.zz1.api.zz1wkm.ZZ1WKM_Monitor.execute(ZZ1WKM_Monitor.java:150) at
com.cmb.zz1.api.zz1wkm.ZZ1WKM_Monitor.execute(ZZ1WKM_Monitor.java:57) at
com.cmb.zz1.api.zz1cms.ZZ1CMS_TCPWrkTask.run(ZZ1CMS_TCPWrkTask.java:39) at
com.cmb.zz1.api.zz1thd.ZZ1THD_ThdExecutor.runWorker(ZZ1THD_ThdExecutor.java:1164)
at
com.cmb.zz1.api.zz1thd.ZZ1THD_ThdExecutor$Worker.run(ZZ1THD_ThdExecutor.java:634)
at java.lang.Thread.run(Thread.java:745)
> Preserve original KeeperException when converted to external exceptions
>
[jira] [Updated] (HBASE-19525) RS side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-19525: - Attachment: HBASE-19525.HBASE-19397.v4.patch > RS side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19525 > URL: https://issues.apache.org/jira/browse/HBASE-19525 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Zheng Hu > Fix For: HBASE-19397 > > Attachments: HBASE-19525.HBASE-19397.v1.patch, > HBASE-19525.HBASE-19397.v2.patch, HBASE-19525.HBASE-19397.v3.patch, > HBASE-19525.HBASE-19397.v4.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19525) RS side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299652#comment-16299652 ] Zheng Hu commented on HBASE-19525: -- After skimmed the code, just need to create/remove hfile-refs znode path when add/remove peer id for peer change operations. so it's easy to do this in patch.v4.an extra issue for bulkload replication will be unnecessary. > RS side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19525 > URL: https://issues.apache.org/jira/browse/HBASE-19525 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Zheng Hu > Fix For: HBASE-19397 > > Attachments: HBASE-19525.HBASE-19397.v1.patch, > HBASE-19525.HBASE-19397.v2.patch, HBASE-19525.HBASE-19397.v3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[ https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299639#comment-16299639 ] stack commented on HBASE-19148: --- Let me try it [~daisuke.kobayashi] > Reevaluate default values of configurations > --- > > Key: HBASE-19148 > URL: https://issues.apache.org/jira/browse/HBASE-19148 > Project: HBase > Issue Type: Bug > Components: defaults >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 2.0.0-beta-1 > > Attachments: > 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch, > HBASE-19148.master.001.patch, HBASE-19148.master.002.patch, > HBASE-19148.master.003.patch, HBASE-19148.master.003.patch, > HBASE-19148.master.004 (1).patch, HBASE-19148.master.004.patch > > > Remove cruft and mythologies. Make descriptions more digestible. Change > defaults given experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19576) Introduce builder for ReplicationPeerConfig and make it immutable
Guanghao Zhang created HBASE-19576: -- Summary: Introduce builder for ReplicationPeerConfig and make it immutable Key: HBASE-19576 URL: https://issues.apache.org/jira/browse/HBASE-19576 Project: HBase Issue Type: Improvement Reporter: Guanghao Zhang Assignee: Guanghao Zhang Will introduce a new ReplicationPeerConfigBuilder. And deprecated the old set* methods in ReplicationPeerConfig. Make the ReplicationPeerConfig we give out be immutable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18625) Splitting of region with replica, doesn't update region list in serverHolding. A server crash leads to overlap.
[
https://issues.apache.org/jira/browse/HBASE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299636#comment-16299636
]
huaxiang sun commented on HBASE-18625:
--
Thanks [~ram_krish], I am commit the patch.
> Splitting of region with replica, doesn't update region list in
> serverHolding. A server crash leads to overlap.
> ---
>
> Key: HBASE-18625
> URL: https://issues.apache.org/jira/browse/HBASE-18625
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.6
>Reporter: Igloo
>Assignee: huaxiang sun
> Fix For: 1.2.8
>
> Attachments: HBASE-18625-branch-1-v001.patch,
> HBASE-18625-branch-1-v002.patch
>
>
> The situation can appear in following steps in release hbase1.2.6
> 1. create 'testtable', 'info', {REGION_REPLICATION=>2}
> 2. write somerecords into 'testtable'
> 3. split the table 'testtable'
> 4. after the spliting, the serverHoldings in RegionStates still holds the
> regioninfo for the replica of parent region
> 5. restart the regionserver where the parent replica-region located
> 6. the offlined replica of parent region will be assigned in
> ServerCrashProcedure.
> hbase hbck 'testtable‘
> ERROR: Region { meta => null, hdfs => null, deployed =>
> qabb-qa-hdp-hbase1,16020,1503022958093;testtable,,1503022907686_0001.42d11cfe195b3cc4d08b2c078a687f6d
> ., replicaId => 1 } not in META, but deployed on
> qabb-qa-hdp-hbase1,16020,1503022958093
> 18 ERROR: No regioninfo in Meta or HDFS. { meta => null, hdfs => null,
> deployed =>
> qabb-qa-hdp-hbase1,16020,1503022958093;testtable,,1503022907686_0001.42d11cfe
>195b3cc4d08b2c078a687f6d., replicaId => 1 }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19525) RS side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299631#comment-16299631 ] Zheng Hu commented on HBASE-19525: -- In patch.v3 , I've not consider much about BulkLoad Replication yet. Let me dig into it. Will create a independent issue if necessary. > RS side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19525 > URL: https://issues.apache.org/jira/browse/HBASE-19525 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Zheng Hu > Fix For: HBASE-19397 > > Attachments: HBASE-19525.HBASE-19397.v1.patch, > HBASE-19525.HBASE-19397.v2.patch, HBASE-19525.HBASE-19397.v3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[
https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299630#comment-16299630
]
Daisuke Kobayashi commented on HBASE-19148:
---
Can we change {{DEFAULT_NORMALIZATION_ENABLED}} to true in
TableDescriptorBuilder.java since the region normalizer seems to be a useful
feature?
{code}
/**
* Constant that denotes whether the table is normalized by default.
*/
public static final boolean DEFAULT_NORMALIZATION_ENABLED = false;
{code}
> Reevaluate default values of configurations
> ---
>
> Key: HBASE-19148
> URL: https://issues.apache.org/jira/browse/HBASE-19148
> Project: HBase
> Issue Type: Bug
> Components: defaults
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments:
> 0002-HBASE-19148-Reevaluate-default-values-of-configurati.patch,
> HBASE-19148.master.001.patch, HBASE-19148.master.002.patch,
> HBASE-19148.master.003.patch, HBASE-19148.master.003.patch,
> HBASE-19148.master.004 (1).patch, HBASE-19148.master.004.patch
>
>
> Remove cruft and mythologies. Make descriptions more digestible. Change
> defaults given experience.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19543) Abstract a replication storage interface to extract the zk specific code
[ https://issues.apache.org/jira/browse/HBASE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299625#comment-16299625 ] Duo Zhang commented on HBASE-19543: --- +1 on immutable and builder. > Abstract a replication storage interface to extract the zk specific code > > > Key: HBASE-19543 > URL: https://issues.apache.org/jira/browse/HBASE-19543 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19543-HBASE-19397.patch, > HBASE-19543-HBASE-19397.patch > > > For now, we will do sanity checks at the same time when updating replication > peer. But this is not a safe way for procedure based replication peer > modification. > For the old zk watcher way, the only thing is updating the data on zk, so if > the data is updated and then we crashes, there is no problem. > For the new procedure way, we need to trigger refresh by ourselves after > updating zk. If we crashes after the updating and before we record the state > change of the procedure, we may fail with IllegalArgumentException when we > execute the procedure next time since the data on zk has already been updated. > So the current way is to do sanity checks in PRE_PEER_MODIFICATION state, and > in UPDATE_STORAGE state we will not do sanity checks any more, just > update(overwrite) the peer storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19543) Abstract a replication storage interface to extract the zk specific code
[
https://issues.apache.org/jira/browse/HBASE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299623#comment-16299623
]
Guanghao Zhang commented on HBASE-19543:
bq. private ReplicationPeerConfig copy(ReplicationPeerConfig peerConfig) {
How about introduce a ReplicationPeerConfigBuilder for this job? Then we can
deprecated the set* methods in ReplicationPeerConfig. For user use case, if he
need update the peer config, he need build a new config and then call
Admin.updateReplicationPeerConfig to update it. We can do this in another issue.
> Abstract a replication storage interface to extract the zk specific code
>
>
> Key: HBASE-19543
> URL: https://issues.apache.org/jira/browse/HBASE-19543
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-19543-HBASE-19397.patch,
> HBASE-19543-HBASE-19397.patch
>
>
> For now, we will do sanity checks at the same time when updating replication
> peer. But this is not a safe way for procedure based replication peer
> modification.
> For the old zk watcher way, the only thing is updating the data on zk, so if
> the data is updated and then we crashes, there is no problem.
> For the new procedure way, we need to trigger refresh by ourselves after
> updating zk. If we crashes after the updating and before we record the state
> change of the procedure, we may fail with IllegalArgumentException when we
> execute the procedure next time since the data on zk has already been updated.
> So the current way is to do sanity checks in PRE_PEER_MODIFICATION state, and
> in UPDATE_STORAGE state we will not do sanity checks any more, just
> update(overwrite) the peer storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19520) Add UTs for the new lock type PEER
[ https://issues.apache.org/jira/browse/HBASE-19520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19520: --- Attachment: HBASE-19520.HBASE-19397.003.patch > Add UTs for the new lock type PEER > -- > > Key: HBASE-19520 > URL: https://issues.apache.org/jira/browse/HBASE-19520 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-19520.HBASE-19397.001.patch, > HBASE-19520.HBASE-19397.002.patch, HBASE-19520.HBASE-19397.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19525) RS side changes for moving peer modification from zk watcher to procedure
[ https://issues.apache.org/jira/browse/HBASE-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-19525: - Attachment: HBASE-19525.HBASE-19397.v3.patch Try Hadoop QA for patch.v3 > RS side changes for moving peer modification from zk watcher to procedure > - > > Key: HBASE-19525 > URL: https://issues.apache.org/jira/browse/HBASE-19525 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Zheng Hu > Fix For: HBASE-19397 > > Attachments: HBASE-19525.HBASE-19397.v1.patch, > HBASE-19525.HBASE-19397.v2.patch, HBASE-19525.HBASE-19397.v3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[ https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299612#comment-16299612 ] Guanghao Zhang commented on HBASE-19559: +1 > Fix TestLogRolling.testLogRollOnDatanodeDeath > - > > Key: HBASE-19559 > URL: https://issues.apache.org/jira/browse/HBASE-19559 > Project: HBase > Issue Type: Bug > Components: test, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19559.patch, HBASE-19559.patch > > > It appears several times on the failing list of pre commit result. > The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19496) Reusing the ByteBuffer in rpc layer corrupt the ServerLoad and RegionLoad
[
https://issues.apache.org/jira/browse/HBASE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299609#comment-16299609
]
stack commented on HBASE-19496:
---
Thanks [~chia7712] Let me add them to dev section in refguide. I have others to
add after this slf4j goes in...
This thing read to commit?
> Reusing the ByteBuffer in rpc layer corrupt the ServerLoad and RegionLoad
> -
>
> Key: HBASE-19496
> URL: https://issues.apache.org/jira/browse/HBASE-19496
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19496.v0.patch, HBASE-19496.v1.patch,
> HBASE-19496.wip.patch
>
>
> {{ServerLoad}} and {{RegionLoad}} store the pb object internally but the
> bytebuffer of pb object may be reused in rpc layer. Hence, the {{ServerLoad}}
> and {{RegionLoad}} which saved by {{HMaster}} will be corrupted if the
> bytebuffer backed is modified.
> This issue doesn't happen on branch-1.
> # netty server was introduced in 2.0 (see HBASE-17263)
> # reusing bytebuffer to read RPC requests was introduced in 2.0 (see
> HBASE-15788)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[ https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299607#comment-16299607 ] Duo Zhang commented on HBASE-19559: --- TestLogRolling passed. I need a +1 here... > Fix TestLogRolling.testLogRollOnDatanodeDeath > - > > Key: HBASE-19559 > URL: https://issues.apache.org/jira/browse/HBASE-19559 > Project: HBase > Issue Type: Bug > Components: test, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19559.patch, HBASE-19559.patch > > > It appears several times on the failing list of pre commit result. > The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19218:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2 and master. Thanks for the help [~easyliangjob]. [~elserj]
et al., I think this should be good now. It was a nice find, a good hole to
plug.
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19218.master.001.patch,
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
> 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223)
> at
> org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731)
> 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=37,
> state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException
> via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace,
> region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException;
> ServerCrashProcedure
>
[jira] [Commented] (HBASE-19543) Abstract a replication storage interface to extract the zk specific code
[
https://issues.apache.org/jira/browse/HBASE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299590#comment-16299590
]
Hadoop QA commented on HBASE-19543:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19397 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} HBASE-19397 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-replication: The patch generated 3 new + 2
unchanged - 0 fixed = 5 total (was 2) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 4 new + 241
unchanged - 5 fixed = 245 total (was 246) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
13s{color} | {color:red} hbase-replication generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}156m 11s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}214m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.replication.TestReplicationDisableInactivePeer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
|
[jira] [Commented] (HBASE-19520) Add UTs for the new lock type PEER
[
https://issues.apache.org/jira/browse/HBASE-19520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299541#comment-16299541
]
Hadoop QA commented on HBASE-19520:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19397 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
29s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} HBASE-19397 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} HBASE-19397 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 24 new + 32
unchanged - 3 fixed = 56 total (was 35) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 1s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903142/HBASE-19520.HBASE-19397.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2f0f5115baac 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19397 / 90d77b5571 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10602/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10602/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10602/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-19566) Fix Checkstyle errors in hbase-client-project
[ https://issues.apache.org/jira/browse/HBASE-19566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299540#comment-16299540 ] Hudson commented on HBASE-19566: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4263 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4263/]) HBASE-19566 Fixed Checkstyle errors in hbase-archetypes and enabled (jan.hentschel: rev d95cf54ba644b18e35a4d88334c674df83e01c7f) * (edit) hbase-archetypes/hbase-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/client/TestHelloHBase.java * (edit) hbase-archetypes/hbase-shaded-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/shaded_client/TestHelloHBase.java * (edit) hbase-archetypes/pom.xml > Fix Checkstyle errors in hbase-client-project > - > > Key: HBASE-19566 > URL: https://issues.apache.org/jira/browse/HBASE-19566 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19566.master.001.patch > > > Fix the remaining Checkstyle errors in the *hbase-client-project* module and > enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19575) add copy constructor to Mutation
Chia-Ping Tsai created HBASE-19575: -- Summary: add copy constructor to Mutation Key: HBASE-19575 URL: https://issues.apache.org/jira/browse/HBASE-19575 Project: HBase Issue Type: Sub-task Reporter: Chia-Ping Tsai Assignee: Chia-Ping Tsai Fix For: 2.0.0-beta-1 Put, Delete, Increment, and Append have similar code in copy constructor. Let us move the copy constructor to the upper class - mutation - so as to remove the duplicate code. BTW, the copy constructor of Put miss the clone of priority. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299522#comment-16299522 ] Hudson commented on HBASE-19491: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #2038 (See [https://builds.apache.org/job/HBase-1.1-JDK8/2038/]) HBASE-19491 Improvements to Nighly runs: Fixes findbugs tests, Exclude (appy: rev bcacdeb212006323c880d70a013de7d7951e2bca) * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/Jenkinsfile * (edit) dev-support/hbase-personality.sh > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[
https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299521#comment-16299521
]
Hadoop QA commented on HBASE-19559:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} hbase-server: The patch generated 0 new + 13
unchanged - 2 fixed = 13 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 50s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 16s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19559 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903143/HBASE-19559.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 781fd0406e3f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d95cf54ba6 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10603/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10603/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10603/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Fix TestLogRolling.testLogRollOnDatanodeDeath
>
[jira] [Commented] (HBASE-18840) Add functionality to refresh meta table at master startup
[
https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299517#comment-16299517
]
Hadoop QA commented on HBASE-18840:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-18477 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} HBASE-18477 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 5m
34s{color} | {color:red} branch has 13 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} HBASE-18477 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
28s{color} | {color:red} hbase-client: The patch generated 2 new + 178
unchanged - 1 fixed = 180 total (was 179) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} hbase-server: The patch generated 12 new + 350
unchanged - 1 fixed = 362 total (was 351) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 4m
7s{color} | {color:red} patch has 13 errors when building our shaded downstream
artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
47m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}117m
34s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9e78c0e |
| JIRA Issue | HBASE-18840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903139/HBASE-18840.HBASE-18477.003%20%282%29%20%281%29.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 47198889d6ae 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299513#comment-16299513 ] Hudson commented on HBASE-19491: FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1953 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1953/]) HBASE-19491 Improvements to Nighly runs: Fixes findbugs tests, Exclude (appy: rev bcacdeb212006323c880d70a013de7d7951e2bca) * (edit) dev-support/hbase-personality.sh * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/Jenkinsfile > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[
https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299509#comment-16299509
]
Hadoop QA commented on HBASE-19491:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
19m 17s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}177m
18s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}231m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903130/test-patch-to-run-precommit-on-one-module.master.002.patch
|
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9d456aa9ec43 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-19543) Abstract a replication storage interface to extract the zk specific code
[
https://issues.apache.org/jira/browse/HBASE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299500#comment-16299500
]
Duo Zhang commented on HBASE-19543:
---
{quote}
1. Can we use ServerName to replace old String(region server name)?
{quote}
Agree.
{quote}
2. Add a addQueue method. The old impl use zk, so it add a new queue by call
ZKUtil.createWithParents when addLog.
{quote}
When will we call this method? When adding a peer? And also at the
initialization of region server? Sounds good.
{quote}
3. Do we need removeReplicatorIfQueueIsEmpty? This can be done by the caller.
{quote}
I do not follow. How do caller do this? Modify zk directly?
{quote}
4. Do we need getAllWALs? This only used by cleaner.
{quote}
Ditto. Then how do the cleaner get the files to delete?
{quote}
5. Method name: use Log or WAL?
{quote}
Let's change to WAL.
For the naming and javadoc, will do.
Thanks.
> Abstract a replication storage interface to extract the zk specific code
>
>
> Key: HBASE-19543
> URL: https://issues.apache.org/jira/browse/HBASE-19543
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-19543-HBASE-19397.patch,
> HBASE-19543-HBASE-19397.patch
>
>
> For now, we will do sanity checks at the same time when updating replication
> peer. But this is not a safe way for procedure based replication peer
> modification.
> For the old zk watcher way, the only thing is updating the data on zk, so if
> the data is updated and then we crashes, there is no problem.
> For the new procedure way, we need to trigger refresh by ourselves after
> updating zk. If we crashes after the updating and before we record the state
> change of the procedure, we may fail with IllegalArgumentException when we
> execute the procedure next time since the data on zk has already been updated.
> So the current way is to do sanity checks in PRE_PEER_MODIFICATION state, and
> in UPDATE_STORAGE state we will not do sanity checks any more, just
> update(overwrite) the peer storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299496#comment-16299496
]
huaxiang sun commented on HBASE-17319:
--
Thanks [~tedyu]!
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17319-branch-1.patch, 17319.stack,
> HBASE-17319-branch-1.patch, HBASE-17319-branch-1.patch, HBASE-17319.patch,
> TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19574) TestBlockReorder fails running against hadoop 3
Ted Yu created HBASE-19574:
--
Summary: TestBlockReorder fails running against hadoop 3
Key: HBASE-19574
URL: https://issues.apache.org/jira/browse/HBASE-19574
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
{code}
testBlockLocationReorder(org.apache.hadoop.hbase.fs.TestBlockReorder) Time
elapsed: 0.031 s <<< ERROR!
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.hbase.fs.TestBlockReorder.(TestBlockReorder.java:84)
testBlockLocation(org.apache.hadoop.hbase.fs.TestBlockReorder) Time elapsed:
0.026 s <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.hbase.fs.TestBlockReorder
{code}
Here is related code:
{code}
((Log4JLogger) DFSClient.LOG).getLogger().setLevel(Level.ALL);
{code}
It seems the {{LOG}} field of DFSClient in hadoop 3 is not Log4JLogger any more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19543) Abstract a replication storage interface to extract the zk specific code
[ https://issues.apache.org/jira/browse/HBASE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299492#comment-16299492 ] Guanghao Zhang commented on HBASE-19543: As we plan to add new storage interface, we can take more time to polish the public methods in storage interface. For interface ReplicationQueueStorage. 1. Can we use ServerName to replace old String(region server name)? 2. Add a addQueue method. The old impl use zk, so it add a new queue by call ZKUtil.createWithParents when addLog. 3. Do we need removeReplicatorIfQueueIsEmpty? This can be done by the caller. 4. Do we need getAllWALs? This only used by cleaner. 5. Method name: use Log or WAL? For interface ReplicationPeerStorage. 1. addReplicationPeer ==> addPeer. As the interface name is ReplicationPeerStorage, so we can simplify the method name. 2. enablePeer, disablePeer, isPeerEnabled or setPeerState,getPeerState 3. Need javadoc for methods. > Abstract a replication storage interface to extract the zk specific code > > > Key: HBASE-19543 > URL: https://issues.apache.org/jira/browse/HBASE-19543 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19543-HBASE-19397.patch, > HBASE-19543-HBASE-19397.patch > > > For now, we will do sanity checks at the same time when updating replication > peer. But this is not a safe way for procedure based replication peer > modification. > For the old zk watcher way, the only thing is updating the data on zk, so if > the data is updated and then we crashes, there is no problem. > For the new procedure way, we need to trigger refresh by ourselves after > updating zk. If we crashes after the updating and before we record the state > change of the procedure, we may fail with IllegalArgumentException when we > execute the procedure next time since the data on zk has already been updated. > So the current way is to do sanity checks in PRE_PEER_MODIFICATION state, and > in UPDATE_STORAGE state we will not do sanity checks any more, just > update(overwrite) the peer storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299460#comment-16299460
]
Yi Liang commented on HBASE-19218:
--
patch is good. +1
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19218.master.001.patch,
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
> 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223)
> at
> org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731)
> 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=37,
> state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException
> via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace,
> region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException;
> ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728,
> splitWal=true, meta=false
> java.lang.UnsupportedOperationException: unhandled state=SERVER_CRASH_FINISH
> at
>
[jira] [Updated] (HBASE-19550) Wrap the Mutation in cp layer to make sure all passed cells are of ExtendedCell
[ https://issues.apache.org/jira/browse/HBASE-19550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19550: --- Summary: Wrap the Mutation in cp layer to make sure all passed cells are of ExtendedCell (was: Wrap the Put in cp layer to make sure all passed cells are of ExtendedCell) > Wrap the Mutation in cp layer to make sure all passed cells are of > ExtendedCell > --- > > Key: HBASE-19550 > URL: https://issues.apache.org/jira/browse/HBASE-19550 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-beta-2 > > > We assume all cells in server are of ExtendedCell. However, cp user can add > their cell impl via Put#add(Cell) in observer. That will cause > UnsupportedOperationException when rs try to update the cell's timestamp and > seq Id. We should do something for cp user...For example, wrap the passed > cells to be a extendcell type. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19496) Reusing the ByteBuffer in rpc layer corrupt the ServerLoad and RegionLoad
[
https://issues.apache.org/jira/browse/HBASE-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299456#comment-16299456
]
Chia-Ping Tsai commented on HBASE-19496:
bq. What is the prescription we should take on going forward?
1) DONT expose any pb object to public user
2) DONT keep the pb object after the call is done
3) the metrics/load/status object should not extend the equals/compare methods
since it mean all counter we add in the future must extend the equals/compare
methods also
I introduce the new classes since the APIs of old classes will get ugly after
we take the prescription.
> Reusing the ByteBuffer in rpc layer corrupt the ServerLoad and RegionLoad
> -
>
> Key: HBASE-19496
> URL: https://issues.apache.org/jira/browse/HBASE-19496
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19496.v0.patch, HBASE-19496.v1.patch,
> HBASE-19496.wip.patch
>
>
> {{ServerLoad}} and {{RegionLoad}} store the pb object internally but the
> bytebuffer of pb object may be reused in rpc layer. Hence, the {{ServerLoad}}
> and {{RegionLoad}} which saved by {{HMaster}} will be corrupted if the
> bytebuffer backed is modified.
> This issue doesn't happen on branch-1.
> # netty server was introduced in 2.0 (see HBASE-17263)
> # reusing bytebuffer to read RPC requests was introduced in 2.0 (see
> HBASE-15788)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19542) fix TestSafemodeBringsDownMaster
[ https://issues.apache.org/jira/browse/HBASE-19542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19542: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for all reviews. > fix TestSafemodeBringsDownMaster > > > Key: HBASE-19542 > URL: https://issues.apache.org/jira/browse/HBASE-19542 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19542.v0.patch, HBASE-19542.v1 (1).patch, > HBASE-19542.v1.patch > > > We need to check the stability of underlay file system when closing async > wal. Otherwise, our hbase can't shutdown gracefully. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19573) Rewrite ReplicationPeer/Queue related classes with the new replication storage interface
Duo Zhang created HBASE-19573: - Summary: Rewrite ReplicationPeer/Queue related classes with the new replication storage interface Key: HBASE-19573 URL: https://issues.apache.org/jira/browse/HBASE-19573 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19567) ClassNotFoundException: org.apache.hadoop.hbase.KeyValue$RawBytesComparator starting 2.0.0 over a 0.98.25 data.
[
https://issues.apache.org/jira/browse/HBASE-19567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299426#comment-16299426
]
Chia-Ping Tsai commented on HBASE-19567:
Thanks for fixing my mistaken. [~stack]
> ClassNotFoundException: org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> starting 2.0.0 over a 0.98.25 data.
> ---
>
> Key: HBASE-19567
> URL: https://issues.apache.org/jira/browse/HBASE-19567
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 19567.patch
>
>
> Starting branch-2 over a 0.98.25 data:
> 382603 2017-12-19 21:25:08,333 DEBUG
> [StoreOpener-8e6e2b71d8a413561d9103b0cb6e4686-1] regionserver.HStore: loaded
> hdfs://ve0524.halxg.cloudera.com:8020/hbase/data/default/ycsb/8e6e2b71d8a413561d9103b0cb6e4686/family/3633fe2e03ad4cceb0d2018f1f0d76da,
> isReference=false, isBulkLoadResult=false, seqid=341101 6,
> majorCompaction=false
> 382604 2017-12-19 21:25:08,339 DEBUG [StoreFileOpenerThread-family-1]
> hfile.HFile: Opening HFile v2 with v3 reader
> 382605 2017-12-19 21:25:08,342 ERROR [StoreFileOpenerThread-family-1]
> regionserver.StoreFileReader: Error reading bloom filter meta for
> GENERAL_BLOOM_META -- proceeding without
> 382606 java.io.IOException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> 382607 at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:583)
> 382608 at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:592)
> 382609 at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.(CompoundBloomFilter.java:79)
> 382610 at
> org.apache.hadoop.hbase.util.BloomFilterFactory.createFromMeta(BloomFilterFactory.java:104)
> 382611 at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.loadBloomfilter(StoreFileReader.java:480)
> 382612 at
> org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:426)
> 382613 at
> org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:461)
> 382614 at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:665)
> 382615 at
> org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$0(HStore.java:533)
> 382616 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 382617 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 382618 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 382619 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 382620 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 382621 at java.lang.Thread.run(Thread.java:745)
> 382622 Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> 382623 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> 382624 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 382625 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> 382626 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 382627 at java.lang.Class.forName0(Native Method)
> 382628 at java.lang.Class.forName(Class.java:264)
> 382629 at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:581)
> 382630 ... 14 more
> Ram suggested a one-liner. I tried it and it seems to work.
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> index 672919d..1a22cd6 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> @@ -570,7 +570,7 @@ public class FixedFileTrailer {
> || (comparatorClassName
>
> .equals("org.apache.hadoop.hbase.CellComparator$MetaCellComparator"))) {
>comparatorKlass = MetaCellComparator.class;
> -} else if
> (comparatorClassName.equals("org.apache.hadoop.hbase.KeyValue.RawBytesComparator")
> +} else if
> (comparatorClassName.equals("org.apache.hadoop.hbase.KeyValue$RawBytesComparator")
> ||
> comparatorClassName.equals("org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator"))
> {
>// When the comparator to be used is Bytes.BYTES_RAWCOMPARATOR, we
> just return null from here
>// Bytes.BYTES_RAWCOMPARATOR is not a CellComparator
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299407#comment-16299407
]
Chia-Ping Tsai edited comment on HBASE-19486 at 12/21/17 2:22 AM:
--
Seems {{autoFlushCount}} is used by test only? If so, could we make
{{autoFlush()}} be protected and then overrride it in test to get the count of
auto flush.
Should we cancel the previous task before invoking new timer task?
{code}
+ autoFlushTimer = new Timer(true); // Create Timer running as Daemon.
+ autoFlushTimer.schedule(new TimerTask() {
+@Override
+public void run() {
+ BufferedMutatorImpl.this.autoFlush();
+}
+ }, writeBufferMaxLingerMs, writeBufferMaxLingerMs);
{code}
was (Author: chia7712):
Seems {{autoFlushCount}] is used by test only? If so, could we make
{{autoFlush()}} be protected and then overrride it in test to get the count of
auto flush.
Should we cancel the previous task before invoking new timer task?
{code}
+ autoFlushTimer = new Timer(true); // Create Timer running as Daemon.
+ autoFlushTimer.schedule(new TimerTask() {
+@Override
+public void run() {
+ BufferedMutatorImpl.this.autoFlush();
+}
+ }, writeBufferMaxLingerMs, writeBufferMaxLingerMs);
{code}
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299421#comment-16299421 ] Hudson commented on HBASE-19491: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1045 (See [https://builds.apache.org/job/HBase-1.2-IT/1045/]) HBASE-19491 Improvements to Nighly runs: Fixes findbugs tests, Exclude (appy: rev 605d92e3c18fd4b1abdcfc882bfd472bfa195f9e) * (edit) dev-support/hbase-personality.sh * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/Jenkinsfile > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299419#comment-16299419 ] Hudson commented on HBASE-19491: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #320 (See [https://builds.apache.org/job/HBase-1.3-IT/320/]) HBASE-19491 Improvements to Nighly runs: Fixes findbugs tests, Exclude (appy: rev b0b4eb217778a603bdb7b09bd47bdf8c9d8631bc) * (edit) dev-support/Jenkinsfile * (edit) dev-support/hbase-personality.sh * (edit) dev-support/hbase_nightly_yetus.sh > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19571) Minor refactor of Nightly run scripts
[ https://issues.apache.org/jira/browse/HBASE-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299410#comment-16299410 ] Appy commented on HBASE-19571: -- Testing singe stage was enough, so i had disabled general checks. Compare artifacts of test build with Nightly master, looks same. Good to go. > Minor refactor of Nightly run scripts > - > > Key: HBASE-19571 > URL: https://issues.apache.org/jira/browse/HBASE-19571 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-19571.master.001.patch > > > Was trying to split out common code in separate lib, but it's not at all > trivial. The way to do it for declarative syntax Jenkinsfile is by using > Shared Libraries which requires separate repo! > The patch ended up being just naming refactors. > Renames OUTPUTDIR to OUTPUT_DIR > Renames OUTPUT_RELATIVE to OUTPUT_DIR_RELATIVE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299407#comment-16299407
]
Chia-Ping Tsai commented on HBASE-19486:
Seems {{autoFlushCount}] is used by test only? If so, could we make
{{autoFlush()}} be protected and then overrride it in test to get the count of
auto flush.
Should we cancel the previous task before invoking new timer task?
{code}
+ autoFlushTimer = new Timer(true); // Create Timer running as Daemon.
+ autoFlushTimer.schedule(new TimerTask() {
+@Override
+public void run() {
+ BufferedMutatorImpl.this.autoFlush();
+}
+ }, writeBufferMaxLingerMs, writeBufferMaxLingerMs);
{code}
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299403#comment-16299403 ] Appy commented on HBASE-19491: -- Thanks for review stack. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19491: - Resolution: Fixed Fix Version/s: 2.0.0-beta-1 Status: Resolved (was: Patch Available) > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299394#comment-16299394 ] Appy commented on HBASE-19491: -- Looks okay. Pushing to all branches. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[ https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299380#comment-16299380 ] Duo Zhang commented on HBASE-19559: --- [~stack] This one is used to fix TestLogRolling. > Fix TestLogRolling.testLogRollOnDatanodeDeath > - > > Key: HBASE-19559 > URL: https://issues.apache.org/jira/browse/HBASE-19559 > Project: HBase > Issue Type: Bug > Components: test, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19559.patch, HBASE-19559.patch > > > It appears several times on the failing list of pre commit result. > The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19543) Abstract a replication storage interface to extract the zk specific code
[ https://issues.apache.org/jira/browse/HBASE-19543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19543: -- Attachment: HBASE-19543-HBASE-19397.patch > Abstract a replication storage interface to extract the zk specific code > > > Key: HBASE-19543 > URL: https://issues.apache.org/jira/browse/HBASE-19543 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Replication >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19543-HBASE-19397.patch, > HBASE-19543-HBASE-19397.patch > > > For now, we will do sanity checks at the same time when updating replication > peer. But this is not a safe way for procedure based replication peer > modification. > For the old zk watcher way, the only thing is updating the data on zk, so if > the data is updated and then we crashes, there is no problem. > For the new procedure way, we need to trigger refresh by ourselves after > updating zk. If we crashes after the updating and before we record the state > change of the procedure, we may fail with IllegalArgumentException when we > execute the procedure next time since the data on zk has already been updated. > So the current way is to do sanity checks in PRE_PEER_MODIFICATION state, and > in UPDATE_STORAGE state we will not do sanity checks any more, just > update(overwrite) the peer storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19559) Fix TestLogRolling.testLogRollOnDatanodeDeath
[ https://issues.apache.org/jira/browse/HBASE-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-19559: -- Attachment: HBASE-19559.patch TestLogRolling passed in pre commit. Try again to see if TestLogRolling can still pass. > Fix TestLogRolling.testLogRollOnDatanodeDeath > - > > Key: HBASE-19559 > URL: https://issues.apache.org/jira/browse/HBASE-19559 > Project: HBase > Issue Type: Bug > Components: test, wal >Reporter: Duo Zhang >Assignee: Duo Zhang > Attachments: HBASE-19559.patch, HBASE-19559.patch > > > It appears several times on the failing list of pre commit result. > The test code is a bit confusing to me. Need to dig more. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19520) Add UTs for the new lock type PEER
[ https://issues.apache.org/jira/browse/HBASE-19520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-19520: --- Attachment: HBASE-19520.HBASE-19397.002.patch > Add UTs for the new lock type PEER > -- > > Key: HBASE-19520 > URL: https://issues.apache.org/jira/browse/HBASE-19520 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-19520.HBASE-19397.001.patch, > HBASE-19520.HBASE-19397.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299336#comment-16299336
]
stack commented on HBASE-19218:
---
[~easyliangjob] Good. We seem to be of same opinion. The fix here will cause us
fail the current request because targetServer is null so we'll go back around
again. It is good by you sir?
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19218.master.001.patch,
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
> 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223)
> at
> org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731)
> 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=37,
> state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException
> via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace,
> region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException;
> ServerCrashProcedure
>
[jira] [Updated] (HBASE-18840) Add functionality to refresh meta table at master startup
[ https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18840: -- Attachment: HBASE-18840.HBASE-18477.003 (2) (1).patch > Add functionality to refresh meta table at master startup > - > > Key: HBASE-18840 > URL: https://issues.apache.org/jira/browse/HBASE-18840 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18840.HBASE-18477.001.patch, > HBASE-18840.HBASE-18477.002.patch, HBASE-18840.HBASE-18477.003 (2) (1).patch, > HBASE-18840.HBASE-18477.003 (2).patch, HBASE-18840.HBASE-18477.003.patch > > > If a HBase cluster’s hbase:meta table is deleted or a cluster is started with > a new meta table, HBase needs the functionality to synchronize it’s metadata > from Storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299334#comment-16299334 ] stack commented on HBASE-19486: --- What [~jerryhe] said. > Automatically flush BufferedMutator after a timeout > > > Key: HBASE-19486 > URL: https://issues.apache.org/jira/browse/HBASE-19486 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Niels Basjes >Assignee: Niels Basjes > Attachments: HBASE-19486-20171212-2117.patch, > HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch, > HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch, > HBASE-19486-20171219-1122-trigger-qa-run.patch, > HBASE-19486-20171220-1612-trigger-qa-run.patch, > HBASE-19486-20171220-2228-trigger-qa-run.patch > > > I'm working on several projects where we are doing stream / event type > processing instead of batch type processing. We mostly use Apache Flink and > Apache Beam for these projects. > When we ingest a continuous stream of events and feed that into HBase via a > BufferedMutator this all works fine. The buffer fills up at a predictable > rate and we can make sure it flushes several times per second into HBase by > tuning the buffer size. > We also have situations where the event rate is unpredictable. Some times > because the source is in reality a batch job that puts records into Kafka, > sometimes because it is the "predictable in production" application in our > testing environment (where only the dev triggers a handful of events). > For these kinds of use cases we need a way to 'force' the BufferedMutator to > automatically flush any records in the buffer even if the buffer is not full. > I'll put up a pull request with a proposed implementation for review against > the master (i.e. 3.0.0). > When approved I would like to backport this to the 1.x and 2.x versions of > the client in the same (as close as possible) way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19218:
--
Assignee: stack (was: Umesh Agashe)
Status: Patch Available (was: Open)
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19218.master.001.patch,
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
> 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223)
> at
> org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731)
> 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=37,
> state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException
> via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace,
> region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException;
> ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728,
> splitWal=true, meta=false
> java.lang.UnsupportedOperationException: unhandled state=SERVER_CRASH_FINISH
> at
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299321#comment-16299321
]
stack commented on HBASE-19218:
---
.001 Fix. One-liner. Test is a behemoth of mocking necessary because error is
deep down in assign.
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Umesh Agashe
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19218.master.001.patch,
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
> 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223)
> at
> org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731)
> 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=37,
> state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException
> via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace,
> region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException;
> ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728,
> splitWal=true, meta=false
>
[jira] [Updated] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-19218:
--
Attachment: HBASE-19218.master.001.patch
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Umesh Agashe
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: HBASE-19218.master.001.patch,
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
> 2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:171)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.addToRemoteDispatcher(RegionTransitionProcedure.java:223)
> at
> org.apache.hadoop.hbase.master.assignment.AssignProcedure.updateTransition(AssignProcedure.java:252)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:82)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:77)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1731)
> 2017-11-08 00:25:27,239 FATAL [ProcExecWrkr-22] procedure2.ProcedureExecutor:
> CODE-BUG: Uncaught runtime exception for pid=37,
> state=FAILED:SERVER_CRASH_FINISH, exception=java.lang.NullPointerException
> via CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
> state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
> table=hbase:namespace,
> region=f147f204a579b885c351bdc0a7ebbf94:java.lang.NullPointerException;
> ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728,
> splitWal=true, meta=false
> java.lang.UnsupportedOperationException: unhandled state=SERVER_CRASH_FINISH
> at
>
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299317#comment-16299317
]
stack commented on HBASE-19218:
---
Some notes first. Patch is coming. To repro the exception above, you need to
comment out the fix like this:
{code}
diff --git
a/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
b/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
index 4cee524a12..35ad553fe8 100644
---
a/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
+++
b/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
@@ -167,7 +167,14 @@ public abstract class RemoteProcedureDispatcher Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Umesh Agashe
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments:
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
>
[jira] [Commented] (HBASE-19567) ClassNotFoundException: org.apache.hadoop.hbase.KeyValue$RawBytesComparator starting 2.0.0 over a 0.98.25 data.
[
https://issues.apache.org/jira/browse/HBASE-19567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299307#comment-16299307
]
Hudson commented on HBASE-19567:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4262 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4262/])
HBASE-19567 ClassNotFoundException: (stack: rev
4d71f4e32fe55fe84e69e745150edd8f6d14c87f)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> ClassNotFoundException: org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> starting 2.0.0 over a 0.98.25 data.
> ---
>
> Key: HBASE-19567
> URL: https://issues.apache.org/jira/browse/HBASE-19567
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0-beta-1
>
> Attachments: 19567.patch
>
>
> Starting branch-2 over a 0.98.25 data:
> 382603 2017-12-19 21:25:08,333 DEBUG
> [StoreOpener-8e6e2b71d8a413561d9103b0cb6e4686-1] regionserver.HStore: loaded
> hdfs://ve0524.halxg.cloudera.com:8020/hbase/data/default/ycsb/8e6e2b71d8a413561d9103b0cb6e4686/family/3633fe2e03ad4cceb0d2018f1f0d76da,
> isReference=false, isBulkLoadResult=false, seqid=341101 6,
> majorCompaction=false
> 382604 2017-12-19 21:25:08,339 DEBUG [StoreFileOpenerThread-family-1]
> hfile.HFile: Opening HFile v2 with v3 reader
> 382605 2017-12-19 21:25:08,342 ERROR [StoreFileOpenerThread-family-1]
> regionserver.StoreFileReader: Error reading bloom filter meta for
> GENERAL_BLOOM_META -- proceeding without
> 382606 java.io.IOException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> 382607 at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:583)
> 382608 at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.createComparator(FixedFileTrailer.java:592)
> 382609 at
> org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.(CompoundBloomFilter.java:79)
> 382610 at
> org.apache.hadoop.hbase.util.BloomFilterFactory.createFromMeta(BloomFilterFactory.java:104)
> 382611 at
> org.apache.hadoop.hbase.regionserver.StoreFileReader.loadBloomfilter(StoreFileReader.java:480)
> 382612 at
> org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:426)
> 382613 at
> org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:461)
> 382614 at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:665)
> 382615 at
> org.apache.hadoop.hbase.regionserver.HStore.lambda$openStoreFiles$0(HStore.java:533)
> 382616 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 382617 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 382618 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 382619 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 382620 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 382621 at java.lang.Thread.run(Thread.java:745)
> 382622 Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.KeyValue$RawBytesComparator
> 382623 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> 382624 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 382625 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> 382626 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 382627 at java.lang.Class.forName0(Native Method)
> 382628 at java.lang.Class.forName(Class.java:264)
> 382629 at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:581)
> 382630 ... 14 more
> Ram suggested a one-liner. I tried it and it seems to work.
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> index 672919d..1a22cd6 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
> @@ -570,7 +570,7 @@ public class FixedFileTrailer {
> || (comparatorClassName
>
> .equals("org.apache.hadoop.hbase.CellComparator$MetaCellComparator"))) {
>comparatorKlass = MetaCellComparator.class;
> -} else if
> (comparatorClassName.equals("org.apache.hadoop.hbase.KeyValue.RawBytesComparator")
> +} else if
> (comparatorClassName.equals("org.apache.hadoop.hbase.KeyValue$RawBytesComparator")
> ||
> comparatorClassName.equals("org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator"))
> {
>
[jira] [Commented] (HBASE-19554) AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
[
https://issues.apache.org/jira/browse/HBASE-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299306#comment-16299306
]
Hudson commented on HBASE-19554:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4262 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/4262/])
HBASE-19554 For debug: Modify HTU.waitUntilAllRegionsAssigned to handle (stack:
rev 084e324fb9f769f221236e31aa97382d2677f911)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> AbstractTestDLS.testThreeRSAbort sometimes fails in pre commit
> --
>
> Key: HBASE-19554
> URL: https://issues.apache.org/jira/browse/HBASE-19554
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-19554.patch
>
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/10554/artifact/patchprocess/patch-unit-hbase-server.txt
> The error message is a bit strange:
> {quote}
> [ERROR] testThreeRSAbort(org.apache.hadoop.hbase.master.TestDLSAsyncFSWAL)
> Time elapsed: 20.627 s <<< ERROR!
> org.apache.hadoop.hbase.TableNotFoundException: Region of
> 'hbase:namespace,,1513320505933.451650152885a3b41d0b1110deca513c.' is
> expected in the table of 'testThreeRSAbort', but hbase:meta says it is in the
> table of 'hbase:namespace'. hbase:meta might be damaged.
> {quote}
> It fails for both FSHLog and AsyncFSWAL. Need to dig more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19563) A few hbase-procedure classes missing @InterfaceAudience annotation
[ https://issues.apache.org/jira/browse/HBASE-19563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299305#comment-16299305 ] Hudson commented on HBASE-19563: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4262 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4262/]) HBASE-19563 A few hbase-procedure classes missing @InterfaceAudience (stack: rev 6c6a9d2d1ca7a8ca79dde2e257ed5f2fdd375770) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/LockStatus.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureMetrics.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/LockAndQueue.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/NoopProcedureStore.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureStateSerializer.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreBase.java > A few hbase-procedure classes missing @InterfaceAudience annotation > --- > > Key: HBASE-19563 > URL: https://issues.apache.org/jira/browse/HBASE-19563 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Labels: beginner, procedure > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19563.master.001.patch, > HBASE-19563.master.002.patch > > > NoopProcedureStore.java > ProcedureStoreBase.java > ProcedureMetrics.java > LockStatus.java > LockAndQueue.java > ProcedureStateSerializer.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299290#comment-16299290 ] Hadoop QA commented on HBASE-19491: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/10599/console in case of problems. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19491: - Attachment: test-patch-to-run-precommit-on-one-module.master.002.patch > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299280#comment-16299280
]
Jerry He commented on HBASE-19486:
--
I try to get a better feel for the word 'linger' to see if it fits better here.
Still like {{write buffer periodic flush interva}} better at the moment to fit
into hbase.
Thanks for the effort to explain it.
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18840) Add functionality to refresh meta table at master startup
[
https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299267#comment-16299267
]
Hadoop QA commented on HBASE-18840:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-18477 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} HBASE-18477 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 4m
30s{color} | {color:red} branch has 13 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} HBASE-18477 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} hbase-client: The patch generated 2 new + 178
unchanged - 1 fixed = 180 total (was 179) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} hbase-server: The patch generated 12 new + 350
unchanged - 1 fixed = 362 total (was 351) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
27s{color} | {color:red} patch has 13 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 10s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 9s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestCompactionInDeadRegionServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9e78c0e |
| JIRA Issue | HBASE-18840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903107/HBASE-18840.HBASE-18477.003%20%282%29.patch
|
| Optional Tests | asflicense shadedjars javac javadoc unit findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ee2fc13fbae3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build
[jira] [Commented] (HBASE-19148) Reevaluate default values of configurations
[
https://issues.apache.org/jira/browse/HBASE-19148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299256#comment-16299256
]
Hadoop QA commented on HBASE-19148:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
8s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 29s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19148 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903108/HBASE-19148.master.004%20%281%29.patch
|
| Optional Tests | asflicense javac javadoc unit xml findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux afd42a51725c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision
[jira] [Commented] (HBASE-19391) Calling HRegion#initializeRegionInternals from a region replica can still re-create a region directory
[
https://issues.apache.org/jira/browse/HBASE-19391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299230#comment-16299230
]
Hadoop QA commented on HBASE-19391:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
43s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 59s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 4s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionOpen |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19391 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903098/HBASE-19391.master.v0.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ee5d092eb5f7 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 4d71f4e32f |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10591/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10591/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10591/console
[jira] [Comment Edited] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299184#comment-16299184 ] Appy edited comment on HBASE-19491 at 12/20/17 11:09 PM: - So the precommit on the 'real patch' passed. Attached 'test patch' to run precommit on single module to see general pre-commit is working fine. Testing nightly mode here: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19491/ fyi [~stack] PTAL. was (Author: appy): So the precommit on script itself passed. Attached test patch to run precommit on single module to see general pre-commit is working fine. Testing nightly mode here: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19491/ fyi [~stack] PTAL. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19491: - Description: Testing infra improvements - Exclude flaky tests from nightly master run (Old nightly master run used to exclude flaky tests, but new nightly one which is based on Jenkins stages wasn't using it. Adding it to new nightly job) - Fixes findbugs check (seems like wasn't working earlier : "0 findbugs 0m 0s Findbugs executables are not available.") was: Testing infra improvements - Exclude flaky tests from nightly master run - Fixes findbugs check (seems like wasn't working earlier : "0 findbugs 0m 0s Findbugs executables are not available.") > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run (Old nightly master run used to > exclude flaky tests, but new nightly one which is based on Jenkins stages > wasn't using it. Adding it to new nightly job) > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299221#comment-16299221
]
Yi Liang commented on HBASE-19218:
--
I have seen this error in my cluster as well
{quote}
2017-11-08 00:25:25,260 INFO [ProcExecWrkr-18]
procedure.MasterProcedureScheduler: pid=40, ppid=37,
state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace,
region=f147f204a579b885c351bdc0a7ebbf94 hbase:namespace
hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.
2017-11-08 00:25:25,263 INFO [ProcExecWrkr-18] assignment.AssignProcedure:
Start pid=40, ppid=37, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure
table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94; rit=OFFLINE,
location=node_5,16020,1510084579728; forceNewPlan=false, retain=true
.
2017-11-08 00:25:26,040 INFO [ProcExecWrkr-23] procedure.ServerCrashProcedure:
pid=42 found RIT pid=40, ppid=37, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
AssignProcedure table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94;
rit=OPENING, location=node_5,16020,1510100695534
2017-11-08 00:25:26,040 WARN [ProcExecWrkr-23]
assignment.RegionTransitionProcedure: Remote call failed pid=40, ppid=37,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94; rit=OPENING,
location=node_5,16020,1510100695534; exception=ServerCrashProcedure pid=42,
server=node_5,16020,1510100695534
2017-11-08 00:25:26,041 INFO [ProcExecWrkr-23] assignment.AssignProcedure:
Retry=1 of max=10; pid=40, ppid=37, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
AssignProcedure table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94;
rit=OPENING, location=node_5,16020,1510100695534
2017-11-08 00:25:26,193 INFO [ProcExecWrkr-25] zookeeper.MetaTableLocator:
Setting hbase:meta (replicaId=0) location in ZooKeeper as
node_2,16020,1510100696388
2017-11-08 00:25:26,195 INFO [ProcExecWrkr-25]
assignment.RegionTransitionProcedure: Dispatch pid=44, ppid=43,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta,
region=1588230740; rit=OPENING, location=node_2,16020,1510100696388
2017-11-08 00:25:26,346 INFO [ProcedureDispatcherTimeoutThread]
procedure.RSProcedureDispatcher: Using procedure batch rpc execution for
serverName=node_2,16020,1510100696388 version=2097152
2017-11-08 00:25:27,187 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor: Updated
table hbase:meta state to ENABLED in META
2017-11-08 00:25:27,187 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator:
Setting hbase:meta (replicaId=0) location in ZooKeeper as
node_2,16020,1510100696388
2017-11-08 00:25:27,209 INFO [ProcExecWrkr-22]
assignment.RegionTransitionProcedure: Dispatch pid=40, ppid=37,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94; rit=OFFLINE,
location=null
2017-11-08 00:25:27,210 INFO [ProcExecWrkr-21]
assignment.RegionTransitionProcedure: Dispatch pid=39, ppid=36,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:acl,
region=24aadcb52fdc43e2ebcffe95d39b43ab; rit=OPENING,
location=node_2,16020,1510100696388
2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace,
region=f147f204a579b885c351bdc0a7ebbf94
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
{quote}
I track the AssignProcedure of hbase:namespace to above log
1. AssignProcedure created for table=hbase:namespace; and state is
RUNNABLE:REGION_TRANSITION_QUEUE
2. ServerCrashProcedure happens and will handle AssignProcedure of
table=hbase:namespace,
and from the log, this AssignProcedure state has already been set to
REGION_TRANSITION_DISPATCH when SCP handle it.
The SCP will set the AssignProcedure back to REGION_TRANSITION_QUEUE and
offline related region.
3. However, The AssignProcedure of hbase:namespace resume from state
REGION_TRANSITION_DISPATCH,
and SCP in step2 has already offline the region and set location as null, so
null pointer exception may happen
The problem happens at step 3 above, the AssignProcedure should resume from
state REGION_TRANSITION_QUEUE, but it actually from REGION_TRANSITION_DISPATCH.
This could happen, since when SCP call remoteCallFailed for AssignProcedure of
hbase:namespace,
ProcedureExecutor is running AssignProcedure of hbase:namespace as state
REGION_TRANSITION_DISPATCH at same time.
and if SCP's handleFailure(which set region location as null) for
hbase:namespace happens before AssignProcedure#addToRemoteDispatcher, and then
null pointer happens
And if we can catch the nullpointerexception and set state back to
REGION_TRANSITION_QUEUE,
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299220#comment-16299220 ] stack commented on HBASE-19491: --- Go for it [~appy]! Patch LGTM (as best as I understand it -- proof is in the deploy!) > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18840) Add functionality to refresh meta table at master startup
[ https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299219#comment-16299219 ] Zach York commented on HBASE-18840: --- [~stack] I agree that it would be better to get this information from hbase:meta if this existed, but this patch is more for the case where the hbase:meta doesn't exist so that wouldn't work. I know that there are some gaps with this functionality (listed below), but this is best effort functionality (usually used when storefiles and regions are stable): 1) Won't handle in process splits 2) Anything not flushed to storefiles will be missed. Also we currently have users utilizing this functionality for their read replica clusters. I think long term we might need to think of a better solution (especially if we want to remove the regioninfo files), but currently this is working. > Add functionality to refresh meta table at master startup > - > > Key: HBASE-18840 > URL: https://issues.apache.org/jira/browse/HBASE-18840 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-18840.HBASE-18477.001.patch, > HBASE-18840.HBASE-18477.002.patch, HBASE-18840.HBASE-18477.003 (2).patch, > HBASE-18840.HBASE-18477.003.patch > > > If a HBase cluster’s hbase:meta table is deleted or a cluster is started with > a new meta table, HBase needs the functionality to synchronize it’s metadata > from Storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299216#comment-16299216 ] Appy commented on HBASE-19491: -- So in the nightly mode run (using separate branch), findbugs is working and i already tested exclusion of flaky tests earlier. Waiting for QA on 'test patch' now. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[
https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299213#comment-16299213
]
Appy commented on HBASE-17436:
--
bq. result = "Still initializing, please wait.";
If an url is json api, it should return json no matter what. Return { error:
"..." }
Design stuff:
# Why just assigned regions? Should be renamed 'Assigned Region States' then.
# What's the reason for excluding regions in splitting, opening, merging,
closing, etc states?
# RegionStates is IA.Private. Maybe we don't have admin fns for these because
we don't want to expose these internal states? Idk. But if so, this jsp is
sneaking into AM and exposing private stuff.
# Official programmatic way of getting info about cluster should be Admin. Json
endpoints are useful, but just exposing one endpoint for region states and
nothing for other 10s of basic functionalities (tables, table states,
regionserver names, etc) is not a good approach. These needs better designing
first. Specially not a button on top banner. If we still decide to add it after
figuring out #3, then let's add it as separate section after 'Tables' on
master, something like following. I hope we can, because i'd like to see it
there.
{panel}
h4. Regions
Number of regions: XX
json endpoint:
Use state=Foo to filter down regions by state.
{panel}
> Add facility to provide more information for Other Regions seen on Master UI
>
>
> Key: HBASE-17436
> URL: https://issues.apache.org/jira/browse/HBASE-17436
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
> Labels: ui, usability
> Fix For: 2.0.0
>
> Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt,
> 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch,
> HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch,
> HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch,
> HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot
> 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png,
> Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35
> PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png
>
>
> [~rpednekar] and I were looking at a case where the count displayed under
> Other Regions was high (~1200).
> Since the table page just maintains a count instead of List of region names,
> it is very difficult for user to determine which regions belong to this
> category.
> We should provide facility to provide more details for this category (LOG,
> JMX, etc).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19566) Fix Checkstyle errors in hbase-client-project
[ https://issues.apache.org/jira/browse/HBASE-19566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel updated HBASE-19566: -- Resolution: Fixed Fix Version/s: 2.0.0-beta-1 3.0.0 Status: Resolved (was: Patch Available) > Fix Checkstyle errors in hbase-client-project > - > > Key: HBASE-19566 > URL: https://issues.apache.org/jira/browse/HBASE-19566 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Trivial > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-19566.master.001.patch > > > Fix the remaining Checkstyle errors in the *hbase-client-project* module and > enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19568) Restore of HBase table using incremental backup doesn't restore rows from an earlier incremental backup
[
https://issues.apache.org/jira/browse/HBASE-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299188#comment-16299188
]
Josh Elser commented on HBASE-19568:
{code}
+ LOG.debug("Added region procedure manager: " +
regionProcedureClass+"\nAdded region observer: " +
+ regionObserverClass);
{code}
Can you change the newline to a ". " (period and space) just to keep this all
on one line (helps in grepping/parsing logs)
{code}
+ private void updateFileLists(List activeFiles, List
archiveFiles)
+ throws IOException {
+FileSystem fs = FileSystem.get(conf);
+List newlyArchived = new ArrayList();
{code}
Should a FileSystem instance be a member to IncrementalTableBackupClient? Or,
pass it in as an argument from {{copyBulkLoadedFiles(..)}}.
{code}
+ private List getFilesRecursively(String fileBackupDir) throws
IllegalArgumentException,
+IOException
+ {
+FileSystem fs = FileSystem.get((new Path(fileBackupDir)).toUri(),
+ new Configuration());
{code}
How about passing in the {{Configuration}} instead of creating a new one? You
have one in {{restoreImages}} already.
{code}
+RemoteIterator it = fs.listFiles(new
Path(fileBackupDir), true);
+while (it.hasNext()) {
+ Path p = it.next().getPath();
+ if (HFile.isHFileFormat(fs, p)) {
+list.add(p);
+ }
+}
{code}
Suggest {{FileSystem.listStatus(Path, PathFilter)}} might be a bit more "terse".
Nice test addition to catch this!
bq. Is the fix mostly done in copyBulkLoadedFiles() ?
Agreed -- any chance I can convince you to move some of the cleanup-related
things (looks mostly like the improvements from the stalled/blocked FT patch
from others)? This seems like a really simple bug to fix, but it gets lost in
the renames. Would be happy to commit the cleanup stuff immediately (given that
was already reviewed).
> Restore of HBase table using incremental backup doesn't restore rows from an
> earlier incremental backup
> ---
>
> Key: HBASE-19568
> URL: https://issues.apache.org/jira/browse/HBASE-19568
> Project: HBase
> Issue Type: Bug
>Reporter: Romil Choksi
>Assignee: Vladimir Rodionov
> Attachments: HBASE-19568-v1.patch
>
>
> Credits to [~romil.choksi]
> Restore of bulk-loaded HBase table doesn't restore deleted rows
> Steps:
> Create usertable and insert a few rows in it
> Create full backup
> Bulk load into usertable, and create first incremental backup
> Bulk load into usertable again, and create second incremental backup
> Delete row each from initial insert, first bulk load and second bulk load
> Restore usertable using second incremental backup
> Verify if each of the deleted rows has been restored
> On restore using second incremental backup id, the test failed as all of the
> rows from first bulk load were not available. Data from initial insertion
> (full backup) and second bulk load were only available.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19218) Master stuck thinking hbase:namespace is assigned after restart preventing intialization
[
https://issues.apache.org/jira/browse/HBASE-19218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299186#comment-16299186
]
stack commented on HBASE-19218:
---
I took another look at the log.
We are doing dispatch to RPC but targetServer is null. The targetServer is
usually allocated by the time we get down to RPC dispatch but what is
interesting here is that this is our second attempt. The first was cancelled by
a concurrently running ServerCrashProcedure:
2017-11-08 00:25:26,040 INFO [ProcExecWrkr-23] procedure.ServerCrashProcedure:
pid=42 found RIT pid=40, ppid=37, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
AssignProcedure table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94;
rit=OPENING, location=node_5,16020,1510100695534
Here just after... is where we go to dispatch. See how target is null
('location='):
2017-11-08 00:25:27,209 INFO [ProcExecWrkr-22]
assignment.RegionTransitionProcedure: Dispatch pid=40, ppid=37,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure
table=hbase:namespace, region=f147f204a579b885c351bdc0a7ebbf94; rit=OFFLINE,
location=null
which is followed a few millis later with:
2017-11-08 00:25:27,211 ERROR [ProcExecWrkr-22] procedure2.ProcedureExecutor:
CODE-BUG: Uncaught runtime exception: pid=40, ppid=37,
state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace,
region=f147f204a579b885c351bdc0a7ebbf94
java.lang.NullPointerException
The nulling of the server happens... when the AssignProcedure does
remoteCallFailed which is what SCP calls when it cancels the AP. This is done
in one thread. Concurrently the assign was in progress on a Procedure executor
thread.
There is a hole then between the check that we have a target.
protected boolean updateTransition(final MasterProcedureEnv env, final
RegionStateNode regionNode)
throws IOException, ProcedureSuspendedException {
if (regionNode.getRegionLocation() == null) {
setTransitionState(RegionTransitionState.REGION_TRANSITION_QUEUE);
return true;
}
... and later in same method where the dispatch is actually done
if (!addToRemoteDispatcher(env, regionNode.getRegionLocation())) {
...
Its a narrow hole but one nonetheless.
A check for a zero'd targetserver when we go to use it seems right thing to do.
If null, fail the dispatch. There are other reasons to dispatch fails. This is
new one.
Let me put up a little patch.
> Master stuck thinking hbase:namespace is assigned after restart preventing
> intialization
>
>
> Key: HBASE-19218
> URL: https://issues.apache.org/jira/browse/HBASE-19218
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Umesh Agashe
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments:
> hbase-hbase-master-ctr-e134-1499953498516-282290-01-03.hwx.site.log.zip,
> hbase-site.xml
>
>
> Our [~romil.choksi] brought this one to my attention after trying to get some
> cluster tests running.
> The Master seems to have gotten stuck never initializing after it thinks that
> hbase:namespace was already deployed on the cluster when it actually was not.
> On a Master restart, it reads the location out of meta and assumes that it's
> there (I assume this invalid entry is the issue):
> {noformat}
> 2017-11-08 00:29:17,556 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> assignment.RegionStateStore: Load hbase:meta entry region={ENCODED =>
> f147f204a579b885c351bdc0a7ebbf94, NAME =>
> 'hbase:namespace,,1510084256045.f147f204a579b885c351bdc0a7ebbf94.', STARTKEY
> => '', ENDKEY => ''} regionState=OPENING
> lastHost=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728
> regionLocation=ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510100695534
> {noformat}
> Prior to this, the RS5 went through the ServerCrashProcedure, but it looks
> like this bailed out unexpectedly:
> {noformat}
> 2017-11-08 00:25:25,187 WARN
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 but
> server not online
> 2017-11-08 00:25:25,187 INFO [ProcExecWrkr-5]
> procedure.ServerCrashProcedure: Start pid=36,
> state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure
> server=ctr-e134-1499953498516-282290-01-03.hwx.site,16020,1510084580111,
> splitWal=t
> rue, meta=false
> 2017-11-08 00:25:25,188 INFO
> [ctr-e134-1499953498516-282290-01-03:2.masterManager]
> master.ServerManager: Processing expiration of
> ctr-e134-1499953498516-282290-01-05.hwx.site,16020,1510084579728 on
> ctr-e134-1499953498516-28
> 2290-01-03.hwx.site,2,1510100690324
> ...
>
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299185#comment-16299185 ] Hadoop QA commented on HBASE-19491: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/10598/console in case of problems. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299184#comment-16299184 ] Appy commented on HBASE-19491: -- So the precommit on script itself passed. Attached test patch to run precommit on single module to see general pre-commit is working fine. Testing nightly mode here: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19491/ fyi [~stack] PTAL. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19491: - Attachment: test-patch-to-run-precommit-on-one-module.master.patch > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch, > test-patch-to-run-precommit-on-one-module.master.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19568) Restore of HBase table using incremental backup doesn't restore rows from an earlier incremental backup
[ https://issues.apache.org/jira/browse/HBASE-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-19568: --- Reporter: Romil Choksi (was: Vladimir Rodionov) > Restore of HBase table using incremental backup doesn't restore rows from an > earlier incremental backup > --- > > Key: HBASE-19568 > URL: https://issues.apache.org/jira/browse/HBASE-19568 > Project: HBase > Issue Type: Bug >Reporter: Romil Choksi >Assignee: Vladimir Rodionov > Attachments: HBASE-19568-v1.patch > > > Credits to [~romil.choksi] > Restore of bulk-loaded HBase table doesn't restore deleted rows > Steps: > Create usertable and insert a few rows in it > Create full backup > Bulk load into usertable, and create first incremental backup > Bulk load into usertable again, and create second incremental backup > Delete row each from initial insert, first bulk load and second bulk load > Restore usertable using second incremental backup > Verify if each of the deleted rows has been restored > On restore using second incremental backup id, the test failed as all of the > rows from first bulk load were not available. Data from initial insertion > (full backup) and second bulk load were only available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[
https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299172#comment-16299172
]
Hadoop QA commented on HBASE-19491:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903116/HBASE-19491.master.002.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 28df7cc2c524 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4d71f4e32f |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| shellcheck | v0.4.4 |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10597/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Exclude flaky tests from nightly master run
> ---
>
> Key: HBASE-19491
> URL: https://issues.apache.org/jira/browse/HBASE-19491
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-19491.master.001.patch,
> HBASE-19491.master.002.patch
>
>
> Testing infra improvements
> - Exclude flaky tests from nightly master run
> - Fixes findbugs check (seems like wasn't working earlier : "0
> findbugs0m 0s Findbugs executables are not available.")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19568) Restore of HBase table using incremental backup doesn't restore rows from an earlier incremental backup
[
https://issues.apache.org/jira/browse/HBASE-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299169#comment-16299169
]
Ted Yu commented on HBASE-19568:
{code}
116 // #4 balk load again
{code}
Correct spelling mistake.
Is the fix mostly done in copyBulkLoadedFiles() ?
> Restore of HBase table using incremental backup doesn't restore rows from an
> earlier incremental backup
> ---
>
> Key: HBASE-19568
> URL: https://issues.apache.org/jira/browse/HBASE-19568
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-19568-v1.patch
>
>
> Credits to [~romil.choksi]
> Restore of bulk-loaded HBase table doesn't restore deleted rows
> Steps:
> Create usertable and insert a few rows in it
> Create full backup
> Bulk load into usertable, and create first incremental backup
> Bulk load into usertable again, and create second incremental backup
> Delete row each from initial insert, first bulk load and second bulk load
> Restore usertable using second incremental backup
> Verify if each of the deleted rows has been restored
> On restore using second incremental backup id, the test failed as all of the
> rows from first bulk load were not available. Data from initial insertion
> (full backup) and second bulk load were only available.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299171#comment-16299171 ] Hadoop QA commented on HBASE-19491: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/10597/console in case of problems. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-19491: - Attachment: HBASE-19491.master.002.patch > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch, > HBASE-19491.master.002.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19568) Restore of HBase table using incremental backup doesn't restore rows from an earlier incremental backup
[ https://issues.apache.org/jira/browse/HBASE-19568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-19568: -- Attachment: HBASE-19568-v1.patch Patch v1. cc: [~te...@apache.org], [~elserj] > Restore of HBase table using incremental backup doesn't restore rows from an > earlier incremental backup > --- > > Key: HBASE-19568 > URL: https://issues.apache.org/jira/browse/HBASE-19568 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-19568-v1.patch > > > Credits to [~romil.choksi] > Restore of bulk-loaded HBase table doesn't restore deleted rows > Steps: > Create usertable and insert a few rows in it > Create full backup > Bulk load into usertable, and create first incremental backup > Bulk load into usertable again, and create second incremental backup > Delete row each from initial insert, first bulk load and second bulk load > Restore usertable using second incremental backup > Verify if each of the deleted rows has been restored > On restore using second incremental backup id, the test failed as all of the > rows from first bulk load were not available. Data from initial insertion > (full backup) and second bulk load were only available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299155#comment-16299155
]
Ted Yu commented on HBASE-17319:
Turns out the branch-1 fix was already in 1.4
If it is not clear what should be done for branch-2, I suggest resolving this.
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17319-branch-1.patch, 17319.stack,
> HBASE-17319-branch-1.patch, HBASE-17319-branch-1.patch, HBASE-17319.patch,
> TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17319:
---
Fix Version/s: (was: 1.5.0)
1.4.0
Status: Open (was: Patch Available)
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.2.4, 1.1.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17319-branch-1.patch, 17319.stack,
> HBASE-17319-branch-1.patch, HBASE-17319-branch-1.patch, HBASE-17319.patch,
> TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[
https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299146#comment-16299146
]
Hadoop QA commented on HBASE-19491:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
0s{color} | {color:red} The patch generated 4 new + 0 unchanged - 0 fixed = 4
total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903111/HBASE-19491.master.001.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 0f9bedb51155 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4d71f4e32f |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| shellcheck | v0.4.4 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/10596/artifact/patchprocess/diff-patch-shellcheck.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10596/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Exclude flaky tests from nightly master run
> ---
>
> Key: HBASE-19491
> URL: https://issues.apache.org/jira/browse/HBASE-19491
> Project: HBase
> Issue Type: Improvement
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-19491.master.001.patch
>
>
> Testing infra improvements
> - Exclude flaky tests from nightly master run
> - Fixes findbugs check (seems like wasn't working earlier : "0
> findbugs0m 0s Findbugs executables are not available.")
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299143#comment-16299143 ] Hadoop QA commented on HBASE-19491: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/10596/console in case of problems. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299142#comment-16299142
]
Hadoop QA commented on HBASE-17319:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-17319 does not apply to branch-1. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.6.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-17319 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903112/HBASE-17319-branch-1.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10595/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.5.0
>
> Attachments: 17319-branch-1.patch, 17319.stack,
> HBASE-17319-branch-1.patch, HBASE-17319-branch-1.patch, HBASE-17319.patch,
> TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17436) Add facility to provide more information for Other Regions seen on Master UI
[
https://issues.apache.org/jira/browse/HBASE-17436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299136#comment-16299136
]
Hadoop QA commented on HBASE-17436:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}115m
27s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-17436 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903089/17436.v12.txt |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux c29b0a34e892 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4d71f4e32f |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10590/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10590/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Add facility to provide more information for Other Regions seen on Master UI
>
>
> Key: HBASE-17436
> URL: https://issues.apache.org/jira/browse/HBASE-17436
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Janos Gub
> Labels: ui, usability
> Fix For: 2.0.0
>
> Attachments: 17436.v10.txt, 17436.v11.txt, 17436.v12.txt,
> 17436.v9.txt, HBASE-17436-v2.patch, HBASE-17436-v5.patch,
> HBASE-17436-v6.patch, HBASE-17436-v6.patch, HBASE-17436-v7.patch,
> HBASE-17436-v8.patch, HBASE-17436.patch, HBASE-17779-v3.patch,
> HBASE-17779-v4.patch, Screen Shot 2017-04-24 at 8.43.31 PM.png, Screen Shot
> 2017-04-26 at 4.39.49 PM.png, Screen Shot 2017-11-08 at 10.29.49 AM.png,
> Screen Shot 2017-11-08 at 10.30.09 AM.png, Screen Shot 2017-11-22 at 4.17.35
> PM.png, initial.patch, regionstates.png, table-selected.png, tableregions.png
>
>
> [~rpednekar] and I were looking at a case where the count displayed under
> Other Regions was high (~1200).
> Since the table page just maintains a count instead of List of region names,
> it is very difficult for user to determine which regions belong to this
> category.
> We should provide facility to provide more details for this category (LOG,
> JMX, etc).
--
This
[jira] [Commented] (HBASE-19571) Minor refactor of Nightly run scripts
[ https://issues.apache.org/jira/browse/HBASE-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299137#comment-16299137 ] Appy commented on HBASE-19571: -- Here's the test build: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19571 > Minor refactor of Nightly run scripts > - > > Key: HBASE-19571 > URL: https://issues.apache.org/jira/browse/HBASE-19571 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-19571.master.001.patch > > > Was trying to split out common code in separate lib, but it's not at all > trivial. The way to do it for declarative syntax Jenkinsfile is by using > Shared Libraries which requires separate repo! > The patch ended up being just naming refactors. > Renames OUTPUTDIR to OUTPUT_DIR > Renames OUTPUT_RELATIVE to OUTPUT_DIR_RELATIVE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19486) Automatically flush BufferedMutator after a timeout
[
https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299135#comment-16299135
]
Niels Basjes commented on HBASE-19486:
--
[~stack] Ok, I understand {{autoflush}} should not be used.
Current name of the setting is {{writeBufferMaxLingerMs}}
The only thing I currently named {{autoflush}} is the timer. I realize that
this is inconsistent with the rest of this change (renaming that to
{{writeBufferMaxLingerTimer}} is easy).
But before I do this: What naming for this feature would you guys prefer?
Something like {{write buffer periodic flush interval}} perhaps?
Or is just getting rid of {{autoflush}} what you want?
> Automatically flush BufferedMutator after a timeout
>
>
> Key: HBASE-19486
> URL: https://issues.apache.org/jira/browse/HBASE-19486
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Niels Basjes
>Assignee: Niels Basjes
> Attachments: HBASE-19486-20171212-2117.patch,
> HBASE-19486-20171218-1229.patch, HBASE-19486-20171218-1300.patch,
> HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
> HBASE-19486-20171219-1122-trigger-qa-run.patch,
> HBASE-19486-20171220-1612-trigger-qa-run.patch,
> HBASE-19486-20171220-2228-trigger-qa-run.patch
>
>
> I'm working on several projects where we are doing stream / event type
> processing instead of batch type processing. We mostly use Apache Flink and
> Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a
> BufferedMutator this all works fine. The buffer fills up at a predictable
> rate and we can make sure it flushes several times per second into HBase by
> tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times
> because the source is in reality a batch job that puts records into Kafka,
> sometimes because it is the "predictable in production" application in our
> testing environment (where only the dev triggers a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to
> automatically flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against
> the master (i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of
> the client in the same (as close as possible) way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19491) Exclude flaky tests from nightly master run
[ https://issues.apache.org/jira/browse/HBASE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299133#comment-16299133 ] Appy commented on HBASE-19491: -- Running precommit just to make sure it's alright after changes in hbase-personality.sh. > Exclude flaky tests from nightly master run > --- > > Key: HBASE-19491 > URL: https://issues.apache.org/jira/browse/HBASE-19491 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy > Attachments: HBASE-19491.master.001.patch > > > Testing infra improvements > - Exclude flaky tests from nightly master run > - Fixes findbugs check (seems like wasn't working earlier : "0 > findbugs0m 0s Findbugs executables are not available.") -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19571) Minor refactor of Nightly run scripts
[ https://issues.apache.org/jira/browse/HBASE-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299132#comment-16299132 ] Appy commented on HBASE-19571: -- Since the changes are only in Jenkinsfile and hbase_nightly_yetus.sh script, there is not point of running precommit on it. Testing it using separate branch, running job manually, and checking results, artifacts, etc > Minor refactor of Nightly run scripts > - > > Key: HBASE-19571 > URL: https://issues.apache.org/jira/browse/HBASE-19571 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-19571.master.001.patch > > > Was trying to split out common code in separate lib, but it's not at all > trivial. The way to do it for declarative syntax Jenkinsfile is by using > Shared Libraries which requires separate repo! > The patch ended up being just naming refactors. > Renames OUTPUTDIR to OUTPUT_DIR > Renames OUTPUT_RELATIVE to OUTPUT_DIR_RELATIVE -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17319:
---
Attachment: HBASE-17319-branch-1.patch
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.5.0
>
> Attachments: 17319-branch-1.patch, 17319.stack,
> HBASE-17319-branch-1.patch, HBASE-17319-branch-1.patch, HBASE-17319.patch,
> TestTruncateTableProcedure-output.tar.gz
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Status: Patch Available (was: Open) > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18294) Reduce global heap pressure: flush based on heap occupancy
[ https://issues.apache.org/jira/browse/HBASE-18294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eshcar Hillel updated HBASE-18294: -- Status: Open (was: Patch Available) > Reduce global heap pressure: flush based on heap occupancy > -- > > Key: HBASE-18294 > URL: https://issues.apache.org/jira/browse/HBASE-18294 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-18294.01.patch, HBASE-18294.02.patch, > HBASE-18294.03.patch, HBASE-18294.04.patch, HBASE-18294.05.patch, > HBASE-18294.06.patch, HBASE-18294.07.patch > > > A region is flushed if its memory component exceed a threshold (default size > is 128MB). > A flush policy decides whether to flush a store by comparing the size of the > store to another threshold (that can be configured with > hbase.hregion.percolumnfamilyflush.size.lower.bound). > Currently the implementation (in both cases) compares the data size > (key-value only) to the threshold where it should compare the heap size > (which includes index size, and metadata). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
