[jira] [Created] (HBASE-15588) Use nonce for checkAndMutate operation

[Jianwei Cui (JIRA)]

Jianwei Cui created HBASE-15588:
---

 Summary: Use nonce for checkAndMutate operation
 Key: HBASE-15588
 URL: https://issues.apache.org/jira/browse/HBASE-15588
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 2.0.0
Reporter: Jianwei Cui


Like {{increment}}/{{append}}, the {{checkAndPut}}/{{checkAndDelete}} operation 
is non-idempotent, so that the client may get incorrect result if there are 
retries, and such incorrect result may lead the application enter an error 
state. A possible solution is using nonce for checkAndMutate operations, 
discussions and suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-15404) PE: Clients in append and increment are operating serially

[Appy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Appy resolved HBASE-15404.
--
Resolution: Invalid

Clients are still working parallely, except that they all process keys from a 
same small range at a time. for eg. fist 0 - 1M, then 1M - 2M, so on..
Thanks stack for explaining.

> PE: Clients in append and increment are operating serially
> --
>
> Key: HBASE-15404
> URL: https://issues.apache.org/jira/browse/HBASE-15404
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Affects Versions: 1.2.0
>Reporter: Appy
>
> On running hbase pe --nomapred increment/append 10,  i see the following 
> output where it seems like threads are executing operations serially. In the 
> UI too, only one RS is getting requests at a time.
> {noformat}
> 6/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-1
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-2
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-8
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-0
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-4
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-6
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-7
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-5
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-9
> 16/03/05 22:26:15 INFO hbase.PerformanceEvaluation: Timed test starting in 
> thread TestClient-3
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.48, min=390.00, max=163444.00, stdDev=892.64, 95th=1361.00, 
> 99th=1605.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.53, min=366.00, max=163400.00, stdDev=885.49, 95th=1361.00, 
> 99th=1605.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.41, min=402.00, max=163436.00, stdDev=891.54, 95th=1359.00, 
> 99th=1602.41
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.51, min=399.00, max=163610.00, stdDev=892.40, 95th=1360.00, 
> 99th=1600.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.59, min=393.00, max=162932.00, stdDev=887.65, 95th=1361.00, 
> 99th=1604.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.26, min=385.00, max=163482.00, stdDev=891.71, 95th=1358.00, 
> 99th=1599.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.51, min=383.00, max=163246.00, stdDev=888.07, 95th=1360.00, 
> 99th=1605.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.45, min=385.00, max=163405.00, stdDev=886.65, 95th=1359.00, 
> 99th=1604.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.38, min=400.00, max=163580.00, stdDev=887.28, 95th=1359.00, 
> 99th=1602.00
> 16/03/05 22:27:54 INFO hbase.PerformanceEvaluation: 0/104857/1048576, latency 
> mean=942.29, min=407.00, max=163403.00, stdDev=889.77, 95th=1357.00, 
> 99th=1597.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.75, min=366.00, max=163400.00, stdDev=817.84, 95th=1363.00, 
> 99th=1605.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.75, min=383.00, max=163246.00, stdDev=821.95, 95th=1363.00, 
> 99th=1604.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.75, min=389.00, max=163444.00, stdDev=824.03, 95th=1364.00, 
> 99th=1603.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.56, min=382.00, max=163403.00, stdDev=822.44, 95th=1363.00, 
> 99th=1603.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.79, min=393.00, max=162932.00, stdDev=818.75, 95th=1365.00, 
> 99th=1601.84
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.70, min=388.00, max=163436.00, stdDev=823.52, 95th=1364.00, 
> 99th=1606.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.72, min=376.00, max=163405.00, stdDev=820.65, 95th=1364.00, 
> 99th=1605.00
> 16/03/05 22:29:35 INFO hbase.PerformanceEvaluation: 0/209714/1048576, latency 
> mean=950.56, min=382.00, max=163482.00, stdDev=823.43, 95th=1363.00, 

[jira] [Created] (HBASE-15587) FSTableDescriptors.getDescriptor() logs stack trace erronously

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-15587:
-

 Summary: FSTableDescriptors.getDescriptor() logs stack trace 
erronously
 Key: HBASE-15587
 URL: https://issues.apache.org/jira/browse/HBASE-15587
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.3.0, 1.4.0


Long time pet-peeve, but it is causing some confusion when looking at the 
master logs for unrelated things. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15586) Unify human readable numbers in the web UI

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-15586:
-

 Summary: Unify human readable numbers in the web UI 
 Key: HBASE-15586
 URL: https://issues.apache.org/jira/browse/HBASE-15586
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 2.0.0, 1.3.0, 1.4.0


I was looking at something else in the regionserver web ui trying to understand 
the WAL size, and we are reporting that as raw bytes, not in MB / GB range. Did 
a quick sweep to unify all the human-readable representations in the UI. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15585) RegionServer coprocessors are not flexible enough

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HBASE-15585:


 Summary: RegionServer coprocessors are not flexible enough
 Key: HBASE-15585
 URL: https://issues.apache.org/jira/browse/HBASE-15585
 Project: HBase
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


While you can do all kinds of things with coprocessors, like arbitrarily 
discard memstore data or replace files randomly during compaction, I believe 
the ultimate power and flexibility is not there. The patch aims to address this 
shortcoming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15584) Revisit handling of BackupState#CANCELLED

[Ted Yu (JIRA)]

Ted Yu created HBASE-15584:
--

 Summary: Revisit handling of BackupState#CANCELLED
 Key: HBASE-15584
 URL: https://issues.apache.org/jira/browse/HBASE-15584
 Project: HBase
  Issue Type: Sub-task
Reporter: Ted Yu
Priority: Minor


During review of HBASE-15411, Enis made the following point:
{code}
nobody puts the backup in cancelled state. setCancelled() is not used. So if I 
abort a backup, who writes to the system table the new state? 

Not sure whether this is a phase 1 patch issue or due to this patch. We can 
open a new jira and address it there if you do not want to do it in this patch. 

Also maybe this should be named ABORTED rather than CANCELLED.
{code}
This issue is to decide whether this state should be kept (e.g. through 
notification from procedure V2 framework in response to abortion).

If it is to be kept, the state should be renamed ABORTED.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15583) Discuss mutable vs immutable HTableDescriptor

[Gabor Liptak (JIRA)]

Gabor Liptak created HBASE-15583:


 Summary: Discuss mutable vs immutable HTableDescriptor
 Key: HBASE-15583
 URL: https://issues.apache.org/jira/browse/HBASE-15583
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Gabor Liptak
Priority: Minor


>From [~enis] in https://issues.apache.org/jira/browse/HBASE-15505:

PS Should UnmodifyableHTableDescriptor be renamed to 
UnmodifiableHTableDescriptor?

It should be named ImmutableHTableDescriptor to be consistent with collections 
naming. Let's do this as a subtask of the parent jira, not here. Thinking about 
it though, why would we return an Immutable HTD in HTable.getTableDescriptor() 
versus a mutable HTD in Admin.getTableDescriptor(). It does not make sense. 
Should we just get rid of the Immutable ones?
We also have UnmodifyableHRegionInfo which is not used at the moment it seems. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Better management of flakies

[Stack]

On Fri, Apr 1, 2016 at 10:05 AM, Sean Busbey  wrote:


> Maybe better if we ensure everything needed to run our jobs is scripts
> in dev-support so that the commiter-only change can just be moving
> jenkins to execute them.
>
>
I like this idea Sean.
St.Ack



> On Fri, Apr 1, 2016 at 12:01 PM, Apekshit Sharma 
> wrote:
> > So who's setting up the jobs? I don't have perms to do it.
> >
> > On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma 
> wrote:
> >
> >> So tackling the two parts individually:
> >> 1) Detection : A good automatic flaky detector will no doubt save manual
> >> effort. The one mentioned by Matteo seems like a good one which we can
> hook
> >> up with post-commit job. I see a minor problem though, we'll should use
> >> about 20 runs at least, but the rate of execution of this post-commit
> >>  job for
> >> trunk is too low which means we'll get stale information from the tool.
> One
> >> way around that would be running the post-commit job back-to-back? It
> can
> >> than trigger another job in the end which will use the tool to recompute
> >> flakies.
> >>
> >> 2) Handling: Ignoring flakies in main build and having separate job just
> >> for flakies. We can run it often enough and get some stats on flaky-ness
> >> rate of each test using the same tool as above.
> >>
> >> However, I feel that we should keep it super simple in the start. Just
> >> have a manual list and a separate job for flakies. It it works out
> fine, we
> >> can add the automatic detection and statistics part later.
> >>
> >> - Appy
> >>
> >
> >
> >
> > --
> >
> > Regards
> >
> > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > 650-963-6311
>
>
>
> --
> busbey
>

Re: Better management of flakies

[Stack]

On Tue, Mar 29, 2016 at 4:46 PM, Matteo Bertozzi 
wrote:

> don't we have already a flakey detector? HBASE-8018
>
> https://github.com/apache/hbase/commit/4dc52261a19ed1055b837b136c97ea36e362ba6e
>
>
I remember that script. It used to work for me but had forgotten about it.
St.Ack




> On Tue, Mar 29, 2016 at 4:39 PM, Sean Busbey  wrote:
>
> > How about if we make a script for determining flakey status, based on
> > rate of failing in either post-commit or in the pre-existing branch
> > check for precommit. The script could live in dev-support and work by
> > effectively grabbing the test reports from those jobs.
> >
> > We can then run that script in a job and save the results as a build
> > artifact.
> >
> > We can then build the changes in your original email: ignoring the
> > list entries in our normal builds and rechecking just them in a
> > dedicated job.
> >
> > On Tue, Mar 29, 2016 at 6:30 PM, Sean Busbey 
> wrote:
> > > Where/How do we maintain the list of tests that are flagged as flakes?
> > >
> > > On Tue, Mar 29, 2016 at 6:25 PM, Apekshit Sharma 
> > wrote:
> > >> Proposal:
> > >> Maintain a list of flaky tests which can be ignored in main builds
> (for
> > >> patches) and setup a new job which will periodically run those flaky
> > tests.
> > >>
> > >> Benefits:
> > >> - Cleaner main builds. Less runs per patch,  ideally no re-runs
> because
> > of
> > >> timeouts and other flakiness. Less frustration. Increased developer
> > >> productivity.
> > >> - Logs from various runs available when anyone tries to fix these
> tests.
> > >>
> > >> Possible con: We start demoting tests to flaky list which never get
> > fixed.
> > >>
> > >> How?
> > >> Internally, we have setup up two jenkins jobs, one for main build and
> > one
> > >> for flaky. In execute shell step, they curl to get the list of flaky
> > tests
> > >> and use surefire plugin flags to ignore/run particular tests.
> > >> I volunteer to set them up. However, not sure if this approach will
> work
> > >> since we have yetus upstream.
> > >>
> > >> - Appy
> > >
> > >
> > >
> > > --
> > > busbey
> >
> >
> >
> > --
> > busbey
> >
>

[jira] [Created] (HBASE-15582) SnapshotManifestV1 too verbose when there are no regions

[Matteo Bertozzi (JIRA)]

Matteo Bertozzi created HBASE-15582:
---

 Summary: SnapshotManifestV1 too verbose when there are no regions
 Key: HBASE-15582
 URL: https://issues.apache.org/jira/browse/HBASE-15582
 Project: HBase
  Issue Type: Bug
  Components: snapshots
Affects Versions: 0.98.16.1, 1.1.4, 1.2.0, 2.0.0, 1.3.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi


swap the INFO for DEBUG in SnapshotManifestV1.loadRegionManifests() otherwise 
the cleaner will spam everyone for no reason



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15581) Reduce garbage created by PB in write path

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-15581:
--

 Summary: Reduce garbage created by PB in write path
 Key: HBASE-15581
 URL: https://issues.apache.org/jira/browse/HBASE-15581
 Project: HBase
  Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Better management of flakies

[Sean Busbey]

If you create a jira that describes what we need to implement, I'll
find time to put in changes.

Maybe better if we ensure everything needed to run our jobs is scripts
in dev-support so that the commiter-only change can just be moving
jenkins to execute them.

On Fri, Apr 1, 2016 at 12:01 PM, Apekshit Sharma  wrote:
> So who's setting up the jobs? I don't have perms to do it.
>
> On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma  wrote:
>
>> So tackling the two parts individually:
>> 1) Detection : A good automatic flaky detector will no doubt save manual
>> effort. The one mentioned by Matteo seems like a good one which we can hook
>> up with post-commit job. I see a minor problem though, we'll should use
>> about 20 runs at least, but the rate of execution of this post-commit
>>  job for
>> trunk is too low which means we'll get stale information from the tool. One
>> way around that would be running the post-commit job back-to-back? It can
>> than trigger another job in the end which will use the tool to recompute
>> flakies.
>>
>> 2) Handling: Ignoring flakies in main build and having separate job just
>> for flakies. We can run it often enough and get some stats on flaky-ness
>> rate of each test using the same tool as above.
>>
>> However, I feel that we should keep it super simple in the start. Just
>> have a manual list and a separate job for flakies. It it works out fine, we
>> can add the automatic detection and statistics part later.
>>
>> - Appy
>>
>
>
>
> --
>
> Regards
>
> Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> 650-963-6311



-- 
busbey

Re: Better management of flakies

[Apekshit Sharma]

So who's setting up the jobs? I don't have perms to do it.

On Wed, Mar 30, 2016 at 11:24 AM, Apekshit Sharma  wrote:

> So tackling the two parts individually:
> 1) Detection : A good automatic flaky detector will no doubt save manual
> effort. The one mentioned by Matteo seems like a good one which we can hook
> up with post-commit job. I see a minor problem though, we'll should use
> about 20 runs at least, but the rate of execution of this post-commit
>  job for
> trunk is too low which means we'll get stale information from the tool. One
> way around that would be running the post-commit job back-to-back? It can
> than trigger another job in the end which will use the tool to recompute
> flakies.
>
> 2) Handling: Ignoring flakies in main build and having separate job just
> for flakies. We can run it often enough and get some stats on flaky-ness
> rate of each test using the same tool as above.
>
> However, I feel that we should keep it super simple in the start. Just
> have a manual list and a separate job for flakies. It it works out fine, we
> can add the automatic detection and statistics part later.
>
> - Appy
>



-- 

Regards

Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
650-963-6311

[jira] [Resolved] (HBASE-12814) Zero downtime upgrade from 94 to 98

[churro morales (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

churro morales resolved HBASE-12814.

Resolution: Not A Problem

Most likely everyone is off the 94 branch.  

> Zero downtime upgrade from 94 to 98 
> 
>
> Key: HBASE-12814
> URL: https://issues.apache.org/jira/browse/HBASE-12814
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 0.94.26, 0.98.10
>Reporter: churro morales
>Assignee: churro morales
> Attachments: HBASE-12814-0.94.patch, HBASE-12814-0.98.patch
>
>
> Here at Flurry we want to upgrade our HBase cluster from 94 to 98 while not 
> having any downtime and maintaining master / master replication. 
> Summary:
> Replication is done via thrift RPC between clusters.  It is configurable on a 
> peer by peer basis and the one caveat is that a thrift server starts up on 
> every node which proxies the request to the ReplicationSink.  
> For the upgrade process:
> * in hbase-site.xml two new configuration parameters are added:
> ** *Required*
> *** hbase.replication.sink.enable.thrift -> true
> *** hbase.replication.thrift.server.port -> 
> ** *Optional*
> *** hbase.replication.thrift.protection {default: AUTHENTICATION}
> *** hbase.replication.thrift.framed {default: false}
> *** hbase.replication.thrift.compact {default: true}
> - All regionservers can be rolling restarted (no downtime), all clusters must 
> have the respective patch for this to work.
> - the hbase shell add_peer command takes an additional parameter for rpc 
> protocol
> - example: {code} add_peer '1' "hbase-101:2181:/hbase", "THRIFT" {code}
> Now comes the fun part when you want to upgrade your cluster from 94 to 98 
> you simply pause replication to the cluster being upgraded, do the upgrade 
> and un-pause replication.  Once you have a pair of clusters only replicating 
> inbound and outbound with the 98 release.  You can start replicating via the 
> native rpc protocol by adding the peer again without the _THRIFT_ parameter 
> and subsequently deleting the peer with the thrift protocol.  Because 
> replication is idempotent I don't see any issues as long as you wait for the 
> backlog to drain after un-pausing replication. 
> Special thanks to Francis Liu at Yahoo for laying the groundwork and Mr. Dave 
> Latham for his invaluable knowledge and assistance.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader

[Rajeshbabu Chintaguntla (JIRA)]

Rajeshbabu Chintaguntla created HBASE-15580:
---

 Summary: Tag coprocessor limitedprivate scope to StoreFile.Reader
 Key: HBASE-15580
 URL: https://issues.apache.org/jira/browse/HBASE-15580
 Project: HBase
  Issue Type: Improvement
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
 Fix For: 2.0.0, 0.98.19, 1.1.5, 1.2.2


For phoenix local indexing we need to have custom storefile reader 
constructor(IndexHalfStoreFileReader) to distinguish from other storefile 
readers. So wanted to mark StoreFile.Reader scope as 
InterfaceAudience.LimitedPrivate("Coprocessor")



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15579) Remove synchronized around nonce in Procedure submit

[Matteo Bertozzi (JIRA)]

Matteo Bertozzi created HBASE-15579:
---

 Summary: Remove synchronized around nonce in Procedure submit
 Key: HBASE-15579
 URL: https://issues.apache.org/jira/browse/HBASE-15579
 Project: HBase
  Issue Type: Sub-task
  Components: proc-v2
Affects Versions: 1.1.4, 1.2.0, 2.0.0, 1.3.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
 Fix For: 2.0.0, 1.3.0, 1.1.5, 1.2.2
 Attachments: HBASE-15579-v0.patch

There is a synchronized in ProcedureExecutor.submitProcedure() that protect the 
nonce insert. We can get rid of that and just use the nonce concurrent map 
pufIfMissing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. If failed, skip to the 
bottom of this email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/185/artifact/website.patch.zip
 | funzip > 25419d8b18dd8f35a102614cd31b274659f747ef.patch
  git fetch
  git checkout -b asf-site-25419d8b18dd8f35a102614cd31b274659f747ef 
origin/asf-site
  git am 25419d8b18dd8f35a102614cd31b274659f747ef.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-25419d8b18dd8f35a102614cd31b274659f747ef branch, and you can review 
the differences by running:

  git diff origin/asf-site

There are lots of spurious changes, such as timestamps and CSS styles in 
tables. To see a list of files that have been added, deleted, renamed, changed 
type, or are otherwise interesting, use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using this 
command:

  git push origin asf-site-25419d8b18dd8f35a102614cd31b274659f747ef:asf-site

Changes take a couple of minutes to be propagated. You can then remove your 
asf-site-25419d8b18dd8f35a102614cd31b274659f747ef branch:

  git checkout asf-site && git branch -d 
asf-site-25419d8b18dd8f35a102614cd31b274659f747ef



If failed, see https://builds.apache.org/job/hbase_generate_website/185/console

[jira] [Created] (HBASE-15578) Handle HBASE-15234 for ReplicationHFileCleaner

[Ashish Singhi (JIRA)]

Ashish Singhi created HBASE-15578:
-

 Summary: Handle HBASE-15234 for ReplicationHFileCleaner
 Key: HBASE-15578
 URL: https://issues.apache.org/jira/browse/HBASE-15578
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 1.3.0
Reporter: Ashish Singhi
Assignee: Ashish Singhi


HBASE-15234 is handled for ReplicationLogCleaner need to handle similarly for 
ReplicationHFileCleaner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15577) there need be a mechanism to enable ZK's ACL check when the authentication strategy is simple

[chenxu (JIRA)]

chenxu created HBASE-15577:
--

 Summary: there need be a mechanism to enable ZK's ACL check when 
the authentication strategy is simple
 Key: HBASE-15577
 URL: https://issues.apache.org/jira/browse/HBASE-15577
 Project: HBase
  Issue Type: Improvement
Reporter: chenxu
Assignee: chenxu


if the hbase.security.authentication is set to simple, the ZKUtil.createACL 
just return Ids.OPEN_ACL_UNSAFE, means that there is no ACL check on the ZK's 
node.
we can refactoring this to enables the ACL's check function



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Coprocessors causes data access delay?

[Anoop John]

Can you provide us the code u have written for the pre hooks + the
custom filter code.  You sure the filter is not filtering out this
recent cell?  Add some logs there and check?

-Anoop-

On Thu, Mar 31, 2016 at 12:57 PM, Jozef Vilcek  wrote:
> Test was done via hbase shell prompt. I am not sure on it's behavior. Can
> it hide some timeouts and return nothing? I hope not.
> I will try to reproduce it with java API.
>
> Coprocessor does not do anything asynchronous in it's hooks. It is hooked
> only on preStoreScannerOpen, preFlushScannerOpen, preCompactScannerOpen.
> What it does is, create a StoreScanner with custom filter, which skips
> certain cells based on fixed rules line qualifier name pattern or how old
> cell is.
>
>
>> Do you wait for the Put to return before you issue the Get?  Or have you
>> disabled auto flushing on your table instance -- HTable.setAutoFlush(false)?
>
>> Coprocessor hooks executed on the Put and Get paths are blocking, so unless
>> you're overriding normal processing in the prePut() hook and doing
>> something asynchronous, the Put should be there by the time it returns to
>> the client.  But it all depends on what is your coprocessor actually
>> doing.  Can you tell us more about that?
>
>> On Tue, Mar 29, 2016 at 6:22 AM Ted Yu  wrote:
>
>> > Is it possible to come up with unit test showing this behavior using> > 
>> > simplified coprocessor ?> >> > Thanks> >> > > On Mar 29, 2016, at 5:42 AM, 
>> > Jozef Vilcek  wrote:> > >> > > Hi,> > >> > > I need 
>> > a help to shed some light on following observation:> > >> > > Through 
>> > hbase shell I do PUT to a non existing cell (create) and> > immediate> > > 
>> > GET.> > > When doing this against hbase table without a coprocessor, every 
>> > GET> > shows> > > the value immediately. When I enable a coprocessor on 
>> > the table and> > > perform> > > the same check, I do not get a value for 
>> > every GET. Sometimes GET returns> > > nothing, like cell is not there. If 
>> > I delay GET few minutes after PUT,> > then> > > data are correctly 
>> > retrieved.> > >> > > I want to understand what is going on.> > > Is 
>> > coprocessor is doing something wrong? Is this somehow normal? or ... ?> > 
>> > >> > > Coprocessor is a RegionObserver doing data filtering / expiration 
>> > based> > on> > > predefined rules.> > > HBase version is 1.0.0 from 
>> > cloudera cdh5.5.1 distibution.> > >> > > Thanks,> > > Jozo
>>

[jira] [Created] (HBASE-15576) Support stateless scanning and scanning cursor

[Phil Yang (JIRA)]

Phil Yang created HBASE-15576:
-

 Summary: Support stateless scanning and scanning cursor
 Key: HBASE-15576
 URL: https://issues.apache.org/jira/browse/HBASE-15576
 Project: HBase
  Issue Type: New Feature
Reporter: Phil Yang
Assignee: Phil Yang


After 1.1.0 released, we have partial and heartbeat protocol in scanning to 
prevent responding large data or timeout. Now for ResultScanner.next(), we may 
block for longer time larger than timeout settings to get a Result if the row 
is very large, or filter is sparse, or there are too many delete markers in 
files.

However, in some scenes, we don't want it to be blocked for too long. For 
example, a web service which handles requests from mobile devices whose network 
is not stable and we can not set timeout too long(eg. only 5 seconds) between 
mobile and web service. This service will scan rows from HBase and return it to 
mobile devices. In this scene, the simplest way is to make the web service 
stateless. Apps in mobile devices will send several requests one by one to get 
the data until enough just like paging a list. In each request it will carry a 
start position which depends on the last result from web service. Different 
requests can be sent to different web service server because it is stateless.

Therefore, the stateless web service need a cursor from HBase telling where we 
have scanned in RegionScanner when HBase client receives an empty heartbeat. 
And the service will return the cursor to mobile device although the response 
has no data. In next request we can start at the position of cursor, without 
the cursor we have to scan from last returned result and we may timeout 
forever. And of course even if the heartbeat message is not empty we can still 
use cursor to prevent re-scan the same rows/cells which has beed skipped.

Obviously, we will give up consistency for scanning because even HBase client 
is also stateless, but it is acceptable in this scene. And maybe we can keep 
mvcc in cursor so we can get a consistent view?

HBASE-13099 had some discussion, but it has no further progress by now.

API:

In Scan we need a new method setStateless to make the scanning stateless and 
need another timeout setting for stateless scanning. In this mode we will not 
block ResultScanner.next() longer than this timeout setting. And we will return 
Results in next() as usual but the last Result (or only Result if we receive 
empty heartbeat) has a special flag to mark it a cursor. The cursor Result has 
only one Cell. Users can scan like this:

{code}
while( r = scanner.next() && r != null && !r.isCursor()){
//just like before
}
if(r != null){
// scanning is not end, it is a cursor
} else {
// scanning is end
}
scanner.close()
{code}

Implementation:

We will have two options to support stateless scanning: 
Only one rpc like small scanning, not supporting batch/partials and cursor is 
row level. It is simple to implementation.
Support big scanning with several rpc requests, supporting batch/partials and 
cursor is cell level. It is a little complex because we need seek at server 
side.

Or we can make it by two phases, support one-shot first?

Any thoughts? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)